Panel issues
Incident Report for QuenchTec
Postmortem

First of all, we are very sorry for the inconvenience that these disturbances may have caused you. Here is a brief description of what happened, what caused it and what we are doing to prevent this from happening again.

Incident

The outages that we experienced this week caused some panels and communities to go offline. This happened on the 5th of February and again on the 6th.

Cause

After analysing the logs, we found that the caching put in place did not work as expected which caused several applications to stop and not being able to restart themselves.

Actions

As as result of this, we have two fixes that we will start rolling out next week.

The first fix deals with the way our cache is setup and will ensure that when items are removed from the cache will not cause the applications to recycle.

The second fix address caching on a back-end level and involves several code changes. This fix is currently going through testing and we expect this to be ready for deployment next week.

Posted Feb 08, 2019 - 11:40 CET

Resolved
This incident has been resolved.
Posted Feb 06, 2019 - 13:28 CET
Monitoring
All panels should now be working as expected. We are very sorry for the inconvenience this has caused you and we will monitor the situation closely.

As soon as the root cause of this problem has been identified and analyzed, we will publish a postmortem here.
Posted Feb 06, 2019 - 11:21 CET
Update
A lot of panels are now back online but we are still working on making sure everything is running as expected.
Posted Feb 06, 2019 - 10:49 CET
Investigating
We are currently investigating an issue with panels loading slowly. We are working hard to resolve this and will keep you updated.
Posted Feb 06, 2019 - 10:23 CET
This incident affected: QuenchTec Panel Management.