Between the hours 13:00 UTC and 16:00 UTC on the 29th of August, CommPeak clients that are using CommPeak PBX Stats encountered login problems and performance issues in the system, and issues with the CommPeak Softphone connection.
The event was triggered by a new feature that was deployed on the last release and created a performance regression.
Our engineering team has identified a performance bottleneck with redis-server and has upgraded the component, that action has achieved partial relief, but the issue was still ongoing.
The root source of the bottleneck was identified and fixed at 15:40 UTC. After the hotfix had been released, it released the bottleneck, and services were restored at 16:00 UTC.
This incident had a widespread impact, and we take availability very seriously. We have identified several areas of improvement and will continue to work on uncovering any other gaps that could cause a recurrence.
