RCA - One of the database connection pooler system configuration settings was incorrectly set to lower limit when we upgraded the servers which resulted in failure of application to db server connections when the traffic was high.
Impact -
System was entirely down from 5:16pm PST to 5:35pm PST - 19min complete outage. System was available with degraded performance (slowness) from 5:35pm PST till about 5:50pm PST - 15min degraded performance.
Posted Feb 15, 2024 - 08:16 PST
Monitoring
On Feb 14, 2024, CallHub users experienced disruption on the CallHub dashboard, encountering HTTP error 403. This issue also affected the performance of the mobile app during the period from 5:16 PM to 5:50 PM PST.
Upon conducting an initial investigation, we identified the root cause related to auto-scaling processes. The issue has been promptly addressed, and as of now, all services have been restored to normal functionality. Our team actively monitors the services to ensure stability.
We apologize for any inconvenience caused during this period.