At 10:24AM, we started receiving reports of outbound call issues. This was immediately escalated to the operations team and CTO. Investigations began immediately, and the issue - caused by a faulty database instance in a high availability database cluster responsible for SIP account authentication - was resolved at 10:41AM.
This database contains sip account credentials, and is used for server and customer authentication for placing outbound calls. It runs as a highly available database cluster within Amazon Web Services (AWS). Although monitoring as well as health checks within AWS did not show any issues with this cluster, one of the server instances was returning invalid username/password on connection requests. The issue was resolved by rebooting this instance within the cluster.