Basic Timeline & Incident Overview:
Starting at approximately 6:18 AM (Mountain) on the morning of 02.26.21, the Splunk On-Call (SpOC) platform began experiencing delays in the outbound delivery of notifications.
These delays were directly related to a critical service disruption associated with one of our primary telecommunications providers.
Although the service disruption was primarily affecting outbound Phone and SMS notifications, the situation also (temporarily) delayed our delivery of Push notifications, as well.
At approximately 7:40 AM (Mountain), actions taken by the SpOC Engineering team alleviated the Push notification delays and returned Push notification delivery to standard operational efficiency.
The proper delivery of SMS notifications resumed at approximately 8:30 AM (Mountain).
Phone notifications were queued and being delivered at a roughly 40-minute delay rate until approximately 9:30 AM (Mountain).
The overall incident timeline was from approximately 6:18 AM (Mountain) - 9:30 AM (Mountain) on the morning of 02.26.21.
After internal testing and direct communication with the telecommunications provider in-question, we deemed the incident resolved at 9:30 AM (Mountain) on the morning of 02.26.21.
. . .
As with any and all such incidents, the appropriate Splunk On-Call (SpOC) will conduct a collaborative and intensive Post Incident Review (PIR) process aimed at both preventative measures and improved responsiveness toward addressing any future issues. If you have any immediate questions or concerns, please contact the Splunk On-Call (SpOC) Support Team at:
Once again, we sincerely apologize for any unintended inconvenience this incident may have caused.