Connections through SDM nodes temporarily disrupted

Summary:

During a routine system update on October 2, a server release was pushed to production that changed how tokens were validated, which led to failed connections to resources. Our internal canary deployment caught the issue and alerted the team. They started to rollback the change when we started to learn about customer impact. There were two compounding issues that caused this outage. 

Cause:

It took some time for the issue to manifest in our Canary environment. By the time the system alerted, the build was deemed OK and pushed to production. Which was the ultimate root cause of the failure.

Impact:

• Disruption to resource connections.
• Users who logged in during the window needed to re-authenticate.
• No unauthorized access or security exposure.

Remediation & Prevention:

• Extending the delay between staging and production deployments for better early detection.
• Updating infrastructure to manage environment