Investigating – AWS Connection Issues with AWS Secret Stores

Incident Report for StrongDM

Postmortem

TIMELINE

  • On January  14, 2026 at 4:59pm PDT, a pull request was merged which changed the way the relays obtain credentials from AWS. We migrated to Go AWS SDK v2 and our migration was missing container credential support which worked out of the box with v1. In EKS environments our code skipped the container endpoint and went directly to IMDS, which could return the wrong IAM role
  • On January 28 2:20am PDT first ticket came into support
  • On January 29 9:53am PDT, the cli release was identified and rolled back.
  • By 9:59am PDT all control planes were rolled back.

CUSTOMER IMPACT

  • Relays running in EKS failed to assume correct IAM roles and resources turned unhealthy

CAUSES

  • Incomplete SDK migration: The migration from AWS SDK v1 to v2 did not account for differences in the default credential provider chain. SDK v1  automatically supported container credentials, while v2 required explicit configuration.
  • In EKS environments, IMDS returns the EC2 node's IAM role rather than the pod's IAM role (configured via IRSA or EKS Pod Identity), causing relays to authenticate with incorrect/insufficient permissions

POSSIBLE REMEDIATIONS

  • Include more testing for AWS credential providers
  • Add monitoring for relay credential failures: Create alerts for spikes in IAM authentication failures or resources transitioning to unhealthy state
Posted Feb 03, 2026 - 16:46 UTC

Resolved

The rollback has been completed. The issue will also be fixed in code.
Posted Jan 29, 2026 - 18:02 UTC

Identified

StrongDM is currently investigating an issue that may impact connectivity to AWS services, specifically AWS Secret Stores, for some customers.

Our team has confirmed a regression and is actively working on a fix. We will be rolling back the affected change to restore service for impacted environments.

We will continue monitoring closely and will provide additional updates as more information becomes available.
Posted Jan 28, 2026 - 17:12 UTC