API
Affected
Major outage from 7:58 PM to 8:47 PM
Operational from 7:58 PM to 8:43 PM, Major outage from 8:43 PM to 8:47 PM
Operational from 7:58 PM to 8:43 PM, Major outage from 8:43 PM to 8:47 PM
- Postmortem · UTC
On September 30, 2026, between approximately 12:09 and 13:38 PDT, requests to our API that required database access failed. Requests not dependent on the database were unaffected. No data was lost or corrupted. Service was fully restored by 13:38 PDT, and we confirmed normal error rates by 13:54 PDT.
What happened
A brief communication interruption between two nodes in our orchestration control plane caused the platform to incorrectly conclude that several infrastructure services had become unhealthy. The platform automatically restarted those services on other hosts, which is the intended recovery behavior.
During that recovery, the database connection layer was placed on a host where it could not bind to its expected network address and instead started with an alternate configuration. The API service came back online and passed its health checks, but was unable to reach the database through this misconfigured layer. As a result, database-backed requests returned errors while the service itself appeared healthy.
Our caching layer was also briefly affected by the same event. It recovered automatically within about one minute and did not contribute to the customer-facing impact.
Why it took time to resolve
Our alerting monitored service reachability but did not monitor elevated error rates. Because the API was reachable and responding, the health checks remained green and no alert was raised. The issue was identified manually about 80 minutes after it began. Once identified, the fix was applied within a few minutes: the database connection layer was moved to a suitable host and the API was restarted to pick up the corrected address.
What we are doing about itAdding error-rate alerting so that elevated API failures trigger an alert within minutes, independent of service reachability.
Preventing the placement conflict that allowed the database connection layer to start with an unusable network configuration.
Making the API resilient to changes in the database connection layer's address without requiring a restart.
Reviewing control plane timeouts so that a short communication interruption does not trigger unnecessary service restarts.
Fixing a storage volume reattachment race that slowed the caching layer's recovery.
- Resolved · UTC
This incident has been resolved.
- Monitoring · UTCWe implemented a fix and are currently monitoring the result.
- Identified · UTC
We are continuing to work on investigating the root cause of the issue.
- Investigating · UTCWe are currently investigating this incident.

