API
Affected
Major outage from 11:30 PM to 12:58 AM
Major outage from 11:30 PM to 12:58 AM
Major outage from 11:30 PM to 12:58 AM
- Postmortem · UTC
Summary
Some requests to our platform failed with server errors. Affected features included runner registration, SDK license check-out, documentation, and session refresh. No customer data was lost.
What happened
During routine infrastructure maintenance, some of our compute nodes were replaced with nodes running a newer operating system image. On that image, the local NVMe storage devices are detected in a different order at startup. One of our startup scripts assumed a specific device identifier, so it failed to set up local storage on the new nodes. Without that storage, an internal service that our platform depends on couldn't start, and requests that relied on it failed.
Resolution
We corrected the storage setup on the affected nodes. The service restarted, and error rates returned to normal.
Preventing a recurrence
- The startup script now identifies storage devices by type instead of by device name, so device order no longer matters.
- We've added an investigation guide to our existing error-rate alerting so engineers can diagnose this kind of failure faster.
- We're reviewing how node replacements are rolled out, so that critical services stay available while nodes are being replaced. - Resolved · UTC
This incident has been resolved.
- Monitoring · UTCWe implemented a fix and are currently monitoring the result.
- Identified · UTC
We are continuing to work on a fix for this incident.
- Investigating · UTCWe are currently investigating this incident.

