Quadric - API – Incident details

All systems operational

API

ResolvedMajor outage1 hour 27 minutes

Affected

API

Major outage from 11:30 PM to 12:58 AM

CLI

Major outage from 11:30 PM to 12:58 AM

Licensing

Major outage from 11:30 PM to 12:58 AM

  • PostmortemUTC

    Summary
    Some requests to our platform failed with server errors. Affected features included runner registration, SDK license check-out, documentation, and session refresh. No customer data was lost.


    What happened
    During routine infrastructure maintenance, some of our compute nodes were replaced with nodes running a newer operating system image. On that image, the local NVMe storage devices are detected in a different order at startup. One of our startup scripts assumed a specific device identifier, so it failed to set up local storage on the new nodes. Without that storage, an internal service that our platform depends on couldn't start, and requests that relied on it failed.


    Resolution
    We corrected the storage setup on the affected nodes. The service restarted, and error rates returned to normal.


    Preventing a recurrence
    - The startup script now identifies storage devices by type instead of by device name, so device order no longer matters.
    - We've added an investigation guide to our existing error-rate alerting so engineers can diagnose this kind of failure faster.
    - We're reviewing how node replacements are rolled out, so that critical services stay available while nodes are being replaced.

  • ResolvedUTC

    This incident has been resolved.

  • MonitoringUTC
    We implemented a fix and are currently monitoring the result.
  • IdentifiedUTC

    We are continuing to work on a fix for this incident.

  • InvestigatingUTC
    We are currently investigating this incident.