Quadric - API – Incident details

All systems operational

API

ResolvedMajor outage49 minutes

Affected

API

Major outage from 7:58 PM to 8:47 PM

CLI

Operational from 7:58 PM to 8:43 PM, Major outage from 8:43 PM to 8:47 PM

Licensing

Operational from 7:58 PM to 8:43 PM, Major outage from 8:43 PM to 8:47 PM

  • PostmortemUTC

    On September 30, 2026, between approximately 12:09 and 13:38 PDT, requests to our API that required database access failed. Requests not dependent on the database were unaffected. No data was lost or corrupted. Service was fully restored by 13:38 PDT, and we confirmed normal error rates by 13:54 PDT.

    What happened

    A brief communication interruption between two nodes in our orchestration control plane caused the platform to incorrectly conclude that several infrastructure services had become unhealthy. The platform automatically restarted those services on other hosts, which is the intended recovery behavior.

    During that recovery, the database connection layer was placed on a host where it could not bind to its expected network address and instead started with an alternate configuration. The API service came back online and passed its health checks, but was unable to reach the database through this misconfigured layer. As a result, database-backed requests returned errors while the service itself appeared healthy.

    Our caching layer was also briefly affected by the same event. It recovered automatically within about one minute and did not contribute to the customer-facing impact.

    Why it took time to resolve

    Our alerting monitored service reachability but did not monitor elevated error rates. Because the API was reachable and responding, the health checks remained green and no alert was raised. The issue was identified manually about 80 minutes after it began. Once identified, the fix was applied within a few minutes: the database connection layer was moved to a suitable host and the API was restarted to pick up the corrected address.


    What we are doing about it

    • Adding error-rate alerting so that elevated API failures trigger an alert within minutes, independent of service reachability.

    • Preventing the placement conflict that allowed the database connection layer to start with an unusable network configuration.

    • Making the API resilient to changes in the database connection layer's address without requiring a restart.

    • Reviewing control plane timeouts so that a short communication interruption does not trigger unnecessary service restarts.

    • Fixing a storage volume reattachment race that slowed the caching layer's recovery.

  • ResolvedUTC

    This incident has been resolved.

  • MonitoringUTC
    We implemented a fix and are currently monitoring the result.
  • IdentifiedUTC

    We are continuing to work on investigating the root cause of the issue.

  • InvestigatingUTC
    We are currently investigating this incident.