Quadric - Notice history

All systems operational

API - Operational

99.89% uptime
Sep 2026 · 99.88%Oct · 100.0%Nov · 99.79%
Sep 2026
Oct 2026
Nov 2026

Runners - Operational

100.0% uptime
Sep 2026 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2026
Oct 2026
Nov 2026

Web app - Operational

RTL Generation - Operational

Operational

Licensing - Operational

SDK - Operational

Notice history

View current status

Nov 2026

API
ResolvedMajor outage1 hour 27 minutes
  • PostmortemUTC

    Summary
    Some requests to our platform failed with server errors. Affected features included runner registration, SDK license check-out, documentation, and session refresh. No customer data was lost.


    What happened
    During routine infrastructure maintenance, some of our compute nodes were replaced with nodes running a newer operating system image. On that image, the local NVMe storage devices are detected in a different order at startup. One of our startup scripts assumed a specific device identifier, so it failed to set up local storage on the new nodes. Without that storage, an internal service that our platform depends on couldn't start, and requests that relied on it failed.


    Resolution
    We corrected the storage setup on the affected nodes. The service restarted, and error rates returned to normal.


    Preventing a recurrence
    - The startup script now identifies storage devices by type instead of by device name, so device order no longer matters.
    - We've added an investigation guide to our existing error-rate alerting so engineers can diagnose this kind of failure faster.
    - We're reviewing how node replacements are rolled out, so that critical services stay available while nodes are being replaced.

  • ResolvedUTC

    This incident has been resolved.

  • MonitoringUTC
    We implemented a fix and are currently monitoring the result.
  • IdentifiedUTC

    We are continuing to work on a fix for this incident.

  • InvestigatingUTC
    We are currently investigating this incident.

Oct 2026

No notices reported this month

Sep 2026

API
ResolvedMajor outage49 minutes
  • PostmortemUTC

    On September 30, 2026, between approximately 12:09 and 13:38 PDT, requests to our API that required database access failed. Requests not dependent on the database were unaffected. No data was lost or corrupted. Service was fully restored by 13:38 PDT, and we confirmed normal error rates by 13:54 PDT.

    What happened

    A brief communication interruption between two nodes in our orchestration control plane caused the platform to incorrectly conclude that several infrastructure services had become unhealthy. The platform automatically restarted those services on other hosts, which is the intended recovery behavior.

    During that recovery, the database connection layer was placed on a host where it could not bind to its expected network address and instead started with an alternate configuration. The API service came back online and passed its health checks, but was unable to reach the database through this misconfigured layer. As a result, database-backed requests returned errors while the service itself appeared healthy.

    Our caching layer was also briefly affected by the same event. It recovered automatically within about one minute and did not contribute to the customer-facing impact.

    Why it took time to resolve

    Our alerting monitored service reachability but did not monitor elevated error rates. Because the API was reachable and responding, the health checks remained green and no alert was raised. The issue was identified manually about 80 minutes after it began. Once identified, the fix was applied within a few minutes: the database connection layer was moved to a suitable host and the API was restarted to pick up the corrected address.


    What we are doing about it

    • Adding error-rate alerting so that elevated API failures trigger an alert within minutes, independent of service reachability.

    • Preventing the placement conflict that allowed the database connection layer to start with an unusable network configuration.

    • Making the API resilient to changes in the database connection layer's address without requiring a restart.

    • Reviewing control plane timeouts so that a short communication interruption does not trigger unnecessary service restarts.

    • Fixing a storage volume reattachment race that slowed the caching layer's recovery.

  • ResolvedUTC

    This incident has been resolved.

  • MonitoringUTC
    We implemented a fix and are currently monitoring the result.
  • IdentifiedUTC

    We are continuing to work on investigating the root cause of the issue.

  • InvestigatingUTC
    We are currently investigating this incident.

Previous

Sep 2026 to Nov 2026

Next