Resolved -
This incident has been resolved.
Oct 21, 06:18 UTC
Update -
In the last three hours we observed a single brief spike in 5xx responses around 20:45 UTC in the US region; since then, 5xx has held at 0% across monitored endpoints. For transparency, we'll keep the incident open in Monitoring with no components marked as affected through the end of the day while we continue to observe.
Oct 20, 22:01 UTC
Monitoring -
Update - Error rates are trending down. Over the last few minutes the US region is averaging about 0.4% 5xx responses. Core paths (Authentication, Viewing Content & Search, Self‑Service, and content edits) are each ≤0.5%, while Integrations and Insights & Analytics fluctuate slightly higher but remain under 1%. We'll keep mitigations in place and continue to monitor.
Oct 20, 20:08 UTC
Update -
Update - This remains a brownout, not a full outage. Most requests in the US region are succeeding, but ongoing upstream degradation is causing intermittent errors and latency. Authentication, Viewing Content & Search, Self‑Service Portals, Emails & Notifications, and Making Changes to Content are largely available; Integrations and Insights & Analytics are the most affected with higher delays/timeouts. We continue to mitigate and monitor.
Oct 20, 17:46 UTC
Identified -
Reopened. Our earlier resolution was premature. Our cloud infrastructure provider continues to report widespread degradation in their US‑East region (about 80 services affected, several in a brownout state). This is causing intermittent errors, elevated latency, and timeouts across multiple Shelf services in the US region, Answer Assist and Search, and integrations. EU and CA regions remain healthy. We are working with the provider and have applied mitigations to reduce impact. Next update within 30 minutes.
Oct 20, 16:13 UTC
Update -
We're still observing occasional timeouts in Agent Assist searches for certain real-time suggestions.
Oct 20, 15:35 UTC
Monitoring -
A fix has been implemented and we are monitoring the results.
Oct 20, 15:07 UTC
Identified -
The issue has been identified and a fix is being implemented.
Oct 20, 14:57 UTC
Monitoring -
A fix has been implemented and we are monitoring the results.
Oct 20, 14:26 UTC
Update -
We are continuing to investigate this issue.
Oct 20, 14:21 UTC
Investigating -
We are currently investigating this issue.
Oct 20, 14:16 UTC