Charts and Detectors are Intermittently Failing to Load

Incident Report for Splunk Observability Cloud US0

Resolved

This incident has been resolved.
Posted Mar 19, 2026 - 12:37 PDT

Update

We have successfully reverted the rate limits to their previous settings and continue to monitor for any issues.
Posted Mar 19, 2026 - 12:32 PDT

Update

We continue to monitor as we slowly reduce the implemented rate limits that partially mitigated the impact to ensure the environment remains stable.
Posted Mar 19, 2026 - 11:57 PDT

Monitoring

The fix has been implemented and we are monitoring the results.
Posted Mar 19, 2026 - 11:25 PDT

Update

Engineering teams continue to apply the following mitigation actions: Fine tuning internal query routing to healthy MTS Store Availability Zones with capacity and limiting expensive and high volume customer queries in order to bring general stability and support the success of scale-out operations.
Posted Mar 19, 2026 - 11:08 PDT

Identified

Charts and Detectors are intermittently failing. Infrastructure monitoring might also see impact. Datapoint Ingest is not impacted. The issue has been identified and a mitigation fix is being implemented.
Posted Mar 19, 2026 - 10:28 PDT

Investigating

Charts and Detectors are intermittently failing. Datapoint ingest is not affected. We are currently investigating.
Posted Mar 19, 2026 - 09:52 PDT
This incident affected: Splunk APM (Splunk APM Interface) and Splunk Observability Cloud Web Interface.