New time series may be delayed in joining detectors in US0

Incident Report for Splunk Observability Cloud US0

Resolved

This incident has been resolved.
Posted Jul 11, 2026 - 11:09 PDT

Update

We continue to investigate delayed inclusion of newly created time series in detectors in US0. New time series may take up to 10 minutes to be included in detectors, which may delay detector evaluations for those time series. Notification delay mean time remains healthy, and no data loss has been observed. Engineering teams are performing a controlled rollout of affected backend components as part of ongoing mitigation work, and this is not expected to introduce additional customer impact. We will provide another update as validation progresses.
Posted Jul 11, 2026 - 08:05 PDT

Update

We continue to investigate delayed inclusion of newly created time series in detectors in US0. New time series may take up to 10 minutes to be included in detectors, which may delay detector evaluations for those time series. Notification delay mean time remains healthy, and no data loss has been observed. Engineering teams are performing a controlled rollout of affected backend components as part of ongoing mitigation work, and this is not expected to introduce additional customer impact. We will provide another update as validation progresses.
Posted Jul 11, 2026 - 04:09 PDT

Update

We continue to investigate delayed inclusion of newly created time series in detectors in US0. New time series may take up to 10 minutes to be included in detectors, which may delay detector evaluations for those time series. Notification delay mean time remains healthy, and no data loss has been observed. Engineering teams are performing a controlled rollout of affected backend components as part of ongoing mitigation work, and this is not expected to introduce additional customer impact. We will provide another update as validation progresses.
Posted Jul 10, 2026 - 23:54 PDT

Update

We have identified that newly created time series in US0 may take up to 10 minutes to be included in detectors, which may delay detector evaluations for those new time series. Notification delay mean time is currently healthy, and no data loss has been observed. Engineering teams are continuing mitigation work and validating whether it clears the remaining delays. We will provide another update once validation is complete.
Posted Jul 10, 2026 - 20:03 PDT

Identified

We have applied mitigations that reduced the impact, but recovery is not yet complete. A small percentage of detectors in US0 may still not be firing as expected due to delayed processing. No data loss has been observed. Engineering teams continue to investigate towards the additional mitigations. We will provide another update as more information becomes available.
Posted Jul 10, 2026 - 19:15 PDT

Update

We are observing continued recovery and are monitoring the environment to ensure the mitigation is fully effective. No data loss has been observed. We will provide another update once recovery is complete.
Posted Jul 10, 2026 - 17:16 PDT

Update

We are observing continued recovery and are monitoring the environment to ensure the mitigation is fully effective. No data loss has been observed. We will provide another update once recovery is complete.
Posted Jul 10, 2026 - 16:06 PDT

Monitoring

We are observing continued recovery and are monitoring the environment to ensure the mitigation is fully effective. No data loss has been observed. We will provide another update once recovery is complete.
Posted Jul 10, 2026 - 15:09 PDT

Update

We are observing continued recovery and are monitoring the environment to ensure the mitigation is fully effective. No data loss has been observed. We will provide another update once recovery is complete.
Posted Jul 10, 2026 - 13:53 PDT

Identified

A mitigation has been applied for the issue affecting a small percentage of detectors in US0 that may not have fired as expected. We are observing continued recovery and are monitoring the environment to ensure the mitigation is fully effective. No data loss has been observed. We will provide another update once recovery is complete.
Posted Jul 10, 2026 - 12:39 PDT

Update

We continue to investigate an issue affecting a small percentage of detectors in US0 that may not have fired as expected. No data loss has been observed. Our engineering teams are actively investigating the issue, and we will provide another update as more information becomes available.
Posted Jul 10, 2026 - 12:02 PDT

Update

We continue to investigate an issue affecting a small percentage of detectors in US0 that may not have fired as expected. No data loss has been observed. Our engineering teams are actively investigating the issue, and we will provide another update as more information becomes available.
Posted Jul 10, 2026 - 11:01 PDT

Update

We continue to investigate an issue affecting a small percentage of detectors in US0 that may not have fired as expected. No data loss has been observed. Our engineering teams are actively investigating the issue, and we will provide another update as more information becomes available.
Posted Jul 10, 2026 - 09:53 PDT

Update

We continue to investigate an issue affecting a small percentage of detectors in US0 that may not have fired as expected. No data loss has been observed. Our engineering teams are actively investigating the issue, and we will provide another update as more information becomes available.
Posted Jul 10, 2026 - 08:52 PDT

Investigating

We have seen a return of a small percentage of detectors that may not be firing. No data is lost. We are continuing to investigate.
Posted Jul 10, 2026 - 07:52 PDT

Update

We are continuing to monitor for any further issues.
Posted Jul 10, 2026 - 06:51 PDT

Update

We are continuing to monitor for any further issues.
Posted Jul 10, 2026 - 05:47 PDT

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Jul 10, 2026 - 05:05 PDT

Investigating

Starting at 1a PDT, a small percentage of detectors may have missed firing. No data has been lost. We are investigating and will provide an update shortly.
Posted Jul 10, 2026 - 05:03 PDT
This incident affected: Alerting.