Resolved -
This incident has been resolved.
Jul 11, 11:09 PDT
Update -
We continue to investigate delayed inclusion of newly created time series in detectors in US0. New time series may take up to 10 minutes to be included in detectors, which may delay detector evaluations for those time series. Notification delay mean time remains healthy, and no data loss has been observed. Engineering teams are performing a controlled rollout of affected backend components as part of ongoing mitigation work, and this is not expected to introduce additional customer impact. We will provide another update as validation progresses.
Jul 11, 08:05 PDT
Update -
We continue to investigate delayed inclusion of newly created time series in detectors in US0. New time series may take up to 10 minutes to be included in detectors, which may delay detector evaluations for those time series. Notification delay mean time remains healthy, and no data loss has been observed. Engineering teams are performing a controlled rollout of affected backend components as part of ongoing mitigation work, and this is not expected to introduce additional customer impact. We will provide another update as validation progresses.
Jul 11, 04:09 PDT
Update -
We continue to investigate delayed inclusion of newly created time series in detectors in US0. New time series may take up to 10 minutes to be included in detectors, which may delay detector evaluations for those time series. Notification delay mean time remains healthy, and no data loss has been observed. Engineering teams are performing a controlled rollout of affected backend components as part of ongoing mitigation work, and this is not expected to introduce additional customer impact. We will provide another update as validation progresses.
Jul 10, 23:54 PDT
Update -
We have identified that newly created time series in US0 may take up to 10 minutes to be included in detectors, which may delay detector evaluations for those new time series. Notification delay mean time is currently healthy, and no data loss has been observed. Engineering teams are continuing mitigation work and validating whether it clears the remaining delays. We will provide another update once validation is complete.
Jul 10, 20:03 PDT
Identified -
We have applied mitigations that reduced the impact, but recovery is not yet complete. A small percentage of detectors in US0 may still not be firing as expected due to delayed processing. No data loss has been observed. Engineering teams continue to investigate towards the additional mitigations. We will provide another update as more information becomes available.
Jul 10, 19:15 PDT
Update -
We are observing continued recovery and are monitoring the environment to ensure the mitigation is fully effective. No data loss has been observed. We will provide another update once recovery is complete.
Jul 10, 17:16 PDT
Update -
We are observing continued recovery and are monitoring the environment to ensure the mitigation is fully effective. No data loss has been observed. We will provide another update once recovery is complete.
Jul 10, 16:06 PDT
Monitoring -
We are observing continued recovery and are monitoring the environment to ensure the mitigation is fully effective. No data loss has been observed. We will provide another update once recovery is complete.
Jul 10, 15:09 PDT
Update -
We are observing continued recovery and are monitoring the environment to ensure the mitigation is fully effective. No data loss has been observed. We will provide another update once recovery is complete.
Jul 10, 13:53 PDT
Identified -
A mitigation has been applied for the issue affecting a small percentage of detectors in US0 that may not have fired as expected. We are observing continued recovery and are monitoring the environment to ensure the mitigation is fully effective. No data loss has been observed. We will provide another update once recovery is complete.
Jul 10, 12:39 PDT
Update -
We continue to investigate an issue affecting a small percentage of detectors in US0 that may not have fired as expected. No data loss has been observed. Our engineering teams are actively investigating the issue, and we will provide another update as more information becomes available.
Jul 10, 12:02 PDT
Update -
We continue to investigate an issue affecting a small percentage of detectors in US0 that may not have fired as expected. No data loss has been observed. Our engineering teams are actively investigating the issue, and we will provide another update as more information becomes available.
Jul 10, 11:01 PDT
Update -
We continue to investigate an issue affecting a small percentage of detectors in US0 that may not have fired as expected. No data loss has been observed. Our engineering teams are actively investigating the issue, and we will provide another update as more information becomes available.
Jul 10, 09:53 PDT
Update -
We continue to investigate an issue affecting a small percentage of detectors in US0 that may not have fired as expected. No data loss has been observed. Our engineering teams are actively investigating the issue, and we will provide another update as more information becomes available.
Jul 10, 08:52 PDT
Investigating -
We have seen a return of a small percentage of detectors that may not be firing. No data is lost. We are continuing to investigate.
Jul 10, 07:52 PDT
Update -
We are continuing to monitor for any further issues.
Jul 10, 06:51 PDT
Update -
We are continuing to monitor for any further issues.
Jul 10, 05:47 PDT
Monitoring -
A fix has been implemented and we are monitoring the results.
Jul 10, 05:05 PDT
Investigating -
Starting at 1a PDT, a small percentage of detectors may have missed firing. No data has been lost. We are investigating and will provide an update shortly.
Jul 10, 05:03 PDT