A fix has been implemented and we are monitoring the results.
Posted Jul 21, 2021 - 22:07 UTC
Update
We have checked in a fix and expect the delay to be resolved in the next 40 minutes. This is only impacting a small portion of the traffic (roughly 1%).
Posted Jul 21, 2021 - 21:56 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Jul 21, 2021 - 21:36 UTC
Investigating
A degradation in the performance of the Splunk APM trace processing pipeline is causing Troubleshooting MetricSets to be delayed by more than five minutes. As a result, the APM Troubleshooting experience, service maps and Tag Spotlight do not have access to the most recent data.
The processing of metrics for Business Workflows, which also depends on this pipeline, are equally delayed. Trace data ingest is not impacted at this time; service-level and endpoint-level Monitoring MetricSets and the detectors built from them are also not impacted.
Posted Jul 21, 2021 - 21:30 UTC
This incident affected: Splunk APM (Splunk APM Troubleshooting MetricSets, Splunk APM Trace Data, Splunk APM Business Workflows).