Final Update: Wednesday, 9/23/2015 08:25 UTC
We’ve confirmed that all systems are back to normal with no customer impact as of 9/23, 7:00 UTC. Our logs show the incident started on 9/23/2015 01:30 UTC and during the issue period customers experienced latency for Trace and Events.
• Root Cause: Our processing pipeline was impacted by additional load from data forking service. We stopped data forking service to mitigate the issue.
We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.
-Application Insights Service Delivery Team
Update: Wednesday, 9/23/2015 06:06 UTC
Root cause has been isolated to one of our processing backend service which was having issue for read and write operations causing customer data latency for Trace and Event data types. We have stopped a telemetry service which was causing additional load to processing pipeline. Lag is significantly going down , however some customers may still experience latency for few more hours.
• Work Around: none
• Next Update: Before 09/23/2015 12:00 UTC
-Application Insights Service Delivery Team
Update: Wednesday, 9/23/2015 04:23 UTC
DevOps team continues to investigate issues within Application Insights. Root cause is not fully understood at this time. Some customers continue to experience latency. We currently have no estimate for resolution.
• Next Update: Before 7:00 UTC
-Application Insights Service Delivery Team
Initial Update: Wednesday, 9/23/2015 02:15 UTC
We are aware of issues within Application Insights and are actively investigating. Some customers may experience Data Latency. The following data types are affected: Customer Event, Trace.
• Work Around: None
• Next Update: Before 4:00 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Application Insights Service Delivery Team