Quantcast
Channel: MSDN Blogs
Viewing all articles
Browse latest Browse all 12366

Experiencing Data Latency for Many Data Types - 9/23 - Resolved

$
0
0

Final Update: Wednesday, 9/23/2015 08:25 UTC

We’ve confirmed that all systems are back to normal with no customer impact as of 9/23, 7:00 UTC. Our logs show the incident started on 9/23/2015 01:30 UTC and during the issue period customers experienced latency for Trace and Events. 

Root Cause: Our processing pipeline was impacted by  additional load from data forking service. We stopped data forking service to mitigate the issue.

We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Application Insights Service Delivery Team


Update: Wednesday, 9/23/2015 06:06 UTC

Root cause has been isolated to  one of our processing backend service  which was having issue for read and write operations causing  customer data latency for Trace and Event data types. We have stopped a telemetry service which was causing additional load to processing pipeline. Lag is significantly going down , however some customers may still experience latency for few more hours.  

Work Around: none
Next Update: Before  09/23/2015 12:00 UTC

-Application Insights Service Delivery Team


Update: Wednesday, 9/23/2015 04:23 UTC

DevOps team continues to investigate issues within Application Insights. Root cause is not fully understood at this time. Some customers continue to experience latency. We currently have no estimate for resolution.

Next Update: Before 7:00 UTC

-Application Insights Service Delivery Team


Initial Update: Wednesday, 9/23/2015 02:15 UTC

We are aware of issues within Application Insights and are actively investigating. Some customers may experience Data Latency. The following data types are affected: Customer Event, Trace.

Work Around: None
Next Update: Before 4:00 UTC 

We are working hard to resolve this issue and apologize for any inconvenience.

-Application Insights Service Delivery Team
 

 

 

Viewing all articles
Browse latest Browse all 12366

Trending Articles