Final Update: Friday, 29 January 2016 04:37 UTC
We've confirmed that all systems are back to normal with no customer impact as of 01/29, 04:20 UTC. Our logs show the incident started on 01/28, 23:08 UTC and that during the 5 Hours 28 minutes that it took to resolve the issue 10% of customers experienced data gaps for multiple data types.
-Application Insights Service Delivery Team
We've confirmed that all systems are back to normal with no customer impact as of 01/29, 04:20 UTC. Our logs show the incident started on 01/28, 23:08 UTC and that during the 5 Hours 28 minutes that it took to resolve the issue 10% of customers experienced data gaps for multiple data types.
- Root Cause: The failure was due to an exception in Application Insights pipeline.
- Lessons Learned: Hot fix has been deployed to handle exceptions of this kind and full RCA regarding these exception is going on.
- Incident Timeline: 5 Hours 28 minutes - 01/28, 23:30 UTC through 01/29, 04:20 UTC
-Application Insights Service Delivery Team
Update: Friday, 29 January 2016 02:47 UTC
Root cause is not fully understood at this moment however as part of mitigation we have applied a hotfix to catch up with backlog data. Processing of backlog data may take few more hours to complete, till then very small subset of customers may continue to experience gaps in thier historic data.
Root cause is not fully understood at this moment however as part of mitigation we have applied a hotfix to catch up with backlog data. Processing of backlog data may take few more hours to complete, till then very small subset of customers may continue to experience gaps in thier historic data.
- Work Around: none
- Next Update: Before 01/29 07:00 UTC
Initial Update: Thursday, 28 January 2016 23:08 UTC
We are aware of issues within Application Insights and are actively investigating. Some customers may experience data gaps while viewing the telemetry data. The following data types are affected: Availability,Customer Event,Dependency,Exception,Metric,Page Load,Page View,Performance Counter,Request,Trace. Recent data is processed successfully without any issue however data for some old time period needs to be back filled.
We are aware of issues within Application Insights and are actively investigating. Some customers may experience data gaps while viewing the telemetry data. The following data types are affected: Availability,Customer Event,Dependency,Exception,Metric,Page Load,Page View,Performance Counter,Request,Trace. Recent data is processed successfully without any issue however data for some old time period needs to be back filled.
- Work Around: None
- Next Update: Before 01/29 03:30 UTC
-Application Insights Service Delivery Team