Final Update: Tuesday, 9/29/2015 06:11 UTC
We have confirmed that all systems are back to normal with no customer impact as of 9/29, 05:47 UTC. Our logs show the incident started on 9/28, 22:24 UTC and that during the approximately 7.5 hours that it took to resolve the issue some customers experienced were unable to create new Application Insights applications.
• Root Cause: The failure was due to performance issues with our back-end database.
• Lessons Learned: We are deploying additional telemetry to assist with a faster diagnosis, should we ever see this failure mode again. We are also investigating several optimizations to our database configuration to improve overall performance.
• Incident Timeline: 7 Hours & 23 minutes - 9/28, 22:24 UTC through 9/29, 05:47 UTC
We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.
-Application Insights Service Delivery Team
Update: Tuesday, 9/29/2015 04:47 UTC
Our DevOps team continues to investigate issues within Application Insights. Root cause is not fully understood at this time, but we are actively engaged and working on mitigating the issue. Some customers continue to experience issues creating new Application Insights applications.
• Work Around: none
• Next Update: Before 07:00 UTC
-Application Insights Service Delivery Team
Update: Tuesday, 9/29/2015 02:27 UTC
Our DevOps team continues to investigate issues within Application Insights. Root cause is not fully understood at this time. Some customers continue to experience issues creating new Application Insights applications. We are working to establish the start time for the issue, initial findings indicate that the problem began at 9/28, 22:24 UTC. We currently have no estimate for resolution.
• Work Around: none
• Next Update: Before 9/29, 04:00 UTC
-Application Insights Service Delivery Team
We have confirmed that all systems are back to normal with no customer impact as of 9/29, 05:47 UTC. Our logs show the incident started on 9/28, 22:24 UTC and that during the approximately 7.5 hours that it took to resolve the issue some customers experienced were unable to create new Application Insights applications.
• Root Cause: The failure was due to performance issues with our back-end database.
• Lessons Learned: We are deploying additional telemetry to assist with a faster diagnosis, should we ever see this failure mode again. We are also investigating several optimizations to our database configuration to improve overall performance.
• Incident Timeline: 7 Hours & 23 minutes - 9/28, 22:24 UTC through 9/29, 05:47 UTC
We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.
-Application Insights Service Delivery Team
Update: Tuesday, 9/29/2015 04:47 UTC
Our DevOps team continues to investigate issues within Application Insights. Root cause is not fully understood at this time, but we are actively engaged and working on mitigating the issue. Some customers continue to experience issues creating new Application Insights applications.
• Work Around: none
• Next Update: Before 07:00 UTC
-Application Insights Service Delivery Team
Update: Tuesday, 9/29/2015 02:27 UTC
Our DevOps team continues to investigate issues within Application Insights. Root cause is not fully understood at this time. Some customers continue to experience issues creating new Application Insights applications. We are working to establish the start time for the issue, initial findings indicate that the problem began at 9/28, 22:24 UTC. We currently have no estimate for resolution.
• Work Around: none
• Next Update: Before 9/29, 04:00 UTC
-Application Insights Service Delivery Team
Initial Update: Tuesday, 9/29/2015 00:15 UTC
We are aware of issues within Application Insights and are actively investigating. Some customers may experience issues when creating new Application Insights applications. Our logs indicate this issue began at approximately 9/28, 22:24 UTC.
• Work Around: none
• Next Update: Before 9/29, 02:00 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Application Insights Service Delivery Team