Final Update: Saturday, 10/3/2015 14:08 UTC
We have confirmed that all systems are back to normal with no customer impact as of approximately 10/3, 09:00 UTC. Our logs show the incident started on 9/29, 23:30 UTC and that during the 81.5 hours that it took to resolve the issue a small number of customers would have seen no data returned for custom metrics when filtering or grouping by custom dimensions.
• Root Cause: The failure was due to a bug in a deployment script that caused an issue with the way the metadata for custom metrics was generated.
• Lessons Learned: We are investigating improvements to monitoring and deployment technologies to prevent further recurrences of this same issue.
• Incident Timeline: 81 Hours & 30 minutes - 9/29, 23:30 UTC through 10/3, 09:00 UTC
We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.
-Application Insights Service Delivery Team
Update: Saturday, 10/3/2015 06:11 UTC
The repair script continues to run without error, but is taking longer than first estimated. Until this script is finished, a very small subset of customers may see no data returned when filtering or grouping data by custom dimension in the Azure portal and we estimate several hours before all access issues are addressed.
• Work Around: none
• Next Update: Before 14:30 UTC
-Application Insights Service Delivery Team
Update: Saturday, 10/3/2015 03:54 UTC
Root cause has been isolated to a bug introduced with an update on approximately 9/29, 23:30 UTC which was impacting custom metadata. To address this issue we are executing a repair script to restore the affected metadata. A very small subset of customers may see no data returned when filtering or grouping data by custom dimension in the Azure portal and we estimate several hours before all access issues are addressed.
• Work Around: none
• Next Update: Before 06:00 UTC
-Application Insights Service Delivery Team
Update: Saturday, 10/3/2015 02:59 UTC
The script regenerating the metadata for custom dimensions is still running. Most customers are repaired at this time, but a very small subset of impacted customers may still see the symptoms described below.
• Work Around: none
• Next Update: Before 04:00 UTC
-Application Insights Service Delivery Team
![]()
We have confirmed that all systems are back to normal with no customer impact as of approximately 10/3, 09:00 UTC. Our logs show the incident started on 9/29, 23:30 UTC and that during the 81.5 hours that it took to resolve the issue a small number of customers would have seen no data returned for custom metrics when filtering or grouping by custom dimensions.
• Root Cause: The failure was due to a bug in a deployment script that caused an issue with the way the metadata for custom metrics was generated.
• Lessons Learned: We are investigating improvements to monitoring and deployment technologies to prevent further recurrences of this same issue.
• Incident Timeline: 81 Hours & 30 minutes - 9/29, 23:30 UTC through 10/3, 09:00 UTC
We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.
-Application Insights Service Delivery Team
Update: Saturday, 10/3/2015 06:11 UTC
The repair script continues to run without error, but is taking longer than first estimated. Until this script is finished, a very small subset of customers may see no data returned when filtering or grouping data by custom dimension in the Azure portal and we estimate several hours before all access issues are addressed.
• Work Around: none
• Next Update: Before 14:30 UTC
-Application Insights Service Delivery Team
Update: Saturday, 10/3/2015 03:54 UTC
Root cause has been isolated to a bug introduced with an update on approximately 9/29, 23:30 UTC which was impacting custom metadata. To address this issue we are executing a repair script to restore the affected metadata. A very small subset of customers may see no data returned when filtering or grouping data by custom dimension in the Azure portal and we estimate several hours before all access issues are addressed.
• Work Around: none
• Next Update: Before 06:00 UTC
-Application Insights Service Delivery Team
Update: Saturday, 10/3/2015 02:59 UTC
The script regenerating the metadata for custom dimensions is still running. Most customers are repaired at this time, but a very small subset of impacted customers may still see the symptoms described below.
• Work Around: none
• Next Update: Before 04:00 UTC
-Application Insights Service Delivery Team
Initial Update: Friday, 10/2/2015 23:44 UTC
We are aware of issues with custom data dimensions in Application Insights and are taking steps to mitigate the impact. A small subset of customers may see no data returned when filtering or grouping data by custom dimensions in the Azure Portal. This data exists, but is not being returned properly after filtering/grouping.
To address this issue, we are regenerating the metadata for the custom dimensions. We expect this effort to take several hours.
• Work Around: none
• Next Update: Before 10/3, 02:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Application Insights Service Delivery Team