Final Update: Thursday, 4 February 2016 15:08 UTC
Our DevOps team has rolled out a full mitigation plan to help unblock our users. At this time, full functionality has been restored to the service.
SUMMARY OF IMPACT: Between 09:10 and 14:28 UTC on 04 Feb 2016, customers attempting to log into their Visual Studio Team Services accounts will have been unable to access their accounts.
PRELIMINARY ROOT CAUSE: A SQL stored procedure that was being called was allocating too much memory in one of the critical backend SQL databases. After an extended period of time, this caused the SQL databases to fall into an unresponsive state and resulted in customers being unable to access their VSTS accounts.
MITIGATION: Engineers attempted to failover the SQL database which allowed for temporary mitigation, however the same procedure was quickly allocating memory to the newly assigned databases, which in turn became unresponsive. Engineers manually assigned allocation limits for the procedure that was being called. This has ensured the backend SQL databases remain in a healthy state.
We sincerely apologize for the inconvenience that this has caused our customers.
VS Team Services Service Delivery Team
--------------------
Update: Thursday, 4 February 2016 13:33 UTC
Our DevOps team has rolled out a partial mitigation plan to help unblock our users. At this time, the following users will be able to access their accounts:
- VS Team Services Accounts that leverage Microsoft Accounts/Windows Live Sign In Credentials are fully unblocked at this time
- VS Team Services Accounts that are backed by Azure Active Directory Sign In Accounts provided their federated authentication tokens are still valid.
New account creations on VS Team Services will not work at this time. Our DevOps team is continuing to actively work on full mitigation options.
We sincerely apologize for the inconvenience that this is causing our customers.
VS Team Services Service Delivery Team
--------------------
Update: Thursday, 4 February 2016 11:40 UTC
Our DevOps team is continuing to actively investigate the root cause of the sign in issues that some of our customers are experiencing currently. We sincerely apologize for the inconvenience that this is causing our customers.
VS Team Services Service Delivery Team
--------------------
Initial Update: Thursday, 4 February 2016 09:30 UTC
We are actively investigating issues with VS Team Services. Subset of customers get Internal Server Error while trying to sign in.
We are working to resolve this issue and apologize for any inconvenience.
Sincerely,
VS Team Services Service Delivery Team