Final Update: Sunday, 23 August 2015 21:30 UTC
The job agents have caught up. We will have to engage in some manual processes until a permanent resolution can be applied but we should be able to avoid any future impact to our customers from this issue.
We apologize for any inconvenience this issue has caused.
Sincerely,
VS Online Service Delivery Team
Update: Sunday, 23 August 2015 20:00 UTC
We have identified the jobs that are causing our job agents to hang, and disabled them. This should stabilize the job agents. Unfortunately there is still a significant backlog of jobs to execute. We hope to have them cleared out within 2 hours.
We'll be analyzing the problematic jobs to determine why they are hanging.
Next Update: 22:00 UTC or earlier if the backlog is worked off
Sincerely,
VS Online Service Delivery Team
Update: Sunday, 23 August 2015 15:45 UTC
The job agents appear to have stopped processing around 02:32 AM UTC. The job agents have been dumped and restarted. Jobs are now processing. Due to the backlog of higher priority jobs it may take a little time for the system to get to the build jobs.
We will have our developers review the dumps for RCA.
Next Update: Before 19:30 UTC or earlier if the backlog is worked off.
Sincerely,
VS Online Service Delivery Team
Initial Update: Sunday, 23 August 2015 16:14 UTC
We are actively investigating issues with queuing Build v.Next builds (Hosted and On Prem) in South Central US. The issue appears to be limited to this scale unit and build pool.
• Work Around: None currently.
• Next Update: Before 17:30 UTC
We are working to resolve this issue and apologize for any inconvenience.
Sincerely,
VS Online Service Delivery Team