Integrating Testing into the CI and CD pipelines

Testing in Continuous Integration and Continuous Deployment Workflows

At a high level DevOps is a focus on increasing customer value by reducing cycle times in the development process.  As a result of these faster cycle times testing as part of the DevOps workflow can be challenge. This section will look at integrating testing into CI/CD pipelines using to ensure less friction in the testing process and a high quality application.   While it probably makes sense to run these tests as part of the CD pipeline at the time of this authoring that is not working so we will implement this as part of the CI Build process.

Step 1.Creating a machine group.  In the Test menu, click on the machines menu.  Click the Action button (Green Plus) to create new machine group.  If we are running low on time consider reusing the Xlab Machine Group that already exists. Even if you do decide to reuse the the preexisting Machine Group as a corner stone of the Testing CI offering please at least open the Test hub and the “Machines” menu.  Select the Test CI machine group and select edit and take a look at the definition.  

Step 2.  Adding Machines to a machine group. To enable testers to very easily run the same test against multiple machines we can add several computers to a machine group.  For user name type in your domain credentials in the DOMAIN/NAME syntax.  For The machine name, please enter an IP Address or FQDN i.e. 4TestCI1.redmond.corp.microsoft.com

There several caveats for computers to be access with this early version of the Test CI agent and workflow.

1. We do not have the support for test machines residing in Azure yet. However this is coming soon.
2. We do not support the Hosted Build controller.
3. The build agent and test agent machines should be on-premises machines and the build machine should be able to communicate with the target test machine.
4. The build agent being used for the Test CI tasks must be domain joined
5. The computer being added the machine group must be domain joined
6. Need to enable winrm on your test agent machine (run winrm quickconfig to enable this)
7. We do not support use of the “hosted” build pool for the Test CI tasks
8. Do not install test agents for visual studio 2015 on a machine with VS2015RC.  There is known issue here that we have fixed for RTM.

Step 3. Edit the Build definition to Add the Test CI Tasks to the CI Pipeline.

Step 4. Add the Test CI tasks

While this is shown in the CD pipleine

Step 5 Configure the Deploy Agent step

Step 6 Configure the Windows Test Agent Deployment

$(Build.Repository.LocalPath)

Step 7 Configure the run with Agent

Step 8 Running the build

Step 9 Monitoring the execution