The customer was having a rough day. They had lots of unit tests, but were seeing an unexpected behavior from Azure DevOps and SonarQube. Because some tests were written in XUnit and others with MSTest, they had separated the execution into two tasks in their build. At the end of the build, results were sent to SonarQube for analysis. The problem was that each time the second set of tests ran in Azure DevOps, the results from the first pass disappeared. The assumption was that the files were being overwritten by the second test run. The fix seemed simple – copy the results before the second set of tests executed.
Unfortunately, it wasn’t that easy – the code coverage files seemed to never actually copy. After tweaking the build tasks a bit, the coverage files were finally copying; unfortunately, the test results were now missing! Another team was having a similar problem, but running a single set of tests. When the SonarQube task ran, it would report that it could not find the coverage files OR the test results.
Taking advantage of the script task, we scanned the folders and examined the files within. In each case, the test files and coverage files were present. For some reason, however, they weren’t being seen. Putting a delay in the script unveiled an even stranger situation. A few seconds after the test ran, the files had been deleted. As a result, SonarQube could not find the files it needed and that task was failing to publish the code coverage, test results, or both.
The Case of the Missing Tests
It turns out there are a few things going on. First, SonarQube relies on specific folder structures to exist within $(Common.TestResultsDirectory). If the files aren’t in that location (and in the expected folder structure), SonarQube won’t be able to gather the results.
The root cause is more challenging. Even though that folder is documented as ‘The local path on the agent where the test results are created’, it turns out that is only partially true. In this issue, the team mentions that the drop location is not actually supported by the tasks, and the task cleans up the files as soon as they are uploaded to Azure DevOps. More challenging, the documented variable may actually be an implementation detail!
In other words, any success the customer had was actually a race condition! They were lucky enough to copy some of the files before they were deleted … by design.
So, it turns out the reason for this somewhat makes sense. The Common Test Results directory is not automatically cleaned up by the agent. For private agents especially, if the vstest task is cancelled or aborted, the results may be left behind. While the vstest task will clean out the folder before running, other tasks may not. As a result, it’s possible to end up with files from a previous run.
At the moment, the task places the files in $(Agent.TempDirectory)\TestResults. Because the agent guarantees that it wil always cleanup that directory, it eliminates the problem of files not being cleaned up. This gives us a few potential options.
We could copy the files from there into the expected location in the Common Test Results directory. That could potentially create a cleanup problem, but it enables SonarQube to work as expected.
We could also point either the .runsettings file or SonarQube to the new directory. The problem with the later approach is that this is an implementation detail. That means it could change in the future.
Additionally, the team stated the tasks leaves the files there until the next vstest task is called in the same build/release. That means the solution only works correctly if only run one vstest per build.
In the next post, I’ll dive into some Azure DevOps APIs that we can use to develop a more future-proof solution to the problem.