Problems Running Tests When Preserving Output Directory Structure with Team Build

Previously I’ve posted about how to override the Team Build default output directory scheme and produce something a bit more sane.

Unfortunately if you do implement this it can break the built-in test run task, and most of the recipes related to it. You’ll get the following error in your build logs:

MSBUILD : warning MSB6003: The specified task executable "MSTest.exe" could not be run. The directory name is invalid

If you run the build with /verbosity:detailed to see the actual args passed to MSTest.exe, and then run MSTest yourself interactively, you’ll see the real underlying error:

Directory "(my build path)\Binaries\Debug" not found.
For switch syntax, type "MSTest /help"

The problem here is that (as detailed on the TestToolsTask doco) the team foundation build targets sets up MSTest.exe with SearchPathRoot="$(OutDir)", ie $(BinariesRoot)\$(Configuration). But if you overrode CustomizableOutDir and never actually copied the binaries out to the output folder that directory will never get created.

Fix 1:

If you’re not really using CustomizableOutDir, remove it. Reverting to the default Team Build directory structure is the simplest way of getting the tests to be located and executed and everything to ‘play nice’.

Fix 2:

Make sure that if your TFBuild.proj says CustomizableOutDir you do actually have the corresponding custom tasks in the individual projects to copy the binaries (see my previous post), otherwise you end up with no output whatsoever, and the test task will fail.

Fix 3:

If you want CustomizableOutDir but want to be robust to the possibility that your project builds may not populate the output directory structures properly, you can hack your build to run the tests out of the source \bin\debug folders.

My first pass was just to add the following to my BeforeTestConfiguration target (that I’d added from the Running Unit Tests without a Test List recipie):

    <!--because this is what the TestTask gets its SearchPath set to, it must exist-->

    <MakeDir Directories="$(OutDir)"/>

But that wasn’t good enough on its own, because now the error was:

CoreTestConfiguration:
File "..\..\(blah)\bin\Debug\Assembly.UnitTests.dll" not found

The relative paths to the test assemblies were correct relative to the $(SolutionDir), but not relative to the $(OutDir). So, for want of a better answer, I just overwrite OutDir for the duration of the test task:

   <!—defined elsewhere-->

   <TestsToRun Include="$(SolutionRoot)\%2a%2a\bin\$(Configuration)\%2a.UnitTests.dll" />

 

  <Target Name="BeforeTestConfiguration">

    <!-- normal bits as per the recipe-->

    <Message Text="Using tests from @(TestsToRun)" Condition=" '$(IsDesktopBuild)'=='true' " />

 

    <CreateItem Include="@(TestsToRun)">

      <Output TaskParameter="Include" ItemName="LocalTestContainer"/>

      <Output TaskParameter="Include" ItemName="TestContainer"/>

    </CreateItem>

 

    <Message Text="LocalTestContainer: @(LocalTestContainer)" Condition=" '$(IsDesktopBuild)'=='true' " />

 

    <!--Fix to allow use of CustomizableOutDir -->

    <MakeDir Directories="$(OutDir)"/>

    <PropertyGroup>

      <OldOutDir>$(OutDir)</OldOutDir>

      <OutDir>$(SolutionDir)</OutDir>

    </PropertyGroup>

  </Target>

 

  <Target Name="AfterTestConfiguration">

    <PropertyGroup>

     <OutDir>$(OldOutDir)</OutDir>

    </PropertyGroup>

  </Target>

Whether this is a good idea or not I’m not sure, but it does seem to work. Note that I put it back the way it was afterwards (using AfterTestConfiguration).

Moral

I think the story here is that using CustomizableOutDir is a complete pain in the arse, which ends up requiring considerable customisation of the rest of the build workflow. I don’t mind a prescriptive process per-se, but I do have a real issue with the ‘flat’ output directory structure that Team Build kicks out. But attempting to change it just seems to cause a heap more trouble than it’s worth.

Actually - as Martin Fowler said years ago - using XML as a build language is a really dumb idea in retrospect. Everyone says TeamCity’s pretty cool: might be time to take a look at that…

 

PS: If you’re trying to get your head around what happens where in Team Build (aren’t we all) there’s a great Team Build Target Map over at the Accentient blog

PS: I notice on Aaron Hallberg’s blog there’s a much simpler approach if you just want to separate per-solution output directory structures, which may not suffer the same problems.