Continuous Integration in TFS 2008: Part 1

Many people now accept the benefits of a regular / continuous integration cycle (even if they don't actually practice it themselves). Picking up the pieces after someone's broken the checked-in source code, especially if it's not picked up for a few days, can be a real time waster.

Like many agile practices, however, the cost / benefit is hard to quantitatively analyse. It's far easier to justify therefore if it's really easy to setup: as the costs tend to zero the benefits become essentially 'free'. And you could argue that tools like CruiseControl.Net have made it pretty easy.

Personally, having spent significant sections of the last 3 years getting CCNet / Nant build cycles going on various projects, I'd beg to differ. Sure, it's really easy to setup CCNet / Nant (or CCNet / MSBuild) to build your solution, but that's only the first step in the process. Typically you also want to do things like:

• Import the latest built version of external dependencies (ie components maintained outside of the solution being built)
• Execute unit tests
• Execute integration tests (so config files pointing at databases etc... have to be in the right place)
• Package the build outputs nicely ('xcopy ready')
• Deploy and install into test environments

CCNet and NAnt don't really give you this stuff 'out of the box'. You spend time gluing bits together, inventing your own build process and so on, and maintaining this stuff seems to get out of control very easily. Deploy and install is a particular minefield, because somewhere in there you have to start doing configuration file substitution (put your test server settings in the web.config etc...). And doing all this in XML just rubs salt into the wound.

You can handle most of this by hand on small projects, but the last app I worked on had five or six deployable parts to it (webservices, windows services, Winforms with ClickOnce manifests and the like), each of which had 20 or so settings to change for each of 7 different environments and the differing systems it integrated with. That's 100's of settings to keep track off, without even getting into the Biztalk artefacts, and that was only one of several projects of similar complexity. Automation's a no brainer at that point.

My solution to try and scale back the per-project cost of managing this was my own open source project SNAK. This attempted to commoditize a standard build / test / package / deploy process that you could implement on your side by pretty much setting a single variable at the top of a build script. And I think it works reasonably well: but it's clearly not the answer, not least because it took a fair amount of my (and others) time, of which I have very little.

So I was very, very hopeful when I started looking at the CI support in TFS 2008. Microsoft were really bashed over CI (lack of) in 2005, but this time round it looks like they've delivered:



You pretty much pick your solution file:



...your output directory...



...and your build frequency, and off you go:



Given how hard it was to deal with VSTS tests under CI in 2005 (because the test file was always in the wrong place), this screen will be a real pleasure to some:



And if you've tried to implement a build output retention policy in NAnt, you'll really appreciate this:



So up until now, absolutely fantastic. But then I had a few issues, which I'll deal with in Part 2 (so as not to take the gloss off the good bits above).

[I was due to present on this topic at the Perth .Net user group the other week, but a failing laptop saw to that (not the way I was expecting the demo to fail!). Since there's now no slots till Xmas, I've recycled some of the content into this post. The laptop was lobotomized and is recovering well...]

[Readify are doing a Dev Day in Perth on the 29th, with TFS as one of the tracks, so I'd be surprised if they didn't cover this there]

CruiseControl.NET RC1 Log-merging issues with nmock via nant

Summary: using a strict mock in a unit test run using nant's nunit2
task breaks RC1 CruiseControl's merging of the nant output [update: This is apparently fixed in later builds of nmock - see comments at end]

When we migrated to the latest version of CruiseControl.NET last week
(the RC1 release), one of our builds stopped merging the logs
properly (so we started getting the dreaded 'Log does not contain any
Xml output from NAnt' error).

This was due to cruisecontrol CDATA-wrapping the nant output in the
log, but at the time I didn't understand why it would do that, and
what had changed in RC1.

It turns out that nmock has a known issue (
http://jira.truemesh.com/secure/ViewIssue.jspa?key=NMO-43 ) where
strict mocks complain when the mocks are Finalise()'d. Since this
only occurs on a GC, this error is normally swallowed by the host
application closing down, but you can see this behaviour if you run a
test including a strict mock from nunit-console.

Normally nant's XmlLogger catches everything going to StdOut and
StdErr, and wraps them nicely in its xml output to catch this kind of
stuff. Unfortunately nant's nunit2 task runs inside nant, rather than
shelling out, so the GC that causes the nmock issue doesn't happen
until nant is closing down, by which time the XmlLogger has already
closed down.

As a result, nmock writes this kind of output directly to StdErr:

Unhandled Exception: NMock.VerifyException: Finalize() called too
many times
expected:0
but was:<1>
at NMock.Mock.Invoke(String methodName, Object[] args)
at ProxyDataFormatter.Finalize()

...the <>'s in which cause CruiseControl to wrap nant's output with
CDATA and hence hide it from the dashboard.

And what's changed in RC1? Well since this wasn't an issue before I
can only infer that RC1 now sucks in StdErr *as well as* StdOut from
nant.

So... Really what it boils down to is:

- CruseControl should probably CDATA the StdErr from nant (rather
than assuming its valid Xml!)

- nmock needs fixing (seems like they're too busy playing with a v2
generic version to look at existing issues)

- tricky to see how nant's XmlLogger could cope better

- should be a FAQ on 'Log does not contain any output from NAnt'

In the intermediary we've hacked our version of nmock as per the link
up above (somewhere) so as to not raise exceptions on Finalize()

Update: This is apparently fixed in later nmock releases, though we found they also broke something else we were doing (which was probably our fault), so for expediency we kept our hack. Later versions of CruiseControl.NET may also handle this more gracefully now too.