Not sold on Database Edition 2008 GDR

Just in time for the end of 2008 Microsoft finally got Database Edition 2008 GDR out the door. This replaces the version that came with VS 2008, primarily bringing with it Sql 2008 compatibility.

So GDR is now Microsoft's 3rd bite at integrating the database development workflow into Visual Studio. Lets start with what it does right:

Obviously the main benefit of bringing the database into Visual Studio is you get source control for free. Sure - you've been able to script out your database and add it to source control before, but this way you can check in database changes and the code that depends on it as one atomic checkin, against a work item. That's actually pretty neat from a tracking / branching point of view.

And because it's a fully model-driven system, all the SQL in the project is tested for consistency and correctness at build time. You could do this before of course (script out and build a structure only database to check it builds), but this way you know any problems you find are local to your workspace, so it really was you that broke them.

The output from the build process is a model that completely describes your database schema, so this can be used to sync up test and production databases, either via the IDE, MSBuild or a redistributable command line utility that can be incorporated into your deployment / install process. Not that you couldn't have done all this with RedGate's SqlComparison SDK of course, but if you struggle to get licences other than MSDN purchased, this does make your life a bit easier, and at least it's been constructed with a build-once-deploy-many philosophy, unlike the clowns behind ClickOnce.

The model also (allegedly) provides a safer platform for database refactoring: it's supposed to 'remember' things like column renames as renames and treat them as such the next time a database is sync'd to the model. I guess only time will tell how well that works in the real world given, for example, multiple databases at different schema versions (ie Prod database 5 versions behind Test etc… ). Are the refactorings going to be journalled away and somehow replayed in the right order? If so, how does it know at which version to start? It's not a simple matter.

But Microsoft has still totally, totally missed the point, because there is still no design support. Not one. Nada. Not even a schema browser.

Hey I can see where this is going, and I can see they're laying foundations here, but in my mind the fundamentals of a database development tool are that they assist in database development. You know - the core stuff: creating objects and sprocs. Source control and a deployment workflow are secondary to that. But with GDR you'd better be hankering after the Sql 6.5 days, because you're going to be writing lots and lots of TSQL. The best I can say is it's colour coded, but didn't we only just get IntelliSense in Management Studio? It just doesn't work for me.

But the alternative - do your dev / design in Management Studio and sync back to the model - is equally flawed, if only because you've got to keep syncing back again (and the sync UI is still a bit ragged). That's a whole lot of friction for relatively nebulous benefits (especially if you already have your database schema under some level of source control), and in my mind questions the whole utility of the product. What's the point of 'integrating' into the Visual Studio IDE if you have to keep tabbing out to do anything even remotely useful (think: execution plans)?

Similarly, whilst there's a Data Compare feature, it's not actually part of your database project, just something you do on an ad-hoc basis. So there's no easy way of designating a certain subset of tables as 'reference data' that should essentially be considered part of the schema, rather than part of the contents, and sync'd up when you redeploy. I'll accept that's probably a vNext feature, but its absence calls into question the viability of using GDR as an end-to-end database deployment manager in the first place, compared to tools like SqlPackager (or scripting using SqlCompare \ SqlDataCompare APIs).

And of course the model parser isn't perfect, so occasionally a valid bit of SQL creates build errors because the model can't fathom what it's doing. In my case it can't seem to resolve the columns in a CTE that queries another (properly referenced) database, and barfs about ambiguous references where there are none. Unfortunately fidelity is always the problem with Model Driven Architectures, even ones that use a DSL as the model, because they have to - at some point - cater for a lowest common denominator, and at any rate they are in an arms race with whatever it is they're modelling. Microsoft is touting it's extensible architecture, and IBM are apparently writing a DB2 provider, but what do you think the chances of 100% functionality coverage are? And for Oracle? I'll wager pretty low. So be prepared to take an 80/20 view, and put some bits into pre/post build scripts and / or re-structure them to keep the parser happy.

Then there are the little things. It creates a schema and object hierarchy when your database is initially imported, but it doesn't enforce it, and the Add \ New (whatever) templates don't pick up on it, so all your new objects end up in the wrong schema unless you're concentrating. If you do anything with any system views you'll need to reference the master database ( %ProgramFiles%\Microsoft Visual Studio 9.0\VSTSDB\Extensions\SqlServer\2008\DBSchemas), so why is it not referenced to start with? And the doco's not been updated for RTM so contains all kinds of outdated information around the command line interfaces, and absolutely no content in the API section other than listing the namespaces.

http://social.msdn.microsoft.com/Forums/en-US/vstsdb/thread/3f0f0888-9b37-4279-81d4-b924595630a8/
http://social.msdn.microsoft.com/Forums/en-US/vstsdb/thread/a966ac3c-c039-4c3f-9b6d-2882d9874282

So this still strikes me as neither fish nor fowl. Given the designers already exist in Management Studio it's a bit pisspoor that the DataDude team haven't borrowed them yet. I imagine I'll still be doing most of my work in Management Studio, and at best attempting to keep the database model in sync. And I suspect that for me, and many others, this will be just too much friction to bother with and so the whole effort will be more or less wasted.

Which is all a bit sad, given it's 2009, I've written a decades worth of database-centric apps, and we've still not got this right. At least no-one's actually paying for this any more.

Sql 2005 Install Woes on Shiny New Big Server

I had a new one this time.

I (of course) ran into the install-hangs-on-setting-file-security issue (KB910070), but I was expecting that. What really threw me was after that then install then just kept dying, leaving this in the logs:

Faulting application sqlservr.exe, version 2005.90.1399.0, faulting module sqlservr.exe, version 2005.90.1399.0, fault address 0x0000000000b323f0.

This really threw the installer too - after an uninstall, and even after a manual cleanup, the installer still though there was an instance hanging around. Which it was. I had to manually delete the SQL Services (using SC), a bunch of instance registry settings and the instance files directory (the MSSQL.1 folder) before I could finally get it to re-install. I guess the uninstall died too.

So then I tried installing again, and again. And again.

So I started speculating. Was it the virus scanner .... No. Could it be the Sql 2005 installer didn't like .Net 3.5 sp1? Uninstall... No. Was I definately using the 64 bit version... Yes. Could I slipstream SP2 and workaround some issue I didn't understand yet... No. Was it the monster 24 cores the server had (4 x hex core)... maybe.

There is a known issue with Sql 2005 instal failing with odd number of cores (ie Phenoms). That (obviously) doesn't count: but maybe Sql can't either. So I used the instructions in KB954835 to criple my monster server down to a single CPU, and then it all installed just fine. I can now install SP2 (3 actually) which allegedly should then make it all work.

Moral:
Obviously I should have been installing Sql 2008 instead
It's clearly becoming way too easy - with multi-multi-core boxes - to drop into some massively unexplored race condition territory in something that's otherwise really quite stable and well tested.
You can have too many cores

Other thoughts:
There must be a better way to restrict an installer or app to run on only one core without farting about with BOOT.INI
Once I put SP3 on it better all work otherwise the boss is going to be really pissed ('What, those other 23 cores? They're um ... spares')

Forcing Web Part pages into Edit Mode

(Before I forget again :)

Even if you've hacked about with a SharePoint master page to remove all the SharePoint chrome (including the edit button and suchlike) there are URL hacks you can use to get the page back into edit mode:

pageurl?ToolPaneView=2

(There's a few more like this, see Sharepoint Tweaks... )

...as well as the hack to get a page into the Web Part Maintanance page:

pageurl?contents=1

Special cleaning instructions for Nokia 6110 Classic

Mine had a bit of a smudge on the screen so I thought I'd pop it through the wash and clean it up. The critical bit here was to make sure the phone was in the pocket of my cycling shorts, so it didn't get bashed / scratched by the washing machine drum.

After an afternoon out on the line drying (still in shorts pocket) bingo: one nice clean phone.

Nokia: tough as.

Recordset to Powershell object collection

I've found this pretty damn useful: it converts an ADO.Net IDbDataReader into a collection of powershell objects, mapping column names to property names so you can sort and filter and whatnot using standard PS syntax.

# Executes an IDbCommand and converts the resultset
# to a Powershell object collection
function ExecuteCommand(
  $command = { throw 'Command must be supplied' }
){
  $rows = @();
  $reader = $command.executereader()
  while($reader.read()){
    $values = new-object object;
    
    for($i = 0; $i -lt $reader.FieldCount; $i++){
      $name = $reader.GetName($i);
      $value = $reader.GetValue($i);
      
      Add-Member -in $values noteproperty $name $value
    }
    $rows+= $values;
  }
  $reader.Close();
  $rows
}

Next week: why I was doing this in the first place (much more interesting)

Frequently Bought Together

Just remember kids: no-one likes a smart arse.

RedGate SqlCompare wins again

...in my highly subjective 'which tool are we going to use for database schema synchronisation?' challenge that is.

RedGate's prices seem to have gone up again ($595 USD for the comparison bundle[1], but if like me you want to use command line interface you're looking at 2x $595 USD, or 3x if you want the API too). And support and maintenance is on top of that.


So I had a good look around and considered the options:

Visual Studio for Database Professionals - easy option since it's now included in our Team Suite SKU. However even the 2008 version is still pretty crude, with very little option to change the generated delta SQL, and as a result scripting out unnecessary changes (like rebuilding your tables via a temp table just to get the column order 'right') and doubtless doing things with role memberships that I didn't want. So that didn't last long.

SqlDelta I like a lot. It does schema and data compare, and it's got a command line interface all for $330 USD - cheaper than RedGate's most basic non-pro compare bundle. And it's from down under. But it choked on my instead-of triggers on a view (either scripted them as CREATE when they needed to be ALTER or vice-versa). So I had to faff about to get a sync to work. That's an immediate fail.


Then there's ApexSQL Diff. But I didn't really get round to using that. Which is where the 'highly subjective' bit of this review comes in, not to mention the 'use the first product that works, stop playing around and get some work done' voice-of-conscience.

So RedGate it is.

[1] There's an option, not available on their website, to get a Sql Compare Pro bundle, which if you need the pro editions + the API basically means you get them for 2/3 price.

Remember to enable MARS when using Snapshot Isolation from SSAS

We started getting this error when processing our cube:

OLE DB error: OLE DB or ODBC error: Cannot create new connection because in manual or distributed transaction mode

It went away when:

• We changed to using ReadCommitted isolation, rather than snapshot
• We processed the cube using Maximum Parallel Tasks = 1

Reading knowlege base article PRB: SQLOLEDB Allows Only One Connection in Scope of Transaction lead me to think that SSAS was trying to open multiple connections within a transaction, which isn't allowed.

Which got me thinking about MARS. Not quite sure why it wasn't on to start with, but I enabled it, and then everything was fine again.

Turns out this is actually a RTFM, if you pay attention when reading How to enable the snapshot transaction isolation level in SQL Server 2005 Analysis Services . Of course it's only in Snapshot mode that SSAS attempts to ensure database consistency for the duration of the entire Process operation, which is why it's not a problem using ReadCommitted isolation.

If McCain wins next week...

...I'll leave the US

and I don't even live there!

Viewing the MDX cellset with WPF

When executing an MDX query there's various bits of useful metadata that can be returned in the cellset over and above the members and dimensions you've explicitly specified in your query. This can include things like the formatted value (as specified by the cube definition) as well as other attribute values for the dimension member you're explicitly querying against.

This kind of stuff can be invaluable if you're writing your own front-end app to access OLAP data, not least because it saves a whole heap of faffing about with WITH clauses (query scoped calculated measures). Something like:

with Member (
  [date].[date].[date].currentmember.properties("Year")
) as TheYear,
select {
  [measures].TheYear, [measures].[measure]
} on Columns,{
  [date].[date].[date]
} on Rows from Cube

...can just become:

select {
  [measures].[measure]
} on Columns,{
  [date].[date].[date]
} dimension properties member_name, member_value,
 [date].[date].[date].Year on Rows
on Rows from Cube

Trouble is you'd really struggle to work this out since all that metadata is helpfully hidden from view when you execute MDX in BIDS (and MDX studio), which makes it all a bit hit-and-miss. So I wrote a little WPF app just to visualise the actual cellset returned. Pretty basic stuff - load the results into a dataset and bind it to a grid. I could fiddle with my DIMENSION PROPERTIES clause with immediate gratification.

But it took me hours to get the binding working. One of the problems is that the MDX columns have names like '[Measures].[MyThing]' and you can't just set that as the property name of your binding and expect the binding infrastructure to cope:

{Binding Path=[Measures].[SomeMeasure]}

The binding infrastructure sees the dot and tries to walk the path, with predictable results:

System.ArgumentException: Measures is neither a DataColumn nor a DataRelation for table Table

[NB: If you had a column simply named SomeMeasure this would work, due to the magic of ICustomTypeDescriptor, but that's another story]

So instead you have to use the indexer syntax on the DataRowView:

{Binding Path=['[Measures].[SomeMeasure]']}

Or (if that made you wince)

{Binding Path=Row['[Measures].[SomeMeasure]']}

But those don't work either:

System.ArgumentException: Column '"[Measures].[SomeMeasure]"' does not belong to table Table

Even whilst the same binding path works 'just fine thanks' in the debugger. It took me a long, long time to realise there's an extra set of quotes in that error message. The WPF binding syntax doesn't require quotes for string indexers:

{Binding Path=Row[[Measures].[SomeMeasure]]}

It looks so wrong but it works.

SharePoint: The final CM frontier

For a long time I thought Biztalk was the elephant in the room when it came to Configuration Management - specifically version control. I'm not talking here about Source Control, I'm talking from a deployment perspective - the 'deploy a given, atomic, integral version into production' problem.

For a traditional .net app (Winforms, ASP.Net, WPF, whatever) it's pretty much sorted. We've had source control integration in the IDE since the dawn of time, CCNet for years, and it's even easier to get CI going now that TFS 2008 supports it out of the box. That's not to say everyone actually does so, but it's there if they want it, right?

Databases are a bit harder, but there are lots of tools around that you can incorporate into your CI cycle, and now that Microsoft's in the space with VSTS for Database Pros (now included with VSTS Dev Edition) the barrier to entry has been dropped again (though it's still a pretty poor story in the SSAS space).

However for Biztalk things are not so rosy. You get source control at least, so you can practice a unified versioning / labelling scheme, but building the project doesn't give you all the artifacts you need to actually deploy into an environment. For that you've got to IDE-deploy to a dev BizTalk instance, configure and then at least export out the binding file (if not a complete MSI). There's NAnt and MSBuild tasks to do the building, but I've not seen anyone wrap up the whole end to end, so Biztalk deployments languish in the 'manual effort with concentration' department. A direct result of this is that integrating your code with biztalk orchestrations (via WS / WCF) is increasingly seen as a more manageable approach than embedding your code, thus alleviating / sidesteping many of the issues.

I thought Biztalk was the frontier of version management - that is until I started working with SharePoint.

I'd previously accepted that working with SharePoint using the Web UI was working uncontrolled, but I'd always imagined that real SharePoint developers used SharePoint Designer, checked things into source control and had established patterns for migrating content between development and production server instances. How wrong I was. Making my first tentative forays into 'how to do this properly' I was struck by the complete absence of any guidance. It appears this is most definitely not a solved problem, which Jeremy pretty much confirmed last night in his great RDN talk on the subject. There's tools out there to manage the problem, but they're pretty new on the block. The book, quite literally, hasn't been written yet.


This is clearly a pretty major failing on Microsoft's part. I appreciate that the main thrust of version-management with SharePoint is version-management of content rather than version-management of configuration, but clearly if they want developers to embrace SharePoint as a platform they're going to have to do a bit better. Time for a 'developers, developers, developers' rant perhaps?


It seems to be that the more productivity-orientated the development environment, the less effort has been put into establishing a viable CM story:



(You could substitute 'maintainability' for 'Ease of CM / CI' if you like: one tends to drive the other)


Clearly it's easier to diff / merge lines of code rather than SharePoint XML manifests (or Workflow XAMLs), but it's a complete abdication to leave these higher-level (ie non-codey) development environments quite so dramatically out the cold.

Surely there's got to be a better answer than using DiffDog for everything?

Getting just the date from Sql Server's datetime

Whilst many still advocate using Convert() to drop the time-bit from a datetime[1], going off to a string like that is nothing like as efficient as the numerical alternative:

CAST( FLOOR( CAST( @dateTime AS FLOAT ) ) AS DATETIME )

Sql stores the days since 1900 in the first 4 bytes, and the time in the last 4 bytes. Throwing away the bit after the decimal place (what the above does) just strips those last 4 bytes right how.

But... why AS FLOAT? Why convert straight to an INT?

CAST( CAST( getdate() AS INT ) as Datetime)

Turns out this gives us tomorrow's date! I guess this is because the starting date is actually Jan 1st 1900, and not Jan 0, which could be a classic off-by-one error if you weren't awake.


[Update]: Because casting direct to an int rounds up in the afternoon! Doh! Doh! Doh!. This is exactly why I'm writing this down, because I knew that once and forgot. Anyway, going via a FLOAT is the go, OR the much more legible alternative that Mitch put in the comments : datediff(day, 0, @yourdate)
[/Update]

That being said, if all you want is an integer value that represents a date, just casting datetime to an int seems like a pretty good way to go, provided you never get carried away and cast it back again. Or I guess you could just compensate for the off-by-one and save yourself[1] a lot of pain later on...

If you're really interested, run this and see for your self:

select
 getdate()
 ,FLOOR( CAST( getdate() AS FLOAT )) -- use floating point
 ,datediff(dd, '19000101', getdate()) -- right answer, by definition
 ,Cast(getdate() as int)
 ,CAST( CAST( getdate() AS INT ) as Datetime) -- tomorrow!

[1] Yes, Sql 2008 has a date-only type. But you and I will still be stripping times from datetimes for many, many years to come.
[2] By which I really mean myself, of course

Continuous Integration in TFS 2008: Part 2

Ok so in Part I I missed two important features due to a schoolboy cut-and-paste error:

• The association of a build with the changesets that triggered it - fantastic for trouble shooting
• Automatically raising a 'bug' work item for 'he who broke the build' to fix it. Sweet.

Maybe I'll go into those in more detail another time, maybe not. They are cool. But here's the not-cool stuff:

Stupid default working directory / issues with path length
Straight out of the box many of my builds failed with excessive path length errors. Unfortunately the build agent starts with its default working directory of C:\Documents and Settings\NetworkService\Local Settings\Temp\ so I've got 61 extra chars on my path before I've even started, and we were already skating pretty close to the 255 chars limit (think BizTalk, over-zealous namespacing, auto-generated code etc...). Easily fixed, but seemed like a silly place to start.

Ok, it's going to be a bit less under Vista / Server 2008: still too long. Is C:\Builds so wrong?

Binaries deployed to shared Staging folder
More problematically, the folder structure created on the staging share is a complete mess, totally useless to use as the source for xcopy deployment to testing environments and the like. I'll show you what I mean:

Here's my solution structure: 4 deployable applications and one shared library:



And here's the staged output from the TFS build:





Pretty disappointing. All the binaries from all of the different applications have been dumped in the same folder. At least the websites have also been spat out in 'xcopy ready' form, but god help me if I wanted to Xcopy the console app one place and the WinForms app somewhere else.

What I'm looking for is more or less what I get out of SNAK right now:



...with any deployable asset (exe, website) neatly packaged to be copied off somewhere.

Same-name content files overwritten
Ok, I guess I could just accept the above and copy everything, only using the EXE that I want, but that's icky, and it does rather suppose there aren't any naming conflicts between the deployable artefacts.

For example: I added some files marked as Content in my HelloWorldConsole app, but they got completely ignored by the deployment process. I had to also mark them as 'Copy to Output Directory' before TFS build stuck them anywhere (which I'm not convinced is correct behaviour, but there you have it), and then it stuck them in the 'right' relative location to the build root folder:



...becomes...



But there are two 'Subfolder1's above, and I only got one out at the end. Predictably, one of them got overwritten.

When would this be a problem? When could different projects possibly have a content file with the same name?! I can think of some examples:

• We always use a file 'log4net.config' to host our logging configuration, so we can change it on the fly without recycling app pools and the like. Only one project would have got the right configuration.
• Bundled licence files (Licence.lic) would get mixed up

I'm sure you can think of some more. And yes, using Embedded Resources works fine, but they're not always an answer (eg: log4net.config).

There is a fix. There's some changes you can make to your *proj files to make them preserve the output path hierarchy when deployed to the build drop folder. But it's a per-project fix, and that's - frankly - a bit lame. I know you don't add new project to your solution on a daily basis, but it's just one more thing that needs to be kept on top of, or things start falling apart. And that's just not how all this is supposed to work.

(Alternatively you could customize everyone's project templates. I guess you could fix it on all their PCs, or you could just put a fixed template on a share somewhere and tell people to use it. Since it's one of the default project types you're amending I guess you probably have to remove the original from their machine too. And hope they don't have to re-install VS anytime...)

Conclusions
Getting VSTS to perform Continuous Integration on your project is now really easy. To be fair, this on it's own was pretty easy with CCNet too, but it's even easier now, and we don't have to fight over the CCNet.config file.

But the staged output from the build strikes me as limited in use. It's possible to go and hack about with the generated MSBuild files that actually perform the build and stage, and bend it to my will, but that's just what I don't want to have to do. I want it to 'just work' and I don't think we're there yet.

New MCCS Certification announced

Microsoft will tommorrow announce their new MCCS - Microsoft Certified Certification Specialist certification. This exam will henceforth be a pre-requisite for embarking on any of the various MCAD / MCSD / MCPD upgrade paths.

The course content is not nailed down yet, but most of the detail is on Gerry's blog. In the comments. Obviously.

Miss VMWorld 2008

Everytime I get this email...

... I get totally the wrong idea.

MyClass in VB.Net

It's always a bit of a shock when you find something you've missed in a language you've used for years. I'm mostly a C# person, but I thought I knew pretty much all of VB.Net's quirks by now. But I totally missed 'MyClass'.

'MyClass' allows a class to access methods and properties as declared on itself, irrespective of them being overridden further down the inheritance heirachy. It's like using 'Me' if all the 'overridable's were removed.

Since there's no C# equivilent this was a big surprise to me, but it shouldn't have been - it's only doing the same as 'MyBase' does (against a type's ancestor): executing properties / methods by specific type address, not via virtual dispatch. As the IL for this sample shows:

Public Class Class1
    Public Overridable ReadOnly Property Name()
        Get
            Return "Class1"
        End Get
    End Property
End Class

Public Class Class2
    Inherits Class1
    Public Overrides ReadOnly Property Name()
        Get
            Return "Class2"
        End Get
    End Property

    Public Function GetNames() As String
        Dim lines(3) As String
        lines(0) = MyBase.Name
        lines(1) = MyClass.Name
        lines(2) = Me.Name
        Return String.Join(",", lines)
    End Function
End Class

Public Class Class3
    Inherits Class2
    Public Overrides ReadOnly Property Name()
        Get
            Return "Class3"
        End Get
    End Property
End Class

Calling new Class3().GetNames() produces the following (edited for brevity)

     // mybase - explicit dispatch to class1
     L_000b: call instance object ConsoleApplication1.Class1::get_Name()

     // myclass - explicit dispatch to class2
     L_001a: call instance object ConsoleApplication1.Class2::get_Name()

     // me - virtual dispatch, will resove to class3's implementation
     L_0029: callvirt instance object ConsoleApplication1.Class2::get_Name()

So the output eventually is 'Class1, Class2, Class3'. Nifty. That being said, I can't honestly say I've ever really needed this, so it might go back into the 'curios' collection. Useful in a pinch maybe, but surely it's a smell? As if designing-for-subclassing wasn't hard enough as it is...


PS: Interestingly the Reflector disassembler doesn't understand this either, so it wasn't just me that missed it: Reflector thinks the VB was:

Public Function GetNames() As String
    Dim lines As String() = New String(4  - 1) {}
    lines(0) = Conversions.ToString(MyBase.Name)
    lines(1) = Conversions.ToString(Me.Name) ' got this wrong
    lines(2) = Conversions.ToString(Me.Name)
    Return String.Join(",", lines)
End Function

Using Extension Methods in .Net 2.0 from VB.Net

So despite what ScottGu originally said, Extension Methods don't 'just work' for VS 2008 projects targeting .Net 2.0.

There's no end of blog posts describing the workaround - add your own ExtensionAttribute class to get it working - but all the samples are in C# (which is interesting in of itself). So here's the VB.Net version:

Namespace System.Runtime.CompilerServices
    <AttributeUsage(AttributeTargets.Method Or AttributeTargets.Assembly Or AttributeTargets.Class)> _
    Public Class ExtensionAttribute
        Inherits Attribute
    End Class
End Namespace

...and why am I bothering to blog about this rather trivial conversion? Because of the key gotcha: make sure you put this in a project with no root namespace set:



That had me banging my head on the table for too long.

As did the next one: extension methods only show up under the 'All' tab in IntelliSense - obviously too advanced for mere Morts. I gotta remember to turn that off: using VB is bad enough without the IDE patronising you as well.

Interestingly, if you get the AttributeUsage declaration wrong on the attribute, you get this error:

"The custom-designed version of System.Runtime.CompilerServices.ExtensionAttribute ... is not valid"

Fascinating. So this hackery works by design, it's just not really supported as such.

More reading: MSDN documentation on Extension Methods in VB

Continuous Integration in TFS 2008: Part 1

Many people now accept the benefits of a regular / continuous integration cycle (even if they don't actually practice it themselves). Picking up the pieces after someone's broken the checked-in source code, especially if it's not picked up for a few days, can be a real time waster.

Like many agile practices, however, the cost / benefit is hard to quantitatively analyse. It's far easier to justify therefore if it's really easy to setup: as the costs tend to zero the benefits become essentially 'free'. And you could argue that tools like CruiseControl.Net have made it pretty easy.

Personally, having spent significant sections of the last 3 years getting CCNet / Nant build cycles going on various projects, I'd beg to differ. Sure, it's really easy to setup CCNet / Nant (or CCNet / MSBuild) to build your solution, but that's only the first step in the process. Typically you also want to do things like:

• Import the latest built version of external dependencies (ie components maintained outside of the solution being built)
• Execute unit tests
• Execute integration tests (so config files pointing at databases etc... have to be in the right place)
• Package the build outputs nicely ('xcopy ready')
• Deploy and install into test environments

CCNet and NAnt don't really give you this stuff 'out of the box'. You spend time gluing bits together, inventing your own build process and so on, and maintaining this stuff seems to get out of control very easily. Deploy and install is a particular minefield, because somewhere in there you have to start doing configuration file substitution (put your test server settings in the web.config etc...). And doing all this in XML just rubs salt into the wound.

You can handle most of this by hand on small projects, but the last app I worked on had five or six deployable parts to it (webservices, windows services, Winforms with ClickOnce manifests and the like), each of which had 20 or so settings to change for each of 7 different environments and the differing systems it integrated with. That's 100's of settings to keep track off, without even getting into the Biztalk artefacts, and that was only one of several projects of similar complexity. Automation's a no brainer at that point.

My solution to try and scale back the per-project cost of managing this was my own open source project SNAK. This attempted to commoditize a standard build / test / package / deploy process that you could implement on your side by pretty much setting a single variable at the top of a build script. And I think it works reasonably well: but it's clearly not the answer, not least because it took a fair amount of my (and others) time, of which I have very little.

So I was very, very hopeful when I started looking at the CI support in TFS 2008. Microsoft were really bashed over CI (lack of) in 2005, but this time round it looks like they've delivered:



You pretty much pick your solution file:



...your output directory...



...and your build frequency, and off you go:



Given how hard it was to deal with VSTS tests under CI in 2005 (because the test file was always in the wrong place), this screen will be a real pleasure to some:



And if you've tried to implement a build output retention policy in NAnt, you'll really appreciate this:



So up until now, absolutely fantastic. But then I had a few issues, which I'll deal with in Part 2 (so as not to take the gloss off the good bits above).

[I was due to present on this topic at the Perth .Net user group the other week, but a failing laptop saw to that (not the way I was expecting the demo to fail!). Since there's now no slots till Xmas, I've recycled some of the content into this post. The laptop was lobotomized and is recovering well...]

[Readify are doing a Dev Day in Perth on the 29th, with TFS as one of the tracks, so I'd be surprised if they didn't cover this there]

Recycling old posts?

Sorry about that. I re-tagged a few articles over the weekend, and I think Blogger has got confused and bounced them into my feed as if they were new posts. Unfortunately some of them were, so it's all a bit of a mess.

New posts were actually:
* Finally: PowerShell as build language
* Using PowerShell to export the Windows Feeds list


Normal service will resume shortly...

Using PowerShell to export the Windows Feeds list

Moved computers recently, and one of the things I realised I lost was my RSS feeds list. It was probably a blessing (I just tend to accumulate subscriptions otherwise), and maybe I should be using a reading service of some nature, but there you are.

Anyway given I'm all Mesh'd up, I though I'd copy my feeds list into my Mesh folder (like my bookmarks), so I'd have a backup and this wouldn't happen again. Only I couldn't find where the feeds list actually lives. Instead there's a whole API for dealing with it...

...which is surprisingly easy to use, and works like a treat in PowerShell (I'm always amazed at it's ability to 'just work' with things like COM objects). So I just exported the list instead:

# Dump the contents of the Windows Feeds store to an XML file

$erroractionpreference="stop";
[xml]$feedsListDocument = "<feeds/>"
$feedsList = $feedsListDocument.get_DocumentElement();
$feedManager = new-object -com "Microsoft.FeedsManager"

@"
<feeds>
$(
 $feedManager.RootFolder.Feeds | % {
  $feed = $_;
  $feedXml = $feed.Xml(-1, 0, 0, 0, 0)
  '<feed Name="{0}">{1}</feed>' -f $feed.Name,$feedXml
 }
)
</feeds>
"@

Easy as. The XML it spits out is overly large (since it includes all the article contents from the cache), but for the MB involved it barely seems worth refining it.

Update 2008-07-17: So like the very next day I realised I could have just sync'd the feed list into Outlook, and asked it to export it as OPML. But syncing into Outlook blew my tiny mailbox quota (these feeds are suprisingly large) so I ended up back doing this again anyway. Then it turned out that IE can export the feed list as OPML too (File \ Import and Export - you'd think I'd have noticed originally) - but I still like having a script because I can schedule it.

Note to self: It is definitely time to find a blog that can cope with XML a bit better

Finally: PowerShell as build language

I've never really got into MSBuild, which surprised some people given how much time in the last four years I've spend mucking about with CCNet / NAnt. It was partly that we did a bit of investigation when MSBuild came out, and saw a couple of things we didn't really like about it and decided to wait for v2 (ie Team Build in TFS 2008). Partly.

More fundamentally however the problem is that MSBuild is just too similar to NAnt, and my considered opinion after years of usage is that NAnt sucks, or to be more specific, XML is a terrible 'language' for writing executable code. Fowler puts it pretty well:

"After all until we tried it I thought XML would be a good syntax for build files"
http://www.martinfowler.com/articles/rake.html

Sure it's fine for the IDE to write that stuff out (though even then you have to look at it and wince, right), but for humans who want to customise their build process? Jesus wept. Variable scope: gone. Explicit parameters for subroutines (targets): gone. Its fine when it's simple, but once you start looping and branching and using temporary variables it's just a great big mess of angle brackets that even it's mother pities. And debugging? Now there's a story...

There's a time and a place for the angle bracket tax, and this isn't it. Square peg, round hole.

So given how amenable for DSLs PowerShell has proven to be, I've been holding my breath for some kinda PowerShell rake-alike.

And here it is: Introducing PSake

(Also Microsoft themselves are thinking about it, and canvassing for opinions about whether it's a good idea or not.)

Sadly (actually quite the opposite) I'm not actually having to deal with the build process on my current project, so I don't really have much excuse to play with it. But I dream of a future in which the TFS Team Build project wizard kicks out a PS1 file instead. It'd certainly make fixing some of it's shortcomings a whole heap easier (that's a subject for a future post)


Edit [14/7/08]: Most requested feature for MSBuild? Debugging. Obviously this'll be interpreted by the MSBuild team as a need for a debugger, but maybe they should have used a language that already had one.

Beyond Compare 3 supports 3 way merge, is totally awesome

Beyond Compare 3 is out in beta. It supports 3 way merges!

I found this out literally minutes before I started what turned into a 2 day mergeathon between two large and divergent branches in TFS, with *lots* of merge conflicts to manually resolve, and I can honestly say I'd probably still be merging if I hadn't downloaded it. It's just fantastic.

I'll probably post some screenshots etc... soon, but if you're struggling merging with BC2 and/or the built-in diff/merge support then you really should check it out.

MSDN Downloads and the fly-out menus trauma

Raymond's just posted about the rationale behind the windows menu show delay, and goes on to point out various web properties that blatantly ignore the underlying usability requirement.

Sadly finding examples is like shooting fish in a barrel. I remember Jakob Nielsen winging about this last millennium, but as the technology moved forwards: Director, DHTML then Flash, the ease with which anyone can design their own UI and distribute it widely over the internet has lead to a flood of bad UI. Even as Vista attempts to move forwards, the new Silverlight version of MSDN Downloads re-re-implements the fly-out-menus concept, with almost unusable results.

Maybe this is a necessary pain we have to move through, but it kinda sucks that we can't explore novel and interesting UI concepts without making them totally unusable. I'm no UI designer, but at least I don't pretend to be, or work as one.

The templating within WPF is a great example of an enabling technology here, where the usability can be codified into a control by 'experts', but still delegate most of the 'funky look' to the end-designer. In this case if WPF / Silverlight had shipped with a decent fly-out menus control, maybe the MSDN Downloads team wouldn't have got it so horribly wrong, and I wouldn't have had to uninstall Silverlight in frustration.

I guess there is hope then that this isn't just another enabling technology that enables people to make a real arse of things.


PS: Check out this bizzarro comment on Raymond's blog:
"Let's not get into the "gynaecologist's interface" that is Vista's Start Menu, shall we?"
WFT?

Don't be Stupid

Years ago I was working on a project and I came up with a fantastic idea to help limit the level of regressions in the codebase I was working on. Rather than write unit tests as little throwaway test harnesses, I moved them into the codebase, and created a little app to execute them. It even did this via reflection, so as we added more tests, they got run too.

I thought I was being pretty clever.

I was being very stupid. I'd just re-invented xUnit, and didn't even know. [1]

It's a particular type of stupidity that manifests itself only in those who'd otherwise regard themselves as anything but: we get so wrapped up in our great idea that we stop to consider that someone else might have done this already. Programmers are particularly badly afflicted by this, mostly because it suits our vanity to create it ourselves.

There was already an automated testing community, that had over time evolved what worked and what didn't, the practical upshot of which - for a .Net developer at the time - was that NUnit already existed. I could have spent the time writing more tests instead. Or better still, more screens, which is what I was actually being paid for.

The last three applications I've worked on have all involved considerable custom frameworks (stupid) including a custom databinding scheme (very stupid). They were written by clever people, most of whom I respect, but they did some stupid things that less able programmers wouldn't have been able to do. Clever isn't always a complement in the agile camp, and this is why.

Of course 5 years of hindsight is a wonderful thing, and I've written my share of head-slappingly dumb code too. And it's all too easy to succumbed to the 'quick fix' fallacy when the boss is breathing down your neck : after all it's so much easier to get started writing your own framework than to learn to use someone else's.

But once you start down the dark path, forever will it haunt your destiny[2]. Which is why I make this plee to you now:

Please, before you put finger to keyboard again, consider whether what you're about to write has already been written.

Don't be stupid.

[1] To be fair to my erstwhile self, at least I was actually doing some testing, which was more than had been done before on that project
[2] Or that particular project at any rate

Enabling multiple RDP sessions in Vista

After many days of frigging around I realised those thegreenbutton.com Vista multiple remote desktop hacks (that you find from google) are all broken by SP1. That page' on missingremote.com that is supposed to draw all this together still hasn’t been updated with this new info.

However add SP1 to your search and you find this other thread, which works: http://thegreenbutton.com/forums/permalink/242509/255166/ShowThread.aspx#255166

Ah, the joys of subsequent-threads-with-lower-page-rank-than-the-original-now-outdated-info.

Running ASP.Net webservices under a service account

Most of the time I run websites and webservices in an app pool that's running as Network Service. It just saves a whole truck load of time and hastle:
* no passwords to worry about
* already trusted for kerberos delegation
* can still use it to talk to a database under integrated security (you just grant access to the machinename$ account in the domain).

Hey - this is what this account was *invented* for.

However, sometimes a specific service account is a must. Reasons include:
* Needing to differentiate access rights between applications running on the same host
* Needing to authenticate back across a one-way domain trust
* Specific policy mandates

Unfortunately you can't just add any account to IIS_WPG and use it, because the ACL on windows\temp is wrong: and grants access to network service rather than to the group. Miss this one, and you'll just get serialization errors left right and center.

So I do this:


Net localgroup iis_wpg /add mydomain\myserviceaccount
cacls %systemroot%\temp /E /G IIS_WPG:C


...then when you change the identity of the app pool you won't get 'Service Unavailable'.

Don't override Equals

A colleague had a problem the other day which turned out to be due to an overridden Equals operator. In this case it was a straightforward bug in the implementation, but after he saw my horror struck face I had to introduce him to the whole 'don't override Equals' philosophy[1]. On the pretext that you've not come across it, here's the argument in full:

• You have two objects that came from different places, and need to know if they represent essentially the same data.
• You can't override Equals unless you also override GetHashCode. If two objects are equal, they must have the same hashcode, or collections are screwed.
  
• GetHashCode must return the same value for an instance throughout it's lifetime, or Hashtable's are screwed
• Your object isn't readonly, so you need an immutable field in the instance to base the hashcode on.
• But if you modify one instance's data to equal another, that field can't change, so the hashcodes are still different.
• You're screwed

And that's without getting into the problems associated with a correct implementation of Equals in the first place (getting the reflexive, symmetric and transitive bit right). Generally speaking some kind of IsEquivilent method is a whole heap less trouble, but it depends what you're up to. You might think about loading your objects through some kind of registry, so references to the 'same' data actually end up pointing to the same instance. Then everything just works...

More reading:

• When (not) to override Equals
• The rules for GetHashCode

UPDATE 10/04/08: Some clarifications: I'm talking about not overriding Equals/GetHashCode for reference types here. It's not such a problem for value types [as IDisposable points out in the comments]. And I've futher clarified some of my assertions about GetHashTable in the comments.

[1] PS: Like all advice, this has exceptions. But the chances are they don't apply in your case. No, really.

Care required passing arrays to .Net methods in Powershell

In Powershell, argument lists for .Net methods are treated as arrays:

$instance.MyMethod($arg1,$arg2);

...which can be confusing if you want to pass an array as a single argument:

$instance.MyMethod($myArray);

New-Object : Cannot find an overload for "MyMethod" and the argument count: ""

Instead, force the array-argument to be contained within a single-member array:

# Note the extra comma below
$instance.MyMethod(,$myArray);

Makes sense when you think about it, but definitely a gotcha.

[In my case, I was caught out with the byte[] overload constructor for a MemoryStream]

Blobs out with SQL 2008

Recently I re-visited the blobs in/blobs out argument with a colleague. You know the one, one of you says blobs shouldn't be stored in database (principally because the last time he tried it 'blobs in' in VB 6 access to the blob data was a pain in the arse), then the other one says no they should be in the database (because the last time they tried it 'blobs out' all the files got mixed up / out of sync / weren't backed up). Etc...

Anyway, not only has Paul Randal posted a good summary of the pros and cons, but he did so as an intro to a new SQL 2008 data type 'FileStream' that attempts to bridge the two approaches (the 'have your cake and eat it' approach).

I'm cautious. Transactions at the filesystem level are a real mess (as some of the OneNote blogs make clear, especially with non-MS implementations of SMB like SAMBA). Your database backup is presumably still huge and unwieldy (or missing the blob data, which is worse?).

The main advantage of this approach seems to be that SQL can access the blob data faster through NTFS than via it's own internal MDF formats. But you've apparently still got to go via SQL to get the data, you can't (for example) just serve up images-stored-as-blobs directly via IIS. Or maybe I've missed something. Either way, the upside all seems to be focused on blob streaming performance, which may or may not be the most relevant factor for your app.

So it's possible that next year's arguments will be blobs in vs blobs out vs filestream, and still no one-size-fits all. Ah well.

Path already mapped in workspace error with CCNet and TFS

Had a problem with CCNet that kept me here till midnight where try as I might, I just couldn't get a build to not fail with the dreaded "Path ... is already mapped in workspace ..." error:

Microsoft.TeamFoundation.VersionControl.Client.MappingConflictException: The path C:\Builds\etc\Working is already mapped in workspace someworkspace

We use a different workspace for every CCNet project to avoid collissions, and to maintain uniqueness we keep the workspace name the same as the CCNet project name. I couldn't find the workspace in question, and was pretty sure I'd already deleted it. In fact I'd used TF Sidekicks to delete all the build user's workspaces, and it still didn't work. So what was up?

Fortunately in a post 'How to handle "The path X is already mapped in workspace Y"' I learnt of the mappings cache file on the client PC, in the user's Local Settings\Application Data\Microsoft\Team Foundation\1.0\Cache\VersionControl.config file. Just nuking workspaces on the server isn't enough!

So to be sure I blew away the build server's local profile entirely, and that finally fixed it.

The new starter experience

I normally avoid 'link-posts', but again Hacknot is right on the money with 'If They Come, How Will they Build It?', an eminently familiar analysis of the plight of a new developer on a project with an oral documentation culture.

In fact I'd go slightly further than Hacknot, and state that the initial experience of a new developer on the project is one of the most important things to get right. First impressions do matter, and if your first impression of a project is the frustration of:

• Not having a login
• Not having internet access
• Not being able to get latest
• Not being able to build
• Not being able to locate any documentation
• Not having clear lines of escalation
  
• Not having clear rules of engagement
• Not knowing what's expected of you
• Not having a mentor
  

...you're going to be hard pushed not to be prejudicing your opinions of the professionalism of the rest of the project. You'll start disheartened, but on the other end of this unfortunate indoctrination you're going to be just like them. You won't regard the absence of the list above as anything other than normal. You'll accept that that's just not how things are done around here. You will love big brother.

[ahem. got carried away there]

I regard the absence of guides and documentation as more than a major time-waster: it's a self-perpetuating morale hole for all future team members to climb into and die.

New staff play a vital part in ensuring a project's approach doesn't atrophy. If you waste their 'fresh' time frustrating them with missing documentation and runaround, you won't get the benefit of seeing things from their eyes. They'll have clammed up and learnt to live with how it is, and by the time you ask them they'll have forgotten that they used to care.

FixBrokenWindows - they're not all in your code

Great Powershell One-liners

Or, 'highly useful things you can do in Powershell':

Watching the last 20 lines of the last (log?) file in a directory:

(gci)[-1] | gc | select-object -last 20

Finding where .Net is installed:

(get-itemproperty HKLM:\software\microsoft\.netframework).InstallRoot

Pinging a URL [1]:

$temp = (new-object net.webclient).DownloadString($url)
if ($?) { write-host Passed -foregroundcolor "green" }

Getting the DNS suffix for a machine:

(Get-WmiObject -class "Win32_ComputerSystem").Domain

[1] Ok, that one's not a one-liner.

Why Powershell Rocks

I try and remain reasonably sanguine / sceptical / cynical about most new technologies, but there's two about at the moment that I just can't find anything to fault with: Vista's Media Center, and Windows Powershell.

So (apart from the fawning) what's so good about Powershell?

The key point is that Powershell is a shell that thinks it's a scripting language. Or is it the other way round? Well it's both anyway. So things you can do in a BAT file, you can do in Powershell:

xcopy /I /F /Y Somefile.abc ..\SomeFolder

[In many cases, like the above, you can just cut and paste the same line into Powershell and it will work]

Ever tried doing that in VBScript/JScript? Either you wrote lines of Scripting.FileSystemObject code, or you shelled out (in which case you've got the line above plus the 'shelling out' code).

But BAT files can only take you so far. Further if you're stubborn or clever, but even then there's a wall. Creating files with todays date in the name is a real swine. Setting IIS properties involves an external VBScript. HTTP-pinging a URL and checking it's up... no chance.

For these you have to write a VBScript or an (.Net?) EXE, where you can take advantage a host of supporting libraries and benefit from 'real' programmer concepts like variable scope, functions and conditionals. But integrating them with your BAT files (ie retting return values back) is quite a challenge. And you end up with your foot in both camps

Powershell does both.

I'll give you an example of the kind of hybrid approach this engenders: setting ACLs for the anonymous user on a website. Now you can assume that it's IUSR_%computername%, but in some scenarios[1] (renamed for security, ghosted image / machine renamed) it's not. So you've got to look up the user first, which is pretty tricky in a BAT file (even with adsutil.vbs), but then set some file permissions, which is nigh-on impossible in VBScript. In PowerShell this is easy:

$iisobj = [wmi]"root/MicrosoftIIsV2:IIsWebVirtualDirSetting.Name='W3SVC/1/Root/MyWebSite'"
$userName=$iisobj.AnonymousUserName
$path=$iisobj.Path

CACLS $path /E /G $userName:R

Note how I did something very 'script' - dealing with objects and properties, and followed it up with something very 'batch' - just calling a shell command (CACLS). And it just worked.

Now in my real script I didn't use CACLS, I used the .Net System.Security.AccessControl classes, because I'm a .Net developer and that's what I thought of first. I wrapped them up into a neat reusable function, so the code above actually didn't look so different. But I ended up writing 20+ lines of code, where CACLS would have done the job just as well. I'm learning too.

There's lots of other great things about Powershell too:

• Some great syntax improvements, like range operators, and reverse-indexing arrays
• Fantastic support for script parameters, and default values
• Here-strings
  

...but they're all secondary to this one, which is that Powershell is all you need[2]. That's pretty compelling to me.


[1] I.E. My scenario.
[2] Ok, unless you're doing something really wacky/Win32 [3] where the .Net BCL doesn't have the support yet.
[3] [4] Is there a difference?
[4] Recursing footnotes? Can I get away with that?

IE Advanced Security Configuration affects Powershell execution policy

Like a few other MS apps, but hardly obvious to us mortals, Powershell determines whether something is 'local' based on the Internet Explorer Security Zones. This means that (by default) scripts run from a UNC are considered part of the Local Intranet zone, and so can run unsigned under the RemoteSigned execution policy:



However, if your Win 2003 server is running the Internet Explorer Enhanced Security Configuration, UNC paths are not included in the zone unless explicitly added:



So your scripts will refuse to run unless you reduce the execution policy down to 'unrestricted', leaving you with lots of nasty messages:

Run only scripts that you trust. While scripts from the Internet can be useful, this script can potentially harm your computer. Do you want to run etc... [D] Do not run [R] Run once [S] Suspend [?] Help (default is "D"):

Yuck (and very frustrating till you work it out).

If IE Enhanced Security Configuration is mandated, the only way round this is to explicitly add the UNCs where your scripts are located into the Local Intranet zone (nb: don't add a http: prefix!)

You can do this manually, or you can just import a set of registry keys if you need to do a whole load in one go (or the same across multiple servers). If you do it through the UI the settings are user-scoped, if you do it via the registry you can add settings either for the current user, or at a machine-wide level. See Adding Sites to the Enhanced Security Configuration Zones in MSDN and Description of Internet Explorer security zones registry entries in the knowlege base for more details.

Log4net in Asp.Net redux: Implement IFixingRequired on your Active Property Values

A while back I noted that the built-in contexts in log4net were broken in the face of ASP.Net's thread agility (and pointed this out to the log4net community).

The workaround I suggested, which was also suggested on the log4net developers list, was to leverage log4net's support for deferred evaluation of logging properties. These are known as Active Property Values, but that's just a fancy/short way of saying 'any object you like, who's ToString() method results in the value that you actually want logged'. This was a pretty neat workaround, eg:

      log4net.GlobalContext.Properties["requestUrl"] = new HttpContextRequestUrlProvider();

    private class HttpContextRequestUrlProvider
    {
        public override string ToString()
        {
            HttpContext context = HttpContext.Current;
            if (context == null) return null;
            return context.Request.RawUrl;
        }
    }

However in one of my logging databases, I recently noticed logging entries attributed to me that had clearly come from someone else, which made me worry for a few hours whether this pattern was broken, or broken when using an appender that buffers (like the AdoNetAppender).

Digging through the source code showed that BufferingAppenderSkeleton does indeed attempt to 'fix' logging entries when they go into the buffer, to guard against exactly these kind of multithreading logging mishaps:

 // Because we are caching the LoggingEvent beyond the
// lifetime of the Append() method we must fix any
// volatile data in the event.
loggingEvent.Fix = this.Fix;

This causes all kinds of intrinsic log4net values (like thread ID, UserName), and the message itself to be 'fixed': i.e. fully evaluated (and rendered via the layout) now, rather than later. Otherwise all the logging events in the buffer would end up being written out with the values in-play at the time the appender flushed (ie potentially another thread/users's context).

This fixing also 'fixes' properties from the log4net contexts (ThreadContext, LogicalThreadContext and GlobalContext) if they implement IFixingRequired. And that's what I missed - one obscure interface:

 // Fix any IFixingRequired objects
IFixingRequired fixingRequired = val as IFixingRequired;
if (fixingRequired != null)
{
 val = fixingRequired.GetFixedObject();
}

But I've missed that for the last couple of years. So… actually I'd rather not think about the implications of that. Meanwhile, if anyone who does any log4net dev would actually like to update the Active Property Value doco, that'd be good, thanks.

Better still, 18 months later, can we have our Adaptive Context now please?

Production Debugging for Hung ASP.Net 2 applications – a crash course

If your app is down right now skip the intro – I’ll understand.

Introduction

First up: debugging – even in the development environment – should never be a substitute to detailed diagnostic logging and appropriate – meaningful – error handling. Any time I fix an exception, if the problem is initially non-obvious, I first fix the logging code. Then and only then do I fix the exception.

Baking in a support mentality early in the development process is the only way to avoid an application that’s hard (and by that I mean expensive) to support. The customer might be paying for the support time, but if you can’t turnaround support issues quickly – and practically any issue in a production business application needs turning around quickly – they start questioning your competence.

But sometimes logs aren’t enough, and even with the best will in the world, sometimes they’re not sufficiently detailed (they won’t - for example – be much help tracking down that memory leak). Sometimes you need to know what’s thrashing right now; to see the smoking gun.

Enter the world of crash dump debugging.

I’d been following Tess’s blog for a while now, but still had a big hole in my understanding. Sure I now knew I could use the !clrstack command to start looking at managed thread stacks, but where the hell was I typing that? This then is my introduction to Production Debugging for .Net applications (or ‘how to understand what Tess is talking about’).

I’m going to be talking about debugging hung .Net 2 ASP.Net applications, but much of this is relevant to debugging any .Net application in a production or testing environment.

Getting the Dump

Assuming you don’t have time to read a more detailed article on this (and the Microsoft guide is 170+ pages!), just do this:

WARNING: Installing the debugging tools on your production server isn’t recommended. That’s not to say you have any choice in the matter, but you should be aware there are security and availability implications that – if your app is hung – you probably don’t care about right now.

Install the Microsoft Debugging Tools for Windows and IISState on the machine where the hang occurs. You’ll need the Debugging Tools on your workstation too, but you can do this later if it’s happening right now.

If your app is thrashing CPU, use TaskManager to get the process ID (PID) of the W3WP process that’s using the most CPU (there may be more than one). NB: PIDs aren’t displayed by default: use View\Select Columns to turn them on.

Otherwise, at the command prompt, enter cscript iisapp.vbs to determine the process ID that matches your app pool.

At the command prompt CD to wherever you installed IISState (C:\IISState is the default). If there is existing content in the output folder, delete or rename the folder. Then execute the following:

IISState –p  <PID> -d

(…replacing <PID> with the PID you determined above)

WARNING: At this point your app will become unresponsive until the dump completes

IISState will put a dump file (.dmp) and a text summary (.log) of the process’s threads etc… in the output directory.

Rename the output folder to something that reflects the date/time, so future dumps don’t overwrite it.

Since we’re doing a ‘hang’, ideally at this point you’d wait a few mins, and then take another dump. If you don’t, you may regret it later and / or have to wait for the issue to re-arise before you can definitively pin it down. However, if the world and his dog want the site back up right now then skip it – it’s your call.

Kill that process. IIS will recycle it, and things should go back to normal for now. Panic over.

Analyzing the Dump

Copy the renamed output directory down onto your machine. For
the process ID you dumped there will also be an IISState summary file, eg IISState-5090.log. This log is a dump of the stack trace of all the threads in the process, and attempts to perform some additional diagnostics for you. Unfortunately, it doesn’t (at present) support .Net 2, so most likely it’s not going to tell you what the problem is. You should at least be able to see a few threads that look something like this:

Thread ID: 13
System Thread ID: d10
Kernel Time: 0:0:4.186
User Time: 0:20:25.702
Thread Type: Managed Thread. Possible ASP.Net page or other .Net worker
No valid SOS data table found.

Begin System Thread Information

# ChildEBP RetAddr  
WARNING: Frame IP not in any known module. Following frames may be wrong.
00 01a9e76c 06ee91cd 0x6eea301
01 01a9eb6c 06cc6e91 0x6ee91cd
02 01a9ee44 06cc6278 0x6cc6e91
03 01a9ee80 06cc3ddd 0x6cc6278
04 01a9eeb8 06cc381c 0x6cc3ddd
05 01a9ef3c 0736fa0f 0x6cc381c
06 01a9ef84 0736f65f 0x736fa0f
07 01a9efec 0725853d 0x736f65f
08 01a9f050 79e9272b 0x725853d
09 01a9f060 79e926ad mscorwks!DllRegisterServerInternal+0x1d6b
0a 01a9f0e0 79e9450b mscorwks!DllRegisterServerInternal+0x1ced
0b 01a9f220 79e943f3 mscorwks!DllRegisterServerInternal+0x3b4b
0c 01a9f37c 07258408 mscorwks!DllRegisterServerInternal+0x3a33
0d 05b30b78 00000008 0x7258408

What does this mean? Well IIState doesn’t understand what’s going on with the stack below that mscorwks dll, which screams .Net 2 stack to you (as does the comment against Thread Type). But the likelihood is there’s not much else we’re going to get out of that file[1]. So it’s time to look at the mini dump.

Open a Visual Studio 2005 Command Prompt window, and from there launch C:\Program Files\Debugging Tools for Windows\windbg.exe (this ensures the .Net 2 framework is in the PATH for the WinDBG process, and makes life simpler –if you don’t have VS2005 installed do it manually). Do File \ Open Crash Dump and brose to the dmp file you created with IISState. If it asks you about saving workspace information just hit no.

It’ll open a text window with some summary information about the dump. This window has a little command bar at the bottom. This is where you do stuff.

The first thing it probably says is “Symbol search path is: *** Invalid ***”. So we type .symfix into the command bar to set it up to dynamically load symbols from Microsoft as required. You can do something to your %path% so you don’t have to do this each time, but I’ve not got round to that yet.

It probably also says

This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(13cc.84): Wake debugger - code 80000007 (first/second chance not available)

…but this is exactly what it says it is – it’s the console debugger attaching (how IISState got the dump), and not a ‘real’ exception per-se.

You can type ~ to get a list of the threads, or ~ *kb to get them with a bit of stack trace too, but we’re still only looking at unmanaged threads here, so pretty meaningless to me.

Type .load SOS to load the managed code debugging extension SOS.dll, then type !threads to see all the managed threads. Now that’s more like it.

NB: If you’re debugging on a different platform to the original crash (eg server was Win2003, you’re on XP), then you’re going to get an error at this point “Failed to load data access DLL, 0x80004005”. The error message contains the fix: execute .cordll -ve -u -l and the correct version of mscordacwks.dll for the dump will automatically be loaded, eg:

0:000> .cordll -ve -u -l
CLRDLL: C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscordacwks.dll:2.0.50727.42
f:0 doesn't match desired version 2.0.50727.91 f:0
CLRDLL: Loaded DLL C:\Program Files\Debugging Tools for Windows\sym\mscordacwks_x86_x86_2.0.50727.91.dll\442337B7562000
  \mscordacwks_x86_x86_2.0.50727.91.dll
CLR DLL status: Loaded DLL C:\Program Files\Debugging Tools for 
  Windows\sym\mscordacwks_x86_x86_2.0.50727.91.dll\442337B7562000\mscordacwks_x86_x86_2.0.50727.91.dll

Your !threads output might look like this:

ThreadCount: 9
UnstartedThread: 0
BackgroundThread: 9
PendingThread: 0
DeadThread: 0
Hosted Runtime: no
      ID OSID Thread   OBJ    State   GC       Context           Domain   Count APT Exception
      12    1  d10 000dce98   1808220 Disabled 00000000:00000000 000d4008     2 Ukn (Threadpool Worker)
      14    2  9f4 000ede10      b220 Enabled  00000000:00000000 000daa20     0 MTA (Finalizer)
      15    3  6d0 00103ad8    80a220 Enabled  00000000:00000000 000daa20     0 MTA (Threadpool Completion Port)
      16    4 14b0 001074d0      1220 Enabled  00000000:00000000 000daa20     0 Ukn
      10    5  4f8 0012e408   880a220 Enabled  00000000:00000000 000daa20     0 MTA (Threadpool Completion Port)
      18    9 1184 06a32e20    800220 Enabled  00000000:00000000 000daa20     0 Ukn (Threadpool Completion Port)
      26    6 15a0 06ada200   180b220 Enabled  00000000:00000000 000daa20     0 MTA (Threadpool Worker)
      28    b   e0 06a5bd40   880b220 Enabled  00000000:00000000 000daa20     0 MTA (Threadpool Completion Port)
      29    c  fd0 06a3c368   180b220 Enabled  00000000:00000000 000daa20     0 MTA (Threadpool Worker)

Notice how the last column tells you what type of thread it is. You can get the managed stack trace for each one of these threads by selecting which thread you’re interested in from the Processes and Threads window (up on the toolbar) and typing !clrstack, but I find it more useful to just look at all of them straight off: ~*e !clrstack

You’ll see a lot of this:

OS Thread Id: 0x84 (0)
Unable to walk the managed stack. The current thread is likely not a managed thread. 
You can run !threads to get a list of managed threads in the process

...because what you're doing is looking at the managed stack trace for all threads (and many of them won't be .Net threads remember), but eventually you’ll see some managed stack traces pop up.

In my case it was a bit of an open-and-shut case, because there was only one .Net thread actually doing anything in the process – kinda a giveaway:

OS Thread Id: 0xd10 (12)
ESP       EIP     
[… trimmed 3rd party vendor stack trace…]
01a9eec4 06cc381c Aspose.Pdf.Pdf.Save(System.IO.Stream)
01a9eed4 069d4a05 MyApp.Server.Utils.WordToPdf.AsposeImpl.Generate(System.IO.MemoryStream)
01a9ef44 0736fa0f MyApp.Server.Document.Implementations.AbstractWordToPdfDocumentGenerationImpl.Generate(...)
01a9ef8c 0736f65f MyApp.Server.Document.GenerationManager.Generate(...)
01a9eff4 0725853d MyApp.Server.ServiceImplementation.DocumentService.GenerateDocumentById(GenerateDocumentByIdRequest)
[… removed ASP.Net framework stack trace …]
01a9f710 65fbe244 System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest)
01a9f744 65fbde92 System.Web.HttpRuntime.ProcessRequestNoDemand(System.Web.HttpWorkerRequest)
01a9f750 65fbc567 System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr, Int32)
01a9f900 79f2d05b [ContextTransitionFrame: 01a9f900] 
01a9f950 79f2d05b [GCFrame: 01a9f950] 
01a9faa8 79f2d05b [ComMethodFrame: 01a9faa8] 

[NB: I've had to edit that for brevity and wrapping]

Ok, so we’re dying in a 3^rd party component. But we’re still going to need to replicate this somehow because (presumably) it doesn’t do this all the time. Fortunately in this case all we need to know is the arguments supplied to our GenerateDocumentById() method to attempt to reproduce in dev. They’re in the dump too, so let’s see what they were.

Select that thread (12) using ~12s (or the Processes and Threads window, and no, I don’t know what the ‘s’ suffix means) and use !clrstack –p to view the stack with parameters. It’s a big list, but in there somewhere is the bit I’m interested in:

0:012> !clrstack –p
[… blah blah …]
01a9eff4 0725853d MyApp.Server.ServiceImplementation.DocumentService.GenerateDocumentById(GenerateDocumentByIdRequest)
     PARAMETERS:
          this = 0x02e596b0
          generateDocumentByIdRequest = 0x02e595b4

!dumpobject (or !do) then tells me the contents of that generateDocumentByIdRequest instance given it’s address as shown above (0x02e595b4):

0:012> !do 0x02e595b4
Name: MyApp.Server.ServiceContracts.GenerateDocumentByIdRequest
MethodTable: 05dcb5e0
EEClass: 06f935ec
Size: 12(0xc) bytes
   (C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\blah blah blah\MyApp.Server.ServiceContracts.DLL)
Fields:
MT        Field    Offset    Type    VT Attr      Value   Name
790fedc4  40001c4  4         System.Int32 0  instance  8229991 documentId

The bit I care about on this is the documentId field itself, and being a value-type field (Int32) I can see the value right there: 8229991 (for a reference type like System.String I'd get another address to the instance itself, ie have to perform another !do).

And in this case, that was that. Well, it turned out to affect more than just this document ID, but at least I had enough to reproduce in the dev environment. And just in case anyone read the stack trace carefully, by the time we got to this stage Aspose already had a fix out for this issue, so I can’t really knock them. Finally whilst I was at it, I had a look at the objects in memory and began to wonder whether I’ve got a memory leak, so there’s probably a follow up to come.

Other SOS commands to look at:

!threadpool (also this is where the CPU utilization hides)
!eeheap –gc (see memory allocations that are subject to the GC)
!dumpheap –stat (see object allocations by type)
!help (gives help on the last extension loaded, so SOS commands in our case)

Other Resources:

Installing and Configuring IISState

SOS: It’s not just an ABBA Song Anymore: A good introductory article on MSDN with a dreadful title

If Broken It Is… Fix it You Should [Tess’s Blog]: This will all make a lot more sense now

Production Debugging for .NET Framework Applications: This 170 page document from Microsoft’s Patterns and Practices team is allegedly out of date, but is an invaluable reference, and what I based most of this article on

Debugging Microsoft® .NET 2.0 Applications: I haven’t read this MSPress book, but the v1 version got quite a lot of citations when I was researching this stuff

---

[1] So why did we take it? Because if it turns out it’s an IIS issue, and not your code, then someone will be able to get something out of it. Just not you.

Nesting Transactions in Stored Procedures (Part 1)

Nesting transactional stored procedures is an absolute minefield.

Many projects I've worked on have just avoided opening transactions within stored procs altogether, and rely entirely on ADO / ADO.Net transactions in the application. This is certainally a simple way to avoid the problem, but isn't always appropriate. At some point there's always a data integrity rule that you want enforced at the lowest possible level (to avoid it being accidentally bypassed - I'm thinking auditing, or the classic 'move money between accounts' scenario). Putting that rule in a stored proc, and making that the only point of call for writes to the table involved makes it pretty safe. This is even more important if it's not just your app using the database.

But nevertheless avoidance is appropriate in many cases, if only because handling the nesting of transactions within nested stored proc calls is a complex matter. To summarize the problems:

Rollback and commit are asymmetric in nested transaction scenarios

Rollback aborts the whole nested stack, whereas commit just decreases the nesting level (only the final commit actually commits anything). So error handling has to behave differently in the proc at the 'root' of the transaction than it does in nested procs. Ulp!

SQL error handling is hard, and poorly-understood

Even basic TSQL error handling is poorly understood: having an exception raised in the calling application but the rest of the stored proc continue executing is pretty counter-intuitive, right? Whadyamean the transaction committed anyway!?!

The fact is that most SQL errors don't affect batch completion without explicit handling, some abort the batch but not the transaction, and some abort both. Furthermore, XACT_ABORT changes which do which. Erk!


Fortunately we don't have to come up with a solution, because there are already well-defined patterns for handling both these issues. There's an excellent article on the subject on Code magazine that's been the basis for my approach for a few years now:

Handling SQL Server Errors in Nested Procedures by Ron Talmage (I use the multi-level model)

Unfortunately barely any developers appear to be aware of the problem, let alone the solution patterns. This is particularly problematic, because either of Ron's solutions must be applied consistently in order to work properly. If that's an issue, maybe you're best going back to the avoidance pattern.

Anyway, the big question for me is "how has this pattern been affected by TRY ... CATCH functionality in Sql Server 2005?". We'll look at that next time...