Avoiding Reporting Services ‘StreamNotFound’ Exception when Viewing Multiple Instances of the Same Report

I’ve been using Reporting Services 2005 to build a series of BI dashboards, initially using the SharePoint reports library, but more recently using straight ASP.Net pages. In essence we’ve been using SSRS as a headless charting engine.

On some of our dashboards we occasionally saw little red crosses rather than our charts – IE’s way of telling us the image hadn’t loaded. Didn’t happen very often, and at the time we had bigger fish to fry, but unfortunately when we moved to our big new multi-core server, it happened a lot more.

In the logs we saw a lot of this:

w3wp!library!f!01/21/2009-12:40:25:: Call to GetItemTypeAction(http://myserver/reports/BlahBlahReport.rdl).

w3wp!library!f!01/21/2009-12:40:26:: e ERROR: Throwing Microsoft.ReportingServices.Diagnostics.Utilities.StreamNotFoundException: The stream cannot be found. The stream identifier that is provided to an operation cannot be located in the report server database., ;

Hotfix 913363 looked promising, but that was included in SP1, and we were SP3, so unless it had regressed, that wasn’t going to help.

What we determined was that this only happened when the same user requested the same report multiple times (with different parameters) at more or less the same time. And that happens a fair bit for us: our dashboards (both the SharePoint versions, and the later ASP.Net ones) basically rendered down to pages of IFRAMES pointing back at SSRS, and sometimes multiple charts on one dashboard would actually just be the same report re-executed with different parameters. So my ‘race condition’ alarm was off and running.

For any report on the dashboard, IE does this:

• Load the HTML in the IFRAME
• Go back to the server and load all the images, javascript etc…

Any generated images in the report have to be cached momentarily on the server between when the HTML is generated, and when the browser comes and ask for them. And of course the browser has to supply some kind of reference back to the server so it gets the right images, and not the images from a different report or a what a different user is viewing. So there’s a Reporting Services Session (which you probably read about the first time you got a rsExecutionContextNotFound exception, right?).

So we speculated that SSRS was getting its sessions in a mess, and somehow a subsequent request to the same report, using different parameters, was throwing away the results of the previous execution (the chart image we are after) in the process of creating a new one. At which point Graeme discovered you can disable session cookies (in the ConfigurationInfo table) which (after an IISReset) indeed fixed the problem entirely.

But why the problem in the first place?

Looking at the HTTP traffic between the browser and the server, one sees that the scoping of the image request is indeed only scoped to an imageID and report as part of the GET url, and within a execution session as part of a cookie:

GET /ReportServer?BlahBlahReport&rs%3aFormat=HTML4.0&rs%3aImageID=d95ebcc7-deba-4734-93c9-270468bd133b HTTP/1.1

Cookie: ConsoleVisible5dd14218-294b-424e-a33a-013236be5290=false; RSExecutionSession%3ahttp%3a%2f%2fserver%2f%2fBlahBlahReport=54m0qr55hytlvn55nzkjxr55

I’ve not followed this through entirely, but basically the trouble is your browser only has one RSExecutionSession cookie, which appears to contains only one execution ID per report URI. You’d imagine then that what you’d get is just the wrong image (outstanding requests for images from the first execution are submitted using the execution ID from the second session), but you don’t, so the most likely scenario I can come up with is that starting the second execution for the report implicitly removes the artefacts (images etc…) still in cache from the first execution. If this happens before the browser’s loaded them then it’s little broken crosses time…

Once you turn cookieless sessions on, however, the GET requests look more like this:

GET /ReportServer?%2fBlahBlahReport&rs%3aSessionID=ucqtvo55wacqugqjln3l2045&rs%3aFormat=HTML4.0&rs%3aImageID=e9a2d08b-23cb-4586-8399-eb42ed5558a7 HTTP/1.1

ie they contain all three of report, execution and image. As such executing the same report multiple times in parallel doesn’t create any race conditions. This seems like such a better way of doing it that I’m surprised this isn’t the default behaviour (I guess it can’t clean up the temporary cache quite as aggressively, but other than that I can’t see the downside).

It’s worth noting that this kind of thing is an absolutely classic pitfall with any form of HTTP session state, which application frameworks like ASP and ASP.Net typically sidestep by serializing requests within a session and thus avoiding order-of-execution issues. However in this case, even if SSRS serialized access it would have to ensure it serialized it in the right sequence to really fix the problem. Tricky.

 

PS: Reporting Services 2005. Haven’t tested on 2008.

Tablet PCs: Still Too Heavy

I went out and bought a nice new HP Touchsmart TX2 the other day. And I’ve been very happy with it. For one thing I’ve hankered after a tablet for some time, but the shear ‘double the price’ barrier has always stopped me. But that’s not the case with the TX2, which goes for under $2k and is fairly comparable with it’s non-tablet brethren.

And it’s a touch-screen as well, though the multi-touch capabilities are pretty much just gimmicks supported by a handful of built-in HP apps until Windows 7 comes along. That being said, I’ve already got very used to flicking the screen for page up / page down operations (though I still think a scroll-wheel on the side would be just as easy).

But it’s just too heavy.

It’s not heavy by current standards: it’s sub 2 kilos, and even the (far more expensive) Fujitsu Lifebooks only go down to 1.6 kg (in a 12”). And it’s definitely not too heavy to carry in a bag on your shoulder. But in my totally non-scientific ‘picking things up with my hand’ testing I’ve determined that the ideal weight for a tablet PC is less than a kilo (the weight of a Dell Mini 9 in fact). Tablets are for holding, right? Well at 2kg (and without much grip on the edges) your thumb is getting a good workout, let me tell you.

So my requirements for a tablet PC that will genuinely shift usage patterns:

• <1kg
• 4 hours battery life minimum, ideally 10, included in the weight above.
• Some decent ergonomics around the frame so one can actually hold onto the damn thing.
• About $600

Various rumours regarding tablet netbooks seem to be coming to fruition, so provided the dual-core Atom has enough grunt to handle the handwriting recognition, then this will be all good. Many a time I’ve sat there with the Mini 9 on my lap thinking ‘if only the screen would just flip round…’

Coming soon: less hardware ramblings. Promise.

I <3 Dell Mini 9

I’ll ‘fess up to being totally unimpressed with progress in the PC industry in the last 10 years. I appreciate we’ve made some fairly big steps technologically speaking (multi-core is right up there), but the growth of power consumption and (as a result) heat and noise dissipation has had me absolutely appalled. Kilowatt PSUs! Heatsinks the size of a fist! 3D graphics may have come on in leaps and bounds, but does it have to sound like a 747 about to take off? It’s all just so crude. I feel like opening the drive bay and shovelling in more coal.

I constantly think back to the Acorn Risc PC I had in uni, which seemed pretty powerful at the time, but whose CPU ran cool to the touch. It’s been downhill ever since.

Similarly I like to complain about laptop battery lives. What’s the point of a mobile computer if you have to be tied to the wall socket every two hours? But then laptops aren’t actually mobile computers, they are portable computers: portable like a portable TV is portable: you can pick them up and carry them around, but realistically to use them you’ve got to plug them in. This drives me *nuts*, particularly as I can easily burn through two hours before I really get started on anything, by which time it’s time for a battery break.

Enter the netbook.

The wife’s old laptop (one of my cast offs) finally went back to the Dell recycling plant in the sky, and so we got her a netbook as a replacement. A Mini 9 to be exact.

And it’s fantastic.

I can’t really say how it compares to it’s other netbook brethren, but if this is what the rest are like then viva the revolution because for on-sofa / in-kitchen[1] / in-bed casual browsing this form factor and weight (1kg) is absolutely on the mark[2]. Not too big, not too small. The Mini is SSD so it’s totally silent (excepting occasional sound card cross-chatter). The Atom CPU runs cool, so it doesn’t make your lap all sweaty. And it’s cheep enough to buy on a whim ($550 currently)[3]. Oh and it’s XP, so it can run .Net.

I don’t totally buy into it as the nirvana of mobile computing, because 3.5 hours is still not enough, but as something you can happily take room to room in the house it’s bang on.

Here it is cosying up to a Dell 15” Latitude: it’s exactly half the size[4]

Here it is dwarfed by my 17” Inspiron 9300, which is only it’s grandfather in Laptop-generations:

Ok, so the 9300 still beats the pants off it in terms of performance, but the Atom team have seriously lit a fire under the rest of the industry because it’s already good enough for most people, and it’s only going to get better. You’d have to imagine when the average punter catches on that Atom-based desktops, like the the EeeBox, can happily handle their Word/Email/Web demands there’s going to be a fairly major correction in the market, whatever the critics say. Just check out the price differential. That 90% of your horsepower you donate to SETI, Folding@Home or whatever: why not just not buy it in the first place and save your kilowatt hours? And that’s before you even consider any cloud computing offload.

In years to come we will look at our current CPU technologies in the same way we view CRTs, incandescent bulbs and the internal combustion engine. Here’s hoping we are making the first footsteps now.

 

[1] For recipes. I mean, no-one really bought an internet fridge, right?
[2] Though it would be better as a tablet
[3] Frankly it’s cheep enough to use as a universal remote control, if only all your devices talked Bluetooth
[4] If it looks like its overhanging on the right, it’s just the perspective

Oracle OLE DB provider precision issues with SSIS 2005

We had this problem[3] before where SSIS would keep complaining that the Oracle client wasn’t giving it the correct precision information for numeric columns:

Error at Import Broken Stock On Hand Levels [DTS.Pipeline]: The "output column "WORKD_TONNES" (42)" has a precision that is not valid. The precision must be between 1 and 38.

We were using Oracle 9r2 client + OLE DB provider against an Oracle 8 database, and I just assumed that one of those wasn’t passing though metadata it should have, leaving SSIS to see the precision and scale as 0:

You can explicitly set the scale/precision in the advanced editor, but it keeps wanting to overwrite it every time you look at the query, which is a major pain in the arse. And explicitly TRUNCing or ROUNDing the column in the source query doesn’t seem to make any difference, even while you’d think that would make the precision explicit. So instead we used the Microsoft OLE DB Provider for Oracle (MSDAORA), and everything was happy.

Well nearly.

Actually it turns out that rather than MSDAORA ‘working’, all it actually does is default the precision and scale as 38,0 in scenarios where it can’t determine the precision/scale. And this happens rather a lot, since an Oracle Numeric doesn’t actually have to have a length or precision specified (though it should): it can be some kind of wierd variable precision float/numerical hybrid, which isn’t something that SSIS 2005 really caters for (nor other parts of Sql for that matter). Anyway, the point is that 38,0 isn’t the right answer either: whilst it’s fine for integer data anything after the decimal is lost, and that can take a while to pick up on.

So I’m not sure which is worse. On the one hand you can use the Oracle OLE DB driver, and all your precisions will be wrong until you fix them, but at least it’s in your face enough you know about it (though sometimes you can’t even OK the dialog, to get into the advanced editor and fix it, and/or it whinges about mis-matches with error output columns).

On the other hand the Microsoft driver makes a good stab at ‘what you probably want’, but as a result you lose precision on decimal data. And even if you set the precision up, it’ll overwrite it again (silently) when you touch the query, so it can go from working to not very inconspicously.

Probably the only safe way here is to use the Microsoft driver, and modify all your queries to cast all non-integer variable Numeric columns to strings[2]. That way the conversion into decimal data has to be explicitly defined, and can’t be lost quite as easily. It’s a bit more work of course, but at least the output is right.

Which brings me back to why I started looking at this again because there is no 64 bit version of MSDAORA [1] (I’m not even sure if MSDAORA is actively supported anymore), and we’ve had all manner of fun getting the 32 bit version running talking to either of the 32 and 64 bit versions of the Oracle client. We failed, but I may have just configured it wrong.

So instead we are in this ridiculous situation where we use MSDAORA at design time, and the Oracle OLE DB driver on the server at execution time, which creates all manner of warnings in the logs and also means some Sql (with inline comments) works on a developers machine but doesn’t work on the server.

It’s a mess.

[1] Nor Jet either, if you were crazy enough to load something from an Excel file. Who would do such a thing?
[2] Interestingly this is exactly how SSIS 2008 gets round this problem: it gives up, maps the variable-Numeric column as a string and leaves the conversion up to you
[3] Obviously the first problem we had was the whinging about the DefaultCodePage issue
[4] Managed to get all the footnotes in backwards for this post, which is pretty poor.

Consistency in SSIS

It just occurred to me that the Derived Column Transformation editor (and other SSIS expressions) take a C (ie C#) style approach to equality:

…whereas SSIS 2005 can only use VB.Net for Script Tasks / Components.

Go figure.

 

[ok, yeah, fixed in SSIS 2008. I know. Lucky you]

GDR Grrr...

I've totally had it with Visual Studio Database Edition 2008 GDR.

CREATE VIEW [dbo].[DateRangeExpanded] WITH SCHEMABINDING
AS
select n.Number, n.Date
from dbo.Numbers n
where n.Date >= CONVERT(DATETIME,'2007-12-01') and
n.Date <= datediff(day, 0, DATEADD(MONTH, 6, GETDATE())) 

Results in

Error TSD03127: Cannot schema bind view '[dbo].[DateRangeExpanded]'
because name 'DATETIME' is invalid for schema binding. Names must be in two-part format and an object cannot reference itself.

WFT? SQL Server is quite happy with the above, it's just GDR getting it's knickers in a twist yet again: DateTime is a type not a column. I have found no way of making it happy with this one[1], other than taking the schemabinding off, and I am pretty pissed about limiting my database design just to accommodate GDR's inadequacies.

If this project wasn't finishing up in a few weeks, GDR would be in the bin by now.


[1] CAST vs CONVERT doesn't make any difference, for one thing.

Dang! SSAS deployment configurations are held in the .user file

I really wish Microsoft would make up their mind where deployment target locations should live (ie setting the destinations for Publish / Deploy)

For SSRS this data is stored in the project file (.rptproj) , per configuration, so anyone can get latest on a project, change their solution configuration to (say) 'Test', hit deploy and away they go, happily deploying to your Test environment. This is how it should be.

For SSAS (2005 / 2008) this data is stored in the .dwproj.user file, so for any new developer / PC / workspace that comes along all those deployment configuration settings have to be re-setup (or copied from another workspace), or all their deployments are to localhost. This is just really irritating. As I discovered today.

For an XBAP (or just a WinForms ClickOnce project) a single deployment target is stored in the project file. Other, previously-used deployment locations, are stored per-user. None are correlated with a project configuration in any way. This 'deploy to prod' design mentality drives me nuts.

For an ASP.Net website the (single) deployment location isn't even stored against in the project at all, but squirreled away in %userprofile%\Local Settings\Application Data\Microsoft\WebsiteCache\Websites.xml. Of course! Web application projects put it soley in the .user file, which is at least less obscure, if no more useful.

Now I'd be the first to say that deployment to test environments etc... is best done from a seperate scripted process, but I accept that not everyone has my affinity for long PowerShell deployment scripts. Plus sometimes I just can't be arsed. So am I completely insane to wish that the in-IDE Publish / Deploy mechanisms would work in a manner that could be actually used would be usable within a team, and not scatter the deployment target information quite so liberally?

Mini USB Convergence Finally Coming

In a rare victory for common sense, it seems that the world’s mobile phone makers have at last agreed to converge on using mini-USB as a universal mobile phone charging standard. Not only that, but chargers will be optional at point of sale, so you won’t end up with a house full of them.

The plan is to do this by 2012, which is a bit slow in my book, but it’s all progress, right?

Substituting view/synonym causes naming collisions in Database Edition GDR

The other day we discovered a duplicated table in the database. We're doing some refactoring, and a certain amount of duplication during this process is inevitable, but to cut a long story short, the two instances of this table needed collapsing into one.

So I dropped one instance, and created a synonym in it's place (pointing to the other). This saves fixing all the views and external references (in the SSAS cube) to that table. Those references will go away over time.

But when I tried to sync the database using GDR it just didn't get it. It's quite happy to create the new synonym, just too dumb to drop the original table first - so there's a naming collision I had to fix by hand. It basically does this:

CREATE SYNONYM [dbo].[Table_2] FOR [dbo].[Table_1];
GO
DROP TABLE [dbo].[Table_2];

Dur! And it does the same thing if you substitute a view for a table too:

CREATE VIEW [dbo].[Table_2]
AS
-- Create view instead of original table
SELECT One FROM dbo.Table_1;

GO
-- Drop original table
DROP TABLE [dbo].[Table_2];

Anyone see the problem here? This is not an edge case guys.

I should point out that this only happens when sync'ing a database with the database model. If you sync two live databases, GDR gets the execution order right.

Not impressed.

(By way of comparison Sql Compare 7 is quite happy with both these scenarios, whether comparing against a live database or a snapshot. Drop table; create synonym - it's not that hard. And Sql Compare 7's been out a while (8 is nearly out))


[Update: 2008-02-20] And another thing: GDR's idea of dropping an object (eg: if you sync to a dev database and the object isn't there any more) is to comment out the object definition in the database project. All well and good, but it's too stupid to also comment out any extended properties the object might have (say the ones the view designer put in to store the view layout), so obviously those extended properties cause the build to crap out as the object they reference doesn't exist.

Not sold on Database Edition 2008 GDR

Just in time for the end of 2008 Microsoft finally got Database Edition 2008 GDR out the door. This replaces the version that came with VS 2008, primarily bringing with it Sql 2008 compatibility.

So GDR is now Microsoft's 3rd bite at integrating the database development workflow into Visual Studio. Lets start with what it does right:

Obviously the main benefit of bringing the database into Visual Studio is you get source control for free. Sure - you've been able to script out your database and add it to source control before, but this way you can check in database changes and the code that depends on it as one atomic checkin, against a work item. That's actually pretty neat from a tracking / branching point of view.

And because it's a fully model-driven system, all the SQL in the project is tested for consistency and correctness at build time. You could do this before of course (script out and build a structure only database to check it builds), but this way you know any problems you find are local to your workspace, so it really was you that broke them.

The output from the build process is a model that completely describes your database schema, so this can be used to sync up test and production databases, either via the IDE, MSBuild or a redistributable command line utility that can be incorporated into your deployment / install process. Not that you couldn't have done all this with RedGate's SqlComparison SDK of course, but if you struggle to get licences other than MSDN purchased, this does make your life a bit easier, and at least it's been constructed with a build-once-deploy-many philosophy, unlike the clowns behind ClickOnce.

The model also (allegedly) provides a safer platform for database refactoring: it's supposed to 'remember' things like column renames as renames and treat them as such the next time a database is sync'd to the model. I guess only time will tell how well that works in the real world given, for example, multiple databases at different schema versions (ie Prod database 5 versions behind Test etc… ). Are the refactorings going to be journalled away and somehow replayed in the right order? If so, how does it know at which version to start? It's not a simple matter.

But Microsoft has still totally, totally missed the point, because there is still no design support. Not one. Nada. Not even a schema browser.

Hey I can see where this is going, and I can see they're laying foundations here, but in my mind the fundamentals of a database development tool are that they assist in database development. You know - the core stuff: creating objects and sprocs. Source control and a deployment workflow are secondary to that. But with GDR you'd better be hankering after the Sql 6.5 days, because you're going to be writing lots and lots of TSQL. The best I can say is it's colour coded, but didn't we only just get IntelliSense in Management Studio? It just doesn't work for me.

But the alternative - do your dev / design in Management Studio and sync back to the model - is equally flawed, if only because you've got to keep syncing back again (and the sync UI is still a bit ragged). That's a whole lot of friction for relatively nebulous benefits (especially if you already have your database schema under some level of source control), and in my mind questions the whole utility of the product. What's the point of 'integrating' into the Visual Studio IDE if you have to keep tabbing out to do anything even remotely useful (think: execution plans)?

Similarly, whilst there's a Data Compare feature, it's not actually part of your database project, just something you do on an ad-hoc basis. So there's no easy way of designating a certain subset of tables as 'reference data' that should essentially be considered part of the schema, rather than part of the contents, and sync'd up when you redeploy. I'll accept that's probably a vNext feature, but its absence calls into question the viability of using GDR as an end-to-end database deployment manager in the first place, compared to tools like SqlPackager (or scripting using SqlCompare \ SqlDataCompare APIs).

And of course the model parser isn't perfect, so occasionally a valid bit of SQL creates build errors because the model can't fathom what it's doing. In my case it can't seem to resolve the columns in a CTE that queries another (properly referenced) database, and barfs about ambiguous references where there are none. Unfortunately fidelity is always the problem with Model Driven Architectures, even ones that use a DSL as the model, because they have to - at some point - cater for a lowest common denominator, and at any rate they are in an arms race with whatever it is they're modelling. Microsoft is touting it's extensible architecture, and IBM are apparently writing a DB2 provider, but what do you think the chances of 100% functionality coverage are? And for Oracle? I'll wager pretty low. So be prepared to take an 80/20 view, and put some bits into pre/post build scripts and / or re-structure them to keep the parser happy.

Then there are the little things. It creates a schema and object hierarchy when your database is initially imported, but it doesn't enforce it, and the Add \ New (whatever) templates don't pick up on it, so all your new objects end up in the wrong schema unless you're concentrating. If you do anything with any system views you'll need to reference the master database ( %ProgramFiles%\Microsoft Visual Studio 9.0\VSTSDB\Extensions\SqlServer\2008\DBSchemas), so why is it not referenced to start with? And the doco's not been updated for RTM so contains all kinds of outdated information around the command line interfaces, and absolutely no content in the API section other than listing the namespaces.

http://social.msdn.microsoft.com/Forums/en-US/vstsdb/thread/3f0f0888-9b37-4279-81d4-b924595630a8/
http://social.msdn.microsoft.com/Forums/en-US/vstsdb/thread/a966ac3c-c039-4c3f-9b6d-2882d9874282

So this still strikes me as neither fish nor fowl. Given the designers already exist in Management Studio it's a bit pisspoor that the DataDude team haven't borrowed them yet. I imagine I'll still be doing most of my work in Management Studio, and at best attempting to keep the database model in sync. And I suspect that for me, and many others, this will be just too much friction to bother with and so the whole effort will be more or less wasted.

Which is all a bit sad, given it's 2009, I've written a decades worth of database-centric apps, and we've still not got this right. At least no-one's actually paying for this any more.

Sql 2005 Install Woes on Shiny New Big Server

I had a new one this time.

I (of course) ran into the install-hangs-on-setting-file-security issue (KB910070), but I was expecting that. What really threw me was after that then install then just kept dying, leaving this in the logs:

Faulting application sqlservr.exe, version 2005.90.1399.0, faulting module sqlservr.exe, version 2005.90.1399.0, fault address 0x0000000000b323f0.

This really threw the installer too - after an uninstall, and even after a manual cleanup, the installer still though there was an instance hanging around. Which it was. I had to manually delete the SQL Services (using SC), a bunch of instance registry settings and the instance files directory (the MSSQL.1 folder) before I could finally get it to re-install. I guess the uninstall died too.

So then I tried installing again, and again. And again.

So I started speculating. Was it the virus scanner .... No. Could it be the Sql 2005 installer didn't like .Net 3.5 sp1? Uninstall... No. Was I definately using the 64 bit version... Yes. Could I slipstream SP2 and workaround some issue I didn't understand yet... No. Was it the monster 24 cores the server had (4 x hex core)... maybe.

There is a known issue with Sql 2005 instal failing with odd number of cores (ie Phenoms). That (obviously) doesn't count: but maybe Sql can't either. So I used the instructions in KB954835 to criple my monster server down to a single CPU, and then it all installed just fine. I can now install SP2 (3 actually) which allegedly should then make it all work.

Moral:
Obviously I should have been installing Sql 2008 instead
It's clearly becoming way too easy - with multi-multi-core boxes - to drop into some massively unexplored race condition territory in something that's otherwise really quite stable and well tested.
You can have too many cores

Other thoughts:
There must be a better way to restrict an installer or app to run on only one core without farting about with BOOT.INI
Once I put SP3 on it better all work otherwise the boss is going to be really pissed ('What, those other 23 cores? They're um ... spares')

Forcing Web Part pages into Edit Mode

(Before I forget again :)

Even if you've hacked about with a SharePoint master page to remove all the SharePoint chrome (including the edit button and suchlike) there are URL hacks you can use to get the page back into edit mode:

pageurl?ToolPaneView=2

(There's a few more like this, see Sharepoint Tweaks... )

...as well as the hack to get a page into the Web Part Maintanance page:

pageurl?contents=1

Special cleaning instructions for Nokia 6110 Classic

Mine had a bit of a smudge on the screen so I thought I'd pop it through the wash and clean it up. The critical bit here was to make sure the phone was in the pocket of my cycling shorts, so it didn't get bashed / scratched by the washing machine drum.

After an afternoon out on the line drying (still in shorts pocket) bingo: one nice clean phone.

Nokia: tough as.

Recordset to Powershell object collection

I've found this pretty damn useful: it converts an ADO.Net IDbDataReader into a collection of powershell objects, mapping column names to property names so you can sort and filter and whatnot using standard PS syntax.

# Executes an IDbCommand and converts the resultset
# to a Powershell object collection
function ExecuteCommand(
  $command = { throw 'Command must be supplied' }
){
  $rows = @();
  $reader = $command.executereader()
  while($reader.read()){
    $values = new-object object;
    
    for($i = 0; $i -lt $reader.FieldCount; $i++){
      $name = $reader.GetName($i);
      $value = $reader.GetValue($i);
      
      Add-Member -in $values noteproperty $name $value
    }
    $rows+= $values;
  }
  $reader.Close();
  $rows
}

Next week: why I was doing this in the first place (much more interesting)

Frequently Bought Together

Just remember kids: no-one likes a smart arse.

RedGate SqlCompare wins again

...in my highly subjective 'which tool are we going to use for database schema synchronisation?' challenge that is.

RedGate's prices seem to have gone up again ($595 USD for the comparison bundle[1], but if like me you want to use command line interface you're looking at 2x $595 USD, or 3x if you want the API too). And support and maintenance is on top of that.


So I had a good look around and considered the options:

Visual Studio for Database Professionals - easy option since it's now included in our Team Suite SKU. However even the 2008 version is still pretty crude, with very little option to change the generated delta SQL, and as a result scripting out unnecessary changes (like rebuilding your tables via a temp table just to get the column order 'right') and doubtless doing things with role memberships that I didn't want. So that didn't last long.

SqlDelta I like a lot. It does schema and data compare, and it's got a command line interface all for $330 USD - cheaper than RedGate's most basic non-pro compare bundle. And it's from down under. But it choked on my instead-of triggers on a view (either scripted them as CREATE when they needed to be ALTER or vice-versa). So I had to faff about to get a sync to work. That's an immediate fail.


Then there's ApexSQL Diff. But I didn't really get round to using that. Which is where the 'highly subjective' bit of this review comes in, not to mention the 'use the first product that works, stop playing around and get some work done' voice-of-conscience.

So RedGate it is.

[1] There's an option, not available on their website, to get a Sql Compare Pro bundle, which if you need the pro editions + the API basically means you get them for 2/3 price.

Remember to enable MARS when using Snapshot Isolation from SSAS

We started getting this error when processing our cube:

OLE DB error: OLE DB or ODBC error: Cannot create new connection because in manual or distributed transaction mode

It went away when:

• We changed to using ReadCommitted isolation, rather than snapshot
• We processed the cube using Maximum Parallel Tasks = 1

Reading knowlege base article PRB: SQLOLEDB Allows Only One Connection in Scope of Transaction lead me to think that SSAS was trying to open multiple connections within a transaction, which isn't allowed.

Which got me thinking about MARS. Not quite sure why it wasn't on to start with, but I enabled it, and then everything was fine again.

Turns out this is actually a RTFM, if you pay attention when reading How to enable the snapshot transaction isolation level in SQL Server 2005 Analysis Services . Of course it's only in Snapshot mode that SSAS attempts to ensure database consistency for the duration of the entire Process operation, which is why it's not a problem using ReadCommitted isolation.

If McCain wins next week...

...I'll leave the US

and I don't even live there!

Viewing the MDX cellset with WPF

When executing an MDX query there's various bits of useful metadata that can be returned in the cellset over and above the members and dimensions you've explicitly specified in your query. This can include things like the formatted value (as specified by the cube definition) as well as other attribute values for the dimension member you're explicitly querying against.

This kind of stuff can be invaluable if you're writing your own front-end app to access OLAP data, not least because it saves a whole heap of faffing about with WITH clauses (query scoped calculated measures). Something like:

with Member (
  [date].[date].[date].currentmember.properties("Year")
) as TheYear,
select {
  [measures].TheYear, [measures].[measure]
} on Columns,{
  [date].[date].[date]
} on Rows from Cube

...can just become:

select {
  [measures].[measure]
} on Columns,{
  [date].[date].[date]
} dimension properties member_name, member_value,
 [date].[date].[date].Year on Rows
on Rows from Cube

Trouble is you'd really struggle to work this out since all that metadata is helpfully hidden from view when you execute MDX in BIDS (and MDX studio), which makes it all a bit hit-and-miss. So I wrote a little WPF app just to visualise the actual cellset returned. Pretty basic stuff - load the results into a dataset and bind it to a grid. I could fiddle with my DIMENSION PROPERTIES clause with immediate gratification.

But it took me hours to get the binding working. One of the problems is that the MDX columns have names like '[Measures].[MyThing]' and you can't just set that as the property name of your binding and expect the binding infrastructure to cope:

{Binding Path=[Measures].[SomeMeasure]}

The binding infrastructure sees the dot and tries to walk the path, with predictable results:

System.ArgumentException: Measures is neither a DataColumn nor a DataRelation for table Table

[NB: If you had a column simply named SomeMeasure this would work, due to the magic of ICustomTypeDescriptor, but that's another story]

So instead you have to use the indexer syntax on the DataRowView:

{Binding Path=['[Measures].[SomeMeasure]']}

Or (if that made you wince)

{Binding Path=Row['[Measures].[SomeMeasure]']}

But those don't work either:

System.ArgumentException: Column '"[Measures].[SomeMeasure]"' does not belong to table Table

Even whilst the same binding path works 'just fine thanks' in the debugger. It took me a long, long time to realise there's an extra set of quotes in that error message. The WPF binding syntax doesn't require quotes for string indexers:

{Binding Path=Row[[Measures].[SomeMeasure]]}

It looks so wrong but it works.

SharePoint: The final CM frontier

For a long time I thought Biztalk was the elephant in the room when it came to Configuration Management - specifically version control. I'm not talking here about Source Control, I'm talking from a deployment perspective - the 'deploy a given, atomic, integral version into production' problem.

For a traditional .net app (Winforms, ASP.Net, WPF, whatever) it's pretty much sorted. We've had source control integration in the IDE since the dawn of time, CCNet for years, and it's even easier to get CI going now that TFS 2008 supports it out of the box. That's not to say everyone actually does so, but it's there if they want it, right?

Databases are a bit harder, but there are lots of tools around that you can incorporate into your CI cycle, and now that Microsoft's in the space with VSTS for Database Pros (now included with VSTS Dev Edition) the barrier to entry has been dropped again (though it's still a pretty poor story in the SSAS space).

However for Biztalk things are not so rosy. You get source control at least, so you can practice a unified versioning / labelling scheme, but building the project doesn't give you all the artifacts you need to actually deploy into an environment. For that you've got to IDE-deploy to a dev BizTalk instance, configure and then at least export out the binding file (if not a complete MSI). There's NAnt and MSBuild tasks to do the building, but I've not seen anyone wrap up the whole end to end, so Biztalk deployments languish in the 'manual effort with concentration' department. A direct result of this is that integrating your code with biztalk orchestrations (via WS / WCF) is increasingly seen as a more manageable approach than embedding your code, thus alleviating / sidesteping many of the issues.

I thought Biztalk was the frontier of version management - that is until I started working with SharePoint.

I'd previously accepted that working with SharePoint using the Web UI was working uncontrolled, but I'd always imagined that real SharePoint developers used SharePoint Designer, checked things into source control and had established patterns for migrating content between development and production server instances. How wrong I was. Making my first tentative forays into 'how to do this properly' I was struck by the complete absence of any guidance. It appears this is most definitely not a solved problem, which Jeremy pretty much confirmed last night in his great RDN talk on the subject. There's tools out there to manage the problem, but they're pretty new on the block. The book, quite literally, hasn't been written yet.


This is clearly a pretty major failing on Microsoft's part. I appreciate that the main thrust of version-management with SharePoint is version-management of content rather than version-management of configuration, but clearly if they want developers to embrace SharePoint as a platform they're going to have to do a bit better. Time for a 'developers, developers, developers' rant perhaps?


It seems to be that the more productivity-orientated the development environment, the less effort has been put into establishing a viable CM story:



(You could substitute 'maintainability' for 'Ease of CM / CI' if you like: one tends to drive the other)


Clearly it's easier to diff / merge lines of code rather than SharePoint XML manifests (or Workflow XAMLs), but it's a complete abdication to leave these higher-level (ie non-codey) development environments quite so dramatically out the cold.

Surely there's got to be a better answer than using DiffDog for everything?

Getting just the date from Sql Server's datetime

Whilst many still advocate using Convert() to drop the time-bit from a datetime[1], going off to a string like that is nothing like as efficient as the numerical alternative:

CAST( FLOOR( CAST( @dateTime AS FLOAT ) ) AS DATETIME )

Sql stores the days since 1900 in the first 4 bytes, and the time in the last 4 bytes. Throwing away the bit after the decimal place (what the above does) just strips those last 4 bytes right how.

But... why AS FLOAT? Why convert straight to an INT?

CAST( CAST( getdate() AS INT ) as Datetime)

Turns out this gives us tomorrow's date! I guess this is because the starting date is actually Jan 1st 1900, and not Jan 0, which could be a classic off-by-one error if you weren't awake.


[Update]: Because casting direct to an int rounds up in the afternoon! Doh! Doh! Doh!. This is exactly why I'm writing this down, because I knew that once and forgot. Anyway, going via a FLOAT is the go, OR the much more legible alternative that Mitch put in the comments : datediff(day, 0, @yourdate)
[/Update]

That being said, if all you want is an integer value that represents a date, just casting datetime to an int seems like a pretty good way to go, provided you never get carried away and cast it back again. Or I guess you could just compensate for the off-by-one and save yourself[1] a lot of pain later on...

If you're really interested, run this and see for your self:

select
 getdate()
 ,FLOOR( CAST( getdate() AS FLOAT )) -- use floating point
 ,datediff(dd, '19000101', getdate()) -- right answer, by definition
 ,Cast(getdate() as int)
 ,CAST( CAST( getdate() AS INT ) as Datetime) -- tomorrow!

[1] Yes, Sql 2008 has a date-only type. But you and I will still be stripping times from datetimes for many, many years to come.
[2] By which I really mean myself, of course

Continuous Integration in TFS 2008: Part 2

Ok so in Part I I missed two important features due to a schoolboy cut-and-paste error:

• The association of a build with the changesets that triggered it - fantastic for trouble shooting
• Automatically raising a 'bug' work item for 'he who broke the build' to fix it. Sweet.

Maybe I'll go into those in more detail another time, maybe not. They are cool. But here's the not-cool stuff:

Stupid default working directory / issues with path length
Straight out of the box many of my builds failed with excessive path length errors. Unfortunately the build agent starts with its default working directory of C:\Documents and Settings\NetworkService\Local Settings\Temp\ so I've got 61 extra chars on my path before I've even started, and we were already skating pretty close to the 255 chars limit (think BizTalk, over-zealous namespacing, auto-generated code etc...). Easily fixed, but seemed like a silly place to start.

Ok, it's going to be a bit less under Vista / Server 2008: still too long. Is C:\Builds so wrong?

Binaries deployed to shared Staging folder
More problematically, the folder structure created on the staging share is a complete mess, totally useless to use as the source for xcopy deployment to testing environments and the like. I'll show you what I mean:

Here's my solution structure: 4 deployable applications and one shared library:



And here's the staged output from the TFS build:





Pretty disappointing. All the binaries from all of the different applications have been dumped in the same folder. At least the websites have also been spat out in 'xcopy ready' form, but god help me if I wanted to Xcopy the console app one place and the WinForms app somewhere else.

What I'm looking for is more or less what I get out of SNAK right now:



...with any deployable asset (exe, website) neatly packaged to be copied off somewhere.

Same-name content files overwritten
Ok, I guess I could just accept the above and copy everything, only using the EXE that I want, but that's icky, and it does rather suppose there aren't any naming conflicts between the deployable artefacts.

For example: I added some files marked as Content in my HelloWorldConsole app, but they got completely ignored by the deployment process. I had to also mark them as 'Copy to Output Directory' before TFS build stuck them anywhere (which I'm not convinced is correct behaviour, but there you have it), and then it stuck them in the 'right' relative location to the build root folder:



...becomes...



But there are two 'Subfolder1's above, and I only got one out at the end. Predictably, one of them got overwritten.

When would this be a problem? When could different projects possibly have a content file with the same name?! I can think of some examples:

• We always use a file 'log4net.config' to host our logging configuration, so we can change it on the fly without recycling app pools and the like. Only one project would have got the right configuration.
• Bundled licence files (Licence.lic) would get mixed up

I'm sure you can think of some more. And yes, using Embedded Resources works fine, but they're not always an answer (eg: log4net.config).

There is a fix. There's some changes you can make to your *proj files to make them preserve the output path hierarchy when deployed to the build drop folder. But it's a per-project fix, and that's - frankly - a bit lame. I know you don't add new project to your solution on a daily basis, but it's just one more thing that needs to be kept on top of, or things start falling apart. And that's just not how all this is supposed to work.

(Alternatively you could customize everyone's project templates. I guess you could fix it on all their PCs, or you could just put a fixed template on a share somewhere and tell people to use it. Since it's one of the default project types you're amending I guess you probably have to remove the original from their machine too. And hope they don't have to re-install VS anytime...)

Conclusions
Getting VSTS to perform Continuous Integration on your project is now really easy. To be fair, this on it's own was pretty easy with CCNet too, but it's even easier now, and we don't have to fight over the CCNet.config file.

But the staged output from the build strikes me as limited in use. It's possible to go and hack about with the generated MSBuild files that actually perform the build and stage, and bend it to my will, but that's just what I don't want to have to do. I want it to 'just work' and I don't think we're there yet.

New MCCS Certification announced

Microsoft will tommorrow announce their new MCCS - Microsoft Certified Certification Specialist certification. This exam will henceforth be a pre-requisite for embarking on any of the various MCAD / MCSD / MCPD upgrade paths.

The course content is not nailed down yet, but most of the detail is on Gerry's blog. In the comments. Obviously.

Miss VMWorld 2008

Everytime I get this email...

... I get totally the wrong idea.

MyClass in VB.Net

It's always a bit of a shock when you find something you've missed in a language you've used for years. I'm mostly a C# person, but I thought I knew pretty much all of VB.Net's quirks by now. But I totally missed 'MyClass'.

'MyClass' allows a class to access methods and properties as declared on itself, irrespective of them being overridden further down the inheritance heirachy. It's like using 'Me' if all the 'overridable's were removed.

Since there's no C# equivilent this was a big surprise to me, but it shouldn't have been - it's only doing the same as 'MyBase' does (against a type's ancestor): executing properties / methods by specific type address, not via virtual dispatch. As the IL for this sample shows:

Public Class Class1
    Public Overridable ReadOnly Property Name()
        Get
            Return "Class1"
        End Get
    End Property
End Class

Public Class Class2
    Inherits Class1
    Public Overrides ReadOnly Property Name()
        Get
            Return "Class2"
        End Get
    End Property

    Public Function GetNames() As String
        Dim lines(3) As String
        lines(0) = MyBase.Name
        lines(1) = MyClass.Name
        lines(2) = Me.Name
        Return String.Join(",", lines)
    End Function
End Class

Public Class Class3
    Inherits Class2
    Public Overrides ReadOnly Property Name()
        Get
            Return "Class3"
        End Get
    End Property
End Class

Calling new Class3().GetNames() produces the following (edited for brevity)

     // mybase - explicit dispatch to class1
     L_000b: call instance object ConsoleApplication1.Class1::get_Name()

     // myclass - explicit dispatch to class2
     L_001a: call instance object ConsoleApplication1.Class2::get_Name()

     // me - virtual dispatch, will resove to class3's implementation
     L_0029: callvirt instance object ConsoleApplication1.Class2::get_Name()

So the output eventually is 'Class1, Class2, Class3'. Nifty. That being said, I can't honestly say I've ever really needed this, so it might go back into the 'curios' collection. Useful in a pinch maybe, but surely it's a smell? As if designing-for-subclassing wasn't hard enough as it is...


PS: Interestingly the Reflector disassembler doesn't understand this either, so it wasn't just me that missed it: Reflector thinks the VB was:

Public Function GetNames() As String
    Dim lines As String() = New String(4  - 1) {}
    lines(0) = Conversions.ToString(MyBase.Name)
    lines(1) = Conversions.ToString(Me.Name) ' got this wrong
    lines(2) = Conversions.ToString(Me.Name)
    Return String.Join(",", lines)
End Function

Using Extension Methods in .Net 2.0 from VB.Net

So despite what ScottGu originally said, Extension Methods don't 'just work' for VS 2008 projects targeting .Net 2.0.

There's no end of blog posts describing the workaround - add your own ExtensionAttribute class to get it working - but all the samples are in C# (which is interesting in of itself). So here's the VB.Net version:

Namespace System.Runtime.CompilerServices
    <AttributeUsage(AttributeTargets.Method Or AttributeTargets.Assembly Or AttributeTargets.Class)> _
    Public Class ExtensionAttribute
        Inherits Attribute
    End Class
End Namespace

...and why am I bothering to blog about this rather trivial conversion? Because of the key gotcha: make sure you put this in a project with no root namespace set:



That had me banging my head on the table for too long.

As did the next one: extension methods only show up under the 'All' tab in IntelliSense - obviously too advanced for mere Morts. I gotta remember to turn that off: using VB is bad enough without the IDE patronising you as well.

Interestingly, if you get the AttributeUsage declaration wrong on the attribute, you get this error:

"The custom-designed version of System.Runtime.CompilerServices.ExtensionAttribute ... is not valid"

Fascinating. So this hackery works by design, it's just not really supported as such.

More reading: MSDN documentation on Extension Methods in VB

Continuous Integration in TFS 2008: Part 1

Many people now accept the benefits of a regular / continuous integration cycle (even if they don't actually practice it themselves). Picking up the pieces after someone's broken the checked-in source code, especially if it's not picked up for a few days, can be a real time waster.

Like many agile practices, however, the cost / benefit is hard to quantitatively analyse. It's far easier to justify therefore if it's really easy to setup: as the costs tend to zero the benefits become essentially 'free'. And you could argue that tools like CruiseControl.Net have made it pretty easy.

Personally, having spent significant sections of the last 3 years getting CCNet / Nant build cycles going on various projects, I'd beg to differ. Sure, it's really easy to setup CCNet / Nant (or CCNet / MSBuild) to build your solution, but that's only the first step in the process. Typically you also want to do things like:

• Import the latest built version of external dependencies (ie components maintained outside of the solution being built)
• Execute unit tests
• Execute integration tests (so config files pointing at databases etc... have to be in the right place)
• Package the build outputs nicely ('xcopy ready')
• Deploy and install into test environments

CCNet and NAnt don't really give you this stuff 'out of the box'. You spend time gluing bits together, inventing your own build process and so on, and maintaining this stuff seems to get out of control very easily. Deploy and install is a particular minefield, because somewhere in there you have to start doing configuration file substitution (put your test server settings in the web.config etc...). And doing all this in XML just rubs salt into the wound.

You can handle most of this by hand on small projects, but the last app I worked on had five or six deployable parts to it (webservices, windows services, Winforms with ClickOnce manifests and the like), each of which had 20 or so settings to change for each of 7 different environments and the differing systems it integrated with. That's 100's of settings to keep track off, without even getting into the Biztalk artefacts, and that was only one of several projects of similar complexity. Automation's a no brainer at that point.

My solution to try and scale back the per-project cost of managing this was my own open source project SNAK. This attempted to commoditize a standard build / test / package / deploy process that you could implement on your side by pretty much setting a single variable at the top of a build script. And I think it works reasonably well: but it's clearly not the answer, not least because it took a fair amount of my (and others) time, of which I have very little.

So I was very, very hopeful when I started looking at the CI support in TFS 2008. Microsoft were really bashed over CI (lack of) in 2005, but this time round it looks like they've delivered:



You pretty much pick your solution file:



...your output directory...



...and your build frequency, and off you go:



Given how hard it was to deal with VSTS tests under CI in 2005 (because the test file was always in the wrong place), this screen will be a real pleasure to some:



And if you've tried to implement a build output retention policy in NAnt, you'll really appreciate this:



So up until now, absolutely fantastic. But then I had a few issues, which I'll deal with in Part 2 (so as not to take the gloss off the good bits above).

[I was due to present on this topic at the Perth .Net user group the other week, but a failing laptop saw to that (not the way I was expecting the demo to fail!). Since there's now no slots till Xmas, I've recycled some of the content into this post. The laptop was lobotomized and is recovering well...]

[Readify are doing a Dev Day in Perth on the 29th, with TFS as one of the tracks, so I'd be surprised if they didn't cover this there]

Recycling old posts?

Sorry about that. I re-tagged a few articles over the weekend, and I think Blogger has got confused and bounced them into my feed as if they were new posts. Unfortunately some of them were, so it's all a bit of a mess.

New posts were actually:
* Finally: PowerShell as build language
* Using PowerShell to export the Windows Feeds list


Normal service will resume shortly...

Using PowerShell to export the Windows Feeds list

Moved computers recently, and one of the things I realised I lost was my RSS feeds list. It was probably a blessing (I just tend to accumulate subscriptions otherwise), and maybe I should be using a reading service of some nature, but there you are.

Anyway given I'm all Mesh'd up, I though I'd copy my feeds list into my Mesh folder (like my bookmarks), so I'd have a backup and this wouldn't happen again. Only I couldn't find where the feeds list actually lives. Instead there's a whole API for dealing with it...

...which is surprisingly easy to use, and works like a treat in PowerShell (I'm always amazed at it's ability to 'just work' with things like COM objects). So I just exported the list instead:

# Dump the contents of the Windows Feeds store to an XML file

$erroractionpreference="stop";
[xml]$feedsListDocument = "<feeds/>"
$feedsList = $feedsListDocument.get_DocumentElement();
$feedManager = new-object -com "Microsoft.FeedsManager"

@"
<feeds>
$(
 $feedManager.RootFolder.Feeds | % {
  $feed = $_;
  $feedXml = $feed.Xml(-1, 0, 0, 0, 0)
  '<feed Name="{0}">{1}</feed>' -f $feed.Name,$feedXml
 }
)
</feeds>
"@

Easy as. The XML it spits out is overly large (since it includes all the article contents from the cache), but for the MB involved it barely seems worth refining it.

Update 2008-07-17: So like the very next day I realised I could have just sync'd the feed list into Outlook, and asked it to export it as OPML. But syncing into Outlook blew my tiny mailbox quota (these feeds are suprisingly large) so I ended up back doing this again anyway. Then it turned out that IE can export the feed list as OPML too (File \ Import and Export - you'd think I'd have noticed originally) - but I still like having a script because I can schedule it.

Note to self: It is definitely time to find a blog that can cope with XML a bit better

Finally: PowerShell as build language

I've never really got into MSBuild, which surprised some people given how much time in the last four years I've spend mucking about with CCNet / NAnt. It was partly that we did a bit of investigation when MSBuild came out, and saw a couple of things we didn't really like about it and decided to wait for v2 (ie Team Build in TFS 2008). Partly.

More fundamentally however the problem is that MSBuild is just too similar to NAnt, and my considered opinion after years of usage is that NAnt sucks, or to be more specific, XML is a terrible 'language' for writing executable code. Fowler puts it pretty well:

"After all until we tried it I thought XML would be a good syntax for build files"
http://www.martinfowler.com/articles/rake.html

Sure it's fine for the IDE to write that stuff out (though even then you have to look at it and wince, right), but for humans who want to customise their build process? Jesus wept. Variable scope: gone. Explicit parameters for subroutines (targets): gone. Its fine when it's simple, but once you start looping and branching and using temporary variables it's just a great big mess of angle brackets that even it's mother pities. And debugging? Now there's a story...

There's a time and a place for the angle bracket tax, and this isn't it. Square peg, round hole.

So given how amenable for DSLs PowerShell has proven to be, I've been holding my breath for some kinda PowerShell rake-alike.

And here it is: Introducing PSake

(Also Microsoft themselves are thinking about it, and canvassing for opinions about whether it's a good idea or not.)

Sadly (actually quite the opposite) I'm not actually having to deal with the build process on my current project, so I don't really have much excuse to play with it. But I dream of a future in which the TFS Team Build project wizard kicks out a PS1 file instead. It'd certainly make fixing some of it's shortcomings a whole heap easier (that's a subject for a future post)


Edit [14/7/08]: Most requested feature for MSBuild? Debugging. Obviously this'll be interpreted by the MSBuild team as a need for a debugger, but maybe they should have used a language that already had one.

Beyond Compare 3 supports 3 way merge, is totally awesome

Beyond Compare 3 is out in beta. It supports 3 way merges!

I found this out literally minutes before I started what turned into a 2 day mergeathon between two large and divergent branches in TFS, with *lots* of merge conflicts to manually resolve, and I can honestly say I'd probably still be merging if I hadn't downloaded it. It's just fantastic.

I'll probably post some screenshots etc... soon, but if you're struggling merging with BC2 and/or the built-in diff/merge support then you really should check it out.

MSDN Downloads and the fly-out menus trauma

Raymond's just posted about the rationale behind the windows menu show delay, and goes on to point out various web properties that blatantly ignore the underlying usability requirement.

Sadly finding examples is like shooting fish in a barrel. I remember Jakob Nielsen winging about this last millennium, but as the technology moved forwards: Director, DHTML then Flash, the ease with which anyone can design their own UI and distribute it widely over the internet has lead to a flood of bad UI. Even as Vista attempts to move forwards, the new Silverlight version of MSDN Downloads re-re-implements the fly-out-menus concept, with almost unusable results.

Maybe this is a necessary pain we have to move through, but it kinda sucks that we can't explore novel and interesting UI concepts without making them totally unusable. I'm no UI designer, but at least I don't pretend to be, or work as one.

The templating within WPF is a great example of an enabling technology here, where the usability can be codified into a control by 'experts', but still delegate most of the 'funky look' to the end-designer. In this case if WPF / Silverlight had shipped with a decent fly-out menus control, maybe the MSDN Downloads team wouldn't have got it so horribly wrong, and I wouldn't have had to uninstall Silverlight in frustration.

I guess there is hope then that this isn't just another enabling technology that enables people to make a real arse of things.


PS: Check out this bizzarro comment on Raymond's blog:
"Let's not get into the "gynaecologist's interface" that is Vista's Start Menu, shall we?"
WFT?

Don't be Stupid

Years ago I was working on a project and I came up with a fantastic idea to help limit the level of regressions in the codebase I was working on. Rather than write unit tests as little throwaway test harnesses, I moved them into the codebase, and created a little app to execute them. It even did this via reflection, so as we added more tests, they got run too.

I thought I was being pretty clever.

I was being very stupid. I'd just re-invented xUnit, and didn't even know. [1]

It's a particular type of stupidity that manifests itself only in those who'd otherwise regard themselves as anything but: we get so wrapped up in our great idea that we stop to consider that someone else might have done this already. Programmers are particularly badly afflicted by this, mostly because it suits our vanity to create it ourselves.

There was already an automated testing community, that had over time evolved what worked and what didn't, the practical upshot of which - for a .Net developer at the time - was that NUnit already existed. I could have spent the time writing more tests instead. Or better still, more screens, which is what I was actually being paid for.

The last three applications I've worked on have all involved considerable custom frameworks (stupid) including a custom databinding scheme (very stupid). They were written by clever people, most of whom I respect, but they did some stupid things that less able programmers wouldn't have been able to do. Clever isn't always a complement in the agile camp, and this is why.

Of course 5 years of hindsight is a wonderful thing, and I've written my share of head-slappingly dumb code too. And it's all too easy to succumbed to the 'quick fix' fallacy when the boss is breathing down your neck : after all it's so much easier to get started writing your own framework than to learn to use someone else's.

But once you start down the dark path, forever will it haunt your destiny[2]. Which is why I make this plee to you now:

Please, before you put finger to keyboard again, consider whether what you're about to write has already been written.

Don't be stupid.

[1] To be fair to my erstwhile self, at least I was actually doing some testing, which was more than had been done before on that project
[2] Or that particular project at any rate

Enabling multiple RDP sessions in Vista

After many days of frigging around I realised those thegreenbutton.com Vista multiple remote desktop hacks (that you find from google) are all broken by SP1. That page' on missingremote.com that is supposed to draw all this together still hasn’t been updated with this new info.

However add SP1 to your search and you find this other thread, which works: http://thegreenbutton.com/forums/permalink/242509/255166/ShowThread.aspx#255166

Ah, the joys of subsequent-threads-with-lower-page-rank-than-the-original-now-outdated-info.

Running ASP.Net webservices under a service account

Most of the time I run websites and webservices in an app pool that's running as Network Service. It just saves a whole truck load of time and hastle:
* no passwords to worry about
* already trusted for kerberos delegation
* can still use it to talk to a database under integrated security (you just grant access to the machinename$ account in the domain).

Hey - this is what this account was *invented* for.

However, sometimes a specific service account is a must. Reasons include:
* Needing to differentiate access rights between applications running on the same host
* Needing to authenticate back across a one-way domain trust
* Specific policy mandates

Unfortunately you can't just add any account to IIS_WPG and use it, because the ACL on windows\temp is wrong: and grants access to network service rather than to the group. Miss this one, and you'll just get serialization errors left right and center.

So I do this:


Net localgroup iis_wpg /add mydomain\myserviceaccount
cacls %systemroot%\temp /E /G IIS_WPG:C


...then when you change the identity of the app pool you won't get 'Service Unavailable'.

Don't override Equals

A colleague had a problem the other day which turned out to be due to an overridden Equals operator. In this case it was a straightforward bug in the implementation, but after he saw my horror struck face I had to introduce him to the whole 'don't override Equals' philosophy[1]. On the pretext that you've not come across it, here's the argument in full:

• You have two objects that came from different places, and need to know if they represent essentially the same data.
• You can't override Equals unless you also override GetHashCode. If two objects are equal, they must have the same hashcode, or collections are screwed.
  
• GetHashCode must return the same value for an instance throughout it's lifetime, or Hashtable's are screwed
• Your object isn't readonly, so you need an immutable field in the instance to base the hashcode on.
• But if you modify one instance's data to equal another, that field can't change, so the hashcodes are still different.
• You're screwed

And that's without getting into the problems associated with a correct implementation of Equals in the first place (getting the reflexive, symmetric and transitive bit right). Generally speaking some kind of IsEquivilent method is a whole heap less trouble, but it depends what you're up to. You might think about loading your objects through some kind of registry, so references to the 'same' data actually end up pointing to the same instance. Then everything just works...

More reading:

• When (not) to override Equals
• The rules for GetHashCode

UPDATE 10/04/08: Some clarifications: I'm talking about not overriding Equals/GetHashCode for reference types here. It's not such a problem for value types [as IDisposable points out in the comments]. And I've futher clarified some of my assertions about GetHashTable in the comments.

[1] PS: Like all advice, this has exceptions. But the chances are they don't apply in your case. No, really.

Care required passing arrays to .Net methods in Powershell

In Powershell, argument lists for .Net methods are treated as arrays:

$instance.MyMethod($arg1,$arg2);

...which can be confusing if you want to pass an array as a single argument:

$instance.MyMethod($myArray);

New-Object : Cannot find an overload for "MyMethod" and the argument count: ""

Instead, force the array-argument to be contained within a single-member array:

# Note the extra comma below
$instance.MyMethod(,$myArray);

Makes sense when you think about it, but definitely a gotcha.

[In my case, I was caught out with the byte[] overload constructor for a MemoryStream]

Blobs out with SQL 2008

Recently I re-visited the blobs in/blobs out argument with a colleague. You know the one, one of you says blobs shouldn't be stored in database (principally because the last time he tried it 'blobs in' in VB 6 access to the blob data was a pain in the arse), then the other one says no they should be in the database (because the last time they tried it 'blobs out' all the files got mixed up / out of sync / weren't backed up). Etc...

Anyway, not only has Paul Randal posted a good summary of the pros and cons, but he did so as an intro to a new SQL 2008 data type 'FileStream' that attempts to bridge the two approaches (the 'have your cake and eat it' approach).

I'm cautious. Transactions at the filesystem level are a real mess (as some of the OneNote blogs make clear, especially with non-MS implementations of SMB like SAMBA). Your database backup is presumably still huge and unwieldy (or missing the blob data, which is worse?).

The main advantage of this approach seems to be that SQL can access the blob data faster through NTFS than via it's own internal MDF formats. But you've apparently still got to go via SQL to get the data, you can't (for example) just serve up images-stored-as-blobs directly via IIS. Or maybe I've missed something. Either way, the upside all seems to be focused on blob streaming performance, which may or may not be the most relevant factor for your app.

So it's possible that next year's arguments will be blobs in vs blobs out vs filestream, and still no one-size-fits all. Ah well.

Path already mapped in workspace error with CCNet and TFS

Had a problem with CCNet that kept me here till midnight where try as I might, I just couldn't get a build to not fail with the dreaded "Path ... is already mapped in workspace ..." error:

Microsoft.TeamFoundation.VersionControl.Client.MappingConflictException: The path C:\Builds\etc\Working is already mapped in workspace someworkspace

We use a different workspace for every CCNet project to avoid collissions, and to maintain uniqueness we keep the workspace name the same as the CCNet project name. I couldn't find the workspace in question, and was pretty sure I'd already deleted it. In fact I'd used TF Sidekicks to delete all the build user's workspaces, and it still didn't work. So what was up?

Fortunately in a post 'How to handle "The path X is already mapped in workspace Y"' I learnt of the mappings cache file on the client PC, in the user's Local Settings\Application Data\Microsoft\Team Foundation\1.0\Cache\VersionControl.config file. Just nuking workspaces on the server isn't enough!

So to be sure I blew away the build server's local profile entirely, and that finally fixed it.

The new starter experience

I normally avoid 'link-posts', but again Hacknot is right on the money with 'If They Come, How Will they Build It?', an eminently familiar analysis of the plight of a new developer on a project with an oral documentation culture.

In fact I'd go slightly further than Hacknot, and state that the initial experience of a new developer on the project is one of the most important things to get right. First impressions do matter, and if your first impression of a project is the frustration of:

• Not having a login
• Not having internet access
• Not being able to get latest
• Not being able to build
• Not being able to locate any documentation
• Not having clear lines of escalation
  
• Not having clear rules of engagement
• Not knowing what's expected of you
• Not having a mentor
  

...you're going to be hard pushed not to be prejudicing your opinions of the professionalism of the rest of the project. You'll start disheartened, but on the other end of this unfortunate indoctrination you're going to be just like them. You won't regard the absence of the list above as anything other than normal. You'll accept that that's just not how things are done around here. You will love big brother.

[ahem. got carried away there]

I regard the absence of guides and documentation as more than a major time-waster: it's a self-perpetuating morale hole for all future team members to climb into and die.

New staff play a vital part in ensuring a project's approach doesn't atrophy. If you waste their 'fresh' time frustrating them with missing documentation and runaround, you won't get the benefit of seeing things from their eyes. They'll have clammed up and learnt to live with how it is, and by the time you ask them they'll have forgotten that they used to care.

FixBrokenWindows - they're not all in your code