Blogsphere faux pas

Don't confuse Atwood and Achewood

Access denied when piping to :Null

It's quite a common thing in my batch files (yes, .BAT is still alive) to limit the verbosity of the output displayed to the user. It's a 'seeing-the-wood-for-the-trees' thing: I don't want to have the user waide through screens of output to know that a CACLS worked ok (and potentially miss something important)

So I quieten things down by piping to null:

aspnet_setreg -k:SOFTWARE\MyCompany\MyApp -u:"%username%" -p:"%password%" >:null

...and listen for ERRORLEVEL instead.

But recently I discovered that this fails with Access Denied if you don't have write permissions to the current directory (yes, you need write permissions to write to :null. Just don't ask). Not only that, but the whole statement fails, not just the piping bit.

So these days I pipe to %temp%\null instead.

When is a type not public?

[Keith's one this]

...when it's a nested type. So you have to do:

if(type.IsPublic || type.IsNestedPublic){
   // do stuff to public types
}

I can't even remember what we were doing when we ran into this, which (to me) indicates I'd better blog it before I forget about it altogether.

Encryption of connection credentials in .Net

In which I discuss techniques to manage the storage of a secret configuration setting (connection credentials) on a production webserver, when the development team (and their automated build/depoy process) shouldn't actually know the secret itself (and especially shouldn't store it in source control).

It's a bit of a ramble to myself as I think this stuff through.

What's new in .Net 2

In .Net 2, the configuration infrastructure includes support for encrypted connection strings out-of the box. One can simply call aspnet_regiis to encrypt (almost) any given section of a configuration file:

aspnet_regiis -pe "connectionStrings" -app "/MachineDPAPI" -prov "DataProtectionConfigurationProvider"

Pasted from <http://msdn2.microsoft.com/en-us/library/ms998280.aspx>

However

• This encrypts the whole section, which makes inspection and verification of other settings harder
• This requires the .config file to have the credentials in in the first place
• So either the credentials sit in source control at the dev end :-(
• Or the install process has to both prompt for them, and add them into the web.config before encryption. This is a pain, but manageable.

A variation on this technique is available for web farm scenarios, where the config is encrypted with a custom exportable RSA key.

aspnet_regiis -pe "connectionStrings" -app "/WebFarmRSA" -prov "CustomProvider"

aspnet_regiis -px "CustomKeys" "C:\CustomKeys.xml" -pri

Pasted from <http://msdn2.microsoft.com/en-us/library/ms998283.aspx>

Apart from being web-farm aware, this technique has the advantage that the public part of the key could be exported to the build server, and the settings encrypted prior to deployment

However this still means the build server (and by inference source control) contains the production credentials in plain text. This is generally what we want to avoid most - internal information leakage is far more likely than a compromise of the production environment, especially in an intranet scenario.

How did we used to manage this in Net 1.1?

Under .Net 1.1 the approach was to use the ASPNET_SETREG tool to move settings into the registry, under DPAPI encryption, then adjust the configuration using a special syntax that said 'read this from the registry instead'.

aspnet_setreg.exe -k:SOFTWARE\MY_SECURE_APP\identity -u:"yourdomainname\username" -p:"password"

Pasted from <http://support.microsoft.com/kb/329290>

identity impersonate="true"
userName="registry:HKLM\SOFTWARE\MY_SECURE_APP\identity\ASPNET_SETREG,userName"
password="registry:HKLM\SOFTWARE\MY_SECURE_APP\identity\ASPNET_SETREG,password"

Pasted from <http://support.microsoft.com/kb/329290>

This has the following advantages:

• Secret only needs to be known once, on initial install, and is thereafter stored in the registry
• Source control and build process doesn't need to know the secret
• Same config file deployed to both boxes in a web farm

Whilst under .net 1.1 this was only supported for specific keys:

identity userName= password=

processModel userName= password=

sessionState stateConnectionString= sqlConnectionString=

Pasted from <http://support.microsoft.com/kb/329290>

…a common pattern, and one I've used a couple of times, was to use aspnet_setreg to put arbitrary settings into the registry in an encrypted manner, and use custom DPAPI code to retrieve them. There was no equivalent of the DPAPI wrapper class 'DataProtection' in .Net 1.1, so Keith Brown provided one:

"Version 2.0 of the .NET Framework introduces a class called DataProtection that wraps DPAPI. It's simple to use; in fact, it looks almost exactly like the wrapper class I provided above. I've shown an example in figure 70.2."

Pasted from <http://pluralsight.com/wiki/default.aspx/Keith.GuideBook.HowToStoreSecretsOnAMachine>

So where are we going with this?

Well to my mind at least the .Net 1.1 registry approach was actually better than the .Net 2 model - it separated the configuration and the secret in a way that perfectly reflected the normal division of labour between development and infrastructure teams (where the devs are expected to provide the configs, but they shouldn't know the secret).

There should be no reason why one cannot use the .Net 1.1 technique (ie use ASPNET_SETREG to store the secret), whilst still using the .Net 2 DataProtection class to unencrypt it.

• This does require careful monitoring of the ACL on the registry key in question
• There's no real requirement to use ASPNET_SETREG over a custom exec (using the DPAPI), bar convenience. That having been said a custom EXE could set the registry key ACL at the same time, which I remember as being a headache otherwise.

Watching files for changes

The moral of the story is, if you want to watch a file for changes, don't just listen to the Changed event on a FileSystemWatcher.

Different applications save files in different ways. Some just overwrite the original file, in which case Changed fires. Others attempt to create a two-phase commit transaction, so either the changes get saved or your original file is preserved (in case of a power outage). However depending on how they do this, they might never raise a Changed event against the file you're looking for - they might (for example) do all the changes to a temp file, then rename it.

Watching C:\*.* with a FileWatcherTestUI.NonBufferedFileSystemWatcher and updating a file 'UpdateSql.txt':

Notepad
[2006-07-10T10:45:27.1245966+08:00] Changed: C:\UpdateSQL.txt
[2006-07-10T10:45:27.1402217+08:00] Changed: C:\UpdateSQL.txt
[2006-07-10T10:45:27.2183472+08:00] Changed: C:\UpdateSQL.txt
 
VS 2003
[2006-07-10T10:45:17.3432840+08:00] Created: C:\ve-329.tmp
[2006-07-10T10:45:17.3589091+08:00] Changed: C:\ve-329.tmp
[2006-07-10T10:45:17.3589091+08:00] Changed: C:\ve-329.tmp
[2006-07-10T10:45:17.3589091+08:00] Changed: C:\ve-329.tmp
[2006-07-10T10:45:17.3745342+08:00] Changed: C:\UpdateSQL.txt
[2006-07-10T10:45:17.4214095+08:00] Changed: C:\UpdateSQL.txt
[2006-07-10T10:45:17.4526597+08:00] Changed: C:\UpdateSQL.txt
[2006-07-10T10:45:17.4682848+08:00] Deleted: C:\ve-329.tmp
 
VS 2005
[2006-07-10T10:45:37.3277869+08:00] Created: C:\ve-32B.tmp
[2006-07-10T10:45:37.3277869+08:00] Changed: C:\ve-32B.tmp
[2006-07-10T10:45:37.3434120+08:00] Changed: C:\ve-32B.tmp
[2006-07-10T10:45:37.3902873+08:00] Changed: C:\ve-32B.tmp
[2006-07-10T10:45:37.4371626+08:00] Changed: C:\ve-32B.tmp
[2006-07-10T10:45:37.4527877+08:00] Created: C:\UpdateSQL.txt~RFf7eb451.TMP
[2006-07-10T10:45:37.4684128+08:00] Deleted: C:\UpdateSQL.txt
[2006-07-10T10:45:37.4684128+08:00] Changed: C:\UpdateSQL.txt~RFf7eb451.TMP
[2006-07-10T10:45:37.4840379+08:00] Renamed: C:\UpdateSQL.txt (from ve-32B.tmp)
[2006-07-10T10:45:37.4996630+08:00] Deleted: C:\UpdateSQL.txt~RFf7eb451.TMP

Prefer Singletons to Static classes

It's a common scenario: you have some core facade class with a load of stateless methods on it that delegate activity down to other layers. There didn't seem much point in having the overhead of instantiating the class to use the methods, so you made all the methods static. You've made a static class.

Trouble is, static classes just aren't as flexible as the alternative: a singleton. For example:

* Static classes can't implement interface contracts
* Static classes can't be passed around or used polymorphically
* Inheritance of static classes is a minefield
* You can't make static members virtual or abstract.

This might not be an issue straight up - in many cases it may never be an issue at all - but when it does you'll be kicking yourself.

Take the scenario where you decide somewhere along the line that your facade isn't the be-all-and-end-all, and that you want to swap implementations in different circumstances. You don't want your objects to know about it, so you refactor the facade into some kind of proxy-by-composition. Still you're tied to having only one implementation active at any one time.

By contrast if you'd started as a singleton, it's pretty easy to change which subclass gets setup on the singleton. Its also far easier to migrate to a dependency-injection type architecture later down the line to support multiple implementations being used in parallel (this also then decouples the objects from the facade/singleton, which is a benefit in plugin-style architectures). And it's far easier to refactor some of the functionality out of the class into a base class, and start replicating your facade functionality in other scenarios.

On a more mundane point, when setting up a static facade you're doing work in the static class constructor. When this throws, it throws TypeInitializationException. With a singleton you get the choice - lazy init via static field init (still throws TypeInitializationException), or via first-property access, which will throw an exception you'll have a chance of debugging.

Go the singleton straight off (a rule I've once again re-learnt the hard way).

VB.Net finally bug (.net 1.1)

Forgot to post this a while back, but a collegue (Matt) showed me this line that I wrote:

        Finally
            If Not cursorScope Is Nothing Then cursorScope.Revert()
        End Try

He was debugging the null reference exception that was thrown when cursorScope.Revert() was called when cursorScope was Nothing.

Yes, that's right, the IF condition was being blatantly ignored, and yes, this did seem to be related to being in the finally clause.

He broke it into a multi-line If / End If to made it all work. Go figure.

Using Generics to improve code readability

Obviously Generics in .Net 2 provides the ability to write type-parameterised code, which is great for collections and the like. But it was only when I started using it a bit I realised the potential for much wider cast-elimination.

Take for example some kind of factory method (or any method that has a type as a parameter, and returns an instance of that type):

SomeType instance = (SomeType)Factory.CreateInstance(typeof(SomeType));

Using generics can cut right through:

SomeType instance = Factory.CreateInstance<SomeType>();

It's particularly noticable in VB.Net (because it has such munted cast syntax in the first place), but the result is normally a lot more legible. Unfortunately the framework is full of missed opportunities to clean up:

thing = DirectCast(Enum.Parse(GetType(SomeType), someValue), SomeType)

could become

thing = Enum.Parse(Of SomeType)(SomeValue)

That's got to be better, right?

WorkflowRuntime.Dispose() doesn't clean up properly

Unfortunately, in the current RC5 build of Windows Workflow (the one that comes with .Net 3 RC1), calling Dispose() on the WorkflowRuntime does not clean up after itself properly. Specifically:

• it doesn't unload workflows in memory
• it doesn't reset the workflow performance counters

To do all that you have to call StopRuntime() first.

Try it for yourself:

          using(WorkflowRuntime workflowRuntime = new WorkflowRuntime())
          {
              workflowRuntime.AddService(new SqlWorkflowPersistenceService(Properties.Settings.Default.WorkflowPersistenceConnectionString));

              WorkflowInstance instance = workflowRuntime.CreateWorkflow(typeof(WorkflowConsoleApplication1.Workflow1));
              instance.Start();

              //workflowRuntime.StopRuntime(); // uncomment me to get the instance to be persisted
          }

This is particularly unfortunate given all of the samples for WF (and the boilerplate code that gets created in new WF projects) just allow the runtime to be disposed when it drops out of scope. And needless to say it goes against the grain of how Dispose() works in practice: as a 'clean up gracefully' rather than just a 'free unmanaged resources' (think SqlConnections getting closed, Transactions being rolled back etc...)

https://connect.microsoft.com/wf/feedback/ViewFeedback.aspx?FeedbackID=224627

Starting Over

A project manager I used to work with had a favorite phrase he liked to bandy around:

"You can't polish a turd"

There's always a point beyond which a particular class or module should just be tossed out and re-written, rather than fixed. A point where something's gone so badly in the wrong direction that it needs to be humanely put down so we can get on with the rest of our lives. But when to start over?

Only the developers who've worked in a given area know really how horrible it is, but even then it's hard to get an objective opinion. Many developers are over-precious, defensive of their own code, and unduly negative of the work of others [1]. Then again, other developers 'at the coal face' can be blind to how screwed something is, happily chiselling away when it's time to grab the dynamite. Even the system's architects can fail to grasp just how unworkable it is from a grunt's perspective. Management just can't win.

Here's some pointers to identify 'start over' candidates:

• It's going to cost more to fix than to build from scratch
• It's going to be quicker to fix than to build from scratch
  
• There's a high defect rate in that area
• There's no test coverage in that area, so any fixes walk a regression minefield
• Even when fixed, it'll be sub-optimal compared to building from scratch
• Even when fixed, it doesn't really do what you need it to
  
• Even when fixed, it imposes significant architectural constraints on the rest of the system (eg perpetuates a coupling you're trying to get rid of, or continues with a depricated metaphor)
• Even when fixed, there's a significant maintanance cost
  
• There's no-one left on the project who understands how it works, yet it's not stable enough to just leave alone, or was left half-done
• There's no documentation
  

Any of these are warning lights, but more than a couple should warrant a long hard look.

For the sake of completeness, all of these can be applied at the project level too. This is a really tough call, but sometimes a clean break is for the best. Salvage what you can and move on. The most reusable part of a project is always the IP anyway, not the implementation. Even if you don't reuse one line of code, you'll still have learnt something.


[1] It happens to us all. Every so often I find myself panning code I wrote myself (and subsequently forgot about)

Why ReSharper is (still) the dog's bollocks

Here's a pretty good post describing why ReSharper is still the dog's bollocks.

I think ReSharper is so utterly brilliant that I'd be deeply suspicious of any C# team that wasn't using it. It's not without it's faults I'll give you, but even then it's a joy to use, and dumps all over the built-in refactoring support in VS2005-C#.

Microsoft and CodeRush just don't get refactoring. They seem to think it's some kind of point-and-click design activity, all green bendy arrows or fiddly tool tips ('making a video game out of your code'). What they don't seem to understand is that I'm typing. If I want to go all clicky-clicky I'll go back to the design pane thanks, or the Class Designer.

ReSharper is zero friction. It's the TestDriven.Net of refactoring tools. Extract Local Variable: Ctrl-Alt-V done. Override method: Alt-Insert, select, done. That thing you were about to type: Ctrl-Shift-Space (pretty much). It's so quick and easy it's not funny. People looking over your shoulder will have difficulty keeping up.

Like many XP techniques, refactoring is one that suddenly makes a lot more sense with the right tool. Using ReSharper has totally changed the way I think about my code.

Dealing with TechnicalDebt

I kind of copped out in a previous post by saying I didn't have any way of dealing with technical debt short of avoiding it in the first place. That was just plain lazy of me.

Dealing with Technical Debt cuts to the heart of the 'debt' metaphor's limitations. Debt implies regular interest payments, but that's not necessarily the case with technical debt. There's quite a continuum, from your 'high street bank' style debt: obvious, regular 'payments' impeding forward momentum; to your loan-shark style debt: maybe you'll get away without paying anything for a long time, but one night in a dark alley he'll want your kneecaps. This latter, unpredictable impact, is the more common scenario.

This then clarifies our approach towards technical debt: treat it as a risk. So, like all risks, you can:

• Mitigate - Have a fall-back position if the debt becomes an issue
• Accept - Assess that the risk is lower than the cost of fixing. You might get lucky
• Reduce - Either deal with the debt, or avoid building new functionality on debt-ridden areas
• Transfer - Make it SomeoneElsesProblem. Tricky.

Handwringing or winging about violating encapsulation doesn't tend to cut the mustard with project managers (and rightly so). Attempting to quantify implementation shortcomings in terms of project risk is far more likely to give the problem the attention it deserves.

Error Reporting - To send or not to send?

Don't think of it as 'can I be bothered to help Microsoft find their bugs' button. Think of it as a 'vote for this f#$%ing issue to be fixed' button. I find that clarifies the decision somewhat...

Broken URLs

Whyohwhyohwhy can't browsers deal with urls with line breaks being pasted into the address bar? All but the shortest urls get broken in emails, and yet the blindingly obvious solution (no it's not Shrinkster) still evades us.

Even a menu option ('Navigate to mangled url in clipboard') would do.

Avoid solution folders in VS2005 for your UserControl projects

In VS2003, whenever you had a form open in the designer, any user controls in your project[*] appeared in a 'My User Controls' tab on the toolbox.

This was essential because (for some reason unbeknownst to man) you can't drag a WinForms user control from the Solution Explorer onto a form, you have to do it from the toolbox (which sucks: the ASP.Net team managed it just fine, so what's the issue?).

In VS2005 the same functionality exists (with the additional feature that it can be disabled for performance reasons with the AutoToolboxPopulate option). But it doesn't work - or not for us at any rate.

Eventually it turns out (thanks Cam) that it doesn't work for projects within solution folders. How crazy is that?

"The fix probably won't make it into Service Pack 1, but I will forward your feedback to the appropriate team for consideration."
[Ben Bradley, MSFT, link above]

However the bug reports says it's fixed already:

http://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=114542

...so I'm crossing fingers for SP1 (whenever that comes out).

[Update: Didn't publish this for ages, but no, SP1 doesn't fix it. Poo]

TechnicalDebt

TechnicalDebt is a term used for all the hacks (in the pejorative sense) in the codebase. It's the shortcuts, the quick-and-dirties that come back to haunt you, the cruft.

The term was coined by Ward Cunningham, and is used in agile circles, though it is essentially at odds to YAGNI. Martin Fowler explains it well.

As I see it, there are two major problems in dealing with Technical Debt:

• Inflexible project schedule. Unless you've got flexibility in your project planning (built in slack, or agile schedule) you've got no time to fix it. It is, after all, work you didn't plan for.
• Inability to articulate benefits. Sure, maybe we shouldn't have done it this way, but we have, so why change it if it means yet more time spent.

This last is particularly pernicious. It's nigh-on impossible to estimate the eventual cost at the time the debt is 'incurred', and in many cases pretty darn hard to measure the cost of the debt whilst you're paying it. Without the data, how can you possibly justify the cost of rework?

So why can't we measure the cost? Well if it's a simple quick-and-dirty that must be fixed for further progress to be made, then it's just the cost of the rework - the debt is repaid. But it's never that clean cut: the decision can have almost subliminal rippling effects on all the code that touches it, and the developers who have to deal with the compromised areas may not be aware that there was a better way available originally. So the debt possibly doesn't ever get 'repaid' as such, and the 'interest payments' such as they are are hidden within the cost of other work.

[Random link that illustrates the costs of unpaid debt]

Plus of course even if you're aware of the hit, unless you do the same activity twice - once paying the tax and once without - it's impossible to measure the true cost. If you are directly aware of the impact, that probably means the impact is severe.

More than often technical debt is actually dealt with just by members of the project team caring enough about their future sanity to just fix it. But that's not a scalable solution: it depends on key people essentially doing work out of hours just to stay on track. Even this isn't viable if the problems involve exceed the 'weekend refactor' threshold (the amount your top developer can get done in a big burst over the weekend). Over this level requires a concerted effort, or at least a staged one, which has to be planned for and so can't be fixed 'under the radar'.

I don't have any magic formula for dealing with TechnicalDebt. My strategy has always centered around avoidance rather than anything else, and being really bullish about having to fix things then and there. It's a far more scalable strategy, but I'm very aware that it's not possible on all projects. How you deal with the debt then I'm not sure. My experience has been that you don't, and that forward progress just becomes progressively more expensive. I guess you hope your project finishes before the debt becomes insurmountable.

Software Quality Characteristics

In CodeComplete, McConnell defines 15 'characteristics of software quality':

External Characteristics

• Correctness
• Usability
• Efficiency
• Reliability
• Integrity
• Adaptability
• Accuracy
• Robustness

Internal Characteristics

• Maintainability
• Flexibility
• Portability
• Reusability
• Readability
• Testability
• Understandability

It's important to work out early on in a project what your priorities are (and especially what they aren't) so that you can steer the project effectively. It's also important not to underestimate the effect of all the unspoken tacit decisions made by developers in the course of their work: the unspoken assumptions. Without clear guidelines, developers will benignly pursue their own ideas about 'the right way' to do something. You can tick off all the use cases you like, but if they favour performance or reusability, and your client's real goals are accuracy and maintainability then you're heading for trouble. It’s always the non-functionals that bit you, IMHO.

Fortunately McConnell also points out that given clear guidance, developers will actually do what they're told, which is reassuring. Setting clear goals is a pretty key management tenant, but one easily forgotten. Some of these characteristics (say performance, code-complexity) can even yeild metrics, so you can deal with the project graphically: BigVisibleCharts.

On my previous project, we used this list as the basis for our peer code reviews, out of which came the categorisation I posted previously, but to re-iterate (slightly re-worded):

• Customer Facing (most of the External Characteristics)
• Maintenance and Design (most of the Internal Characteristics)
• Defects (Just hunting for bugs)

I think it’s a good split. Each review role had a cheat sheet which consisted of the relevant characteristics (as defined above) to look for, and some check lists. This helps ensure visibility of the quality priorities into the code review, which makes them doubly hard to ignore.


PS: If you weren't sure what some of the characteristics meant, maybe you should re-read your copy of CodeComplete.

Focused Code Reviews

I was going to write something laying out the arguments for code reviews, but Hacknot beat me to it (and did a better job):

Hacknot: In Praise Of Code Review

To which I'll only add that not only is Code Review cheaper than formal testing, but it picks up whole classes of defects that can never be found by traditional testing (eg: plain unmaintainable code)

On my last project we did peer code reviews, with three random reviewers for the week, each performing a different role in the review:

• Customer Focused & Usability (do the messages make sense, does the flow work, are we accurate and not robust (or viceversa if required))
• Maintenance and Legibility (self-documenting code, documentation comments, architectural synergies)
• Bugslaying (checking for obvious foobars, and reviewing test coverage)

We found this split worked pretty well at keeping reviewers focused, without which people can tend to miss the hard-to-spot stuff. Additionally having multiple people review each unit ensures that someone's bound to pick up on any serious issues, whilst also maximizing the cross-pollination of approaches.

Also I'm all for paper reviews. It's easier to annotate on paper, plus I tend to think along the lines that if someone missed it once on-screen, maybe printing it out reduces the chance someone else will do the same thing.

When to Automate

Generally speaking there are two main reasons to automate a process:

• It's going to be done many times (so it'll save time), or
  
• It needs to be done exactly the same each time

Doing a release build falls into both those categories, so from that perspective automated server builds (ala NAnt / CruiseControl.Net) start to look like a bit of a no brainer (especially if they can do the depoyment for you).

Interestingly, the Braidy Tester points out there's another important reason to automate, one that - as a developer - we invoke all the time: Morale. Even if I'm only going to do something once, if it doesn't take me much longer to automate it, then I will, for the sake of my own sanity. I'm a developer, not a computer.

It's worth factoring in the 'hidden cost' of not automating when making leadership calls, and err on the side of automation when it's a close call. Is not automating worth the morale cost of making people do really boring work?

Fluent Interfaces vs Currying

Martin Fowler's written recently about Fluent Interfaces, which are basically class API's designed from the perspective of creating highly-legible, flowing code. I've been dallying a bit with this over the last year, eg:

    TaskUtils.CopySettingsFrom(source).To(destination);

or

    Assert(someString, new Contains("some substring"));

... having been inspired by the NMock designers. There's even some examples in the .Net class library (eg: the SqlParameterCollection's Add method returns the parameter just added).

I'd only just got round to thinking of it as Currying (and prior to that, Syntactic Sugar). Terminology explosion! Martin's example is a bit more complicated than mine, and the more complicated schemes have their disadvantages, but it's always nice to see people going the extra mile to make their code usable. It adds polish, finesse.

A sharp tool[1] like ReSharper, or VS2005 (C# only) can really make this kind of stuff a breeze - you just type

    someobject.Add(thing).Substract(thing)

...and ReSharper will prompt you to create the appropriate Add and Subtract methods, with it's best guess as to the parameters you want. The result - clearer, more usable API's - is one of the reasons why the TDD guys have been advocating this approach all the time....


[1] As in Chapter 12, 'The Mythical Man Month', Frederick P. Brookes