'Specified wildcard pattern is invalid' with Octopus Deploy and PSake

So we upgraded to Octopus Deploy v2, and all our deployments are now failing with this error:

22/07/2014 11:51:08 AM: An Error Occurred:

Info 11:51:08

Test-Path : Cannot retrieve the dynamic parameters for the cmdlet. The

Info 11:51:08

specified wildcard pattern is not valid:

Info 11:51:08

Octopus.Environment.MachinesInRole[myproject-sql-node]

At D:\Octopus\Applications\Test\ORDW.Staging.TPPS\7.0.310-trunk_1\Tools\Install

Info 11:51:08

\psake.psm1:357 char:17

Info 11:51:08

+ if (test-path "variable:\$key") {

Our install process is quite complex, so we use Psake to wrangle it. Integration between the two is relatively straightforward (in essence we just bind $octopusParameters straight onto psake's -properties), and I could see from the stack trace that the failure was actually happening within the PSake module itself. And given the error spat out the variable that caused the issue, I figured it was to do with the variable name.

Most of the variable names are the same as per Octopus Deploy v1, but we do now get some extra ones, in particular the 'Octopus.Environment.MachinesInRole[role]' one. But that's not so different from the type of variables we've always got from Octopus, eg: 'Octopus.Step[0].Name', so what's different?

Where psake is failing is where it pushes properties into variable scope for each of the tasks it executes as part of the 'build', and apparently it's choking because test-path doesn't like it. So I put together some tests to exercise test-path with different variable names, and find out when it squealed. This all works against the same code as runs in psake, ie:

test-path "variable:\$key"

$key	Result
a	Ok
a.b	Ok
a-b	Ok
b.a	Ok
b-a	Ok
a.b.c	Ok
a-b-c	Ok
c.b.a	Ok
c-b-a	Ok
a[a]	Ok
a[a.b]	Ok
a[a-b]	Ok
a[b.a]	Ok
a[b-a]	Cannot retrieve the dynamic parameters for the cmdlet.  The specified wildcard pattern is not valid: a[b-a]
a[a.b.c]	Ok
a[a-b-c]	Ok
a[c.b.a]	Ok
a[c-b-b]	Cannot retrieve the dynamic parameters for the cmdlet.  The specified wildcard pattern is not valid: a[c-b-b]
a[b-b-a]	Ok

I've highlighted the failure cases, but what's just as interesting is which cases pass. This gives a clue as to the underlying implementation, and why the failure happens.

To cut a long story short, it appears that any time you use square brackets in your variable name, PowerShell uses wildcard matching to parse the content within the brackets. If that content contains a hypen, then the letters before and after the first hyphen are used as a range for matching, and the range end is prior to the range start (ie: alphabetically earlier), you get an error.

Nasty.

It's hard to know who to blame here. Octopus makes those variables based on what roles you define in your environment, though I'd argue the square brackets is potentially a bad choice. PSake is probably partly culpable, though you'd be forgiven for thinking that what they were doing was just fine, and there's no obvious way of supressing the wildcard detection. Ultimately I think this is probably a PowerShell bug. Whichever way you look at it, the chances of me getting it fixed soon are fairly slight.

In this case I can just change all my Octopus role names to use dots not hypens, and I think I'll be out of the woods, but this could be a right royal mess to fix otherwise. I'd probably have to forcefully remove variables from scope just to keep PSake happy, which would be ugly.

Interestingly the documentation for Test-Path is a bit confused as to whether wildcard matching is or isn't allowed here - the description says they are, but Accept wildcard characters' claims otherwise:

PS C:\> help Test-Path -parameter:Path

-Path 
    Specifies a path to be tested. Wildcards are permitted. If the path includes spaces, enclose it in quotation marks. The parameter 
    name ("Path") is optional.
    
    Required?                    true
    Position?                    1
    Default value                
    Accept pipeline input?       True (ByValue, ByPropertyName)
    Accept wildcard characters?  false

Also interesting is that Set-Variable suffers from the same issue, for exactly the same cases (and wildcarding definitely doesn't make any sense there). Which means you can do this:

${a[b-a]} = 1

but not this

Set-Variable 'a[b-a]' 1

Go figure.


Update 23/7:

• You can work around this by escaping the square brackets with backticks, eg Set-Variable 'a`[b-a`]'
• I raised a Connect issue for this, because I think this is a bug
• I've raised an issue with Psake on this, because I think they should go the escaping route to work around it.

Another PowerShell Gotcha - Order of precidence and type inference

There was a fun moment at last week's Perth Dot Net User Group when the presenter (demonstrating Octopus Deploy's new scripting console), put in the following PowerShell:

> Write-Host [Environment]::MachineName

And got back (literally)

[Environment]::MachineName

...which wasn't what he expected at all. It's worth pausing to consider why this happens.

Remember of course that when passing strings as arguments in PowerShell, the quotes are optional - it would make a fairly poor shell replacement otherwise. So for example the following is completely valid:

> Write-Host Hello!

This is the reason^[1] why variables in PowerShell have the dollar prefix - it makes their use in binding fairly unambiguous (just as the @ symbol in Razor achieves the same thing).

> $a = 'Moo'
> Write-Host $a
Moo

If you really wanted to write '$a' you'd have to enclose it in single quotes (as I just did) or escape the dollar symbol.

Anyway back to the original problem, you can see that PowerShell has two possible ways of interpreting

> Write-Host [Environment]::MachineName

...and since it doesn't start with a $, you get the 'bind to as an object' behavior, which - in this case - gives you a string (since it's clearly not a number).

What you really wanted was one of the following:

> Write-Host ([Environment]::MachineName)
> Write-Host $([Environment]::MachineName)
SOMECOMPUTERNAME

They both give the intended result, by forcing the expression within the brackets to be evaluated first (which on its own is unambiguous to the parser), and then passing the result of that as an argument to the bind for Write-Host.

This is really important trick to know, because will otherwise bite you again and again when you try and call a .Net method, and attempt to supply a parameter via an expression, for example:

$someClass.SomeMethod($a.length -1)
...when what you need to say is

$someClass.SomeMethod(($a.length -1))
Key take-home: When in doubt, add more brackets^[2]

[1] presumably
[2] parenthesis

Get-Member on empty collections

PowerShell's pipeline just loves to unravel collections, with the result that sometimes, when you want to do something on the collection itself, you can't. Like with Get-Member:

$blah.Catalogs | Get-Member
Get-Member : No object has been specified to the get-member cmdlet.

What happened? Did $object.Catalogs return null, or did it return an empty IEnumerable? This has bitten me a few times, especially when poking around in an API for the first time (ie: at this point I have no idea what 'Catalogs' is, whether it's ICollection or whatever).

The answer, I realize, is to avoid the pipeline:

Get-Member -inputObject:$blah.Catalogs

   TypeName: BlahNamespace.BlahCollection

Name           MemberType            Definition                                                      
----           ----------            ----------                                                      
Add            Method                System.Void Add(Microsoft.SqlServer.Management.IntegrationSer...
Clear          Method                System.Void Clear()                                             

Much better

Analysis Services 2008 R2 breaking change when deploying from the command line

As collegues of mine will attest, I will script anything that has to be deployed. Some things are easier than others.

In the case of Analysis Services, the .asdatabase file that comes out of the build needs to be futher transformed to create the XMLA that you need to run on the server to deploy your (updated) cube definition. Rather than attempt to replicate this transformation, I have previously chosen to get the Analysis Services deployment utility to do this for me, since this can supplied with command line arguments:

write-host "Generating XMLA"
$asDeploy = "$programfiles32\Microsoft SQL Server\90\Tools\Binn\VSShell\Common7\IDE\Microsoft.AnalysisServices.Deployment.exe"
& $asDeploy "$pwd\..\bin\MyCube\MyCube.asdatabase" /d /o:"$pwd\MyCube.xmla"

Which works just nicely. Except when we migrated that project to SQL 2008 R2, when it stopped working.

Well, actually that's not true. We'd been deploying to a 2008 R2 server for ages, it was when we changed the deployment script to use the 2008 version of the deployment tool that it all broke.

Basically the next line in the script kept complaining that 'MyCube.xmla' didn't exist, but I'd look in the folder after the script had run and the file was there. So it seemed like maybe there was a race condition.

Which there was.

If you examine the PE headers for the Sql 2005 version of the deployment util (using a tool like dumpbin) you'll see it's marked as a command line application:

C:\>dumpbin /headers "C:\Program Files (x86)\Microsoft SQL Server\90\Tools\Binn\VSShell\Common7\IDE\Microsoft.AnalysisServices.Deployment.exe" | find /i "subsystem"
            4.00 subsystem version
               3 subsystem (Windows CUI)

...but the 2008 R2 version is marked as a gui application:

C:\>dumpbin /headers "C:\Program Files (x86)\Microsoft SQL Server\100\Tools\Binn\VSShell\Common7\IDE\Microsoft.AnalysisServices.Deployment.exe" | find /i "subsystem"
            4.00 subsystem version
               2 subsystem (Windows GUI)

What? Can't see the difference? One is marked CUI with a 'C', the other is GUI with a 'G'. An unfortunately high degree of visual similarity given what a fundamental difference it makes: launch the first from the command line and you wait for it, launch the second and you don't. When scripting it's pretty important to know which one you've got, or you're going to get race conditiions.

In this case the answer was to control the process launching, so we can explicitly decide to wait:

     start-process -FilePath:$asDeploy -ArgumentList:$asdatabase,"/d","/o:$xmla" -Wait;

Maybe I should just do that all the time to be safe, but just being able to use other command line tools within a script without a whole lot of ceremony is one of the really nice bits about powershell, so I tend not to. In this case the launch semantics of an existing utility changing between versions seems like a really nasty thing to be caught out by.
Good reference sources:
Stack Overflow: Can one executable be both a console and GUI app?
MSDN: A Tour of the Win32 Portable Executable File Format

Configuring SSAS 2008 HTTP access on IIS 7.5 using PowerShell

…was way too much like hard work / blindly poking about in the dark, and there’s still a nasty hacky bit in the script where I gave up trying to make the IIS snap-in work, and just used appcmd, but it does actually work:

<#
.Synopsis
Configures HTTP access for SSAS on the local box

.SeeAlso
http://technet.microsoft.com/en-sg/library/gg492140(en-us).aspx
http://learn.iis.net/page.aspx/436/powershell-snap-in-changing-simple-settings-in-configuration-sections/
http://www.iis.net/ConfigReference/system.webServer/security/isapiCgiRestriction
#>
$ErrorActionPreference = 'stop'
$scriptDir = split-path $myInvocation.MyCommand.Path;
$isapiPath = "$scriptDir\msmdpump.dll"

import-module WebAdministration
cd IIS:\

# Register msmdpump as a (globally) acceptable ISAPI handler
$isapiRestrictions = Get-WebConfiguration /system.webServer/security/isapiCgiRestriction
$handler = @($isapiRestrictions.Collection | ? { $_.description -eq 'OLAP' } )
if(-not $handler){
    # This way the object always appears locked?
    # but should be fine according to http://forums.iis.net/t/1158906.aspx
   $null = @"
    $restrictions = $isapiRestrictions.GetCollection();
    $handler = $restrictions.CreateElement('add');
    $handler.SetAttributeValue('path', $isapiPath);
    $handler.SetAttributeValue('allowed',$true);
    $handler.SetAttributeValue('description','OLAP');
    $restrictions.Add($handler);
"@

    # instead try
    # http://serverfault.com/questions/134361/programmatically-add-an-isapi-extension-dll-in-iis-7-using-adsi
    & "$env:windir\system32\inetsrv\appcmd.exe" set config /section:isapiCgiRestriction /+"[path='$isapiPath', description='OLAP',allowed='True']"

}

# Create the app pool
cd IIS:\AppPools
if(-not (Test-Path OLAP)){
    $olapAppPool = New-WebAppPool OLAP
}else{
    $olapAppPool = Get-Item OLAP
}
$olapAppPool.managedPipelineMode = 'classic'
$olapAppPool | Set-Item

# Create the website
cd IIS:\Sites
$defaultSite = @(dir)[0]
cd $defaultSite.Name

if(-not (Test-Path OLAP)){
    $olapApp = New-WebApplication OLAP -physicalPath:$scriptDir
}else{
    $olapApp = Get-Item OLAP
}
$olapLocation = '{0}/{1}' -f $defaultSite.Name,$olapApp.Name
cd OLAP

# Setup the web application: first associate the app pool
set-itemproperty $pwd -name applicationPool -value 'OLAP'

# ...then enable anonymous access
$basicAuth = Get-WebConfiguration -Filter /system.webServer/security/authentication/anonymousAuthentication
$basicAuth.Enabled = $true
$basicAuth | Set-WebConfiguration -Filter /system.webServer/security/authentication/anonymousAuthentication -PSPath IIS:\ -Location $olapLocation

# ...and create a mapping for that handler for this web app
$mapping = Get-WebHandler OLAP
if(-not $mapping){
    $mapping = New-WebHandler OLAP -ScriptProcessor:$isapiPath -Path:*.dll -Verb:* -Location:$olapLocation -PSPath:IIS:\ -Modules:IsapiModule
}
$mapping

Wow. IIS administration is just as bizarre and arcane as it always was. Did someone say ‘where is the setup wizard’?

Extract Zip from PowerShell

Adapted from an old Powershell Guy post, and flipped around to unzip the files:

function unzip($zipPath, $destination){
    $shell = new-object -com shell.application;
    $zip = $shell.NameSpace($zipPath);
    foreach($item in $zip.items()){
        $shell.Namespace($destination).copyhere($item)
    }
}

Evil use of shell object (where is the managed API?), but seems to work quite nicely actually. Note carefully the brackets after $zip.Items().

Now I’ve worked it out it seems really easy to find others doing the same thing. Oh well.

• http://www.powershellcommunity.org/Forums/tabid/54/aft/5239/Default.aspx



• http://learningpcs.blogspot.com/2010/07/powershell-working-with-zip-files.html



• http://www.vistax64.com/powershell/166287-re-copyhere-ignores-voptions-windows-powershell.html

PowerDbg is search result #7 for ‘WinDbg’

Ok, this is only on MSDN search, but still that seems pretty damn high:

Mind you, we’re #38 on Bing, and #14 on Google so we’re not completely inconspicuous.

Time to pull our fingers out and finish off v6 I think.

Converting to Int

You wouldn’t have thought that such as basic operation as turning a double into an integer would be so poorly understood, but it is. There are three basic approaches in .Net:

• Explicit casting, i.e. (int)x
• Format, using String.Format, or x.ToString(formatString)
• Convert.ToInt32

What’s critical to realise is that all of these do different things:

    var testCases = new[] {0.4, 0.5, 0.51, 1.4, 1.5, 1.51};

    Console.WriteLine("Input  Cast   {0:0}  Convert.ToInt32");

    foreach (var testCase in testCases)

    {

        Console.WriteLine("{0,5} {1,5} {2,5:0} {3,5}", testCase, (int)testCase, testCase, Convert.ToInt32(testCase));

    }

Input  Cast   {0:0} Convert.ToInt32
  0.4     0     0     0
  0.5     0     1     0
 0.51     0     1     1
  1.4     1     1     1
  1.5     1     2     2
 1.51     1     2     2

As my basic test above shows, just casting is the equivalent of Math.Floor – it looses the fraction. This surprises some people.

But look again at the results for 0.5 and 1.5. Using a format string rounds up[1], to 1 and 2, whereas using Convert.ToInt32 performs bankers rounding[2] (rounds to even) to 0 and 2. This surprises a lot of people, and you’d be forgiven for missing it in the doco (here vs. here):

Even more interesting is that PowerShell is different, in that the [int] cast in PowerShell is the same as a Convert.Int32, not a Math.Floor():

> $testCases = 0.4,0.5,0.51,1.4,1.5,1.51
> $testCases | % { "{0,5} {1,5} {2,5:0} {3,5}" -f $_,[int]$_,$_,[Convert]::ToInt32($_) }

Input  Cast   {0:0} Convert.ToInt32
  0.4     0     0     0
  0.5     0     1     0
 0.51     1     1     1
  1.4     1     1     1
  1.5     2     2     2
 1.51     2     2     2

This is a great gotcha, since normally I’d use PowerShell to test this kind of behaviour, and I’d have seen the wrong thing (note to self: use LinqPad more)

 

[1] More precisely it rounds away from zero, since negative numbers round to the larger negative number.

[2] According to Wikipedia bankers rounding is a bit of a misnomer for ‘round to even’, and even the MSDN doco on Math.Round seems to have stopped using the term.

Splatting Hell

Recently both at work and at home I was faced with the same problem: a PowerShell ‘control’ script that needed to pass parameters down to an arbitrary series of child scripts (i.e. enumerating over scripts in a directory, and executing them in turn).

I needed a way of binding the parameters passed to the child scripts to what was passed to the parent script, and I thought that splatting would be a great fit here. Splatting, if you aren’t aware of it, is a way of binding a hashtable or array to a command’s parameters:

# ie replace this:
dir -Path:C:\temp -Filter:*

# with this:
$dirArgs = @{Filter="*"; Path="C:\temp"}
dir @dirArgs

Note the @ sign on the last line. That’s the splatting operator (yes, its also the hashtable operator as @{}, and the array operator as @(). It’s a busy symbol). It binds $dirArgs to the parameters, rather than attempting to pass $dirArgs as the first positional argument.

So I thought I could just use this to pass any-and-all arguments passed to my ‘master’ script, and get them bound to the child scripts. By name, mind, not by position. That would be bad, because each of the child scripts has different parameters. I want PowerShell to do the heavy lifting of binding the appropriate parameters to the child scripts.

Gotcha #1

I first attempted to splat $args, but I’d forgotten that $args is only the ‘left over’ arguments after all the positional arguments had been taken out. These go into $PSBoundParameters

Gotcha #2

…but only the ones that actually match parameters in the current script/function. Even if you pass an argument to a script in ‘named parameter’ style, like this:

SomeScript.ps1 –someName:someValue

…if there’s no parameter ‘someName’ on that script, this goes into $args as two different items, one being ‘-someName:’ and the next being ‘someValue’. This was surprising. Worse, once the arguments are split up in $args they get splatted positionally, even if they would otherwise match parameters on what’s being called. This seems like a design mistake to me (update: there is a Connect issue for this).

Basically what this meant was that, unless I started parsing $args myself, all the parameters on all the child scripts had to be declared on the parent (or at least all the ones I wanted to splat).

Gotcha #3

Oh, and $PSBoundParameters only contains the named parameters assigned by the caller. Those left unset, i.e. using default values, aren’t in there. So if you want those defaults to propagate, you’ll have to add them back in yourself:

function SomeFunction(
    $someValue = 'my default'
){
    $PSBoundParameters['someValue'] = $someValue

Very tiresome.

Gotcha #4

$PSBoundParameters gets reset after you dotsource another script, so you need to capture a reference to it before that :-(

Gotcha #5

Just when you thought you were finished, if you’re using [CmdLetBinding] then you’ll probably get an error when splatting, because you’re trying to splat more arguments than the script you’re calling actually has parameters.

To avoid the error you’ll have to revert to a ‘vanilla’ from an ‘advanced’ function, but since [CmdLetBinding] is implied by any of the [Parameter] attributes, you’ll have to remove those too :-( So back to $myParam = $(throw ‘MyParam is required’) style validation, unfortunately.

(Also, if you are using CmdLetBinding, remember to remove any [switch]$verbose parameters (or any others that match the ‘common’ cmdlet parameters), or you’ll get another error about duplicate properties when splatting, since your script now has a –Verbose switch automatically. The duplication only becomes an issue when you splat)

What Did We Learn?

Either: Don’t try this at home.

Or: Capture PSBoundParameters, put the defaults back in, splat it to child scripts not using CmdLetBinding or being ‘advanced functions’

Type your parameters, and put your guard throws back, just in case you end up splatting positionally

Have a lie down

One-line TODO Extractor in PowerShell

I previously wrote a PowerShell TODO extractor, that blasts through an entire source hierarchy looking for TODOs, and reports them to the console, complete with a few lines of context either side so you can tell what you’re looking at. It was like, 20 lines of code.

Well blow me if v2 just doesn’t do it out of the box:

PS > dir . -filter:*.cs -recurse | select-string "\sTODO\s" -context:4 -CaseSensitive

What’s New In PowerShell 2

At work, where I do most of my PowerShell, we’ve only just shifted off XP, so until recently I’d not really looked much into the differences between PowerShell 1 and 2. The ISE is pretty good (its a debugger!), support for webservices is a few years too late (but very welcome) and I can see Remote PowerShell being pretty useful.

So I’d not really been keeping up. If anything I was deliberately ignoring it, to avoid the temptation to write something that would require upgrading the server. But eventually, I cracked[1].

Oh My God.

Put aside for the moment the absolute avalanche[2] of new cmdlets (write-verbose, out-gridview, select-xml[3], measure-object etc…), and put aside for the moment support for background jobs, the wonderful -split and -join operators, and even put aside how tab-completion now works for .net static methods...

Tab completion now works for script functions and their parameters. You can type in a function on one line, and be happily tab-completing it on the next. You can even add comment-based or XML help, though probably not at the console.

Once again, PowerShell rocks

 

[1] Blame PowerDbg

[2] Some guy[4] is writing a blog series on every new cmdlet!

[3] Select-Xml: Here’s one I used today at work to get all the references from all the C# project files within a folder hierarchy. Sure you could do it all before with XmlDocument, but check this out:

PS > dir . -filter:*.csproj -Recurse | `
Select-Xml -XPath:'//*[local-name() = "Reference"]' | `
Select-Object -ExpandProperty Node

Include
-------
System
System.Core
System.Xml.Linq
System.Data.DataSetExtensions
System.Data
System.Xml

[4] He’s called Jonathan Medd, but the ‘some guy’ thing has a certain ring to it…

[5] Oh, and proper try{}catch{}finally{} error handling. I missed that

WinDbg Pain Points

Previously I talked about PowerDbg, what an awesome idea it was, but how it lacked some things. Well I spoke to the author, Roberto[1], who asked me to put my code where my mouth was, and now I am working with him on the next version.

So… if there’s anything particularly painful that you do in WinDBG now is the time to shout. You can comment on this blog if you like, but better would be to raise a ‘proposed feature’ on the Codeplex site itself.

A good example would be just how hard it is to work with a .Net Dictionary in WinDBG (except PowerDbg already handles that, and even better in the new version). Anything where you want a slightly ‘higher level’ view of the raw SOS data.

 

[1] Yes, that Roberto.

PowerShell 2 Breaking Change When Shelling Out

Whilst PowerShell 2 is by-and-large backwards compatible, I’ve discovered at least one breaking change that appears to be undocumented: the behaviour of argument parsing when calling another executable seems to have changed.

Previous behaviour:

PowerShell has effectively parsed the argument as if it were calling a PowerShell script: splitting it into two parts along the colon, and passing the second part ‘intact’ because it was quote wrapped.

New behaviour in v2:

PowerShell has treated the arguments as completely opaque and passed them to the exe using ‘normal’ command line parsing semantics (split on spaces etc…). It has not split the argument along the colon (which was the breaking change for us). In the second case, because the argument didn’t start with a quote (it starts with ‘-test’) the argument is broken in half at the space.

I think this is a good change, in that PowerShell shouldn’t make assumptions about how the exe you are calling likes it’s parameters (I got badly burnt that way trying to call an SSIS package). But it’s certainly one to watch out for.

 

PS: Not sure at all about this behaviour, which is the same in both v1 and v2:

Surely the fact you pass the argument as a string variable indicates you want it as one argument. Surely.

Working Directory Independence for PowerShell Scripts

pushd (split-path $myInvocation.MyCommand.Path);

Not quite as simple or memorable as the batch file version sadly…

Automating WinDBG with PowerShell

I’ve been doing a bit of WinDBG work recently after a long hiatus, and I’ve been blown away by some of the things I’ve missed.

One of them was PowerDBG: a Powershell (2) module for working with WinDBG in Powershell. How awesome is that? No really, how freaking awesome.

But I couldn’t help but feel the execution was lacking something. It wasn’t, for want of a better word, very Powershelly. For example, this is what you’d do in PowerDBG to look at an object:

PS C:\> connect-windbg "tcp:port=10456,server=mr-its-64-vds"
PS C:\> Send-PowerDbgCommand ".loadby sos mscorwks"
PS C:\> Send-PowerDbgCommand "!do 0000000001af7680"
# Glance at the original WinDBG output make sure it looks ok
PS C:\> $global:g_commandOutput
0:000> Name: MyNamespace.Services.MyService
MethodTable: 000007ff002b1fd8
EEClass: 000007ff002af238
Size: 72(0x48) bytes
 (c:\Program Files (x86)\SomeFolder\SomeDll.dll)
Fields:
              MT    Field   Offset                 Type VT     Attr
Value Name
0000000000000000  4000148        8                       0 instance 00000000024
09858 _laneGroups
0000000000000000  4000149       10                       0 instance 00000000024
04490 _lanes
0000000000000000  400014a       18                       0 instance 00000000026
c7730 _routes
0000000000000000  400014b       20                       0 instance 00000000024
d4f78 _roadSections
0000000000000000  400014c       28                       0 instance 00000000026
cc668 _devices
000007ff007222e0  400014d       30 ...gDatabaseProvider  0 instance 0000000001a
f76c8 _provider
0000000000000000  400014e       38                       0 instance 00000000023
16b30 MappingsUpdated

# Call the dump-object parser to stick it in a CSV file
PS C:\> Parse-PowerDbgDSO

# look in the CSV file
PS C:\> type .\POWERDBG-PARSED.LOG
key,value

Name:,MyNamespace.Services.MyService#$#@
:,000007ff002b1fd8#$#@
:,000007ff002af238#$#@
72(0x48),bytes#$#@
4000148,8                       0 instance 0000000002409858 _laneGroups#$#@
4000149,10                       0 instance 0000000002404490 _lanes#$#@
400014a,18                       0 instance 00000000026c7730 _routes#$#@
400014b,20                       0 instance 00000000024d4f78 _roadSections#$#@
400014c,28                       0 instance 00000000026cc668 _devices#$#@
400014d,30 ...gDatabaseProvider  0 instance 0000000001af76c8 _provider#$#@
400014e,38                       0 instance 0000000002316b30 MappingsUpdated#$#
@
PS C:\>

That’s a bit ugh. Commands share state via the global 'g_commandoutput' rather than the pipeline, and the end-goal of most operations seems to be to spit out a CSV file POWERDBG-PARSED.Log.

I think we can do better.

I want objects, preferably ones that look like my original objects. I want to be able to send them down the pipeline, filter on them, sort them and maybe pipe some back to the debugger to pick up more details. And I want cmdlets for common WinDBG /SOS operations like !dumpobject rather than pass every command as a string. In short, I want a real PowerShell experience.

More like this:

PS C:\> $o = dump-object 0000000001af7680
PS C:\> $o

__Name          : Mrwa
__MethodTable   : 000007ff002b1fd8
__EEClass       : 000007ff002af238
__Size          : 72
_laneGroups     : 0000000002409858
_lanes          : 0000000002404490
_routes         : 00000000026c7730
_roadSections   : 00000000024d4f78
_devices        : 00000000026cc668
_provider       : 0000000001af76c8
MappingsUpdated : 0000000002316b30
__Fields        : {System.Object, System.Object, System.Object, System.Object..
                  .}

Note how I've mapped the field value/addresses onto a synthetic PowerShell object that uses the same names for the properties as the original fields (which were underscore prefixed, as you can see in the original WinDBG output above). I can then work with the object in the debugger in a more natural way:

PS C:\> $o._lanes | dump-object


__0           : 000
__MethodTable : 000007ff0072b8c8
__EEClass     : 000007feede6ba30
__Size        : 88
buckets       : 00000000024044e8
entries       : 00000000024050e8
count         : 688
version       : 688
freeList      : -1
freeCount     : 0
comparer      : 00000000013da180
keys          : 0000000000000000
values        : 0000000000000000
_syncRoot     : 0000000000000000
m_siInfo      : 0000000000000000
__Fields      : {System.Object, System.Object, System.Object, System.Object...}

Note also that I've kept the metadata originally available about the object by mapping those WinDBG output lines to double underscore-prefixed properties on the object. And I've not lost all that extra metadata about the fields either: whilst the properties above 'shortcut' to the field value/address, you can look in the __Fields collection to find the details if you need them (it's just much harder to pipeline stuff this way):

PS C:\> $o.__Fields


MT     : 0000000000000000
Field  : 4000148
Offset : 8
Type   :
VT     : 0
Attr   : instance
Value  : 0000000002409858
Name   : _laneGroups

MT     : 0000000000000000
Field  : 4000149
Offset : 10
Type   :
VT     : 0
Attr   : instance
Value  : 0000000002404490
Name   : _lanes

# ... etc...

Normally looking in arrays and dictionaries via WinDBG is a massive pain in the arse (find the backing array for the dictionary, find the key-value pair for each item, find the object that the value points to). PowerDBG has a script to automate this, and again I've tried to implement a more 'pipeliney' one:

PS C:\> $items = dump-dictionary $o._lanes
PS C:\> $items[0..2]

key                                     value
---                                     -----
00000000024098f8                        00000000024098d0
0000000002409a10                        00000000024099e8
0000000002409a68                        0000000002409a40

You can easily pipe this to dump-object to inspect the objects themselves. In my case I wanted to know if any of the objects in the dictionary had a given flag set, which ended up looking something like this:

PS C:\> $items | 
	% { Dump-Object $_.value } | 
	? { $_.MyFlag -eq 1 } | 
	% { $_.MyId } | 
	Dump-Object | 
	% { $_.__String }

That's a mouthful, but basically what I'm doing is getting doing a !do for all the values in that dictionary, and for all those that have the MyFlag set true I send the MyId field down the pipeline. That's a string, so I do a dump-object on it, and then send the actual string value to the output.

With a large dictionary this stuff can take quite some time (and seriously chew memory in WinDBG) but then you wouldn’t be worrying about any of this if the dictionary only had two items – you’d do it by hand.

At the moment all this is unashamedly sitting atop PowerDBG’s basic ‘channel’ to WinDBG, but that should probably go too. PowerDBG grabs lines from the console out and concatenates them into a string, but actually want line-by-line output from the debugger, because I want to leverage PowerShell’s streaming pipeline (i.e. emit objects as they are ready, rather than when the command finishes). Maybe another day.

You can get the script for this from my SkyDrive. It’s definitely a first pass, but.

PowerShell <3

I was in a deep directory tree, and I wanted to CD into the same relative location in another tree (another parallel TFS workspace as it happens):

cd ($pwd -replace 'ITS_Spare','ITS')

Ah, bliss[1]

 

[1] In a deeply sad and nerdy way, admittedly

3 PowerShell Array Gotchas

I love PowerShell, and use it for pretty much everything that I’m not forced to compile. As a result I’ve got fairly competent, and people have suggested to me that I should pull my finger out and do more blogging about PowerShell tricks and tips and suchlike. And they are right.

As a first pass, here are 3 PowerShell gotchas, all which revolve around array handling. PowerShell does some funky stuff here to make certain command line operations more intuitive, which can easily throw you if you are still thinking C# [1]

Converting Function Arguments To An Array By Mistake

When you call a .Net method, you use the familiar obj.Method(arg1,arg2) syntax, with a comma between each parameter.

However when you call a script function (or a cmdlet), you must omit the commas. If you don’t, you pass your arguments as an array to the first parameter, and many times the resulting error won’t immediately tip you off.

PS > function DumpArgs($one,$two){ write-host "One: " $one; write-host "Two: " $two }

PS > dumpargs 1 2  # Correct
One:  1
Two:  2

PS > dumpargs 1,2  # Incorrectly pass both to 1st parameter
One:  1 2
Two:

Subtle, yes? Anyone ever who has ever done any PowerShell ever has done this at least once. Ever.

Passing An Array to a .Net Constructor or Method

When you call a .Net method, as described above, that familiar comma syntax is actually still creating an array from your arguments, it’s just that’s what PowerShell uses to call .Net methods (and ctors) via the reflection APIs.

So there is a reverse gotcha. How do you call a .Net method (or ctor) that takes an array as its single parameter? Whether you create an array in-line (using comma syntax) or up-front as a variable, you will likely be told ‘Cannot find an overload for "xxx" and the argument count: "n"’, as PowerShell fails to find a method with the same number of parameters as the length of your array:

PS > $bytes = [byte[]]0x1,0x2,0x3,0x4,0x5,0x6
PS > $stream = new-object System.IO.MemoryStream $bytes
New-Object : Cannot find an overload for "MemoryStream" and the argument count: "6".

If you make the byte array smaller you’ll get other errors, as the length matches one of the ctor overloads, but not the types, or you may get semantic errors when the values bind to an overload, but fail imperative validation logic. eg:

PS > $bytes = [byte[]]0x1,0x2,0x3
PS > $stream = new-object System.IO.MemoryStream $bytes
New-Object : Exception calling ".ctor" with "3" argument(s): "Offset and length were out of bounds…

Worst of all, sometimes it can ‘work’ but not in the way you intended.

As an exercise, imagine what would happen if the array was 0x1,0x0,0x1 [3].

The (somewhat counter-intuitive) solution here is to wrap the array – in an array. This is easily done using the ‘comma’ syntax (a comma before any variable creates a length-1 array containing the variable):

PS > $bytes = 0x1,0x2,0x3,0x4,0x5,0x6
PS > $stream = new-object System.IO.MemoryStream (,$bytes)
PS > $stream.length
6

Indexing into a String, Expecting a String[]

PowerShell love to unpack things: it’s like kids at Christmas. So if a function ‘returns’ a collection with only one item in it (only one line in the file, or one file in the directory) you will get the item back, and not the collection.

Since a string itself can be indexed (as if it were char[]), this can lead to weird behaviour:

PS C:\Users\Piers> (dir .\Links | % { $_.Name })[0]
Desktop.lnk
PS C:\Users\Piers> (dir .\Links\desk* | % { $_.Name })[0]
D

In the first case the indexer retrieves the first file name as expected. However in the second only one item matched the wildcard. As a result we didn’t get back an array, but an item (the string name), and that's what we indexed into (yielding the first character). Not what we wanted.

The easy fix is to always use @ to ensure an expression produces an array, even if it only evaluates to a single item:

PS C:\Users\Piers> @(dir .\Links\desk* | % { $_.Name })[0]
Desktop.lnk

(NB: This is different from the ‘comma’ syntax described above that always introduces a parent array)

Bonus: Enumerating Collections of Arrays Without Unpacking

On a similar note, if you have a collection of arrays, and you pipe it, you will only ‘see’ the individual items, not the arrays. Again, the answer here is to wrap the arrays in arrays: only one level of unravelling is performed.

Bonus: Enumerating a Hashtable

By contrast, Hashtables don’t unravel automatically, though you might imagine they do. For example:

PS > $items = @{One=1;Two=2;Three=3}
PS > $items | % { $_ }

Name                           Value
----                           -----
Two                            2
Three                          3
One                            1

PS > # and yet...
PS > $items | % { $_.Name }
PS > # returns nothing. 
PS > $items | % { $_.ToString() }
System.Collections.Hashtable

We're not actually enumerating the hashtable's *contents*, rather we are enumerating the hashtable as if it were a single item in a list. It just has very specific default rendering behaviour (which is why we see the contents spat out).

This normally happens for non-IEnumerable types, but presumably happens deliberately for Hashtable (which is enumerable) because it's quite 'special' within PowerShell. Anyway, to get round this you have to make the enumeration explicit:

PS > $items.GetEnumerator() | % { $_.Name }
Two
Three
One

Enough Bonus Already!

 

[1] In Hanselminutes 200 [2], Jon Skeet makes the point that C# and Java are – syntax-wise – similar enough for it to be confusing, as opposed to obvious ‘in your face’ transitions (eg between Java and VB#). I had the same experience when I travelled to the USA having spent 6 months in South America: when everyone was speaking Spanish you expected things to be different, but somehow in the US because they spoke the same language (nearly) my guard was down, and so every so often you’d be totally thrown by something being different. Anyway, I recon it’s like that between C# and PowerShell. It’s .Net and has {}’s so you are lured into a false sense of security and end up with trap all over your face. Or something.

[2] What is wrong with this URL? For show 200. That just freaks me out.

[3] No, the answer’s not down here. It’s an exercise for the reader.

Active Directory Powershell Utils

These four basic fuctions make life so much better when dealing with AD:

$ds = new-object system.directoryservices.directorysearcher

function Find-Group($samName){
 $ds.filter = ('(samaccountname={0})' -f $samName)
 $ds.FindAll()
}

function Find-User($samName){
 $ds.filter = ('(samaccountname={0})' -f $samName)
 $ds.FindAll()
}

function Find-Members($groupName){
 $group = Find-Group $groupName
 $group.Properties.member | Get-Entry
}

function Get-Entry($cn){
  begin{
    if($cn){
      new-object System.directoryservices.DirectoryEntry("LDAP://$cn")
    }   
  }
  process{
    if($_){
      new-object System.directoryservices.DirectoryEntry("LDAP://$_")
    }
  }
  end{
  }
}

(ok, three functions. Two are the same, I know)

When to Automate (Revisited)

I'm going to add another scenario to my 'When to Automate' list: when the execution time is more important than the preperation time.

The classic example of this is a typical data migration effort. Months of work goes into creating a one-off artefact, primarily to ensure the system down-time is minimized. There are other advantages (testing, pre-transition sign-off, auditing), but in most cases these just exist to mitigate against the risk of downtime, so it's still all about the execution interval.

What I'd not really thought about until the other day was that this also applies to refactoring efforts on code. Any large refactoring burns a considerable amount of time just keeping up with the changes people check-in whilst you're still working away. Given a sizable enough refactoring, and enough code-churn, you get to a point beyond which you actually can't keep up, and end up trapped in merge integration hell.

There are two normal responses to this. The first is to keep the size down in the first place, and bite off smaller, more manageable chunks. This can involve some duplication of effort, which is why many team-leads will take the second option: come in in the evening or over the weekend for a big-bang macho refactorfest.

But there is a 'third way': the refactoring can be as big as you like, as long as you keep it brief. If you can take the latest version of the codebase, refactor it and check it in before anyone else commits any more changes then you only have to worry about the pained expressions on the face of your co-workers as the entire solution restructures in front of their eyes. But you'll get that anyway, right?

Right now I've got some 50 projects, spread over 4 solutions, most of which have inconsistent folder name to project names to assemblyname to base namespace, some of which we just want to rename as the solution has matured.

To do this by hand would be an absolute swine of a job, and highly prone to checkin-integration churn (all those merges on .csproj / .sln files! the pain! the pain!). Fortunately however Powershell just eats this kind of stuff up. It's taken the best part of the day to get the script ready, but now it has the whole thing can be done and checked in in some 15 mins, solutions and source control included. And there's no point-of-no-return: any point up till checkin all I've done is improve the script for next time, no time has been wasted.

And by de-costing[1] the whole process, we can do it again some day...

[1] ok, I'm sorry for 'de-costing'. But I struggled to come up with a better term for 'reducing the overhead to next-to-nothing' that wasn't unnecessarily verbose, or had 'demean' connotations.

Birthday Present

Whilst I was unwrapping my other presents, Microsoft RTM’d Windows 7 and Powershell 2. Just for me! You shouldn’t have, etc… Only, apparently I can’t unwrap it until Aug 6th.

But who to send the ‘thank you’ to?