Wednesday, November 02, 2005

ThreadStatic, CallContext and HttpContext in ASP.Net

Summary:
Even if you think you know what you're doing, it is not safe to store anything in a ThreadStatic member, CallContext or Thread Local Storage within an ASP.Net application, if there is the possibilty that the value might be setup prior to Page_Load (eg in IHttpModule, or page constructor) but accessed during or after.

[Update: Aug 2008 In view of the fairly large number of people continuing to link to this post I feel the need to clarify that this thread-swapping behaviour happens at a very specific point in the page lifecycle and not whenever-it-feels-like-it. My wording after the Jef Newson quote was unfortunate. That aside, I've been immensely gratified (and flattered) by the number of times I've seen this post cited within design discussions around dealing appropriately with HttpContext. I'm glad people found it useful.]

There's a lot of confusion about using how to implement user-specific singletons in ASP.Net - that is to say global data that's only global to one user or request. This is not an uncommon requirement: publishing Transactions, security context or other 'global' data in one place, rather than pushing it through every method call as tramp data can make for a cleaner (and more readable) implementation. However its a great place to shoot yourself in the foot (or head) if you're not careful. I thought I knew what was going on, but I didn't.

The preferred option, storing your singletons in HttpContext.Current.Items, is simple and safe, but ties the singleton in question to being used within an ASP.Net application. If the singleton's down in your business objects, this isn't ideal. Even if you wrap the property-access in an if statement
if(HttpContext.Current!=null){
/* store in HttpContext */
}else{
/* store in CallContext or ThreadStatic */
}
... then you've still got to reference System.Web from that assembly, which tends to encorage more 'webby' objects in the wrong place.

The alternatives are to use a [ThreadStatic] static member, Thread local storage (which pretty much amounts to the same thing), or CallContext.

The problems with [ThreadStatic] are well documented, but to summarize:
Scott Hanselman gets it right, that ThreadStatic doesn't play well with ASP.Net, but doesn't fully explain why.

Storage in CallContext alleviates some of these problems, since the context dies off at the end of the request and GC will occur eventually (though you can still leak resources until the GC happens if you're storing Disposables). Additionally CallContext is how HttpContext gets stored, so it must be ok, right?. Irrespective, you'd think (as I did) that provided you cleaned up after yourself at the end of each request, everthing would be fine:
"If you initialize a ThreadStatic variable at the beginning of a request, and you properly dispose of the referenced object at the end of the request, I am going to go out on a limb and claim that nothing bad will happen. You're even cool between contexts in the same AppDomain

"Now, I could be wrong on this. The clr could potentially stop a managed thread mid-stream, serialize out its stack somewhere, give it a new stack, and let it start executing. I seriously doubt it. I suppose that it is conceivable that hyperthreading makes things difficult as well, but I also doubt that."
Jef Newsom

Update: This was the misleading bit. I do explain further later on that this thread-swap can only happen between the BeginRequest and the Page_Load, but Jef's quote creates a very powerful image I failed to immediately correct. My bad.

Trouble is that's exactly what happens. Trouble is that's almost what happens. Under load ASP.Net can migrate inbound requests from its IO thread pool to a queue taken up by it's worker process thread pool:
So at some point ASP.NET decides that there are too many I/O threads processing other requests. [...] It just takes the request and it queues it up in this internal queue object within the ASP.NET runtime. Then, after that's queued up, the I/O thread will ask for a worker thread, and then the I/O thread will be returned to its pool. [...] So ASP.NET will have that worker thread process the request. It will take it into the ASP.NET runtime, just as the I/O thread would have under low load.
Microsoft ASP.NET Threading Webcast
Now I always knew about this, but I assumed it happened early enough in the process that I didn't care. It appears however that I was wrong. We've been having a problem in our ASP.Net app where the user clicks one link just after clicking another, and our app blows up with a null reference exception for one of our singletons (I'm using CallContext not ThreadStatic for the singleton, but it turns out it doesn't matter).

I did a bit of research about how exactly ASP.Net's threading works, and got conflicting opinions-masquerading-as-fact (requests are thread-agile within a request vs requests are pinned to a thread for their lifetime) so I replicated my problem in a test application with a slow page (sleeps for a second) and a fast page. I click the link for the slow page and before the page comes back I click the link for the fast page. The results (a log4net dump of what's going on) surprised me.

What the output shows is that - for the second request - the BeginRequest events in the HttpModule pipeline and the page constructor fire on one thread, but the Page_Load fires on another. The second thread has had the HttpContext migrated from the first, but not the CallContext or the ThreadStatic's (NB: since HttpContext is itself stored in CallContext, this means ASP.Net is explicitly migrating the HttpContext across). Let's just spell this out again:
  • The thread switch occurs after the IHttpHandler has been created
  • After the page's field initializers and constructor run
  • After any BeginRequest, AuthenticateRequest, AquireSessionState type events that your Global.ASA / IHttpModules are using.
  • Only the HttpContext migrates to the new thread
This is a major PITA, because as far as I can see it mean the only persistence option for 'ThreadStatic'esque behavior in ASP.Net is to use HttpContext. So for your business objects, either you're stuck with the if(HttpContext.Current!=null) and the System.Web reference (yuck) or you've got to come up with some kind of provider model for your static persistence, which will need setting up prior to the point that any of these singletons are accessed. Double yuck.

Please someone say it ain't so.

Appendix: That log in full:
[3748] INFO  11:10:05,239 ASP.Global_asax.Application_BeginRequest() - BEGIN /ConcurrentRequestsDemo/SlowPage.aspx
[3748] INFO 11:10:05,239 ASP.Global_asax.Application_BeginRequest() - threadid=, threadhash=, threadhash(now)=97, calldata=
[3748] INFO 11:10:05,249 ASP.SlowPage_aspx..ctor() - threadid=3748, threadhash=(cctor)97, threadhash(now)=97, calldata=3748, logicalcalldata=3748
[3748] INFO 11:10:05,349 ASP.SlowPage_aspx.Page_Load() - threadid=3748, threadhash=(cctor)97, threadhash(now)=97, calldata=3748, logicalcalldata=3748
[3748] INFO 11:10:05,349 ASP.SlowPage_aspx.Page_Load() - Slow page sleeping....

[2720] INFO 11:10:05,669 ASP.Global_asax.Application_BeginRequest() - BEGIN /ConcurrentRequestsDemo/FastPage.aspx
[2720] INFO 11:10:05,679 ASP.Global_asax.Application_BeginRequest() - threadid=, threadhash=, threadhash(now)=1835, calldata=
[2720] INFO 11:10:05,679 ASP.FastPage_aspx..ctor() - threadid=2720, threadhash=(cctor)1835, threadhash(now)=1835, calldata=2720, logicalcalldata=2720, threadstatic=2720

[3748] INFO 11:10:06,350 ASP.SlowPage_aspx.Page_Load() - Slow page waking up....
[3748] INFO 11:10:06,350 ASP.SlowPage_aspx.Page_Load() - threadid=3748, threadhash=(cctor)97, threadhash(now)=97, calldata=3748, logicalcalldata=3748
[3748] INFO 11:10:06,350 ASP.Global_asax.Application_EndRequest() - threadid=3748, threadhash=97, threadhash(now)=97, calldata=3748
[3748] INFO 11:10:06,350 ASP.Global_asax.Application_EndRequest() - END /ConcurrentRequestsDemo/SlowPage.aspx

[4748] INFO 11:10:06,791 ASP.FastPage_aspx.Page_Load() - threadid=2720, threadhash=(cctor)1835, threadhash(now)=1703, calldata=, logicalcalldata=, threadstatic=
[4748] INFO 11:10:06,791 ASP.Global_asax.Application_EndRequest() - threadid=2720, threadhash=1835, threadhash(now)=1703, calldata=
[4748] INFO 11:10:06,791 ASP.Global_asax.Application_EndRequest() - END /ConcurrentRequestsDemo/FastPage.aspx


The key bit is what happens when FastPage's Page_Load fires. The ThreadID is 4748, but the threadID I stored in HttpContext in the ctor is 2720. The hash code for the logical thread is 1703, but the one I stored in the ctor is 1835. All data I stored in the CallContext is gone (even that marked ILogicalThreadAffinative), but HttpContext is still there. As you'd expect, my ThreadStatic is gone too.

27 comments:

James Koch said...

Could you clarify if this was reproduced in .NET 1.1 or 2.0? Sorry if I missed it in the details.

I'm anxious to hear if anyone else has come up with any other details/alternatives. For a custom, portable "CurrentRequest.User" singleton I was also looking at Thread.CurrentPrincipal, but I shyed away due to SQL Server Windows auth and CAS issues. For simplicity's sake I'd rather developers know their code is running in a single configurable service account, with a business-domain-specific CurrentRequest.User context fully separated... I wonder if Thread.CurrentPrincipal has any similar issues w/ agile threads...

piers7 said...

Everything's 1.1 unless mentioned otherwise at the moment. Sorry I should have made that clearer.

You're right of course that this would apply to the Thread.CurrentPrincipal as well. From memory this gets setup at the end of OnAuthenticate, in which case it would have to be re-setup if the thread migrates in order not to get lost. I'll have another play...

The workaround we've been using is to store all per-user context in a UserContext object. This class just delegates storage either to Thread local storage, or to HttpContext.Items, depending on whether HttpContext.Current!=null. As a result this class insulates our business objects from having to know the mechanics of how per-user context is stored, and doesn't tie them to a web UI (so they can still participate in unit tests, even if there's no WinForms UI on the horizon).

piers7 said...

The Thread.CurrentPrincipal gets setup from the HttpContext.User in two different places: in HttpApplication.SetPrincipalOnThread, which fires after the authentication modules (if any) have done their work, and in HttpApplication.OnThreadEnter, which is what's called when a HttpApplication instance starts its execution, or resumes it's execution on a different thread.

So - as you'd expect - there's no issues with Thread.CurrentPrincipal being lost when you migrate to another thread, though if you've overwritten the Thread.CurrentPrincipal directly, rather than setting HttpContext.User then you're not going to get what you want.

I guess you could use the IPrincipal as your user-singleton if you wanted to, but it would tie your business objects to knowing about a particular IPrincipal (or at least a particular interface), so I'm not convinced its a great solution.

Erktjul said...

Assuming System.Web.HttpApplication.OnThreadEnter is called anytime an HttpApplication instance resumes its execution,

Then: OnThreadEnter of System.Web.HttpApplication calls SwitchContext(System.Web.HttpContext) of System.Web.HttpContextWrapper, which calls SwitchContext(System.Web.HttpContext) of System.Web.Hosting.ContextBase, which switches the HostContext of System.Runtime.Remoting.Messaging.CallContext.

By looking at HostContext getter and setter in CallContext, and since the HttpContext does not implement ILogicalThreadAffinative, we derive that the HostContext will be set via set_HostContext(...) of whatever is returned by Thread.CurrentThread.GetIllogicalCallContext().

GetIllogicalCallContext() returns a System.Runtime.Remoting.Messaging.IllogicalCallContext. Which stores its HostContext in a private m_HostContext field and NOT in its inner HashTable, where data accessed via SetData/GetData go. So what OnThreadEnter does is initialize the HostContext of a new IllogicalCallContext -- hence making sure that the HttpContext is available. But it does NOT copy the whole IllogicalCallContext, and therefore whatever has been set with SetData is lost.

Again: CallContext does not migrates, only CallContext.HostContext.

Now, I'm not quite sure what we can do with this. In an ASP.NET app, HostContext will be of (sealed) type HttpContext anyway so to access its Items collection we'd have to reference System.Web anyway. Unless we use reflection?

What's scary -- unless we're missing something obvious here -- is that CallContext is used by all sorts of services including transactions, log4net, some IoC frameworks...??

Ian said...

What's scary -- unless we're missing something obvious here -- is that CallContext is used by all sorts of services including transactions, log4net, some IoC frameworks...??

Yeah. That was my first thought as I read through all this. I've used CallContext as a persistence mechanism more times than I can remember based on a combination of "that's how HttpContext works" and "that's how all these IoC frameworks do it."

Any further thoughts on this? Obviously if this were a huge in-the-field problem we'd be seeing alternate approaches. Has anyone seen one?

Ian

piers7 said...

We're still using the UserContext approach I suggested above. This is just a class that delegates to CallContext or HttpContext depending on if HttpContext is available or not.

Putting this class in it's own assembly prevents the System.Web reference leakage.

Anonymous said...

I just got bit by this myself. I am using .NET 2.0. I have changed by code to use HttpContext. Fun fun.

Anonymous said...

Wow, what a pain. I am so glad I found this post though - I thought I was going crazy. Thank you.

Fernando said...

I am using threadstatic field to store my Connection object.

The idea is that all conection objects are the same in all threads but I need them to be thread static to make sure that the connection object is not use simultaneous for opening two datareaders for example.

So I don´t care in which thread a request is procesed since all thread have a smilar connection. What is important to me is that the same connection is not use to do two or more task at the same time.

Do you consider that this is a case where threaStatic is OK?

piers7 said...

]] Do you consider that this is a case where threaStatic is OK?

Maybe, but I'd probably still avoid it, for a couple of reasons:

Primarily, leaving your connection instances floating about like this is a resource drain. Expensive resources like this should be released as early as possible, not just left open for the duration. Ok, so it's only 20 connections, but imagine if everyone did that.

You've also got to be very careful about keeping the connection instances 'unsullied': avoid using SQL context, certain security schemes etc... I've seen this thread-connection-reuse scheme combine with a transaction leak to bring down an entire system (rather than just a certain request always bombing out, which would have been easier to diagnose).

And you've still got the issue that the connection instance you start with in the HTTPHandlers isn't neccesarily the one you use in your Page_Load etc... so woe betide you if you try and use a transaction that spans the two (though, for various reasons, this would be a bad thing to do anyway).

Anonymous said...

piers7, am trying to reproduce the same scenario (Thread switching), which is discussed. I am using Asp.net 2.0 (AS.net development server). From your log it is clear that the issue is reproducible. But i could not able to do that in 2.0? Can you please confirm that it is not the problem in 2.0. Also it would be helpful if you can post your code with which you able to reproduce.

Thanks.

weingrass@yahoo.com said...

Hi,

im working on a business framework shared between web and windows applications

im in need to store objects like facades and searched for thread based solutions coz these objects lifecycle is bound to the executing thread

these facades are created for the httprequest or windows lifetime and i ve no need to share them with other threads

i thought callcontext was for me the best way in my study case

doing a little search on the web bring me to ur post and i didnt find anything wrong about callcontext in ur post

is there in ur opinion anything wrong in using callcontext to do that ?

thx for ur reply

weingrass@yahoo.com said...

Hi Jeff

im working on a business framework shared between web and windows applications

im in need to store objects like facades and searched for thread based solutions coz these objects lifecycle is bound to the executing thread

these facades are created for the httprequest or windows lifetime and i ve no need to share them with other threads

i thought callcontext was for me the best way in my study case

doing a little search on the web bring me to ur post and i didnt find anything wrong about callcontext in ur post

is there in ur opinion anything wrong in using callcontext to do that ?

thx for ur reply

Mike Nelson said...

Thanks for your article, and solution which i will now use, really appreciate it and can't see much in the way of MS docs on this. I think it is stupid that asp.net even allows the "static" keyword - why would you ever need it? Really it should be configurable so you can change the meaning of your statics to what you need, with the default being per-request. The only reason i can think of for not doing this is too hard to implement, which is a crappy reason for making life difficult for everyone! This is a hangover from c++ single threaded pre web apps, and should have been brought into the 21st century by now.

Serhiy said...
This comment has been removed by the author.
Nimesh – Perception System said...

Nice Post
Informative and useful one
I am .Net Developer and I am looking for this
Thanks for the great stuff.

Anonymous said...
This comment has been removed by a blog administrator.
Anonymous said...

What do you do in case HttpContext.Current is null?

If you need the information stored in it, how do you gain access to it?

Regards

Ecommerce developer said...

I was also looking at Thread.CurrentPrincipal, but I shyed away due to SQL Server Windows auth and CAS issues. For simplicity's sake I'd rather developers know their code is running in a single configurable service account, with a business-domain-specific CurrentRequest.

oscommerce development said...

I guess you could use the IPrincipal as your user-singleton if you wanted to, but it would tie your business objects to knowing about a particular IPrincipal (or at least a particular interface), so I'm not convinced its a great solution.

Jim Toast said...

One alternative option for ThreadStatic type behavior that only lasts the length of a request is a follows:

- Create Custom IPrinciple and IIdentity Classes.
- Add a property to one of these that contains a Dictionary of objects
- Each request at the PostAuthenticateRequest stage, replace the existing Principle with the custom Principle (you can easily just wrap the existing one to retain the values)
(Set Thread.CurrentPrincipal AND the HttpContext.Current.User to the new principal object)

After this, you can access your dictionary in a non webspecific way by: ((CustomPrincipal)Thread.CurrentPrincipal).MyDictionary

It seems like a bit of work up front, but it really is not too bad, and performs well. The advantage of this method is that:
- The dictionary will only last the length of the ThreadContext (i.e. the length of a request). There is no need to manually clean at the end
- ASP.Net takes care of transferring the Principal to any new threads if the thread changes for a request
- .Net takes care of transferring a reference to the principal to all child threads, this is available from child threads also.

I have used this method successfully before. Once the initial pain of implementing this pattern, I really never had to think about much again - it just seems to work.

PHP Web Development said...

Good to read this. Well written post. Thanks a lot!

Nazish Ali Rizvi said...

Very informative and unleashed article Thanks and the all the comments.I am a newbie in this type of problem. kindly someone explain me How can I do it with .net framework 4 in WebApplication.I want to store my EntityFrameWork context I want to ensure it one context per request not share context..Any Help would be appreciable.. Thanks

中華民國虛擬總統 said...

I've conducted my own test using the same slow/fast page concept in .NET 4.0 but wasn't able to see any thread switching behavior. The only time I can make it happen it's when I mark the page as Async and use AddOnPreRenderCompleteAsync to add my own being/end request handler. In summary, thread switching happens in an async operation is perfectly understandable and the author needs to publish how he made it happen in a synchronous operation.

piers7 said...

中華民國虛擬總統: 7 years later, given you're almost certainally not using a single core PC, it's going to be hard to reproduce under the same level of trival load. Try ramping it up a bit (a lot). Also, the threshold for request queuing / handoff has probably been tuned a fair bit between ASP.Net 2 and 4.

JT said...

I'm looking at similar behaviour under .NET 4.5 / IIS Express 8.

If anyone is looking to try to reproduce it, make sure you're using IIS or IIS Express and not Cassini (the built in web server) because Cassini is extremely different.

For me, a thread context variable is set during BeginRequest, and the thread sometimes changes between there and the handling of the request. My options appear to be (a) don't do it, or (b) try to make sure it happens right before the request is processed so there's more chance that it will stay.

Dewy said...

Hi Piers,

Is this still true with 4.5 and using Set/GetLogicalData on the CallContext?

Popular Posts