The term was coined by Ward Cunningham, and is used in agile circles, though it is essentially at odds to YAGNI. Martin Fowler explains it well.
As I see it, there are two major problems in dealing with Technical Debt:
- Inflexible project schedule. Unless you've got flexibility in your project planning (built in slack, or agile schedule) you've got no time to fix it. It is, after all, work you didn't plan for.
- Inability to articulate benefits. Sure, maybe we shouldn't have done it this way, but we have, so why change it if it means yet more time spent.
So why can't we measure the cost? Well if it's a simple quick-and-dirty that must be fixed for further progress to be made, then it's just the cost of the rework - the debt is repaid. But it's never that clean cut: the decision can have almost subliminal rippling effects on all the code that touches it, and the developers who have to deal with the compromised areas may not be aware that there was a better way available originally. So the debt possibly doesn't ever get 'repaid' as such, and the 'interest payments' such as they are are hidden within the cost of other work.
[Random link that illustrates the costs of unpaid debt]
Plus of course even if you're aware of the hit, unless you do the same activity twice - once paying the tax and once without - it's impossible to measure the true cost. If you are directly aware of the impact, that probably means the impact is severe.
More than often technical debt is actually dealt with just by members of the project team caring enough about their future sanity to just fix it. But that's not a scalable solution: it depends on key people essentially doing work out of hours just to stay on track. Even this isn't viable if the problems involve exceed the 'weekend refactor' threshold (the amount your top developer can get done in a big burst over the weekend). Over this level requires a concerted effort, or at least a staged one, which has to be planned for and so can't be fixed 'under the radar'.
I don't have any magic formula for dealing with TechnicalDebt. My strategy has always centered around avoidance rather than anything else, and being really bullish about having to fix things then and there. It's a far more scalable strategy, but I'm very aware that it's not possible on all projects. How you deal with the debt then I'm not sure. My experience has been that you don't, and that forward progress just becomes progressively more expensive. I guess you hope your project finishes before the debt becomes insurmountable.