On Rail Franchises, Public Sector Pay and Modelling Failures

[danieldwilliam]

I’m going to talk a little bit about the rail franchise issue, public sector finance and public sector pay.

I don’t work for the Department for Transport and I’m not a hardcore modeller but I have spent a bit of time in the public sector finance teams and some time working with hardcore models. This is some educated guesswork.

I suspect that what’s happened with the rail franchises is a modelling failure.  When I use the word model I mean an excel spreadsheet which seeks to replicate the terms and conditions, prices and costs of a particular business or contract as a way on understanding how various decisions and events will affect the financial performance of that venture. The more complex the undertaking the larger and more complex the model. These are not simple spreadsheets.  They are about as far from back of the envelope as you can get.

The model I used to work with was for a power station.  The power station had four large contracts, each of which was over 100 pages long.  2 for power sales, 1 for gas purchases, 1 for maintenance of the gas turbines.  It also traded gas and power and carbon certificates in the open market and participated in a very technical power industry thing called the Balancing Mechanism. These all interacted with each other. As did the air and water temperature, rainfall, air pressure, water quality and a dozen or so other variables.

It even contained a model of what the contracts looked like from the other side, so we could guess what our counterparties would do.

Every business, contractual or operating decision was run through the model. It had to be able to answer any question asked of it.

The model ran to about 50 sheets on excel and weighed in at about 100 mb.

You put one number or one formula wrong in 1 cell out of several hundred thousand and your model will tell you lies.

I think the rail franchise models are going to be no less complex. The contracts involved are at least as big as a power station, perhaps an order of magnitude bigger.

In my experience working with models like this they are a full time job. You have one modeller working full time on one model.  If you are modelling one of these bad boys you live in it constantly.  It’s a very difficult job.  The skills and the attention to detail required are rare and hard to acquire. The introductory course in how to do it costs £2k. It takes months to build a model. It takes months to learn how to work with one.

Modellers at my old employer cost in the region of £70k a year, salary, pension, bonus and benefits.  A team of five of them were managed by a director level post who cost about £100,00. These are not cheap individuals to employ. However, compared to getting it wrong and costing your employer, say £40m, paying this money might seem cheap. In hindsight. (Which is somewhat defeats the point of a model.)

As I say, this sounds to me like a modelling failure. It sounds like either some duff information has been fed into the model, or the contracts involved haven’t been properly understood, or the  architecture of the model, the way it calculates different elements of the franchise agreements was built incorrectly. It could be that someone has just put a decimal point it the wrong place. These things happen. Everyone I know who has worked with models has made one howler and found someone else’s howler.

It may be the case that one error is to blame for the whole franchise problem. That the first model had an error buried deep in it and this was carried to each subsequent model and that the Department of Transport’s modelling team are entirely competent, just really, really unlucky.  Or it could be that the modelling team have been systematically under-staffed, with team members coming and going, staff not properly trained, or trying to handle too many models, with insufficient help, leadership and support. In which case, a howler was bound to happen, and probably had already and been ignored.

I don’t know for sure but I’m willing to bet that anyone who was good at modelling the rail franchise contracts for the Department of Transport was offered a modelling job somewhere else for more money. Probably, by one of the rail operators.

Here government are undertaking complex contractual arrangements which are difficult and therefore expensive to manage.  The public are rightly concerned that something has gone wrong.  I’d make this comment.

You can complain about the government’s ability to manage complex contracts and the costs of getting them wrong.  You can complain about public sector pay.  You can’t complain about both.

EDITED: to add

 

And it looks like the problem lies not in the modelling per se but in the assumptions going into the model.

 

http://www.bbc.co.uk/news/business-19816359

and a further Peston article on the capital requirements assumptions. via andrewducker
http://www.bbc.co.uk/news/business-19881240

Comments

[f4f3.livejournal.com]

I've just realised that you were doing the same job as my sister does now (she is at ScottishPower in their Energy Trading department).
From what I've read this doesn't look like a decimal point error. It looks like a failure to weight the risk of projected income correctly when working out the total value of a bid. This meant that promises of what would be paid out towards the end of a 15 year deal were treated as being worth as much as early, less risky payments. From what I can see FirstStagecoach put in a bid heavily weighted on future payments - allowing them a top-line figure which was much higher than anyone else's bid. I suspect that they were clever enough to be gaming the model, based on past DoT awards, and their decision not to sue on failure to perform late into contracts (with franchises being allowed to "return the keys" without incurring full contractual penalties.
Whether this was a policy decision on their quantative risk analysis, to allow more reliance to be put on future promises than was sensible, or (as I suspect we'll be told) a cock-up in the weighting, we'll probably never know.

[widgetfox.livejournal.com]

Paul's post and my comment here.

[andrewducker]

Some more information on this:
http://www.bbc.co.uk/news/business-19881240

I'm finding it all rather fascinating (and also don't have time to write proper posts).