On Rail Franchises, Public Sector Pay and Modelling Failures

[danieldwilliam]

I’m going to talk a little bit about the rail franchise issue, public sector finance and public sector pay.

I don’t work for the Department for Transport and I’m not a hardcore modeller but I have spent a bit of time in the public sector finance teams and some time working with hardcore models. This is some educated guesswork.

I suspect that what’s happened with the rail franchises is a modelling failure.  When I use the word model I mean an excel spreadsheet which seeks to replicate the terms and conditions, prices and costs of a particular business or contract as a way on understanding how various decisions and events will affect the financial performance of that venture. The more complex the undertaking the larger and more complex the model. These are not simple spreadsheets.  They are about as far from back of the envelope as you can get.

The model I used to work with was for a power station.  The power station had four large contracts, each of which was over 100 pages long.  2 for power sales, 1 for gas purchases, 1 for maintenance of the gas turbines.  It also traded gas and power and carbon certificates in the open market and participated in a very technical power industry thing called the Balancing Mechanism. These all interacted with each other. As did the air and water temperature, rainfall, air pressure, water quality and a dozen or so other variables.

It even contained a model of what the contracts looked like from the other side, so we could guess what our counterparties would do.

Every business, contractual or operating decision was run through the model. It had to be able to answer any question asked of it.

The model ran to about 50 sheets on excel and weighed in at about 100 mb.

You put one number or one formula wrong in 1 cell out of several hundred thousand and your model will tell you lies.

I think the rail franchise models are going to be no less complex. The contracts involved are at least as big as a power station, perhaps an order of magnitude bigger.

In my experience working with models like this they are a full time job. You have one modeller working full time on one model.  If you are modelling one of these bad boys you live in it constantly.  It’s a very difficult job.  The skills and the attention to detail required are rare and hard to acquire. The introductory course in how to do it costs £2k. It takes months to build a model. It takes months to learn how to work with one.

Modellers at my old employer cost in the region of £70k a year, salary, pension, bonus and benefits.  A team of five of them were managed by a director level post who cost about £100,00. These are not cheap individuals to employ. However, compared to getting it wrong and costing your employer, say £40m, paying this money might seem cheap. In hindsight. (Which is somewhat defeats the point of a model.)

As I say, this sounds to me like a modelling failure. It sounds like either some duff information has been fed into the model, or the contracts involved haven’t been properly understood, or the  architecture of the model, the way it calculates different elements of the franchise agreements was built incorrectly. It could be that someone has just put a decimal point it the wrong place. These things happen. Everyone I know who has worked with models has made one howler and found someone else’s howler.

It may be the case that one error is to blame for the whole franchise problem. That the first model had an error buried deep in it and this was carried to each subsequent model and that the Department of Transport’s modelling team are entirely competent, just really, really unlucky.  Or it could be that the modelling team have been systematically under-staffed, with team members coming and going, staff not properly trained, or trying to handle too many models, with insufficient help, leadership and support. In which case, a howler was bound to happen, and probably had already and been ignored.

I don’t know for sure but I’m willing to bet that anyone who was good at modelling the rail franchise contracts for the Department of Transport was offered a modelling job somewhere else for more money. Probably, by one of the rail operators.

Here government are undertaking complex contractual arrangements which are difficult and therefore expensive to manage.  The public are rightly concerned that something has gone wrong.  I’d make this comment.

You can complain about the government’s ability to manage complex contracts and the costs of getting them wrong.  You can complain about public sector pay.  You can’t complain about both.

EDITED: to add

 

And it looks like the problem lies not in the modelling per se but in the assumptions going into the model.

 

http://www.bbc.co.uk/news/business-19816359

and a further Peston article on the capital requirements assumptions. via andrewducker
http://www.bbc.co.uk/news/business-19881240

Comments

[danieldwilliam.livejournal.com]

By “weight the risk” I think you mean taking the expected value of the payment e.g. there is a 10% chance that these guys just won’t pay, let’s only count 90% of the final year payments, rather than adjusting the payment expected in a net present value calculation for the time value of money.

The former would be pretty amatuer, the latter unforgivable.

My observation of the public sector is that seems to have an inbuilt refusal to assume that the other side will play dirty.

[f4f3.livejournal.com]

My first formal experience with risk management was through a Big American Firm, who bid on Big Federal Contracts. the assumption was that the government would sue you for every penny possible, and keep you to every promise you made. In return, you made sure you got every penny out of them that the contract would bear. It was a massively adversarial model, and I didn't enjoy cranking it much, but it does give me a good tool-set to use in the UK.

[danieldwilliam.livejournal.com]

Maybe the US and UK state sectors are different.

Maybe my observation is wrong, or just localised.

Massively adversarial relationships are not so great.

Part of the issue is that I think it is difficult for the state, with an eye to transperency to award a contract to a firm on the basis of a good working relationship in previous dealing.

I recall dad telling me about the contracting strategy at a large oil firm he may have worked for – if we see you trying to do a good job and something goes wrong, we’ll see you right, if you take the piss or try and screw us over YOU WILL NEVER WORK IN OIL AND GAS AGAIN.