On Rail Franchises, Public Sector Pay and Modelling Failures

[danieldwilliam]

I’m going to talk a little bit about the rail franchise issue, public sector finance and public sector pay.

I don’t work for the Department for Transport and I’m not a hardcore modeller but I have spent a bit of time in the public sector finance teams and some time working with hardcore models. This is some educated guesswork.

I suspect that what’s happened with the rail franchises is a modelling failure.  When I use the word model I mean an excel spreadsheet which seeks to replicate the terms and conditions, prices and costs of a particular business or contract as a way on understanding how various decisions and events will affect the financial performance of that venture. The more complex the undertaking the larger and more complex the model. These are not simple spreadsheets.  They are about as far from back of the envelope as you can get.

The model I used to work with was for a power station.  The power station had four large contracts, each of which was over 100 pages long.  2 for power sales, 1 for gas purchases, 1 for maintenance of the gas turbines.  It also traded gas and power and carbon certificates in the open market and participated in a very technical power industry thing called the Balancing Mechanism. These all interacted with each other. As did the air and water temperature, rainfall, air pressure, water quality and a dozen or so other variables.

It even contained a model of what the contracts looked like from the other side, so we could guess what our counterparties would do.

Every business, contractual or operating decision was run through the model. It had to be able to answer any question asked of it.

The model ran to about 50 sheets on excel and weighed in at about 100 mb.

You put one number or one formula wrong in 1 cell out of several hundred thousand and your model will tell you lies.

I think the rail franchise models are going to be no less complex. The contracts involved are at least as big as a power station, perhaps an order of magnitude bigger.

In my experience working with models like this they are a full time job. You have one modeller working full time on one model.  If you are modelling one of these bad boys you live in it constantly.  It’s a very difficult job.  The skills and the attention to detail required are rare and hard to acquire. The introductory course in how to do it costs £2k. It takes months to build a model. It takes months to learn how to work with one.

Modellers at my old employer cost in the region of £70k a year, salary, pension, bonus and benefits.  A team of five of them were managed by a director level post who cost about £100,00. These are not cheap individuals to employ. However, compared to getting it wrong and costing your employer, say £40m, paying this money might seem cheap. In hindsight. (Which is somewhat defeats the point of a model.)

As I say, this sounds to me like a modelling failure. It sounds like either some duff information has been fed into the model, or the contracts involved haven’t been properly understood, or the  architecture of the model, the way it calculates different elements of the franchise agreements was built incorrectly. It could be that someone has just put a decimal point it the wrong place. These things happen. Everyone I know who has worked with models has made one howler and found someone else’s howler.

It may be the case that one error is to blame for the whole franchise problem. That the first model had an error buried deep in it and this was carried to each subsequent model and that the Department of Transport’s modelling team are entirely competent, just really, really unlucky.  Or it could be that the modelling team have been systematically under-staffed, with team members coming and going, staff not properly trained, or trying to handle too many models, with insufficient help, leadership and support. In which case, a howler was bound to happen, and probably had already and been ignored.

I don’t know for sure but I’m willing to bet that anyone who was good at modelling the rail franchise contracts for the Department of Transport was offered a modelling job somewhere else for more money. Probably, by one of the rail operators.

Here government are undertaking complex contractual arrangements which are difficult and therefore expensive to manage.  The public are rightly concerned that something has gone wrong.  I’d make this comment.

You can complain about the government’s ability to manage complex contracts and the costs of getting them wrong.  You can complain about public sector pay.  You can’t complain about both.

EDITED: to add

 

And it looks like the problem lies not in the modelling per se but in the assumptions going into the model.

 

http://www.bbc.co.uk/news/business-19816359

and a further Peston article on the capital requirements assumptions. via andrewducker
http://www.bbc.co.uk/news/business-19881240

Comments

[f4f3.livejournal.com]

I've just realised that you were doing the same job as my sister does now (she is at ScottishPower in their Energy Trading department).
From what I've read this doesn't look like a decimal point error. It looks like a failure to weight the risk of projected income correctly when working out the total value of a bid. This meant that promises of what would be paid out towards the end of a 15 year deal were treated as being worth as much as early, less risky payments. From what I can see FirstStagecoach put in a bid heavily weighted on future payments - allowing them a top-line figure which was much higher than anyone else's bid. I suspect that they were clever enough to be gaming the model, based on past DoT awards, and their decision not to sue on failure to perform late into contracts (with franchises being allowed to "return the keys" without incurring full contractual penalties.
Whether this was a policy decision on their quantative risk analysis, to allow more reliance to be put on future promises than was sensible, or (as I suspect we'll be told) a cock-up in the weighting, we'll probably never know.

[danieldwilliam.livejournal.com]

By “weight the risk” I think you mean taking the expected value of the payment e.g. there is a 10% chance that these guys just won’t pay, let’s only count 90% of the final year payments, rather than adjusting the payment expected in a net present value calculation for the time value of money.

The former would be pretty amatuer, the latter unforgivable.

My observation of the public sector is that seems to have an inbuilt refusal to assume that the other side will play dirty.

[f4f3.livejournal.com]

My first formal experience with risk management was through a Big American Firm, who bid on Big Federal Contracts. the assumption was that the government would sue you for every penny possible, and keep you to every promise you made. In return, you made sure you got every penny out of them that the contract would bear. It was a massively adversarial model, and I didn't enjoy cranking it much, but it does give me a good tool-set to use in the UK.

[danieldwilliam.livejournal.com]

Maybe the US and UK state sectors are different.

Maybe my observation is wrong, or just localised.

Massively adversarial relationships are not so great.

Part of the issue is that I think it is difficult for the state, with an eye to transperency to award a contract to a firm on the basis of a good working relationship in previous dealing.

I recall dad telling me about the contracting strategy at a large oil firm he may have worked for – if we see you trying to do a good job and something goes wrong, we’ll see you right, if you take the piss or try and screw us over YOU WILL NEVER WORK IN OIL AND GAS AGAIN.

[danieldwilliam.livejournal.com]

I've just realised that you were doing the same job as my sister does now (she is at ScottishPower in their Energy Trading department).


Could be, the structure at my former employer was a bit a different from SP’s and staffed with significantly fewer people, so rather broader jobs.

[f4f3.livejournal.com]

It was the energy trading part which rang a bell for me.

[danieldwilliam.livejournal.com]

Energy trading is a fascinating business.

[widgetfox.livejournal.com]

Paul's post and my comment here.

[danieldwilliam.livejournal.com]

Thanks for the blogpimpage and also thanks for the link to Paul’s post. Very interesting.

I’d overlooked the hiring in specialists angle. Generally I’m in favour of not having rarely, specialist, expensive skills on the books unless they are a core part of your business or you are a very large organisation and will be using them all the time.

I am left wondering if sometimes the government is not really sure what its core business actually is.

[widgetfox.livejournal.com]

What do you think its core business is?

[danieldwilliam.livejournal.com]

That is the question of the age.

A longer answer will follow shortly.

In the particular case of the rail franchise I think the business they were actually in was active counter-party for a series of geographically spread rail operations, rather than the business I think they thought they were in, which was the occassional letter of rail franchises and receiver of rents.

If I may draw an analogy, they thought they were letting a number of small flats on long leases, what they were actually doing was running a small hotel.

[widgetfox.livejournal.com]

I take the point of your analogy, but I question it, because in my world the point of running a hotel is to turn a profit and the point of running a rail service is to help people to get to where they need to go.

[danieldwilliam.livejournal.com]

One might argue that the point of running a hotel was to provide people with short-term short-notice accommodation and the point of running a railway was to make money.

[widgetfox.livejournal.com]

One might. But I wouldn't, in either case.

[danieldwilliam.livejournal.com]

And that’s a political position. Which for the record I don’t wholly disagree with. In that my politics suggests that running a hotel should be a) more about making money therefore b) a private sector activity and running a railway might be a) less about making money per se and therefore b) might sit better in the public sector.

The boundary between public and private enterprise is pretty ragged. Unless the state is going to go down a wholly vertically integrated nationalised model, at some point it is going to bump up against some interaction with the private sector. In which case the state then becomes in the business of acting as a counter-party for a private business. If the railways were wholly owned and operated by that the state and they were only buying manufactured equipment from the private sector this might be fairly straightforward. However, the state have gotten themselves involved in some form of revenue sharing arrangement here. Having gotten themselves involved in the business of private sector provision of railways they appear not to have realised the implications.

[danieldwilliam.livejournal.com]

These are, to me, not either or but very much both and.

In that a hotel makes money by being good at providing short-term short-notice accomodation. A railway makes money by being good at moving people around.

Or try swapping out good for better than someone else.

[widgetfox.livejournal.com]

I've just reread your comment and you were talking not about their purpose (as I'd initially read) but "what business they're in" so on reflection I think you're right and I'm wrong.

[danieldwilliam.livejournal.com]

Very gracious.

I think if one is going to be in *business* one should remember one is in business. As to whether the purposes of government are best served by being in business or not is a moot point.

There are a couple of posts bubbling around for me on this but I don’t have time to write them properly in October.

[widgetfox.livejournal.com]

Yes, there could be several meanings of this, I think. I think it's possible to be "in the business of" something without actually being "in business" but it might be that in practice this distinction is too difficult to maintain.

[danieldwilliam.livejournal.com]

I think there is actually a useful distinction there.

Thinking back to my days as FD for an archaeology unit, we were certainly in the business of archaeology. We did archaeology. Money was an issue, both in that we needed money to do archaeology and that we often did archaeology in order to earn money. So we had to mindful of the financial implications and we also had to be mindful that we were dealing with people who were financially motivated in their transactions with us. There were deals to be done.

We arguably weren’t in business doing archaeology, the money wasn’t a significant motivating factor. We wouldn’t move into say, mine clearing, where our skill set might be useful to boost profits.

I would contrast this with, say, the bulk of the NHS, where they have a budget to be mindful of but most of the work doesn’t bump up against counter-parties and so on.

[andrewducker]

Some more information on this:
http://www.bbc.co.uk/news/business-19881240

I'm finding it all rather fascinating (and also don't have time to write proper posts).