The overconfidence trap

Plus - why we should give installers free heat pumps

I’ve wanted to write some sort of newsletter for ages. As a CEO, it’s often harder to make time for personal writing, and it’s exposing to share half baked ideas. But the discipline of writing accelerates thinking, even more so if you get robust feedback. So here we are.

Every two weeks, I’ll do two things: I’ll share a policy idea, and I’ll share advice on how to innovate, drawing on the experiments, methods and tools we use at Nesta and BIT.

I enjoy feedback, even if it's a spirited disagreement - so please do leave a comment, share or you can find me on Bluesky, X, LinkedIn (or pop me an email at ravi.gurumurthy@nesta.org.uk).

The overconfidence trap

If you’re watching Traitors, you’ll have seen the dangers of overconfidence, and how the loudest voices often prevail over the wise.

Now you might assume the participants are just not that smart. After all, the well-known Dunning-Kruger effect (or the Truss-Kwarteng effect as I call it), suggests overconfidence particularly afflicts those with limited skills and expertise.

But recent research from my brilliant colleague Mark Egan at BIT shows that overconfidence is a much more widespread problem. We are prone to making bad decisions and judgement calls because we tend to rate our own abilities more highly than is warranted.

Much of the previous research in this area has been done through lab studies with university students, largely in North America. Real world studies have concentrated more on showing the limitations of experts, rather than looking at the general population.

So we decided to run what we call a ‘calibration quiz’ with 2000 members of the British public. We asked them to rate whether 30 statements were true or false. We then also asked them to rate how confident they were that they were right, ranging from 50% (no idea) to 100% confidence.

The findings are intriguing (covered in the Guardian here). 81% of people were over-confident - they believed they knew the answers significantly more often than they actually did. 3% were under confident. 16% were ‘well calibrated’. There were no differences by gender, but substantial variation by age. Those aged over 55 were far more likely to be overconfident than those aged 18 to 24 - and the difference between them was more than twice the difference between the most and least-educated respondents.

The good news is that you can improve how well calibrated you are. You can start by testing your own calibration here to work out whether you are under- or over- confident. You’ll get your results instantly, and I'll share our collective results in a future newsletter. Think of this as basic cognitive hygiene – just as you expect to get check-ups for your teeth or eyes, regular calibration tests can monitor the accuracy of your judgments.

Once you know where you stand, there are all sorts of ways of improving your calibration (this is true for both individual and collective decision making). For example, training to debias decision-making has been shown to increase the accuracy of forecasters.

Here are some of the things I try to apply:

• try to keep score of whether you are routinely overconfident. I used to have such optimism bias when starting projects, that I’d double the time and cost of my original estimates

• make space for systematic challenge from diverse voices. We use pre-mortems and red teams quite often. We imagine why an idea has died, and then work out what caused the failure. (If I was a minister or permanent secretary, I’d want red-teaming from people implementing a policy)

• this might sound rather basic, but conduct evidence reviews. We start with a theory of change, look at the evidence, and then set out our levels of confidence based on the quality of evidence.

• roughly model solutions, forcing the conversation to be had with numbers rather than adjectives, so you get a sense of the scale of change.

• tap into the collective wisdom of your staff and stakeholders as forecasters. We once turned our ‘risk register’ into a forecasting process where staff could update their predictions and we could see which individuals or groups were better forecasters.

Finally, think about the diversity of teams not just in terms of viewpoint but also calibration levels. While we might individually strive to be better calibrated, overly confident people can helpfully stretch the boundaries, while those who are naturally more sceptical can bring critical challenge.

Want more heat pumps? Start with installers

Ed Miliband is causing problems in my marriage. Seriously.

His removal of the ‘1 metre rule’ has taken away my last possible excuse for not getting a heat pump. How can I credibly lead an organisation that has net zero homes as one of its three missions while still owning a boiler?

My wife is unconvinced. We have a badly installed heating system, but, so far plumbers and boiler engineers we’ve talked to have given us a litany of reasons why a heat pump is a bad idea: it won’t work in the middle of winter so you might need a hybrid; you’ll spend a fortune replacing every radiator and getting extra insulation; they’ll be putting hydrogen through our pipes in future so why bother changing…

So how do I convince my wife? I started by listing out some basic facts. In the biggest ever survey of heat pump owners, they were just as satisfied as owners of gas boilers. Installations of heat pumps by Heat Geek are now getting an average energy efficiency of 4.3 on their installs - that means for every one unit of electricity they use, they generate over 4 units of energy in the form of heat. (I think this is the point boiler chat was banned from the table).

I tried to pull in the big guns, and said that Ed Miliband himself has a heat pump, and Greg Jackson reckons that with the Octopus agile tariff, a heat pump is likely to be cheaper overall. But to no avail. In desperation, I’m planning on a surprise Valentine’s day trip to visit a heat pump so she can see it in action and talk to the owner.

The serious point amid my domestic travails is that the messenger really matters. I know more than most about heat pumps but I’m just not as credible a promoter as someone who does home heating for a living - nor are politicians and energy company bosses. We need the plumbers and boiler installers who come into people’s houses every day to become advocates of heat pumps. To do that we need them to really trust the technology.

The government is currently offering £7500 to anyone who gets a heat pump. So here’s an additional suggestion: why not offer the 150,000 people who are on the ‘Gas Safe Register’ (e.g. boiler installers) a free heat pump. Let’s make it a financial no-brainer for the people with perhaps the biggest influence on people’s heating choices to go green. If they own a heat pump themselves - assuming they perform well (the evidence says they do), we’ll have 150,000 heat pump owners leading the government’s word-of-mouth campaign. If they become installers themselves, then they’ll also be much better at selling heat pumps to customers as they can talk credibly about their own experience.

Nesta is now prototyping the idea at a small scale. In partnership with the Scottish and Northern Ireland Plumbing Employers’ Federation (SNIPEF), we are giving around 20 experienced plumbing and heating professionals, who have just undertaken heat pump training, a heat pump to install in their own home.

I’ll keep you posted on what we learn from this, but I think the idea has a lot of potential. Although it might seem a bit leftfield, I think we are much more likely to shift social norms about desirable heating sources by targeting installers.

There’s also a general lesson for policymakers about shaping behaviours. When behavioural science became popular 15 years ago, it was heavily associated with quite an individualistic approach - sending text message reminders, demonstrating social proof, adjusting the choice architecture. That works well for many behavioural issues, particularly when what matters is keeping an issue salient or tackling inertia. But when dealing with trickier problems, we need to capitalise on the social networks and group dynamics that shape norms. I’ll share further posts about experiments we’re doing that mine that seam.

What I’m reading / listening to / thinking about

The New India by Rahul Bhattia (reviewed here). My wife got me this. She’s much more interested in India than low carbon heating solutions.

Nesta Signals: Robots behaving badly - why manners matter in a world designed for humans. Watching my kids interact with the Amazon Echo for kids that they got for Christmas has got me thinking about human-robot etiquette.

Hew Locke’s What Have We Here exhibition at the British Museum. I got the pleasure of speaking to Hew about his work and his exhibition has really stuck with me.