Tag Archives: forecasting

Do you know RIO?

Infrared image of group of people in meetingDuring the pandemic many political leaders have been heard to justify their decisions by telling us that they were following advice from scientists.  I think it was Thomas Kuhn who proposed that the views of a group of scientists will be normally distributed if the group is large enough, i.e., a bell-shaped curve with a few scientists providing outlying opinions on either end and the majority in the middle of the distribution [see ‘Uncertainty about Bayesian methods’ on June 7th, 2017].  So, it depends which scientist you consult as to what advice you will receive.  Of course, you can consult a group of experts in order to identify the full range of advice and seek a consensus; however, this is notoriously difficult because some voices will be louder than others and some experts will be very certain about their predictions of the future while others will be very cautious about predicting anything.  This is often because the former group are suffering from meta-ignorance, i.e., failing to even consider the possibility of being wrong, while the latter are so aware of the ontological or deep uncertainties that they prefer to surround their statements with caveats that render them difficult or impossible to interpret or employ in decision-making [see ‘Deep uncertainty and meta ignorance’ on July 21st 2021].  Politicians prefer a simple message that they can explain to the media and tend to listen to the clear but usually inaccurate message from the confident forecasters [see ‘Forecasts and chimpanzees throwing darts’ on September 2nd, 2020].  However, with time and effort, it is possible to make rational decisions based on expert opinion even when the opinions appear to diverge.  There are several recognised protocols for expert elicitation which are used in a wide range of engineering and scientific activities to support decision-making in the absence of comprehensive information.  I frequently use a form of the Sheffield protocol developed originally to elicit a probability distribution for an unknown uncertainty from a group of experts.  Initially, the group of experts are asked individually to provide private, written, independent advice on the issue of concern.  Subsequently, their advice is shared with the group and a discussion to reach a consensus is led by a facilitator. This can be difficult if the initial advice is divergent and individuals hold strong views.  This is when RIO can help.  RIO stands for Rational Impartial Observer and an expert group often rapidly reach a consensus when they are asked to consider what RIO might reasonably believe after reading their independent advice and listening to their discussion.


Anthony O’Hagan, Expert knowledge elicitation: subjective but scientific, The American Statistician, 73:Sup.1, 69-81, 2019.

Certainty is unattainable and near-certainty unaffordable

The economists John Kay and Mervyn King assert in their book ‘Radical Uncertainty – decision-making beyond numbers‘ that ‘economic forecasting is necessarily harder than weather forecasting’ because the world of economics is non-stationary whereas the weather is governed by unchanging laws of nature. Kay and King observe that both central banks and meteorological offices have ‘to convey inescapable uncertainty to people who crave unavailable certainty’. In other words, the necessary assumptions and idealisations combined with the inaccuracies of the input data of both economic and meteorological models produce inevitable uncertainty in the predictions. However, people seeking to make decisions based on the predictions want certainty because it is very difficult to make choices when faced with uncertainty – it raises our psychological entropy [see ‘Psychological entropy increased by ineffective leaders‘ on February 10th, 2021].  Engineers face similar difficulties providing systems with inescapable uncertainties to people desiring unavailable certainty in terms of the reliability.  The second law of thermodynamics ensures that perfection is unattainable [see ‘Impossible perfection‘ on June 5th, 2013] and there will always be flaws of some description present in a system [see ‘Scattering electrons reveal dislocations in material structure‘ on November 11th, 2020].  Of course, we can expend more resources to eliminate flaws and increase the reliability of a system but the second law will always limit our success. Consequently, to finish where I started with a quote from Kay and King, ‘certainty is unattainable and the price of near-certainty unaffordable’ in both economics and engineering.