Category Archives: uncertainty

Credibility is in the eye of the beholder

Last month I described how computational models were used as more than fables in many areas of applied science, including engineering and precision medicine [‘Models as fables’ on March 16th, 2016]. When people need to make decisions with socioeconomic and, or personal costs, based on the predictions from these models, then the models need to be credible. Credibility is like beauty, it is in the eye of the beholder. It is a challenging problem to convince decision-makers, who are often not expert in the technology or modelling techniques, that the predictions are reliable and accurate. After all, a model that is reliable and accurate but in which decision-makers have no confidence is almost useless. In my research we are interested in the credibility of computational mechanics models that are used to optimise the design of load-bearing structures, whether it is the frame of a building, the wing of an aircraft or a hip prosthesis. We have techniques that allow us to characterise maps of strain using feature vectors [see my post entitled ‘Recognising strain‘ on October 28th, 2015] and then to compare the ‘distances’ between the vectors representing the predictions and measurements. If the predicted map of strain is an perfect representation of the map measured in a physical prototype, then this ‘distance’ will be zero. Of course, this never happens because there is noise in the measured data and our models are never perfect because they contain simplifying assumptions that make the modelling viable. The difficult question is how much difference is acceptable between the predictions and measurements . The public expect certainty with respect to the performance of an engineering structure whereas engineers know that there is always some uncertainty – we can reduce it but that costs money. Money for more sophisticated models, for more computational resources to execute the models, and for more and better quality measurements.

Models as fables

3 Replies

In his book, ‘Economic Rules – Why economics works, when it fails and how to tell the difference‘, Dani Rodrik describes models as fables – short stories that revolve around a few principal characters who live in an unnamed generic place and whose behaviour and interaction produce an outcome that serves as a lesson of sorts. This seems to me to be a healthy perspective compared to the almost slavish belief in computational models that is common today in many quarters. However, in engineering and increasingly in precision medicine, we use computational models as reliable and detailed predictors of the performance of specific systems. Quantifying this reliability in a way that is useful to non-expert decision-makers is a current area of my research. This work originated in aerospace engineering where it is possible, though expensive, to acquire comprehensive and information-rich data from experiments and then to validate models by comparing their predictions to measurements. We have progressed to nuclear power engineering in which the extreme conditions and time-scales lead to sparse or incomplete data that make it more challenging to assess the reliability of computational models. Now, we are just starting to consider models in computational biology where the inherent variability of biological data and our inability to control the real world present even bigger challenges to establishing model reliability.

Sources:

Dani Rodrik, Economic Rules: Why economics works, when it fails and how to tell the difference, Oxford University Press, 2015

Patterson, E.A., Taylor, R.J. & Bankhead, M., A framework for an integrated nuclear digital environment, Progress in Nuclear Energy, 87:97-103, 2016

Hack, E., Lampeas, G. & Patterson, E.A., An evaluation of a protocol for the validation of computational solid mechanics models, J. Strain Analysis, 51(1):5-13, 2016.

Patterson, E.A., Challenges in experimental strain analysis: interfaces and temperature extremes, J. Strain Analysis, 50(5): 282-3, 2015

Patterson, E.A., On the credibility of engineering models and meta-models, J. Strain Analysis, 50(4):218-220, 2015

Unexpected bad news for turkeys

1 Reply

Risk is defined as the probability of something happening multiplied by consequences of its occurrence. Engineers expend considerable time and effort in reducing the consequences of the quite probable such as ensuring the passenger compartment of a car will remain largely intact during a crash. The installation of automatic breaking systems on some cars is also an attempt to reduce the probability of an impact, i.e. the occurrence of a crash. It is difficult to anticipate the very low probability event with catastrophic consequences and so cars are not designed to deal with meteors dropping out the sky or even elevated highways collapsing in an earthquake. These are what Nassim Taleb, labels Black Swan events but would perhaps be better named after another bird, the more humble turkey. An American turkey is fed daily by a friendly human and has no idea in November that Thanksgiving is about happen with fatal consequences, or substitute December and Christmas for a British turkey. From the turkey’s perspective everything is fine down on the farm until it isn’t; in other words, the very low probability event with catastrophic consequences occurs. From the farmer’s perspective, turkey for dinner at Thanksgiving is close to a certainty, i.e. a probability of one. The problem for engineers designing machines is to have the perspective of the farmer and not the turkey.

Happy Thanksgiving to my American readers!

Source:

Nassim Nicholas Taleb, The Black Swan – the impact of the highly probable, London: Penguin Books, 2008.

Greed overwhelms ethics

3 Replies

My car but not my house!

The scandal about Volkswagen emissions has already caused journalists and others to wring their hands or to preach sermons, or both, about the ethical standards of the engineering profession, see for example the Editor’s blog in Professional Engineer where he reminds us that professional engineers should conduct their professional work and relationships with “integrity and objectivity and with due regard for the welfare of the people, the organisations and the environment with which they interact”. The quotation is from the Royal Charter of the Institution of Mechanical Engineers. Fine words written to induce good intentions that rarely survive in the face of greed or simply the need to keep your job so that you can feed and house your family.

In my view, the emissions scandal seems to have parallels with the banking scandals of the past decade, in which corporate greed has trampled over ethics and morals in the pursuit of ever larger profits while government regulators through incompetence or acquiescence have allowed it to continue. Volkswagen were wrong to design a device to cheat the Government emission tests, but Government regulators were naive to design a test in which it was so easy to cheat. Senior executives at Volkswagen have blamed their employees following other recent examples. These company leaders were paid gigantic salaries to provide both leadership and management and, in my opinion, they have failed in both by blaming the people they are supposed to be leading and by allowing the scandals to happen in the first place.

The evidence would suggest that we can trust neither corporations nor governments to take care of the environment. One solution is for all vehicles to provide real-time information on the dashboard about NOX and Carbon emissions as well fuel consumption then we can make our own choices. When I moved to the US more than a decade ago, I was surprised to find that fuel consumption data was not available on the dashboard of most cars as it was already commonplace in Europe. Perhaps its presence was a factor in the development of fuel efficient cars in Europe although clearly higher fuel prices play a large role. However, in the absence of a tax on emissions, real-time emission data on the dashboard would motivate engineering ingenuity to compete to produce lower emission designs instead of wasting creativity on cheating in useless Government tests.

Here are some of facts to support my statements above about large profits and gigantic salaries:

Volkswagen profits rose 21% in 2014 to more than $12billion on an annual turnover of $230 billion which is comparable to the GDP of Portugal.

The CEO of Volkswagen was paid almost $18million per year which is about 250 times the average salary of a Volkswagen engineer.

Sources:

Hibbert, L., Editor’s comment: October 2015 The Volkswagen emissions testing scandal has put the issue of professional ethics in the spotlight, Professional Engineer, October 07, 2015.

Wall Street Journal, Feb 27th, 2015

Volkswagen AG Annual Report 2014

World Bank Data Bank

Reuters, March 12th, 2012

glassdoor.com