Category Archives: FACTS

Deep uncertainty and meta-ignorance

4 Replies

Decorative imageThe term ‘unknown unknowns’ was made famous by Donald Rumsfeld almost 20 years ago when, as US Secretary of State for Defense, he used it in describing the lack of evidence about terrorist groups being supplied with weapons of mass destruction by the Iraqi government. However, the term was probably coined by almost 50 years earlier by Joseph Luft and Harrington Ingham when they developed the Johari window as a heuristic tool to help people to better understand their relationships.  In engineering, and other fields in which predictive models are important tools, it is used to describe situations about which there is deep uncertainty.  Deep uncertainty refers situations where experts do not know or cannot agree about what models to use, how to describe the uncertainties present, or how to interpret the outcomes from predictive models.  Rumsfeld talked about known knowns, known unknowns, and unknown unknowns; and an alternative simpler but perhaps less catchy classification is ‘The knowns, the unknown, and the unknowable‘ which was used by Diebold, Doherty and Herring as part of the title of their book on financial risk management.  David Spiegelhalter suggests ‘risk, uncertainty and ignorance’ before providing a more sophisticated classification: aleatory uncertainty, epistemic uncertainty and ontological uncertainty.  Aleatory uncertainty is the inevitable unpredictability of the future that can be fully described using probability.  Epistemic uncertainty is a lack of knowledge about the structure and parameters of models used to predict the future.  While ontological uncertainty is a complete lack of knowledge and understanding about the entire modelling process, i.e. deep uncertainty.  When it is not recognised that ontological uncertainty is present then we have meta-ignorance which means failing to even consider the possibility of being wrong.  For a number of years, part of my research effort has been focussed on predictive models that are unprincipled and untestable; in other words, they are not built on widely-accepted principles or scientific laws and it is not feasible to conduct physical tests to acquire data to demonstrate their validity [see editorial ‘On the credibility of engineering models and meta-models‘, JSA 50(4):2015].  Some people would say untestability implies a model is not scientific based on Popper’s statement about scientific method requiring a theory to be refutable.  However, in reality unprincipled and untestable models are encountered in a range of fields, including space engineering, fusion energy and toxicology.  We have developed a set of credibility factors that are designed as a heuristic tool to allow the relevance of such models and their predictions to be evaluated systematically [see ‘Credible predictions for regulatory decision-making‘ on December 9th, 2020].  One outcome is to allow experts to agree on their disagreements and ignorance, i.e., to define the extent of our ontological uncertainty, which is an important step towards making rational decisions about the future when there is deep uncertainty.

References

Diebold FX, Doherty NA, Herring RJ, eds. The Known, the Unknown, and the Unknowable in Financial Risk Management: Measurement and Theory Advancing Practice. Princeton, NJ: Princeton University Press, 2010.

Spiegelhalter D,  Risk and uncertainty communication. Annual Review of Statistics and Its Application, 4, pp.31-60, 2017.

Patterson EA, Whelan MP. On the validation of variable fidelity multi-physics simulations. J. Sound and Vibration. 448:247-58, 2019.

Patterson EA, Whelan MP, Worth AP. The role of validation in establishing the scientific credibility of predictive toxicology approaches intended for regulatory application. Computational Toxicology. 100144, 2020.

Credible predictions for regulatory decision-making

3 Replies

detail from abstract by Zahrah ReshRegulators are charged with ensuring that manufactured products, from aircraft and nuclear power stations to cosmetics and vaccines, are safe.  The general public seeks certainty that these devices and the materials and chemicals they are made from will not harm them or the environment.  Technologists that design and manufacture these products know that absolute certainty is unattainable and near-certainty in unaffordable.  Hence, they attempt to deliver the service or product that society desires while ensuring that the risks are As Low As Reasonably Practical (ALARP).  The role of regulators is to independently assess the risks, make a judgment on their acceptability and thus decide whether the operation of a power station or distribution of a vaccine can go ahead.  These are difficult decisions with huge potential consequences – just think of the more than three hundred people killed in the two crashes of Boeing 737 Max airplanes or the 10,000 or so people affected by birth defects caused by the drug thalidomide.  Evidence presented to support applications for regulatory approval is largely based on physical tests, for example fatigue tests on an aircraft structure or toxicological tests using animals.  In some cases the physical tests might not be entirely representative of the real-life situation which can make it difficult to make decisions using the data, for instance a ground test on an airplane is not the same as a flight test and in many respects the animals used in toxicity testing are physiologically different to humans.  In addition, physical tests are expensive and time-consuming which both drives up the costs of seeking regulatory approval and slows down the translation of new innovative products to the market.  The almost ubiquitous use of computer-based simulations to support the research, development and design of manufactured products inevitably leads to their use in supporting regulatory applications.  This creates challenges for regulators who must judge the trustworthiness of predictions from these simulations.  [see ‘Fake facts & untrustworthy predictions‘ on December 4th, 2019]. It is standard practice for modellers to demonstrate the validity of their models; however, validation does not automatically lead to acceptance of predictions by decision-makers.  Acceptance is more closely related to scientific credibility.  I have been working across a number of disciplines on the scientific credibility of models including in engineering where multi-physics phenomena are important, such as hypersonic flight and fusion energy [see ‘Thought leadership in fusion energy‘ on October 9th, 2019], and in computational biology and toxicology [see ‘Hierarchical modelling in engineering and biology‘ on March 14th, 2018]. Working together with my collaborators in these disciplines, we have developed a common set of factors which underpin scientific credibility that are based on principles drawn from the literature on the philosophy of science and are designed to be both discipline-independent and method-agnostic [Patterson & Whelan, 2019; Patterson et al, 2021]. We hope that our cross-disciplinary approach will break down the subject-silos that have become established as different scientific communities have developed their own frameworks for validating models.  As mentioned above, the process of validation tends to be undertaken by model developers and, in some sense, belongs to them; whereas, credibility is not exclusive to the developer but is a trust that needs to be shared with a decision-maker who seeks to use the predictions to inform their decision [see ‘Credibility is in the eye of the beholder‘ on April 20th, 2016].  Trust requires a common knowledge base and understanding that is usually built through interactions.  We hope the credibility factors will provide a framework for these interactions as well as a structure for building a portfolio of evidence that demonstrates the reliability of a model. 

References:

Patterson EA & Whelan MP, On the validation of variable fidelity multi-physics simulations, J. Sound & Vibration, 448:247-258, 2019.

Patterson EA, Whelan MP & Worth A, The role of validation in establishing the scientific credibility of predictive toxicology approaches intended for regulatory application, Computational Toxicology, 17: 100144, 2021.

Image: Extract from abstract by Zahrah Resh.

35 years later and still working on a PhD thesis

3 Replies

It is about 35 years since I graduated with my PhD.  It was not ground-breaking although, together with my supervisor, I did publish about half a dozen technical papers based on it and some of those papers are still being cited, including one this month which surprises me.  I performed experiments and computer modelling on the load and stress distribution in threaded fasteners, or nuts and bolts.  There were no digital cameras and no computer tomography; so, the experiments involved making and sectioning models of nuts and bolts in transparent plastic using three-dimensional photoelasticity [see ‘Art and Experimental Mechanics‘ on July 17th, 2012].  I took hundreds of photographs of the sections and scanned the negatives in a microdensitometer.  The computer modelling was equally slow and laborious because there were no graphical user interfaces (GUI); instead, I had to type strings of numbers into a terminal, wait overnight while the calculations were performed, and then study reams of numbers printed out on long rolls of paper.  The tedium of the experimental work inspired me to work on utilising digital technology to revolutionise the field of experimental mechanics over the following 15 to 20 years.  In the past 15 to 20 years, I have moved back towards computer modelling and focused on transforming the way in which measurement data are used to improve the fidelity of computer models and to establish confidence in their predictions [see ‘Establishing fidelity and credibility in tests and simulations‘ on July 25th, 2018].  Since completing my PhD, I have supervised 32 students to successful completion of their PhDs.  You might think that was a straightforward process of an initial three years for the first one to complete their research and write their thesis, followed by one graduating every year.  But that is not how it worked out, instead I have had fallow years as well as productive years.  At the moment, I am in a productive period, having graduated two PhD students per year since 2017 – that’s a lot of reading and I have spent much of the last two weekends reviewing a thesis which is why PhD theses are the topic of this post!

Footnote: the most cited paper from my thesis is ‘Kenny B, Patterson EA. Load and stress distribution in screw threads. Experimental Mechanics. 1985 Sep 1;25(3):208-13‘ and this month it was cited by ‘Zhang D, Wang G, Huang F, Zhang K. Load-transferring mechanism and calculation theory along engaged threads of high-strength bolts under axial tension. Journal of Constructional Steel Research. 2020 Sep 1;172:106153‘.

The blind leading the blind

3 Replies

Three years after it started, the MOTIVATE project has come to an end [see ‘Getting smarter’ on June 21st, 2017].  The focus of the project has been about improving the quality of validation for predictions of structural behaviour in aircraft using fewer, better physical tests.  We have developed an enhanced flowchart for model validation [see ‘Spontaneously MOTIVATEd’ on June 27th, 2018], a method for quantifying uncertainty in measurements of deformation in an industrial environment [see ‘Industrial uncertainty’ on December 12th, 2018] and a toolbox for quantifying the extent to which predictions from computational models represent measurements made in the real-world [see ‘Alleviating industrial uncertainty’ on May 13th, 2020].  In the last phase of the project, we demonstrated all of these innovations on the fuselage nose section of an aircraft.  The region of interest was the fuselage skin behind the cockpit window for which the out-of-plane displacements resulting from an internal pressurisation load were predicted using a finite element model [see ‘Did cubism inspire engineering analysis?’ on January 25th, 2017].  The computational model was provided by Airbus and is shown on the left in the top graphic with the predictions for the region of interest on the right.  We used a stereoscopic imaging system  to record images of a speckle pattern on the fuselage before and after pressurization; and from these images, we evaluated the out-of-plane displacements using digital image correlation (DIC) [see ‘256 shades of grey‘ on January 22, 2014 for a brief explanation of DIC].  The bottom graphic shows the measurements being made with assistance from an Airbus contractor, Strain Solutions Limited.  We compared the predictions quantitatively against the measurements in a double-blind process which meant that the modellers and experimenters had no access to one another’s results.  The predictions were made by one MOTIVATE partner, Athena Research Centre; the measurements were made by another partner, Dantec Dynamics GmbH supported by Strain Solutions Limited; and the quantitative comparison was made by the project coordinator, the University of Liverpool.  We found that the level of agreement between the predictions and measurements changed with the level of pressurisation; however, the main outcome was the demonstration that it was possible to perform a double-blind validation process to quantify the extent to which the predictions represented the real-world behaviour for a full-scale aerospace structure.

The content of this post is taken from a paper that was to be given at a conference later this summer; however, the conference has been postponed due to the pandemic.  The details of the paper are: Patterson EA, Diamantakos I, Dvurecenska K, Greene RJ, Hack E, Lampeas G, Lomnitz M & Siebert T, Application of a model validation protocol to an aircraft cockpit panel, submitted to the International Conference on Advances in Experimental Mechanics to be held in Oxford in September 2021.  I would like to thank the authors for permission to write about the results in this post and Linden Harris of Airbus SAS for enabling the study and to him and Eszter Szigeti for providing technical advice.

For more on the validation flowchart see: Hack E, Burguete R, Dvurecenska K, Lampeas G, Patterson E, Siebert T & Szigeti, Steps towards industrial validation experiments, In Multidisciplinary Digital Publishing Institute Proceedings (Vol. 2, No. 8, p. 391) https://www.mdpi.com/2504-3900/2/8/391

For more posts on the MOTIVATE project: https://realizeengineering.blog/category/myresearch/motivate-project/

The MOTIVATE project has received funding from the Clean Sky 2 Joint Undertaking under the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 754660 and the Swiss State Secretariat for Education, Research and Innovation under contract number 17.00064.

The opinions expressed in this blog post reflect only the author’s view and the Clean Sky 2 Joint Undertaking is not responsible for any use that may be made of the information it contains.