Regulators are charged with ensuring that manufactured products, from aircraft and nuclear power stations to cosmetics and vaccines, are safe. The general public seeks certainty that these devices and the materials and chemicals they are made from will not harm them or the environment. Technologists that design and manufacture these products know that absolute certainty is unattainable and near-certainty in unaffordable. Hence, they attempt to deliver the service or product that society desires while ensuring that the risks are As Low As Reasonably Practical (ALARP). The role of regulators is to independently assess the risks, make a judgment on their acceptability and thus decide whether the operation of a power station or distribution of a vaccine can go ahead. These are difficult decisions with huge potential consequences – just think of the more than three hundred people killed in the two crashes of Boeing 737 Max airplanes or the 10,000 or so people affected by birth defects caused by the drug thalidomide. Evidence presented to support applications for regulatory approval is largely based on physical tests, for example fatigue tests on an aircraft structure or toxicological tests using animals. In some cases the physical tests might not be entirely representative of the real-life situation which can make it difficult to make decisions using the data, for instance a ground test on an airplane is not the same as a flight test and in many respects the animals used in toxicity testing are physiologically different to humans. In addition, physical tests are expensive and time-consuming which both drives up the costs of seeking regulatory approval and slows down the translation of new innovative products to the market. The almost ubiquitous use of computer-based simulations to support the research, development and design of manufactured products inevitably leads to their use in supporting regulatory applications. This creates challenges for regulators who must judge the trustworthiness of predictions from these simulations. [see ‘Fake facts & untrustworthy predictions‘ on December 4th, 2019]. It is standard practice for modellers to demonstrate the validity of their models; however, validation does not automatically lead to acceptance of predictions by decision-makers. Acceptance is more closely related to scientific credibility. I have been working across a number of disciplines on the scientific credibility of models including in engineering where multi-physics phenomena are important, such as hypersonic flight and fusion energy [see ‘Thought leadership in fusion energy‘ on October 9th, 2019], and in computational biology and toxicology [see ‘Hierarchical modelling in engineering and biology‘ on March 14th, 2018]. Working together with my collaborators in these disciplines, we have developed a common set of factors which underpin scientific credibility that are based on principles drawn from the literature on the philosophy of science and are designed to be both discipline-independent and method-agnostic [Patterson & Whelan, 2019; Patterson et al, 2021]. We hope that our cross-disciplinary approach will break down the subject-silos that have become established as different scientific communities have developed their own frameworks for validating models. As mentioned above, the process of validation tends to be undertaken by model developers and, in some sense, belongs to them; whereas, credibility is not exclusive to the developer but is a trust that needs to be shared with a decision-maker who seeks to use the predictions to inform their decision [see ‘Credibility is in the eye of the beholder‘ on April 20th, 2016]. Trust requires a common knowledge base and understanding that is usually built through interactions. We hope the credibility factors will provide a framework for these interactions as well as a structure for building a portfolio of evidence that demonstrates the reliability of a model.
References:
Patterson EA & Whelan MP, On the validation of variable fidelity multi-physics simulations, J. Sound & Vibration, 448:247-258, 2019.
Patterson EA, Whelan MP & Worth A, The role of validation in establishing the scientific credibility of predictive toxicology approaches intended for regulatory application, Computational Toxicology, 17: 100144, 2021.
Image: Extract from abstract by Zahrah Resh.
About four years ago I wrote about living in bubbles and rarely coming into contact with people outside of our bubble [see ‘You’re all weird‘ on February 8th, 2017]. This was in the context of our experience of the media and our surprise when electorates make apparently irrational decisions. Since early this year we have been encouraged to live in more literal bubbles in order to slow down the spread of COVID-19; so, for example, we have created bubbles of researchers using our research labs in shifts to avoid a total shutdown of research when someone tests positive for coronavirus. For many people, the pandemic has isolated them in a bubble of one that has created concerns about the well-being and happiness of individuals living and working alone. When asked about the place he is happiest, the artist Ai Weiwei responded ‘Every place is equal for me. Even in detention I could still find joyful moments’. He finds ways to connect to other people and their emotions by reflecting on who he is, which leads to moments of joy. He believes that success in life is about finding yourself in way that ‘doesn’t need ambition or talent. It just needs a functioning mind, emotion and simple judgment.’ During lockdowns induced by the COVID-19 pandemic, I believe that it has become more important to maintain the life of mind through reading and discovering new ideas. As Jarvis Cocker said in a recent interview: ‘I don’t want to spend the rest of my life thinking the same thoughts and feeling the same things, rechewing the same thing. I find that really boring.’ I hope that these posts have brought you new ideas and ways of thinking during 2020; writing them has certainly kept my mind active and stimulated. So, I plan to continue in 2021 and hope that you will continue to read them. Best wishes for a happy New Year!
Realize Engineering 