Tag Archives: validation

Fake facts & untrustworthy predictions

I need to confess to writing a misleading post some months ago entitled ‘In Einstein’s footprints?‘ on February 27th 2019, in which I promoted our 4th workshop on the ‘Validation of Computational Mechanics Models‘ that we held last month at Guild Hall of Carpenters [Zunfthaus zur Zimmerleuten] in Zurich.  I implied that speakers at the workshop would be stepping in Einstein’s footprints when they presented their research at the workshop, because Einstein presented a paper at the same venue in 1910.  However, as our host in Zurich revealed in his introductory remarks , the Guild Hall was gutted by fire in 2007 and so we were meeting in a fake, or replica, which was so good that most of us had not realised.  This was quite appropriate because a theme of the workshop was enhancing the credibility of computer models that are used to replicate the real-world.  We discussed the issues surrounding the trustworthiness of models in a wide range of fields including aerospace engineering, biomechanics, nuclear power and toxicology.  Many of the presentations are available on the website of the EU project MOTIVATE which organised and sponsored the workshop as part of its dissemination programme.  While we did not solve any problems, we did broaden people’s understanding of the issues associated with trustworthiness of predictions and identified the need to develop common approaches to support regulatory decisions across a range of industrial sectors – that’s probably the theme for our 5th workshop!

The MOTIVATE project has received funding from the Clean Sky 2 Joint Undertaking under the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 754660 and the Swiss State Secretariat for Education, Research and Innovation under contract number 17.00064.

The opinions expressed in this blog post reflect only the author’s view and the Clean Sky 2 Joint Undertaking is not responsible for any use that may be made of the information it contains.

Image: https://www.tagesanzeiger.ch/Zunfthaus-Zur-Zimmerleuten-Wiederaufbauprojekt-steht/story/30815219

 

Spatial-temporal models of protein structures

For a number of years I have been working on methods for validating computational models of structures [see ‘Model validation‘ on September 18th 2012] using the full potential of measurements made with modern techniques such as digital image correlation [see ‘256 shades of grey‘ on January 22nd 2014] and thermoelastic stress analysis [see ‘Counting photons to measure stress‘ on November 18th 2015].  Usually the focus of our interest is at the macroscale, for example the research on aircraft structures in the MOTIVATE project; however, in a new PhD project with colleagues at the National Tsing Hua University in Taiwan, we are planning to explore using our validation procedures and metrics [1] in structural biology.

The size and timescale of protein-structure thermal fluctuations are essential to the regulation of cellular functions. Measurement techniques such as x-ray crystallography and transmission electron cryomicroscopy (Cryo-EM) provide data on electron density distribution from which protein structures can be deduced using molecular dynamics models. Our aim is to develop our validation metrics to help identify, with a defined level of confidence, the most appropriate structural ensemble for a given set of electron densities. To make the problem more interesting and challenging the structure observed by x-ray crystallography is an average or equilibrium state because a folded protein is constantly in motion undergoing harmonic oscillations, each with different frequencies and amplitude [2].

The PhD project is part of the dual PhD programme of the University of Liverpool and National Tsing Hua University.  Funding is available in form of a fee waiver and contribution to living expenses for four years of study involving significant periods (perferably two years) at each university.  For more information follow this link.

References:

[1] Dvurecenska, K., Graham, S., Patelli, E. & Patterson, E.A., A probabilistic metric for the validation of computational models, Royal Society Open Society, 5:180687, 2018.

[2] Justin Chan, Hong-Rui Lin, Kazuhiro Takemura, Kai-Chun Chang, Yuan-Yu Chang, Yasumasa Joti, Akio Kitao, Lee-Wei Yang. An efficient timer and sizer of protein motions reveals the time-scales of functional dynamics in the ribosome (2018) https://www.biorxiv.org/content/early/2018/08/03/384511.

Image: A diffraction pattern and protein structure from http://xray.bmc.uu.se/xtal/

Establishing fidelity and credibility in tests & simulations (FACTS)

A month or so ago I gave a lecture entitled ‘Establishing FACTS (Fidelity And Credibility in Tests & Simulations)’ to the local branch of the Institution of Engineering Technology (IET). Of course my title was a play on words because the Oxford English Dictionary defines a ‘fact’ as ‘a thing that is known or proved to be true’ or ‘information used as evidence or as part of report’.   One of my current research interests is how we establish predictions from simulations as evidence that can be used reliably in decision-making.  This is important because simulations based on computational models have become ubiquitous in engineering for, amongst other things, design optimisation and evaluation of structural integrity.   These models need to possess the appropriate level of fidelity and to be credible in the eyes of decision-makers, not just their creators.  Model credibility is usually provided through validation processes using a small number of physical tests that must yield a large quantity of reliable and relevant data [see ‘Getting smarter‘ on June 21st, 2017].  Reliable and relevant data means making measurements with low levels of uncertainty under real-world conditions which is usually challenging.

These topics recur through much of my research and have found applications in aerospace engineering, nuclear engineering and biology. My lecture to the IET gave an overview of these ideas using applications from each of these fields, some of which I have described in past posts.  So, I have now created a new page on this blog with a catalogue of these past posts on the theme of ‘FACTS‘.  Feel free to have a browse!

Spontaneously MOTIVATEd

Some posts arise spontaneously, stimulated by something that I have read or done, while others are part of commitment to communicate on a topic related to my research or teaching, such as the CALE series.  The motivation for a post seems unrelated to its popularity.  This post is part of that commitment to communicate.

After 12 months, our EU-supported research project, MOTIVATE [see ‘Getting Smarter‘ on June 21st, 2017] is one-third complete in terms of time; and, as in all research it appears to have made a slow start with much effort expended on conceptualizing, planning, reviewing prior research and discussions.  However, we are on-schedule and have delivered on one of our four research tasks with the result that we have a new validation metric and a new flowchart for the validation process.  The validation metric was revealed at the Photomechanics 2018 conference in Toulouse earlier this year [see ‘Massive Engineering‘ on April 4th, 2018].  The new flowchart [see the graphic] is the result of a brainstorming [see ‘Brave New World‘ on January 10th, 2018] and much subsequent discussion; and will be presented at a conference in Brussels next month [ICEM 2018] at which we will invite feedback [proceedings paper].  The big change from the classical flowchart [see for example ASME V&V guide] is the inclusion of historical data with the possibility of not requiring experiments to provide data for validation purposes. This is probably a paradigm shift for the engineering community, or at least the V&V [Validation & Verification] community.  So, we are expecting some robust feedback – feel free to comment on this blog!

References:

Hack E, Burguete RL, Dvurecenska K, Lampeas G, Patterson EA, Siebert T & Szigeti E, Steps toward industrial validation experiments, In Proceedings Int. Conf. Experimental Mechanics, Brussels, July 2018 [pdf here].

Dvurcenska K, Patelli E & Patterson EA, What’s the probability that a simulation agrees with your experiment? In Proceedings Photomechanics 2018, Toulouse, March 2018.