Tag Archives: model validation

From strain measurements to assessing El Niño events

Figure 11 from RSOS 201086One of the exciting aspects of leading a university research group is that you can never be quite sure where the research is going next.  We published a nice example of this unpredictability last week in Royal Society Open Science in a paper called ‘Transformation of measurement uncertainties into low-dimensional feature space‘ [1].  While the title is an accurate description of the contents, it does not give much away and certainly does not reveal that we proposed a new method for assessing the occurrence of El Niño events.  For some time we have been working with massive datasets of measurements from arrays of sensors and representing them by fitting polynomials in a process known as image decomposition [see ‘Recognising strain‘ on October 28th, 2015]. The relatively small number of coefficients from these polynomials can be collated into a feature vector which facilitates comparison with other datasets [see for example, ‘Out of the valley of death into a hype cycle‘ on February 24th, 2021].  Our recent paper provides a solution to the issue of representing the measurement uncertainty in the same space as the feature vector which is roughly what we set out to do.  We demonstrated our new method for representing the measurement uncertainty by calibrating and validating a computational model of a simple beam in bending using data from an earlier study in a EU-funded project called VANESSA [2] — so no surprises there.  However, then my co-author and PhD student, Antonis Alexiadis went looking for other interesting datasets with which to demonstrate the new method.  He found a set of spatially-varying uncertainties associated with a metamodel of soil moisture in a river basin in China [3] and global oceanographic temperature fields collected monthly over 11 years from 2002 to 2012 [4].  We used the latter set of data to develop a new technique for assessing the occurrence of El-Niño events in the Pacific Ocean.  Our technique is based on global ocean dynamics rather than on the small region in the Pacific Ocean which is usually used and has the added advantages of providing a confidence level on the assessment as well as enabling straightforward comparisons of predictions and measurements.  The comparison of predictions and measurements is a recurring theme in our current research but I did not expect it to lead into ocean dynamics.

Image is Figure 11 from [1] showing convex hulls fitted to the cloud of points representing the uncertainty intervals for the ocean temperature measurements for each month in 2002 using only the three most significant principal components . The lack of overlap between hulls can be interpreted as implying a significant difference in the temperature between months.

References:

[1] Alexiadis, A. and Ferson, S. and  Patterson, E.A., , 2021. Transformation of measurement uncertainties into low-dimensional feature vector space. Royal Society Open Science, 8(3): 201086.

[2] Lampeas G, Pasialis V, Lin X, Patterson EA. 2015.  On the validation of solid mechanics models using optical measurements and data decomposition. Simulation Modelling Practice and Theory 52, 92-107.

[3] Kang J, Jin R, Li X, Zhang Y. 2017, Block Kriging with measurement errors: a case study of the spatial prediction of soil moisture in the middle reaches of Heihe River Basin. IEEE Geoscience and Remote Sensing Letters, 14, 87-91.

[4] Gaillard F, Reynaud T, Thierry V, Kolodziejczyk N, von Schuckmann K. 2016. In situ-based reanalysis of the global ocean temperature and salinity with ISAS: variability of the heat content and steric height. J. Climate. 29, 1305-1323.

Out of the valley of death into a hype cycle?

Fig 5 from Middleton et al with full captionThe capability to identify damage and track its propagation in structures is important in ensuring the safe operation of a wide variety of engineering infrastructure, including aircraft structures. A few years ago, I wrote about research my group was performing, in the INSTRUCTIVE project [see ‘INSTRUCTIVE final reckoning‘ on January 9th, 2019] with Airbus and Strain Solutions Limited, to deliver a new tool for monitoring the development of damage using thermoelastic stress analysis (TSA) [see ‘Counting photons to measure stress‘ on November 18th, 2015].  We collected images using a TSA system while a structural component was subject to cycles of load that caused damage to initiate and propagate during a fatigue test. The series of images were analysed using a technique based on optical flow to identify apparent movement between the images which was taken as indication of the development of damage [1]. We demonstrated that our technique could indicate the presence of a crack less than a millimetre in length and even identify cracks initiating under the heads of bolts using experiments performed in our laboratory [see ‘INSTRUCTIVE update‘ on October 4th, 2017].  However, this technique was susceptible to errors in the images when we tried to use low-cost sensors and to changes in the images caused by flight cycle loading with varying amplitude and frequency of loads.  Essentially, the optical flow approach could be fooled into identifying damage propagation when a sensor delivered a noisy image or the shape of the load cycle was changed.  We have now overcome this short-coming by replacing the optical flow approach with the orthogonal decomposition technique [see ‘Recognising strain‘ on October 28th, 2015] that we developed for comparing data fields from measurements and predictions in validation processes [see ‘Million to one‘ on November 21st, 2018] .  Each image is decomposed to a feature vector and differences between the feature vectors are indicative of damage development (see schematic in thumbnail from [2]).  The new technique, which we have named the differential feature vector method, is sufficiently robust that we have been able to use a sensor costing 1% of the price of a typical TSA system to identify and track cracks during cyclic loading.  The underpinning research was published in December 2020 by the Royal Society [2] and the technique is being implemented in full-scale ground-tests on aircraft structures as part of the DIMES project.  Once again, a piece of technology is emerging from the valley of death [see ‘Slowly crossing the valley of death‘ on January 27th, 2021] and, without wishing to initiate the hype cycle [see ‘Hype cycle‘ on September 23rd, 2015], I hope it will transform the use of thermal imaging for condition monitoring.

Logos of Clean Sky 2 and EUThe INSTRUCTIVE and DIMES projects have received funding from the Clean Sky 2 Joint Undertaking under the European Union’s Horizon 2020 research and innovation programme under grant agreements No. 685777 and No. 820951 respectively.

The opinions expressed in this blog post reflect only the author’s view and the Clean Sky 2 Joint Undertaking is not responsible for any use that may be made of the information it contains.

References

[1] Middleton CA, Gaio A, Greene RJ & Patterson EA, Towards automated tracking of initiation and propagation of cracks in Aluminium alloy coupons using thermoelastic stress analysis, J. Non-destructive Testing, 38:18, 2019.

[2] Middleton CA, Weihrauch M, Christian WJR, Greene RJ & Patterson EA, Detection and tracking of cracks based on thermoelastic stress analysis, R. Soc. Open Sci. 7:200823, 2020.

Reduction in usefulness of reductionism

decorative paintingA couple of months ago I wrote about a set of credibility factors for computational models [see ‘Credible predictions for regulatory decision-making‘ on December 9th, 2020] that we designed to inform interactions between researchers, model builders and decision-makers that will establish trust in the predictions from computational models [1].  This is important because computational modelling is becoming ubiquitous in the development of everything from automobiles and power stations to drugs and vaccines which inevitably leads to its use in supporting regulatory applications.  However, there is another motivation underpinning our work which is that the systems being modelled are becoming increasingly complex with the likelihood that they will exhibit emergent behaviour [see ‘Emergent properties‘ on September 16th, 2015] and this makes it increasingly unlikely that a reductionist approach to establishing model credibility will be successful [2].  The reductionist approach to science, which was pioneered by Descartes and Newton, has served science well for hundreds of years and is based on the concept that everything about a complex system can be understood by reducing it to the smallest constituent part.  It is the method of analysis that underpins almost everything you learn as an undergraduate engineer or physicist. However, reductionism loses its power when a system is more than the sum of its parts, i.e., when it exhibits emergent behaviour.  Our approach to establishing model credibility is more holistic than traditional methods.  This seems appropriate when modelling complex systems for which a complete knowledge of the relationships and patterns of behaviour may not be attainable, e.g., when unexpected or unexplainable emergent behaviour occurs [3].  The hegemony of reductionism in science made us nervous about writing about its short-comings four years ago when we first published our ideas about model credibility [2].  So, I was pleased to see a paper published last year [4] that identified five fundamental properties of biology that weaken the power of reductionism, namely (1) biological variation is widespread and persistent, (2) biological systems are relentlessly nonlinear, (3) biological systems contain redundancy, (4) biology consists of multiple systems interacting across different time and spatial scales, and (5) biological properties are emergent.  Many engineered systems possess all five of these fundamental properties – you just to need to look at them from the appropriate perspective, for example, through a microscope to see the variation in microstructure of a mass-produced part.  Hence, in the future, there will need to be an increasing emphasis on holistic approaches and systems thinking in both the education and practices of engineers as well as biologists.

For more on emergence in computational modelling see Manuel Delanda Philosophy and Simulation: The Emergence of Synthetic Reason, Continuum, London, 2011. And, for more systems thinking see Fritjof Capra and Luigi Luisi, The Systems View of Life: A Unifying Vision, Cambridge University Press, 2014.

References:

[1] Patterson EA, Whelan MP & Worth A, The role of validation in establishing the scientific credibility of predictive toxicology approaches intended for regulatory application, Computational Toxicology, 17: 100144, 2021.

[2] Patterson EA &Whelan MP, A framework to establish credibility of computational models in biology. Progress in biophysics and molecular biology, 129: 13-19, 2017.

[3] Patterson EA & Whelan MP, On the validation of variable fidelity multi-physics simulations, J. Sound & Vibration, 448:247-258, 2019.

[4] Pruett WA, Clemmer JS & Hester RL, Physiological Modeling and Simulation—Validation, Credibility, and Application. Annual Review of Biomedical Engineering, 22:185-206, 2020.

Slowly crossing the valley of death

A view of a valleyThe valley of death in technology development is well-known amongst research engineers and their sponsors. It is the gap between discovery and application, or between realization of an idea in a laboratory and its implementation in the real-world. Some of my research has made it across the valley of death, for example the poleidoscope about 15 years ago (see ‘Poleidoscope (=polariscope+kaleidoscope)‘ on October 14th, 2020).  Our work on quantitative comparisons of data fields from physical measurements and computer predictions is about three-quarters of the way across the valley.  We published a paper in December (see Dvurecenska et al, 2020) on its application to a large panel from the fuselage of an aircraft based on work we completed as part of the MOTIVATE project.  I reported the application of the research in almost real-time in a post in December 2018 (see ‘Industrial Uncertainty‘ on December 12th, 2018) and in further detail in May 2020 as we submitted the manuscript for publication (‘Alleviating industrial uncertainty‘ on May 13th, 2020).  However, the realization in the laboratory occurred nearly a decade ago when teams from Michigan State University and the University of Liverpool came together in the ADVISE project funded by EU Framework 7 programme (see Wang et al, 2011). Subsequently, the team at Michigan State University moved to the University of Liverpool and in collaboration with researchers at Empa developed the technique that was applied in the MOTIVATE project (see Sebastian et al 2013). The work published in December represents a step into the valley of death; from a university environment into a full-scale test laboratory at Empa using a real piece of aircraft.  The MOTIVATE project involved a further step to a demonstration on an on-going test of a cockpit at Airbus which was also reported in a post last May (see ‘The blind leading the blind‘ on May 27th, 2020).  We are now working with Airbus in a new programme to embed the process of quantitative comparison of fields of measurements and predictions into their routine test procedures for aerospace structures.  So, I would like to think we are climbing out of the valley.

Image: not Death Valley but taken on a road trip in 2008 somewhere between Moab, UT and Kanab, UT while living in Okemos, MI.

Sources:

Dvurecenska, K., Diamantakos, I., Hack, E., Lampeas, G., Patterson, E.A. and Siebert, T., 2020. The validation of a full-field deformation analysis of an aircraft panel: A case study. The Journal of Strain Analysis for Engineering Design, p.0309324720971140.

Sebastian, C., Hack, E. and Patterson, E., 2013. An approach to the validation of computational solid mechanics models for strain analysis. The Journal of Strain Analysis for Engineering Design, 48(1), pp.36-47.

Wang, W., Mottershead, J.E., Sebastian, C.M. and Patterson, E.A., 2011. Shape features and finite element model updating from full-field strain data. International Journal of Solids and Structures, 48(11-12), pp.1644-1657.

For more posts on the MOTIVATE project: https://realizeengineering.blog/category/myresearch/motivate-project/

The MOTIVATE project has received funding from the Clean Sky 2 Joint Undertaking under the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 754660 and the Swiss State Secretariat for Education, Research and Innovation under contract number 17.00064.

The opinions expressed in this blog post reflect only the author’s view and the Clean Sky 2 Joint Undertaking is not responsible for any use that may be made of the information it contains.