Tag Archives: innovation

Ancient models and stochastic parrots

Leave a reply

Decorative image of a parrot in the parkIn 2021 Emily Bender and her colleagues published a paper suggesting that the Large Language Models (LLMs) underpinning many Artificial Intelligence applications (AI apps) were little more than stochastic parrots.  They described LLMs as ‘a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning’.  This has fuelled the ongoing debate about the real capabilities of AI apps versus the hype from the companies trying persuade us to use them.  Most AI apps are based on statistical analysis of data as stated by Bender et al; however, there is a trend toward physics-based machine learning in which known laws of physics are combined with machine-learning algorithms trained on data sets [see for example the recent review by Meng et al, 2025].  We have been fitting data to models for a very long time.  In the fifth century BC, the Babylonians made perhaps one of the greatest breakthroughs in the history of science, when they realized that mathematical models of astronomical motion could be used to extrapolate data and make predictions.  They had been recording astronomical observations since 3400 BC and the data was all collated in cuneiform in the library at Nineveh belonging to King Ashurbanipal who ruled from 669-631 BC.  While our modern-day digital storage capacity in data centres might far exceed the clay tablets with cuneiform symbols found in Ashurbanipal’s library, it seems unlikely that they will survive five thousand years as part of the data from the Babylonians’ astronomical observations has done and still be readable.

References:

Bender, E.M., Gebru, T., McMillan-Major, A. and Shmitchell, S., 2021, March. On the dangers of stochastic parrots: Can language models be too big?🦜. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency (pp. 610-623).

Meng C, Griesemer S, Cao D, Seo S, Liu Y. 2025. When physics meets machine learning: A survey of physics-informed machine learning. Machine Learning for Computational Science and Engineering. 1(1):20.

Wisnom, Selena, The library of ancient wisdom.  Penguin Books, 2025.

Image: Parrot in the park – free stock photo by Pixabay on Stockvault.net

Are we individuals?

Leave a reply

It has been estimated that there are 150 species of bacteria in our gut with a megagenome correspondingly larger than the human genome; and that 90% of the cells in our bodies are bacterial [1].  This challenges a simple understanding of individual identity because on one level we are a collection of organisms, mainly bacteria, rather than a single entity.  The complexity is almost incomprehensible with 30 trillion cells in the human body each with about a billion protein molecules [2].  Each cell is apparently autonomous, making decisions about its role in the system based on information acquired through communicating and signalling with its neighbours, the rest of the system and the environment.  Its autonomy would appear to imbue it with a sense of individual identity which is shaped by its relationships within the network of cells [3].  This also holds for human beings within society although you could argue the network is simpler because the global population is only about 8 billion; however the quantity of information being communicated is probably greater than between cells, so perhaps that makes the network more complex.  Networks are horizontal hierarchies with no one or thing in overall control and they can adapt to cope with changes in the environment.  By contrast, vertical hierarchies depend on top-down obedience and tend to eliminate dissent, yet without dissent there is little or no innovation or adaptation.  Hence, vertical hierarchies can appear to be robust but are actually brittle [4].  In a network we can build connections and share knowledge leading to the development of a collective intelligence that enables us to solve otherwise intractable problems.  Our ability to acquire knowledge not just from own our experiences but also from the experience of others, and hence to progressively grow collective intelligence, is one of the secrets of our success as a species [5].  It also underpins the competitive advantage of many successful organisations; however, it needs a horizontal, stable structure with high levels of trust and mutual dependence, in which our sense of individual identity is shaped by our relationships.

References:

  1. Gilbert SF, Sapp J, Tauber AI, A symbiotic view of life: we have never been individuals, Quarterly Review of Biology, 87(4):325-341, 2012.
  2. Ball P, How Life Works, Picador, 2023.
  3. Wheatley M, Leadership and the New Science: Discovering Order in a Chaotic World, 2nd Edition, Berrett-Koehler Publishers Inc, San Francisco, 1999.
  4. McWilliams D, Money – A Story of Humanity, Simon & Schuster, London, 2024.
  5. Henrich J, The secret of our success: how culture is driving human evolution, domesticating our species, and making us smarter, Princeton, NJ: Princeton University Press, 2015.

Reproducibility in science and technology

Leave a reply

Schematic diagram from cited paper in Open Research EuropeIt has been suggested that there is crisis in science concerning the reproducibility of data [1].  New research findings are usually published based on data collected only by the group reporting the new findings, which raises the probability of bias in the results as well as reducing their likely validity.  It also creates a temptation to tamper with or falsify data given the incentives to publish.  It is unlikely that any prestigious journal would publish work that simply demonstrates that previously published findings can be reproduced consistently.  Yet, when they have tried to reproduce published data from experiments, many researchers have been unable to do so [2], which perhaps perversely makes the attempt to reproduce results publishable.  However, if no one has attempted to reproduce a published dataset then it stands until demonstrated to not be reproducible, which implies that much of the data in the published literature could be irreproducible and hence of dubious value.  This is a bigger problem than it might seem, because most scientific and technological innovation is built on the findings of fundamental research.  Hence, we are building on shaky foundations if results are not reproducible. Similarly, the transition from prototypes to reliable products is dependent on achieving reproducibility in the real-world of results obtained with a prototype in the laboratory.  I have been discussing these issues with a close collaborator for a number of years and last week we published a letter, in Open Research Europe, summarizing our views.  In ‘Achieving reproducibility in the innovation process’ [3], we propose that a different approach to reproducibility is required for each phase of the innovation process, i.e., discovery, translation and application, because reproducibility has different implications in each phase.  The diagram, reproduced from the paper (CC-BY-4.0), shows our ideas schematically but follow the link to read and comment on them.

References

[1] Baker, M. (2016). Reproducibility crisis. Nature, 533(26), 353-66.

[2] Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T. H., Huber, J., Johannesson, M., … & Wu, H. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2(9), 637-644.

[3] Whelan M & Patterson EA, (2025). Achieving reproducibility in the innovation process, Open Research Europe, 5:25. https://doi.org/10.12688/openreseurope.19408.1

Emergence of ideas leading to a lack of deep insights

Leave a reply

Decorative imageIn Surrealism, which emerged after World War 1, artists attempted to allow the subconscious mind to express itself and resulted in illogical montages or dreamlike scenes and ideas.  Some surrealists championed the subconscious because they thought it would release society from the oppressive rationality of capitalism.  Anna Wiele Kjaer of the University of Copenhagen has suggested that instead our subconscious has been colonised by capitalism and is being shaped the endless of streams of disconnected images flowing from our phones, which are as incongruous as any surrealist montage.  To decolonise our subconscious and to replenish our creativity many of us need a digital detox involving time away from our electronic devices [see ‘Digital detox with deep vacation’ on August 10th, 2016] allowing our brains to switch into mind wandering mode for long uninterrupted periods [see ‘Mind wandering’ on September 3rd, 2014].  Cormac McCarthy has described how ideas struggle against their own realisation and come with their own innate scepticism that acts like a steering mechanism for their emergence from our subconscious.  He also suggests that all ideas come to an end when they lose lustre becoming a tool, perhaps as a theory, strategy or plan, and you can no longer entertain the illusion that they hold some deep insight into reality.  Many of my thoughts never coalesce into an emergent idea but remain as illogical and disconnected as a surrealist montage and the few that do emerge don’t provide deep insights into reality that I recognise.

Sources:

Anya Harrison, Another Surrealism, 2022

Cormac McCarthy, The Passenger, Pan MacMillan, 2023.

Jackie Wullschläge, Surrealism at 100: does it still have the power to disrupt?, FT Weekend, 27 January 2024.

Image: Ceramic tile by Pablo Picasso in museum in Port de Sóller Railway Station, Mallorca.