Tag Archives: Engineering

Feedback on feedback

Feedback on students’ assignments is a challenge for many in higher education.  Students appear to be increasingly dissatisfied with it and academics are frustrated by its apparent ineffectiveness, especially when set against the effort required for its provision.  In the UK, the National Student Survey results show that satisfaction with assessment and feedback is increasing but it remains the lowest ranked category in the survey [1].  My own recent experience has been of the students’ insatiable hunger for feedback on a continuing professional development (CPD) programme, despite receiving detailed written feedback and one-to-one oral discussion of their assignments.

So, what is going wrong?  I am aware that many of my academic colleagues in engineering do not invest much time in reading the education research literature; perhaps because, like the engineering research literature, much of it is written in a language that is readily appreciated only by those immersed in the subject.  So, here is an accessible digest of research on effective feedback that meets students’ expectations and realises the potential improvement in their performance.

It is widely accepted that feedback is an essential component [2] in the learning cycle and there is evidence that feedback is the single most powerful influence on student achievement [3, 4].  However, we often fail to realise this potential because our feedback is too generic or vague, not sufficiently timely [5], and transmission-focussed rather than student-centered or participatory [6].  In addition, our students tend not to be ‘assessment literate’, meaning they are unfamiliar with assessment and feedback approaches and they do not interpret assessment expectations in the same way as their tutors [5, 7].  Student reaction to feedback is strongly related to their emotional maturity, self-efficacy and motivation [1]; so that for a student with low self-esteem, negative feedback can be annihilating [8].  Emotional immaturity and assessment illiteracy, such as is typically found amongst first year students, is a toxic mix that in the absence of a supportive tutorial system leads to student dissatisfaction with the feedback process [1].

So, how should we provide feedback?  I provide copious detailed comments on students’ written work following the example of my own university tutor, who I suspect was following example of his tutor, and so on.  I found these comments helpful but at times overwhelming.  I also remember a college tutor who made, what seemed to me, devastatingly negative comments about my writing skills, which destroyed my confidence in my writing ability for decades.  It was only restored by a Professor of English who recently complimented me on my writing; although I still harbour a suspicion that she was just being kind to me.  So, neither of my tutors got it right; although one was clearly worse than the other.  Students tend to find negative feedback unfair and unhelpful, even when it is carefully and politely worded [8].

Students like clear, unambiguous, instructional and direction feedback [8].  Feedback should provide a statement of student performance and suggestions for improvement [9], i.e. identify the gap between actual and expected performance and provide instructive advice on closing the gap.  This implies that specific assessment criteria are required that explicitly define the expectation [2].  The table below lists some of the positive and negative attributes of feedback based on the literature [1,2].  However, deploying the appropriate attributes does not guarantee that students will engage with feedback; sometimes students fail to recognise that feedback is being provided, for example in informal discussion and dialogic teaching; and hence, it is important to identify the nature and purpose of feedback every time it is provided.  We should reduce our over-emphasis on written feedback and make more use of oral feedback and one-to-one, or small group, discussion.  We need to take care that the receipt of grades or marks does not obscure the feedback, perhaps by delaying the release of marks.  You could ask students about the mark they would expect in the light of the feedback; and, you could require students to show in future work how they have used the feedback – both of these actions are likely to improve the effectiveness of feedback [5].

In summary, feedback that is content rather than process-driven is unlikely to engage students [10].  We need to strike a better balance between positive and negative comments, which includes a focus on appropriate guidance and motivation rather than justifying marks and diagnosing short-comings [2].  For most of us, this means learning a new way of providing feedback, which is difficult and potentially arduous; however, the likely rewards are more engaged, higher achieving students who might appreciate their tutors more.

References

[1] Pitt E & Norton L, ‘Now that’s the feedback that I want!’ Students reactions to feedback on graded work and what they do with it. Assessment & Evaluation in HE, 42(4):499-516, 2017.

[2] Weaver MR, Do students value feedback? Student perceptions of tutors’ written responses.  Assessment & Evaluation in HE, 31(3):379-394, 2006.

[3] Hattie JA, Identifying the salient facets of a model of student learning: a synthesis of meta-analyses.  IJ Educational Research, 11(2):187-212, 1987.

[4] Black P & Wiliam D, Assessment and classroom learning. Assessment in Education: Principles, Policy & Practice, 5(1):7-74, 1998.

[5] O’Donovan B, Rust C & Price M, A scholarly approach to solving the feedback dilemma in practice. Assessment & Evaluation in HE, 41(6):938-949, 2016.

[6] Nicol D & MacFarlane-Dick D, Formative assessment and self-regulatory learning: a model and seven principles of good feedback practice. Studies in HE, 31(2):199-218, 2006.

[7] Price M, Rust C, O’Donovan B, Handley K & Bryant R, Assessment literacy: the foundation for improving student learning. Oxford: Oxford Centre for Staff and Learning Development, 2012.

[8] Sellbjer S, “Have you read my comment? It is not noticeable. Change!” An analysis of feedback given to students who have failed examinations.  Assessment & Evaluation in HE, DOI: 10.1080/02602938.2017.1310801, 2017.

[9] Saddler R, Beyond feedback: developing student capability in complex appraisal. Assessment & Evaluation in HE, 35(5):535-550, 2010.

[10] Hounsell D, Essay writing and the quality of feedback. In J Richardson, M. Eysenck & D. Piper (eds) Student learning: research in education and cognitive psychology. Milton Keynes: Open University Press, 1987.

Getting smarter

A350 XWB passes Maximum Wing Bending test [from: http://www.airbus.com/galleries/photo-gallery%5D

Garbage in, garbage out (GIGO) is a perennial problem in computational simulations of engineering structures.  If the description of the geometry of the structure, the material behaviour, the loading conditions or the boundary conditions are incorrect (garbage in), then the simulation generates predictions that are wrong (garbage out), or least an unreliable representation of reality.  It is not easy to describe precisely the geometry, material, loading and environment of a complex structure, such as an aircraft or a powerstation; because, the complete description is either unavailable or too complicated.  Hence, modellers make assumptions about the unknown information and, or to simplify the description.  This means the predictions from the simulation have to be tested against reality in order to establish confidence in them – a process known as model validation [see my post entitled ‘Model validation‘ on September 18th, 2012].

It is good practice to design experiments specifically to generate data for model validation but it is expensive, especially when your structure is a huge passenger aircraft.  So naturally, you would like to extract as much information from each experiment as possible and to perform as few experiments as possible, whilst both ensuring predictions are reliable and providing confidence in them.  In other words, you have to be very smart about designing and conducting the experiments as well as performing the validation process.

Together with researchers at Empa in Zurich, the Industrial Systems Institute of the Athena Research Centre in Athens and Dantec Dynamics in Ulm, I am embarking on a new EU Horizon 2020 project to try and make us smarter about experiments and validation.  The project, known as MOTIVATE [Matrix Optimization for Testing by Interaction of Virtual and Test Environments (Grant Nr. 754660)], is funded through the Clean Sky 2 Joint Undertaking with Airbus acting as our topic manager to guide us towards an outcome that will be applicable in industry.  We held our kick-off meeting in Liverpool last week, which is why it is uppermost in my mind at the moment.  We have 36-months to get smarter on an industrial scale and demonstrate it in a full-scale test on an aircraft structure.  So, some sleepness nights ahead…

Bibliography:

 

ASME V&V 10-2006, Guide for verification & validation in computational solid mechanics, American Society of Mech. Engineers, New York, 2006.

European Committee for Standardisation (CEN), Validation of computational solid mechanics models, CEN Workshop Agreement, CWA 16799:2014 E.

Hack E & Lampeas G (Guest Editors) & Patterson EA (Editor), Special issue on advances in validation of computational mechanics models, J. Strain Analysis, 51 (1), 2016.

http://www.engineeringvalidation.org/

Listening with your eyes shut

I am in the London Underground onboard a train on my way to a conference on ‘New Approaches to Higher Education’ organised by the Institution of Engineering and Technology and the Engineering Professors’ Council.  The lady opposite has her eyes closed but she is not asleep because she opens them periodically as we come into stations to check whether it’s her stop.  I wonder if she is trying to reproduce John Hull’s experience of the depth of sounds as a blind person [see my post entitled ‘Rain brings out the contours in everything‘ on February 22, 2017].  For the second time in recent weeks, I close my eyes and try it for myself.  It is surprising how in a crowded train, I can’t hear anyone, just the noise made by the train.  It’s like a wobble board that’s joined by a whole percussion section of an orchestra when we go around a bend or over points.  The first time I closed my eyes was at a concert at the Philharmonic Hall in Liverpool.  My view of the orchestra was obstructed by the person in front of me so, rather than stare at the back of their head, I closed my eyes and allowed the music to dominate my mind.  Switching off the stream of images seemed to release more of my brain cells to register the depth and richness of Bach’s Harpsichord Concerto No. 5.  I was classified as tone deaf at school when I was kicked out of the choir and I learned no musical instruments, so the additional texture and dimensionality in the music was a revelation to me.

Back to the London Underground – many of my fellow passengers were plugged into their phones or tablets via their ears and eyes.  I wondered if any were following the MOOC on Understanding Super Structures that we launched recently.  Unlikely I know, but it’s a bit different, because it is mainly audio clips and not videos.  We’re trying to tap into some of the time many people spend with earbuds plugged into their ears but also make the MOOC more accessible in countries where internet access is mainly via mobile phones.  My recent experiences of listening with my eyes closed, make me realize that perhaps we should ask people to close their eyes when listening to our audio clips so that they can fully appreciate them.  If they are sitting on the train then that’s fine but not recommended if you are walking across campus or in town!

We are all citizens of the world

A longer post this week because I was invited to write an article for the Citizens of Everywhere project being organised by the Centre for New and International Writing at the University of Liverpool. The article is reproduced below:

Scientists seek to discover and describe knowledge, while engineers seek to apply and deploy the same knowledge by creating technology that supports our global society.  In their quests, both scientists and engineers are dependent on each other and on those that have gone before them.  On each other, because scientists increasingly need technology in order make discoveries, and because engineers need new scientific discoveries to drive innovation; and both groups stand on the shoulders of their predecessors, to mis-quote Isaac Newton who said he was able to see further by standing on the shoulders of his predecessors.  Scientists and engineers have to build on the achievements of their predecessors, otherwise nothing would be achieved in a single lifetime.  This process is enabled by the global dissemination of knowledge and understanding in our society, which does not recognise any boundaries and flows around the world largely unimpeded by the efforts of nation states and private corporations.  As Poincaré is reputed to have said ‘the scientist does not study nature because it is useful; he studies it because he delights in it, and he delights in it because it is beautiful’.  The feeling of delight is a reward for hours of intense study; but, the realization that you are the first to recognise or discover a new scientific fact generates so much excitement that you want to tell everyone.  Scientists have always met to share their findings and discuss the implications.  As a young researcher, I had a postcard above my desk showing a photograph of the attendees at the 5th Solvay Conference in 1927 at which 29 scientists from around the world met to debate the latest discoveries relating to electrons and photons.  Seventeen of the 29 attendees at this conference went on to receive Nobel prizes.  Not all scientific meetings are as famous, or perhaps as significant, as the Solvay conference; but, today they are happening all around the world involving thousands of researchers from scores of countries.  Besides the bureaucratic burden of obtaining visas, national boundaries have little impact on these exchanges of scientific and technological knowledge and understanding.  If you are a researcher working in the subject with sufficient funding then you can attend; and if your work is sufficiently novel, rigorous and significant, as judged by your peers, then you can present it at one of these meetings.  You can also listen to the world’s leading experts in the field, have a discussion over a coffee, or even a meal, with them before going back to your laboratory or office and attempting to add to society’s knowledge and understanding.  Most scientists and engineers work as part of a global community contributing to, and exploiting, a shared knowledge and understanding of natural and manufactured phenomena; and in this process, as global citizens, we are relatively unaware and uninfluenced by the national boundaries drawn and fought over by politicians and leaders.  Of course, I have described a utopian world to which reality does not conform, because in practice corporations attempt to protect their intellectual property for profit and national governments to classify information in the national interests and sometimes restrict the movement of scientists and technologist to and from states considered to be not playing by the right set of rules.  However, on the timescale of scientific discovery, these actions are relatively short-term and rarely totally effective.  Perhaps this is because the delight in the beauty of discovery overcomes these obstacles, or because the benefits of altruistic sharing outweigh the selfish gain from restrictive practices.  (Of course, the scientific community has its charlatans, fraudsters and free-loaders; but, these counterfeiters tend to operate on a global stage so that even their fake science impacts on the world-wide community of scientists and engineers.)  Participation in this global exchange of ideas and information makes many of us feel part of a world-wide community, or citizens of the world, who are enfranchised by our contributions and interactions with other citizens and international organisations.  Of course, along with everyone else, we are also inhabitants of the world; and these two actions, namely enfranchisement and inhabiting, are key characteristics of a citizen, as defined by the Shorter Oxford English Dictionary.  Theresa May in her speech last October, at the Conservative party conference said: ‘If you believe you’re a citizen of the world, you’re a citizen of nowhere.’  If she is right, then she rendered many scientists and engineers as aliens; however, I don’t think she is, because citizenship of the world does not exclude us from also being citizens of other, local communities; even though politicians may want to redraw the boundaries of these communities and larger unions to which they belong.  However, in practice, it is hard to avoid the fact that we are all inhabitants of planet Earth and have a responsibility for ensuring that it remains habitable for our grand-children and great-grandchildren; so, we are all citizens of the world with its associated responsibilities.

When I was a student, thirty years ago, James Lovelock published his famous book, ‘Gaia’ in which he postulated that the world was a unified living system with feedback control that preserved its own stability but not necessarily the conditions for the survival of the human race.  More recently, Max Tegmark, in his book ‘Our Mathematical Universe’, has used the analogy of spaceship Earth stocked with large but limited supplies of water, food and fuel, and equipped with both an atmospheric shield and a magnetic field to protect us from life-threatening ultra-violet and cosmic rays, respectively.  Our spaceship has no captain; and we spend next to nothing on maintenance such as avoiding onboard explosions, overheating, ultra-violet shield deterioration or premature depletion of supplies.  Lovelock and Tegmark are part of a movement away from a reductionist approach to science that has dominated since Descartes and Newton, and towards systems thinking, in which it is recognised that the whole is more than the sum of the parts.  It’s hard for most of us to adopt this new thinking, because our education was configured around dividing everything into its smallest constituent parts in order to analyse and understand their function; but, this approach often misses, or even destroys, the emergent behaviour of the complex system – it’s like trying to understand the functioning of the brain by physically dissecting it.  Recently reported statements about citizens of the world and about climate change, suggest that some world leaders and politicians find it easier, or more convenient, to use reductionism to ignore or deny the potential for complex systems, such as our global society and planet Earth, to exhibit emergent behaviour.

Thomas L. Friedmann in his book, ‘The World is Flat’ warned that ‘every young American would be wise to think of themselves competing against every young Chinese, Indian or Brazilian’.  He was right; we cannot turn back the globalisation of knowledge.  The hunger for knowledge and understanding is shared by all and courses provided over the internet are democratizing knowledge to an unprecedented level.  For instance, I recently taught a course on undergraduate thermodynamics – not normally a popular subject; but, it was made available globally as a massive open on-line course (MOOC) and taken by thousands of learners in more than 130 countries.  The citizens of the world are becoming empowered by knowledge and simultaneously more networked.  Large complex networks are systems that exhibit emergent behaviour, which tends to be unexpected and surprising, especially if you only consider their constituents.