Evidence-based practice and instructional coaching - why the research evidence is not enough.

Just this week, @DrSamSims wrote a very well-argued blogpost giving four reasons why  instructional coaching is the most well-evidenced form of CPD,  and which concludes that All schools that aspire to be evidence-based should be giving it (instructional coaching ) a go.  However,  from the perspective of what evidence-based practice is all about – the last sentence in the blogpost is badly flawed.   First, it’s not for researchers to tell school leaders and teachers what they should or should not be doing in their schools and classrooms .  What school leaders and teachers prioritise in their schools and classroom is down to their professional judgment.  Indeed, the last sentence of Sam’s blogpost is grist to the mill for the opponents of evidence-based education.  Second, the role of research evidence in evidence-based practice is to provide the backing for warrants.  As such, research evidence plays an indirect role in developing an arguments supporting the use of the intervention - Kvernbekk (2016).  Third, even if you think that instructional coaching meets an obvious priority for  your school i.e. supporting the improvement of teaching learning – that does not mean your school should automatically do it.  Ideally, school leaders would use a disciplined process to work out whether what worked ‘there’ is going to work ‘here’.

So in the rest of this post I am, once again, going to lean on the work of Kvernbekk (2016)  to examine the process you might wish to undertaken before adopting even the ‘most evidenced’ intervention.

Read More

Models of organisational development - What type of culture does your school or college have?


Earlier this week I came across this headline - which although describing further education colleges - I also get the impression that many teachers in schools would recognise similar 'Soviet' style cultures.


So in this post I’m going to write about Patterson, Nolan, et al. (2011) and two models of organisational development - perform or perish and responsive and relational  which were developed from research undertaken in the health sector and which are described in Table 1.

Table 1 Two models of organisational development – Perform or Perish -  Responsive and Relational

Perform or perish
Responsive and relational
Pace: Quick fix, short term, process driven, pushing and fixing

Relational and responsive Complexity: Longer term, focus on
people and perceptions, brokering

External: Top down agenda, local context largely overlooked, off-the- shelf, one-size fits all approaches applied

Locally contextual factors fully acknowledged and addressed, solutions tailored to situation, existing models modified accordingly

Select few determine goals and direction of change
All groups including users/carers involved in deciding goals and direction of change

Punitive and transactional leadership style from top, little unit level leadership

Empowering, inspiring and transformational leadership style at all levels, especially unit

Metrics matter: Superficial, often quantitative targets for success, e.g. patient flow

Meaning matters, relational, dynamic qualitative ‘indicators’ of success, peoples’ experiences

Scored

Profiled
Impoverished change environment results and the ‘senses’ are reduced

Enriched change environment results and the ‘senses’ are enhanced


What’s the relevance for school leadership and management?

Whilst both of these models are ‘idealised’ extreme cases they do provide a useful way for thinking about different approaches to organisational development.  Although for use in the health-care setting; the various dimensions;  be it focus – short-term vs long-term; external demands vs internal context; scale of engagement – the few vs the many; impoverished vs enriched may ring a few alarm bells for colleagues working in schools.  Especially if my Twitter timeline is as anything to go by.  So what are the implications for teachers and school-leaders who wish to bring about evidence-based/informed/enriched schools.

First, the perform or perish model of organisational development seems culturally at odds with developing and evidence-based school.  Punitive and transactional leadership is not going to create the conditions of trust necessary for teachers and schools leaders to challenge existing ways of working.

Second, to develop and evidence-based school will take time, it’s not a quick fix, and will require a recognition of the complexity of the issues to be addressed. It will require multiple iterations of what we think might work to come up with ways of working which are tailored to the particular requirements of a school.

Third, the evidence-based school is not just about teachers accessing, reading and implementing the latest reports of the Education Endowment Foundation.  Just as important, if not more so, is to engage in genuine and meaningful partnerships with pupils, parents and the community.  Their views and perceptions are an integral part of the evidence-based use to help make decisions.

Fourth, no one action or group of actions will lead to a school becoming responsive and relational, it will require a combination of interventions and challenges across the whole-range of the school’s activities, in order to ensure such an approach is part of the organisational DNA of the school.

And finally  

This model and way of thinking about organisational development may be particularly relevant for those in involved in the senior leadership and management of multi-academy trusts and help those leaders avoid the pitfalls of a perform or perish organisational culture.

References

Andrews, N., Gabbay, J., Le May, A., Miller, E., O'Neill, M. and Petch, A. (2015). Developing Evidence Enriched Practice in Health and Social Care with Older People.

Patterson, M., Nolan, M., Rick, J., Brown, J., Adams, R. and Musson, G. (2011). From Metrics to Meaning: Culture Change and Quality of Acute Hospital Care for Older People. Report for National Intsitue for Health Research Service Delivery and Oranisation programme.

The school research lead and critical thinking

Attending conferences and critically appraising what you hear, see and read is on its own not enough to make better decisions about teaching, learning and the leadership and management of schools.  What you see and hear needs to be linked to your own expertise, but also the context your school and the needs and preferences of pupils, teacher, parents, governing bodies and other stakeholders.  Drawing upon the work of Jenicek, Croskerry, et al. (2011) in health-care I am going to adopt their argument, that critical thinking and decision-making can be seen as the means for making that link.

Critical thinking

Jenicek et al use a definition of critical thinking which was developed by a range of experts in critical thinking from a number of subject, fields and disciplines and which states that it should be defined as “purposeful self-regulatory judgment which results in interpretation, analysis, evaluation and inferences, as well as the explanation of the evidential, conceptual, methodological, criteriological, or contextual considerations upon which that judgment is based.’ (p 12-14).

However, Jenicek at al argue that what’s even more useful than a definition of critical thinking are the skills and abilities which underpin critical thinking in practice.  Table 1 provides a list of the specific abilities underlying critical thinking in medical practice, which should be equally applicable to schools and other educational settings

Table 1 Specific abilities underlying critical thinking in medical practice
  • Understand the principles of argumentation
  • Knowing and understandings dual Systems 1 and System 2 thinking processes and their interactions
  • Awareness and understanding of evolutionary influences on decision-making
  • Recognizing distracting stimuli, propaganda, bias and irrelevance
  • Identifying, analysing, and challenging assumptions in arguments
  • Awareness and understandings of the impact of the cognitive fallacies and poor reasoning
  • Awareness and understanding of the major impact of the cognitive and affective biases on thinking
  • Recognizing deception, deliberate or otherwise
  • Capacity for assessing credibility of information
  • Understanding the needs for monitoring and control of one’s own thought processes
  • Understanding the importance of monitoring and control of one’s own affective states
  • Awareness of the critical impact of fatigue and sleep deprivation on decision-makings
  • Imagining and exploring alternatives
  • Capacity for effectively thinking through problems
  • Understanding the importance of the context in which decisions are made
  • Systematic and effective decision-making
  • Understanding the dynamics and properties of individual vs group decision-making
  • Capacity for anticipating the consequences of decisions


Jenicek et al p14

What are the implications of this model of critical thinking for teachers, school research champions, and senior school leaders?

First, it would be wrong to assume that all staff have the same level of critical thinking skills. Both new and existing educators will need to be trained in critical thinking – so as to help them become more able in integrating different sources of evidence into a coherent decision and plan of action.

Second, teachers, school research champions and senior school leaders may wish to engage in some form of reflection about their current level of expertise in critical thinking.  A couple of useful places to start would be to ask yourself – what do I know about both argumentation and the ladder of inference?  If the answer is nothing, then exploring both these concepts will all likelihood be rewarded with improved critical thinking and decision-making.

Third, the opportunities for developing critical thinking can be easily found throughout, the school day, week or year.  School leaders can use conversations in staff rooms to help understanding the underlying principles of both their own and other peoples’ perspectives.  Departmental meetings may be used to challenge the assumptions of how certain subjects and topics are taught.  Journal clubs may provide opportunities for developing critical thinking, although it should be emphasised that critical thinking is not limited to critically appraising research

Four, new interventions and existing programmes should be subject to critical review to help understand whether there are cognitive biases or affective states which have had an inadvertent negative impact on decisions and associated outcomes.

And finally

Future posts will explore in more detail the principles of argumentation and in doing so will be looking at the Toulmin model of arguments and will look at the work of Kvernbekk (2013, Kvernbekk (2016) in how this model can be applied in educational contexts.

References

Jenicek, M., Croskerry, P. and Hitchcock, D. L. (2011). Evidence and Its Uses in Health Care and Research: The Role of Critical Thinking. Medical science monitor: international medical journal of experimental and clinical research. 17. 1. RA12.
Kvernbekk, T. (2013). Evidence-Based Practice: On the Function of Evidence in Practical Reasoning. Studier i Pædagogisk Filosofi. 2. 2. 19-33.
Kvernbekk, T. (2016). Evidence-Based Practice in Education: Functions of Evidence and Causal Presuppositions. London. Routledge.


The school research lead, confirmation bias and unknown unknowns

A few weeks ago I had the privilege of attending Professor Chris Brown's @ChrisBrown1475 inaugural lecture at Portsmouth University.  One of the great things about attending such events is that you get the chance to talk to some very interesting people, for example, Ruth Luzmore a primary headteacher at an inner London all through school.  Now my conversations with Ruth - by now online -  led onto a discussion about known unknowns and unknown unknowns has led me to a re-visit some work I had done on this topic in trying to discover the things that we don't know we don't know.For as US Secretary of Defense Donald Rumseld famously, or should I say infamously, said:

There are things we know that we don’t know. (And) there are known unknowns.  That is to say there are things we know we don’t know.  That is to say they are things that we now know what we don’t know.  But there are also unknown unknowns.  These are things that we do not know that we don’t’ know.’  

Feduzi & Runde (2014) argue that we are particularly bad at looking for things beyond what we already know, in that we are prone to confirmation bias i.e. we look for evidence that will confirm what we already know.  In addition, we also tend to be too conservative in our predictions of likely outcomes, which can lead to a clustering of predictions which tend to be over optimistic.

In order to address these problems associated with cognitive bias Feduzi & Runde (2014) have put forward a technique which seeks to expand the number of possible scenarios and at the same time, look for evidence to support those scenarios.   The process is made up of several steps
  • Think of three main scenarios which you can envisage being the outcome of a decision – things improve, pretty much stay the same, or get worse
  • Place those three outcomes on a favourability scale  
  • Now try and imagine the worst possible scenario, where things don’t just get worse there is a total collapse - in other words a scenario which is 'completely off the-scale' 
  • Having imagined this scenario, try and find evidence that might make this worst possible scenario a possibility
  • Then try and imagine a scenario where success is beyond your wildest dreams,  then go and search for evidence that would make this scenario a possibility.
Feduzi and Runde argue that by doing this then you are likely to discover information that you did not previously know about, and in doing so, you will have uncovered some unknown unknowns.  This uncovering of unknown unknowns is the product of seeking to discover evidence that confirms alternatives, rather than seeking out evidence which rules out alternatives.  By searching out for information which seeks to confirms alternatives provides a counter-point to confirmation bias associated with disproving hypotheses.

So what are the implications for you the school research lead.
  • Recognise that there are things that you don't know that you don't know
  • When thinking through scenarios - it's important to engage a range of individuals - so that the constituent components of either total success or failure can be explored.
  • Try and use a range of techniques - that can help you mitigate the impact of cognitive biases - be it premortems or decision check-lists 
And finally none of the material I've referred to in this post have their origins in education - maybe if we spent a little more time looking at other fields - be it knowledge management, improvement science or implementation science - may we can get to know a lot more about what others know.