Applied Theatre – measuring what works and what matters (and really believing it)

Believe me>

I know, I feel and I see that what Odd Arts does – works. It changes people’s way of thinking and seeing the world; it opens up new opportunities through self-belief and realisation of personal ability; it empowers people to find ways to challenge the things or people that harm us. Proving this is another matter.  Of course, we should have to prove we make a difference. There is ever growing scepticism of charity morals and we are at a time of national austerity – proving our impact is necessary and right.

Within its 15 year history, Odd Arts has of course evaluated it’s work. We have used a lot of techniques: Before and after questionnaires;  Case-studies; Recording indicators for change (all the usual suspects). Often showing a journey of real change and impact. However, there have been a few things happen in 2018 that made me really question how authentic this is, resulting in a complete shift in attitude towards data collection, evidence and evaluation.

Did I always believe what our data told us?  No.  Did it mean we did our job better and served our beneficiaries better? No. Did it help us tick boxes and gain funding? Yes. Our past response:  “It’s out of our hands”

This response is not reflective of the values of our organisation.

This is why we have changed.
 

Telling us one thing, but knowing another…

Let me explain. Recently I took part in Social Value UK’s Impact Management Training. It gave me a little time to sit back and think about the WHAT and the WHY of evaluation and reflect on some of our findings. It made me think in particular about a recent project and the data it produced:

In this project, we worked with men in a community rehabilitation unit (Langley House Trust) delivering workshops around issues relating to resettlement. The male ex-offenders all developed their own interactive forum theatre piece based on their lives and experiences, in order to re-imagine their future and resolve past issues. The project was everything we hope for as applied theatre practitioners: Meaningful; personal; supportive and with the capacity to be life-changing. However, the questionnaires from the men told me that they had less confidence; fewer employment skills and a reduced ability to work with others. I knew this simply wasn’t true.

Similarly, I have worked on projects where participants on paper show a zero to hero transition, suggesting they had no ability at the beginning of a project, yet on completion were now demonstrating exemplary employment skills and sky high confidence, when you really know this journey was no where near as black and white and was, in fact, far more subtle. The self-evaluation questions we have used for years rely on the ability to honestly and effectively reflect on our personal state of being, participants mark themselves between 0 and 6 on their skills in different areas. But what if, at the beginning of a course, the participant has never reflected on their well-being and confidence, and has no idea what critical thinking skills are? Surely by participating in a project that builds these skills enables them to understand what they mean. Surely part of that journey is the forming the ability to effectively and realistically evaluate their own being and behaviour. Sometimes we complete evaluations at the end of a performance and presentation, when spirits are high and participants are buzzing … but what about 5 days later when they are alone in their prison cell, would the answers be the same?

What does this tell us about the traditional questionnaires? Maybe they’re meaningless. Maybe they’re not. Definitely they’re unreliable as standalone evidence.

This is why we have changed.
 

It’s out of our hands … or is it?

Another experience influencing our shift in attitudes in responding to data was being part of two research reports (with Manchester Metropolitan University and UCLAN). The researchers from UCLAN’s The Psychosocial Research Unit recognised real benefits to the participants involved in our Creative Leadership programme including:

• ‘Raised self-esteem, enabling re-evaluation of past ‘failures’ and future opportunities;
• Developed awareness of imaginative approaches to life and problem-solving;
• Demonstrated to participants that creative activities offer opportunities for self expression and peer recognition;
• Enabled participants’ voices to be heard through theatre increasing their ability to communicate;
• Enabled participants to share and reflect upon apparently intractable issues that might otherwise have remained unexpressed;
• Developed a feeling of empowerment and personal authority, highlighting aspects of leadership other than rule-bound discipline;
• Increased appreciation of the nature and benefits of teamwork;
• Provided an opportunity for pleasure, freedom of thought and a sense of release that was obtained through being able to participate in the creative process’;

HOWEVER, they also raised points regarding things they thought could be better or were providing potential setbacks in the programme, these being:

1. The style of how we encouraged participants to complete accreditation was in complete opposition to the creative, inclusive and exciting style of delivery we pride ourselves on. We always aim to make learning fun and experiential embedded completely in the creative process. Our partners asked us to use workbooks they provided to take participants through an accreditation module. It was evident in our delivery we didn’t ‘believe’ or ‘buy in’ to the workbooks we used. Odd Arts facilitators often tried to persuade learners to complete them through statements like “just write this out quickly and then we can do something fun” …  The underlying message:  Education and qualifications are dull. How did we respond to this feedback? “It’s out of our hands, their books have nothing to do with us”
2. Partner staff often posed challenges and could even hold back the achievements of our programme by making obstacles, negative or destructive comments or mocking the young people when they joined in (an experience many applied theatre practitioners are up against). How did we respond to this feedback? “It’s out of our hands, their staff have nothing to do with us”
3. Partners often banned particularly contentious discussion topics, whilst allowing others.  To be more specific, in one unit where we were working with children who had been victim and perpetrators of sexual violence we were told to under no circumstances talk about this issue.  We were however allowed to discuss issues such as gang violence, drugs, bullying and other offending behaviour.  Everyone (including the children) knew we must know about the sex-related crimes and we all knew no-one would bring it up…  It became the elephant in the room, especially when discussing issues around ‘personal barriers’.  The underlying message:  This is something to be ashamed of and that should never be spoken about, and that other violent or gang related behaviour was more acceptable.  How did we respond to this feedback? “It’s out of our hands, their policy has nothing to do with us”
4. Partners might feel threatened or fearful of our work:  Will it damage the young people they work with?  Will it show them up?  Will it mean their job or role has less meaning?  How did we respond to this feedback? “Nothing to do with us …”

Our response to all of the above points is not ‘restorative’, is not ‘strengths based’ and is  absolutely not reflective of the values of our organisation.  This is why we have changed.
 

“It is in our hands and has everything to do with us” …

So, we say we have changed. Here’s how.

Accreditation: What do we do now?

• Follow the approach we have to wider learning which throws out of the window traditional desk based learning and use experiential, visual, audio, and creative ways to measure and record the criteria needed for gaining accreditation
• If the qualification can only be achieved through desk based write ups then it’s not for Odd Arts
• We have created a series of art-work postcards that help participants complete the evidence based element of the accreditation.  They are nice to touch and look at and something to be proud of and keep

Evaluation: What do we do now?

• Don’t evaluate anything we believe is meaningless or box ticking
• Record things we feel are meaningful but may not be obvious
• Evaluate to better ourselves and our service and not only for funding
• Don’t promise to ‘prove’ change beyond our means
• Scrap a ‘blanket approach’
• Select data collection methods from a series of evaluation mechanisms that can swap and change to meet the needs of our participants
• Ensure evaluation is inclusive, embedded and creative
• When we know we can’t provide evidence to the level needed, stick to what we’re good at (delivering the programme) and find funding for external researchers
• Ensure our staff understand the aims of Odd Arts and their projects so that data collection can be steered towards answering these
• Be open to learning things we didn’t anticipate: The unexpected outcomes
• If we don’t care we don’t record it
• Be open to change in response to what the data tells us (we will make mistakes and from this, get better)

 

Partnerships: What do we know now?

Before delivering any key programme of work we now incorporate an introductory training and vision sharing session with partner staff. The aims of this are to:

• Build trust and positive relationships with partners
• Outline expectations and roles from both sides
• Include partners in the programme plan, value their experience and knowledge
• Listen and respond to the needs of partners
• Be willing to learn from partners
• Provide an experiential understanding of how we work and the theories behind it to reduce fear (with the view that we might even be allowed to address the elephant in the room and reduce risk of reaffirming shame and stigma)
• Allow space for potential challenges to be discussed and find potential solutions to it
• Inspire creativity with partners

 

Why did this take so long?

Reflecting on the impact of the external factors influencing your work is something many of us (including myself) rarely find the time to do.  With the best intentions we focus on programme content and learner needs. But if the impact of external partners, accreditation and data collection is done in a way that is detrimental to our delivery then we need to find the time to respond to this. We also need to be honest about which data has the potential for real impact.

Odd Arts is an organisation that prides itself on a restorative approach, creativity, innovation and inclusivity. As long as we extend our values to every part of our organisation – even the boring bits 😉 – then we create more opportunities for ourselves, the people and the organisations we work with.