I have recently volunteered to help an embryonic Domestic Abuse project develop and implement an evaluation framework.
This got me thinking about the value of evaluation and writing a blog on this very subject.
Perhaps you’ll find something in this blog to encourage you as a domestic abuse or community safety practitioner to build self evaluation into your projects?
Do let me know your thoughts!
What is Evaluation?
“informing the development of policy and practice …. a means of testing and developing social theory …… the question asked ….. is; What works for whom in what circumstances?” (Nick Tilley (2000) presentation at Founding Conference of the Danish Evaluation Society)
“programme evaluation is the systematic collection of information about the activities, characteristics, and outcomes (or results) of programmes to
- Make judgements about the programmer
- Improve programme effectiveness, and/pr
- Inform decisions about future programming” (M Q Patton)
“evaluation is all about questions …..[it] has a reputation for being technical and difficult- in practice it is part of every day life and we do it many times a day without realising ….. The kind of questions that the evaluation of projects seeks to answer include:
- Did we do what we planned to do? If not, why not?
- What worked? Why?
- What did not work? Why?
- What difference, if any, does the project make? How have things changed over time and for whom ?
- Is the project meeting needs? Whose needs?
- What do users/members/beneficiaries think about the work?
- Is the donor’s money well spent? Is it achieving what the donor intended?
- What could we do differently?
- How can we use the learning to develop the work?” (www.evaluationtrust.org)
When to Evaluate?
Evaluation is appropriate at any time before during or after a programme or intervention. This is different to a monitoring or assessment process which necessarily must take place after activity has been undertaken.
There are lots of benefits from conducting evaluation
- provide information about what has been achieved
- judge and demonstrate the merits or worth of a programme or intervention (did it make a difference and to who?)
- provide recommendations for change or improvements
- provide information to inform organisational development, planning and management
- set service standards
An evaluation can also help to
- justify costs and expenditure (value for money)
- help in the development of cost benefit analysis and business cases
- support the case for sustainable funding
The evaluation process
The evaluation trust has this useful summary of the process:
- engage key stakeholders
- define project aims/outcomes
- define purpose of evaluation
- determine focus – the questions it needs to answer
- specify the timescale, resources and who is involved
- describe the work of the projects and how you will know you are achieving your aims and outcomes
- chose evaluators
- select information collection methods
- collect information
- analyse and write up results
- use results to make changes internally
- disseminate results
Domestic Abuse Partnership Area Self Assessment (Audit Commission)
Evaluation can look at different questions of needs, process, outputs, outcomes and impact.
The Audit Commission published an excellent toolkit Domestic Abuse Partnership Area Self Assessment 6 days ago.
This tool is self-evaluation tool which ask questions mainly relating to process and outputs. In other words it asks whether and how certain activities are being undertaken and what type and nature of programmes and interventions are being carried out.
Considering the toolkit in terms of the evaluation process set out, it is possible to identify the following stages:
Stage 2. The aim the evaluation is to review local approaches in the context of financial and organisational change and Home Office ambition to end violence against women and girls.
Stage 3. The purpose of the evaluation is to ensure that partnerships have clear priorities.
The detailed theory is that partnerships
- may want to focus more on using mainstream staff and resources.
- will want to show existing managers as well as new health and police commissioners how domestic abuse services can support mainstream statutory work
- will want to demonstrate how domestic abuse services supporting mainstream statutory work save money for partners as well as helping victims.
Stage 4. The audit commission has determined focus by reviewing evidence and literature reports, information and comments from individuals at the Audit Commission and other inspectorates with experience of aspects of domestic abuse work and visits to eight areas to interview local practitioners involved in tackling domestic abuse.
Stage 8. The questionnaire part of the toolkit is one information collection method. Partnerships could enhance this evaluation with a “rich picture” by undertaking some semi-structured interviews and considering quantitative data.
Where evaluation could have made a difference?
Policy Exchange published the report Cost of the Cops, Manpower and Deployment in Policing two days ago.
As reports go this generated controversy and debate. It argued, amongst other things for the increased “civilianisation” of the police service stating “There is still substantial room for increasing the use of civilians in Policing inEnglandandWales”.
Greater Manchester Police Chief Constable Peter Fahy on the other hand speaking for the Association of Chief Police Officers (Acpo), said to the BBC “many office-based jobs required trained officers. It is crucially important the police officers are used in roles which require their expertise, powers and experience. That said, this doesn’t just apply to the front line, there are many office-based jobs where police officers are required, including handling intelligence, delivering training, or processing offenders through the criminal justice system.”
Why is there such a mismatch of positions on some of the key findings of the report?
There’s nothing fundamentally wrong with the analysis or data. Had the project team taken a more evaluative approach some of the controversy could have been avoided. In particular:
- there is nothing to suggest that the project engaged key stakeholders at the outset (Stage 1)
- although there is no detailed methodology set out the information collection methods seem limited to secondary analysis of quantitative data so instead of asking challenging questions (eg could police officers wear their uniforms to work?) in qualitative research these are being made as recommendations which are now the subject of criticism from many (Stage 8).
- Results are unlikely to result in changes to the police service (Stage 11)
- Evaluation is an excellent tool to use in difficult times of limited resources to consider what really works locally and in support of sustainable funding for projects.
- Evaluation can be long and complex and can also be short and simple depending on available resources and information collection methods.
- Evaluation which is built into a project can ask questions as the project is delivered which is a cost effective approach.
- Effective evaluation requires a partnership between the evaluators, the project team and service users