Research methods

Research aims and methods

Summary of research activity

The report works with the theory of change pdf=328kb developed for Big Local (2013) in that it aims to capture:

  • the diversity of the initiative in terms of structures, processes, local contexts and goals. More detail on that diversity is presented in Section One of the full report which offers brief pen portraits of the 15 Big Local partnerships involved in Our Bigger Story,
  • the factors which enable and/or potentially hinder progress against outcomes and long term goals,
  • the flexible, non-linear nature of the programme. Change takes place within resident-led timescales, encourages risk taking and is not ‘measured’ against specific annual targets or spend patterns,
  • Big Local as a learning system: what works in resident-led regeneration and, crucially, why does it work?

The findings in this report are based on:

Table 8: Our Bigger Story: Summary of evaluation activity 2015-16

Evaluation Activity Year 1 (2015) Evaluation Activity Year 2 (2016)
Review of existing Big Local data – including community profiles and plans as well as materials (such as film, podcasts and photographs) available in other media Ongoing reviews of Big Local data (e.g. Plan Reviews).
102 individual interviews with key stakeholders - including resident partnership members, workers, delivery partners, Locally Trusted Organisations, volunteers and ‘beneficiaries’ (as described by a number of partnerships), 55 individual interviews and conversations, with partners, delivery partners, recipients of small grants and local people accessing Big Local supported projects, and Local Trust
Observation sessions at partnership meetings/plan reviews and Big Local events, ten focus groups involving 54 resident partnership members, delivery partners and beneficiaries. 15 facilitated thematic discussions (i.e. all 15 Big Local case study sites), involving 157 participants 13 observation sessions at partnership meetings/plan reviews and Big Local events
Four networking workshops in Ramsey, Leeds, Birmingham and Hanwell to explore emerging Big Local themes and outcomes involving 14 areas
Focus Groups with Big Local reps (two) and Local Trust staff
Resident partner diaries in seven areas

Our Bigger Story has also built a website which brings together:

  • Materials (in different media) produced by Big Local areas themselves (www.ourbiggerstory.com). There are currently 238 of posts across areas on the website. (A ‘post’ may be a single item such as a film or podcast, or multiple materials (e.g. up to 12 photographs per event.)
  • 34 films, available on a dedicated vimeo channel produced by the evaluation team in local areas
  • Seven summative films addressing the issues raised in this report.

This site, therefore, allows for the tracking of change in Big Local areas over the life of the programme and can be searched either by media format (e.g. film/audio etc.) or against a time-line of when things happened.

Research aims

This report maps progress in, and additional learning from, the evaluation sample of 15 Big Local areas 2015 to 2016. It takes the story of Big Local forwards from the Early Years pdf=2.76mb research undertaken by NCVO during 2014 (NCVO, 2015) and builds on other subsequent evaluation reports such as Community Engagement in Big Local pdf=595kb (NCVO 2016) and People places and health agencies pdf=458kb, (Institute for Voluntary Action Research report 2016).

This report aims to explore:

  • the five overarching elements of evaluation identified by Local Trust: namely, the approach, delivery, outcomes, impact and influence of Big Local
  • the principles which underpin Big Local as a resident-led, asset-based approach to neighbourhood development and regeneration
  • the specific tasks outlined for Our Bigger Story within the integrated evaluation framework developed by Local Trust (March 2015) in terms of understanding Big Local processes and outcomes.

These are summarised in Table 9, below.

Table 9: Our Bigger Story: Core Evaluation focus

EVALUATION - PROCESS FOCUS EVALUATION - OUTCOME FOCUS
Resident-led (approach) Residents better able to identify and prioritise needs (outcomes)
Asset-based (approach) Residents better able to respond to needs (outcomes)
Taking place at community pace (approach) People have increased skills and confidence (outcomes)
Opportunities for reflection (approach) Residents improve Big Local areas in ways that matter to them (outcomes)
Taking risks (approach) A better place to live (outcomes)
Local Trust makes funding available in ways that are accessible and user friendly (delivery) Infrastructure supports resident decision making (impact)
Lasting and sustainable change in communities (impact)
Government policy facilitates resident-led decision making (influence)

In addition, in 2016, the evaluation team introduced five key research themes to ‘unpack’ learning around process and outcomes, as agreed with Local Trust. These related to Big Local areas capacity to negotiate change, the extent to which they are a catalyst for change in their areas, the development of community leadership and influence, their experience in dealing with expectations around ‘achievement,’ and understanding about sustainability and legacy. These are explained further in Section 4 of the current report.

Research methods

Big Local aims to bring about positive change in communities. The aim of Our Bigger Story, as a longitudinal multi-media evaluation, is to record that change, as well as the learning involved, in multiple ways in 15 Big Local areas over time. Considerable attention was given during 2015, the first year of the evaluation, to building the architecture to support what is an ambitious, complex, and long-term study. This has involved:

  • Selection and recruitment of a sample of 15 Big Local areas to participate in the research
  • Web-site and data-base development
  • Multi-media training and support for participating Big Local areas.

Selecting and recruiting evaluation sites

The evaluation is being undertaken in and with 15 Big Local areas, representing 10% of all the communities covered by the Big Local programme. These were selected through a short baseline survey of all 150 Big Local areas carried out between February and May 2015. The survey was designed to assess interest in involvement in Our Bigger Story and to identify the range of social media technologies currently being used by areas. It was accompanied by taster workshops at the 2015 Big Local spring events, attracting 51 participants.

In total 35 Big Local areas expressed an interest in participating in the evaluation. These were assessed against criteria agreed with Local Trust which were designed to ensure a balanced sample covering, for example, all regions in England, a mix in terms of urban/rural/coastal areas and differing population sizes. The final group of 15 selected evaluation sites includes:

  • Big Local areas in all regions in England
  • nine urban Big Local areas, three rural and three coastal
  • five ‘wave 1’ Big Local areas, six from ‘wave 2’ and four from ‘wave 3’
  • Big Local areas with populations ranging from around 2,400 residents to around 11,500 and an average of 6,500.

Inception meetings were held with each of the 15 selected Big Local areas, in order to ensure that each had taken a strategic, collective and informed decision to participate. Big Local areas asked questions around confidentiality, time expectations and the availability of support from the evaluation team, ownership and archiving of evaluation materials. Residents were particularly concerned about the film making aspect of Our Bigger Story as the period of negotiating access to areas coincided with the airing of Benefit’s Street on television and substantial time has been dedicated to :

  • building trust with partnerships and residents in terms of the content of any public facing multi-media material
  • delivering multi-media training and offering grants of up to £500 to purchase equipment as a way of encouraging areas to develop their own materials. Media training events were held in September 2015 in Birmingham, London and York, involving 69 participants in total, to give Big Local areas hands-on experience of film machining, creating digital stories and podcasts as well as being introduced to creative visual/written methods of evaluation. All 15 evaluation areas attended with representatives from an additional five areas that had also expressed initial interest through the baseline survey. All also received the equipment grant (up to £500) for the amount requested by early 2016.

Web-site and data-base development

Our Bigger Story has started to construct an integrated public facing website which will document multi-media evaluation materials from all 15 Big Local areas over the life of the programme (see www.ourbiggerstory.com). This is searchable by a time-line, by the media used and against Big Local outcomes. It enables the evaluation team and individual Big Local areas to up-load materials (video, audio, photographs and reports) that are for wider circulation and do not include information that could be deemed to be confidential. There are currently 238 ‘posts’7 from the 15 Big Local areas on the website alongside a vimeo channel (The Residents Stories) with 34 films made by OBS over 2015-16.

There have been a number of teething problems with the website. For Big Local areas with slow broadband speeds, uploading films can be extremely time consuming. Internally, the practice of uploading materials has identified a number of layout problems – in particular the matching of text with photographs of different sizing. Both these issues are being addressed by Creative Media through advice to the areas and web-redesign.

Behind the public facing website, systems for secure online data storage and management have been developed as part of building the foundations for the longitudinal study. Given the volume (and file size) of visual materials additional secure storage has been purchased from the University. This has the capacity to preserve a decade’s worth of written, audio and visual data.

Evaluation activities year one (2015)

Evaluation activity over year one consisted of:

  • a review of existing data from the selected Big Local evaluation areas, including community profiles and plans as well as materials available in other media (such as existing films, podcasts and photographs),
  • individual semi-structured interviews with 102 key stakeholders involved in the Big Local areas - including resident partnership members, workers, delivery partners, Locally Trusted Organisations, volunteers, local businesses and beneficiaries,
  • 85 filmed interviews/observations in ten Big Local areas.
  • observations of six meetings and events, such as partnership meetings, plan reviews and wider stakeholder and community events. Filming and observation sessions have not been ‘double counted’ in the description of research activity,
  • ten focus groups involving 54 resident partnership members, delivery partners and beneficiaries, and
  • pilot social media analysis in three Big Local areas.

The focus in year one was on gaining a baseline understanding of each of the 15 areas: their histories, the patterns of (and learning from) development behind Big Local and what they hoped to achieve. The methods adopted relied largely on more traditional evaluation methods (e.g. taped/transcribed individual and focus group interviews) supplemented by film making and other multi-media activity (e.g. podcasts). This data informed the Interim Summary Report pdf=592kb (with a more detailed paper submitted to Local Trust) published in April 2016.

Evaluation activities year two (2016)

Year one involved substantial learning for the team around the integration of multi-media methods into a longitudinal evaluation. In discussion with Local Trust and the 15 areas involved, a slightly different approach was adopted in 2016 in that Our Bigger Story:

  • made more use of film as a means of recording both individual interviews and group activities/meetings. There are now a series of ‘In Conversation’ films on the OBS website.
  • integrated film into the evaluation process as a way of stimulating discussions between the Big Local areas involved.

In addition, the focus for data gathering (post base-line) changed. In addition to gathering information on area’s progress against outcomes a more thematic approach was adopted to capture data around the themes of leadership, influence, expectations, legacy and Big Local as a change mechanism.

Research activity has, therefore, focused more on ‘whole partnership’ facilitated workshops (with fewer individual interviews with partners and residents), recording activities supported by Big Local areas (e.g. the use of small grants) and the use of ‘cross partnership’ events to bot stimulate discussion and gather data. Over the year this involved:

  • audio or video recording partnership meetings in all 15 areas
  • whole partnership thematic workshops in each area attended by 163 people, predominantly local residents
  • cross partnership events were attended by 48 members of local partnerships. Each session lasted up to three hours (using films from each Big Local area to stimulate discussion). Edited films of these sessions in Ramsey, Leeds, Birchfield and Hanwell have been made and are available on the OBS website
  • observations at 13 partnership meetings
  • two workshops with Big Local reps attended by 35 people. The first of these reflected on the wider applicability of the findings from the Interim report whilst the second, specifically, focused of aspects of risk and risk management in Big Local areas.
  • One focus group with Local Trust staff to reflect on the emerging findings from thematic analysis
  • One interview with Debbie Ladds, prior to her leaving her post as Local Trust chief executive
  • 62 interviews in the areas with partnership members, delivery partners, recipients of small
  • grants and local residents accessing services supported or delivered by Big Locals.

In addition, residents in seven areas kept reflective diaries on their experience of being Big Local partners. An edited film version of these diaries is available as ‘Diary Lines’.

DOWNLOAD 2015-2016 REPORT

You can download the full printed report as a pdf document. The pdf document is 1.5MB. You will require Adobe Acrobat Reader to view and read.