Back to all blogs

Measuring What Matters: The S²Cities iMEL Framework

Insights from EAFIT University, the global monitoring, evaluation and learning (MEL) partner for S²Cities

S²Cities is a global programme aiming to improve the safety and well-being of young people in urban environments in secondary cities. The programme seeks to better understand urban systems and their influence on safety and well-being, build capacity within these systems, and enable youth innovation to solve critical challenges for safer urban environments. S²Cities will also establish a global learning network of cities, young people, thought leaders, and practitioners to foster a better understanding of urban safety for young people.

 

To ensure that the programme is meeting these objectives and achieving its desired impact, its implementation needs to be monitored and evaluated regularly against specific quantitative and qualitative indicators. Moreover, since S²Cities is a multi-phase programme, it is critical to periodically collect learnings and feedback to improve future phases of the programme. 

To achieve these objectives, S²Cities utilizes an integrated Monitoring, Evaluation and Learning (iMEL) framework, created and implemented by EAFIT University in Medellín, Colombia.

In this article, Manuela Ramirez, a member of the EAFIT MEL framework team, explains the iMEL framework and how it tracks improvement and change. 

 

The iMEL team discusses the framework with members from Fondation Botnar and GIB.

 

Figure 1: The EAFIT team discusses the iMEL framework with GIB, ICLEI and Fondation Botnar (Source: Kelly Donovan)

 

What is the iMEL system? 

The iMEL system is the integrated monitoring, evaluation, and learning scheme for S²Cities. It unites, manages, and analyzes the programme’s data, integrating evidence-based and participatory insights on the drivers and limitations to building urban safety for young people. The iMEL framework is evidence-based, relying on insights from the statistical analysis of quantitative data. At the same time, its participatory approach aims to generate “knowledge for action” through bottom-up data collection that focuses on locally defined priorities and perspectives. 

 

Who is the framework for?

The iMEL serves as a guide and articulation point for donors, programme partners, participating cities, and youth beneficiaries to track the progress of the programme towards achieving expected outcomes for each pathway of change defined by S²Cities.

 

What are these pathways of change?

The S²Cities Theory of Change (ToC) considers four pathways of change to improve young people’s safety.

Figure 2: The four pathways of change highlighted in the S²Cities Theory of Change

 

1. System understanding: 

 

To design interventions that effectively address young people’s inclusion and safety, it is crucial to first understand how young people act in public spaces, how they build relationships with other community members, how public and educational institutions support or limit these interactions, etc.

 

Therefore, the first pathway of change is identifying these pre-contextual elements, which will allow programme partners to identify and implement interventions that respond to young people’s needs and their unique limitations. 

The System Understanding pathway also encourages young people to participate in identifying these pre-contextual elements. 

After all, who could be better to determine what affects young people’s safety than young people themselves? In this stage, partners apply different diagnostic tools like surveys and Geographical Information Systems (GIS) to gather data on youth behaviour in relation to their urban environments, among other aspects. This approach allows local partners to identify tangible requirements for improving youth safety. 

 

Additionally, the programme recognizes that the characteristics, actions, and relations of people and environments are dynamic in nature. The flexibility of the ToC helps S²Cities to adapt to the particularities of these systems and account for this dynamic nature when analysing the factors that catalyse change.

 

2. Capacity building and empowerment: 

 

Capacity-building is a self-explanatory term — it is the act of building the capacities of people and groups to “perform functions, solve problems, and achieve objectives [1].” The previous stage of System Understanding identifies weaknesses in young people’s urban safety along with potential opportunities for change. Insights from this stage can help programme partners design capacity development to equip young people to:

 

i) Overcome limitations and have a voice in urban decision-making.
ii) Create and promote innovative solutions to the prioritised safety issues in the city.

These tools are meant to unleash youth creativity and curiosity, developing their capacity to innovate and their potential to influence, lead, and educate subsequent generations.

In this manner, these tools posit young leaders to create sustainable strategies that address diverse global issues [2]. Therefore, capacity building empowers young people by facilitating deliberative and problem-solving spaces and equipping them to use tools to devise solutions to urban safety.

 

 

Figure 3: Local partner GERAK conducts a capacity-building session for young people in Bandung, Indonesia (Source: Kelly Donovan)

 

3. Innovation and youth-driven action: 

 

Innovation is one of the most crucial tools for driving social change. Pairing innovation with urban planning opens up a plethora of potential opportunities — from new developments and improvements that solve urban issues, to new ways of adapting to the complex challenges of today’s cities.  

The main activity of this stage is establishing an innovation incubator for young people to develop and share ideas that address safety challenges. 

This is a space where youth can collaborate with their peers, receive advice from experts, access valuable tools to develop their projects, and often get grants to materialise them. These spaces for innovation refer not only to physical spaces but also to the time, opportunities, and tools necessary for innovation [3]. An innovation incubator empowers young people by providing experiences, tools, and opportunities that help them explore and apply their capacities as change-makers [4].

 

4. System change: 

 

The four and last pathway is System Change, which employs learnings from the other three pathways. The purpose of this pathway is to serve as an evaluative moment for the programme to identify what worked (and what did not) in promoting young people’s inclusion and safety in the selected urban environments. This pathway aims to escalate the lessons learned from the local level to the global level and build a better understanding of how to improve young people’s safety in diverse contexts. 

 

The System Change pathway comprises four main activities:

 

i) Building knowledge through research: Learning from other experiences on academic and non-academic grounds to apply their methods and tools in the programme. 

 

ii) Promoting and engaging young people in the evaluation of the programme: Through the participatory approach, the program desires that its beneficiaries not only act as subjects of interventions but actively participate in developing insights and results. 

 

iii) Contributing to a global learning network: This is done naturally with the local, regional, and global partners of the programme, but at the same time is expected to connect with other audiences too. 

 

iv) Sharing knowledge to capitalize on and consolidate change: This focuses on creating strategies to communicate lessons learned by different programme stakeholders. 

 

When will programme data be collected?

 

The iMEL system will be constantly gathering the programme’s data and generating analysis reports for the programme’s decision-making. There are specifically six important milestones for measuring the impact of interventions, evaluating changes, and identifying improvements in the processes. These are:

 

Milestone 1 – City profile: This milestone comprises a set of ‘contextual’ indicators which record current sociodemographic data on young people in the programme region/city. 

 

Milestone 2 – Intervention baseline: Here, data will be collected from the programme’s participants to define a starting point to measure the potential impact that the programme will have on young people.

 

Milestone 3 – Intervention midline: This milestone is a follow-up measurement point that will record and demonstrate how the capacity-building and empowerment of participants have evolved. At the same time, it will serve as feedback for the programme’s implementation partners, informing them about their progress in pursuing programme objectives and allowing them to make adjustments, if necessary. 

 

Milestone 4 – Intervention endline evaluation: The fourth milestone is the endline evaluation (ex-post). It will measure the different outcomes of the intervention using Relational Well-being (RWB) dimensions. It will also evaluate the scale of the intervention and the possibilities of replication in other contexts.
This impact evaluation will also collect significant experiences and lessons learned in a Knowledge Management System, which will help the programme improve constantly. This measurement point reflects the ‘Impact’ level of the Theory of Change (ToC) and will allow us to compare programme outcomes with the baseline to see if the expected change was achieved.

 

Milestone 5 – Post-intervention city profile: Similar to the first milestone, this is a set of ‘contextual’ indicators that will record post-intervention sociodemographic data on young people in the programme region/city. It is meant to be a record of young people’s safety perceptions and issues for local urban administrations.

 

Monitoring reports: The monitoring system runs parallel to the evaluation milestones.  It collects and reflects data from the management indicators tied to each programme implementation activity in the ToC.

 

Timeline of iMEL milestones. Milestone 1 (City Profile) and Milestone 2 (Baseline) are recorded at the start of the programme. Milestone 3 is a midline evaluation of program progress. Milestone 4 records the endline indicators and milestone 5 documents the city profile after the programme's completion. The programme is monitored throughout its length for certain indicators.

 

Figure 4: iMEL Milestones during the programme duration

 

How will the iMEL Framework be implemented? 

 

Through a Knowledge Management System that: 

  1. Coordinates monitoring, evaluation, and learning activities (and deliverables) between regional and local partners.
  2. Analyses the data gathered by regional and local partners.
  3. Transfers knowledge to programme partners, urban stakeholders, and global learning networks for decision-making across contexts.
  4. Communicates highlights and lessons learned in alliance with communication partner City Collab.

 

 

Figure 5: Manuela conducts a capacity building session on the iMEL framework for Fundación Mi Sangre, the local implementation partner in Envigado, Colombia.

 

Bio:

 

Manuela Ramirez is supporting the development of S²Cities’ global Monitoring, Evaluation and Learning framework, with a focus on qualitative methods. She is a politologist finishing her graduate studies in Behavioral Science at Universidad EAFIT. She has worked as a consultant for the Political Analysis Center (CAP) at Universidad EAFIT and as a MEL coordinator for Fundación Prolongar in Bogotá. Through her work, she has focused on projects for trust-building, peace-building, and human development through education and culture.

 

EAFIT University, located in Medellín, Colombia, is S²Cities’ global monitoring, evaluation and learning (MEL) partner. In this capacity, EAFIT University will carry out the following activities, among others: (1) Develop the program’s global MEL framework; (2) Ensure implementing and management partners are informed and aware of MEL activities and processes; and (3) Develop and ensure timely delivery of evaluation reports as scheduled in workplans.

 

[1] Simmons, Annie, et al. “Defining community capacity building: is it possible?” Prev Med., vol. 52, no. 3-4, 2011, pp. 193-9. National Library of Medicine, https://pubmed.ncbi.nlm.nih.gov/21320524/. Accessed 12 July 2022.

[2] Bastien, Sheri, and Halla B. Holmarsdottir, editors. Youth as Architects of Social Change: Global Efforts to Advance Youth-Driven Innovation. Springer International Publishing, 2017. Springer Link, https://link.springer.com/book/10.1007/978-3-319-66275-6. Accessed 12 July 2022.

[3] Sebba, Judy, et al. Youth-Led Innovation: Enhancing the Skills and Capacity of the Next Generation of Innovators. National Endowment for Science, Technology and the Arts (NESTA), 2009. ResearchGate, https://www.researchgate.net/publication/278666609_Youth-Led_Innovation_Enhancing_the_Skills_and_Capacity_of_the_Next_Generation_of_Innovators. Accessed 12 July 2022.

[4] Tolstad, Ingrid M., et al. “The Amplifier Effect: Oslo Youth Co-creating Urban Spaces of (Be)longing.” Youth as Architects of Social Change: Global Efforts to Advance Youth-Driven Innovation, edited by Sheri Bastien and Halla B. Holmarsdottir, Springer International Publishing, 2017, pp. 215–242. Springer Link, https://link.springer.com/chapter/10.1007/978-3-319-66275-6_9#author-information. Accessed 12 July 2022.

Further Reading