TCI’s TARINA Center of Excellence in New Delhi provides capacity building on project monitoring and evaluation (M&E) to senior- and mid-level managers of organizations interested in advancing data-based solutions for development effectiveness. Recently, TCI conducted two M&E workshops, one for managers of nongovernmental organizations (NGOs) and another for managers of corporate social responsibility (CSR) projects. The former was organized on February 16-17, 2018 with Grameen Development Services (GDS) and the Tata Institute of Social Sciences (TISS) in Lucknow, while the latter was organized with TISS on December 21-22, 2017 in Mumbai.
Prof. Prabhu Pingali, Director of TCI, delivering a lecture on M&E systems at the capacity building workshop in Luckow, Uttar Pradesh.
Spread over nine technical sessions, both workshops focused on M&E system fundamentals, from establishing a Theory of Change (ToC) and a proejct Management Information System (MIS) to understanding evaluation methods, sampling methods, and data interpretation. A session was also dedicated to teaching managers how to draft Terms of Reference (ToR) for evaluation studies and examples of what a good evaluation looks like. Overall, the workshops aimed to provide practical guidance on how managers can improve the implementation and efficiency of their projects to maximize impact.
Getting the fundamentals right
The first session of both workshops focused on the following questions around the fundamentals of M&E:
- Where are we going?
- How do we get there?
- What will tell us if we are moving in the right direction?
- What will tell us that we have arrived?”
The purpose of these questions is to help managers define the primary goal of a project, as well as secondary objectives which are intended to provide a detailed strategy for accomplishing the desired goal. Furthermore, the session provdied an overview of commonly used M&E terms, such as project inputs, outputs, outcomes, performance indicators, and benchmarks. This helped participants understand and appreciate the various components of monitoring, evaluation, and impact assessment, as well as the differences between these three concepts.
Establishing a Theory of Change for efficient M&E
TCI’s workshops emphasized the need for managers to establish a ToC at the onset of a project, which delineates the main outcomes or results the project aims to achieve and the main pathways by which it will achieve them. A clearly defined ToC is key to developing a strong project and M&E system.
Participants worked in groups to develop and present a ToC for an existing or new project. By the end of the session, they could begin to appreciate how the ToC lays out the foundation of any project and becomes the reference point for understanding the successes and/or failures of the project as it is implemented.
Identifying performance indicators for measuring change
Measuring change and the impact of a project over time is often tricky. Defining precise and consistent indicators is essential for effective M&E. TCI dedicated a full technical session to the basic rules of measurement. Discussions focused on: identifying project input, output, and outcome indicators; choosing clear data sources for these indicators; and determining the frequency of data collection.
Converting data into information for better project management
Strategic improvements can be made to project implementation with consistent, reliable, and quality data. Participants discussed the various methods used within their respective organizations for gathering data and extracting lessons to improve project management. These included unstructured field monitoring visits and semi-structured, need-based visits to project sites, as well as periodic progress reporting. Among the methods shared, TCI highlighted the need for organizations to build and maintain a flexible, cost-effective, and participatory project MIS for collecting real-time data in the field on a regular basis. The granular data and indicators gathered through an MIS can provide revealing information, allowing for mid-course corrections to ensure better outcomes.
Comparing changes: Methods for conducting an evaluation study
The purpose of an evaluation study is to measure and assess the changes that can be attributed to the various activities or interventions implemented under a particular project, which is typically achieved by comparing changes observed between a treatment group and a control group. This technical session exposed participants to various evaluation methods and addressed questions such as the following:
- How can changes be compared if no baseline survey was conducted?
- Is there an ideal comparison group?
The choice of an evaluation method largely depends on the availability of resources, type of information needed, degree of interruption to the project, and the optimal way to reach respondents given the literacy, social, and cultural context of the project location.
From data collection to data analysis
Data is a critical component of any project and so administration of data becomes an important task for managers. This session reviewed the technical aspects of collecting, cleaning, managing, and converting data into information. Discussions focused on the following questions raised by participants:
- What is an ideal sample size?
- What data collection tools should be adopted?
During this session, participants learned about the advantages of commonly used data collection tools, namely Paper Assisted Personal Interview (PAPI) and Computer Assisted Personal Interview (CAPI) tools. They also learned the importance of distinguishing between “good” data and “bad” data once it has been collected and were asked to engage in an exercise where they had to identify “good” data from a field survey conducted. Finally, basic rules for data cleaning were discussed, including the need to ensure units are consistent, data labels are clear and easy to understand, and duplicate records are removed.
The why, what, and how of evaluation management
It is important to define the expectations of an evaluation study at the outset, and so it is equally important to develop a clear ToR for the evaluator(s) that will undertake the study. A ToR defines the objectives, scope, and responsibilities of the evaluator(s). It also provides a description of the resources available to conduct the study. Systematic M&E reporting forms an integral part of the evaluation and involves the identification of which findings are to be reported, to whom, at what interval, and in what format.
From M&E to a learning organization
Project assessments are not always easy to undertake or to digest and act upon the results when complete. Therefore, in the final technical session, participants were asked to deliberate on the following questions:
- What is the M&E budget in their respective organizations?
- Who has access to M&E results?
- With whom are the results shared?
- Does the organization encourage learning from failures?
During this session, TCI explained that upon finalizing M&E results, organizations should translate the results into knowledge for improving project implementation, management, and processes. TCI also emphasized that in order to initiate change, organizations should aim to become a “learning organization” by acquiring knowledge and modifying its behaviour to reflect new insights.
TCI to continue capacity building efforts
Building the M&E capacities of different stakeholders forms an integral part of TCI’s work in India. Technical workshops organized so far have helped to broaden stakeholders’ understanding of M&E systems for development projects. Similar workshops will continue to be offered to organizations interested in strengthening their M&E capacities to become effective “learning organizations.”
By Ruchira Boss
Ruchira Boss (firstname.lastname@example.org) is an Assistant Program Officer for TCI-TARINA, based in New Delhi, India.