Six problems in Operationalising Monitoring, Evaluation and Learning (and their solutions)

pksddn Operational MEL, Posts 12 Comments

Does your job description include the words – Monitoring, Evaluation and Learning (MEL)? Are you supposed to design and/or implement MEL in a non-profit, Development organisation? And you have no problems? Then this post is not for you! 🙂

For the vast majority of those who do face problems in MEL, you may be surprised to know that you are not alone. A LOT of persons face similar problems, as I found out when I moderated an e-discussion on this subject.

Here is what happened. I was working for a massive livelihood project in the Indian Himalayas. The project supported women’s credit groups to save money and use these savings to run value chains. These ranged from fruits, vegetables, backyard poultry, spices, bee-keeping …. you get the idea. We worked with 47,000 women in a huge geographical area. And I was the ONLY MEL officer for the whole project.

A heavy workload was only ONE of my problems. I faced many other problems, which I shall describe shortly. Looking for help, I decided to run an e-discussion with fellow MEL officers in other similar projects in Asia to ask them how they were faring. After long and passionate discussions, we found that ALL of us faced similar problems, and were all suffering alone! We also came up with a few solutions.

Here is what we learnt:

Problem 1. MEL is seen more as a reporting and policing system than as a learning system

This is probably THE core problem faced by most MEL staff. In a vast majority of development organisations, MEL is rarely seen as something that supports project planning and implementation. An information retrieval system? Yes. Source for data when the donor report is due? Yes. A learning tool to improve project performance? REALLY?

In some cases, the MEL system (and officer) is also perceived as the internal policeman, keeping tabs on non-performing staff. Mostly, the MEL system is deliberately kept only in a minimal state of activation. I found this out in one of my assignments, when my Director refused to give me permission to carry out planned monitoring visits, talk to project staff or hold review meetings. The reason? Fear. Fear that I would “uncover something” that would be detrimental to the project reputation. This is just one example of the many fears and misconceptions that surround MEL in most projects.

Problem 2. Lack of clarity about when MEL work really starts

A common (mis) perception among project staff (and even in the minds of some MEL officers) is that “MEL comes after we have started implementing a project. There is nothing for the MEL officer to do at the early stages of the project”. However, setting up robust and sensitive MEL systems needs to be done right at the beginning of the project. This ensures clarity on the objectives that are to be achieved, and the metrics to track the same.

Problem 3. Lack of clarity on why a Baseline is needed and when to do it

A corollary to the above apathy about integrating MEL from the beginning of the project is the all-pervasive confusion around the word “Baseline”. Most people consider “Baseline” as an all-encompassing survey of target households covering ALL aspects of their lives. Often, this baseline report is too general to be used or misses information on key project objectives.

In one project, I was setting up a system to monitor the Sustainable Development Goals at Gram Panchayat (local government) level. When I looked at the Baseline, I found that data had been collected on everything except on the relevant SDG indicators which the project was supposed to impact! In such cases, where baseline data is missing, the only solution is to “construct” a baseline (more on that in a subsequent post).

Good MEL practice is to ensure that BEFORE the project starts, data on key parameters and indicators that the project wishes to change is collected. Often, this requires targeted data collection only on a few parameters and can be infinitely more useful than voluminous data on non-essential points. As the old adage says, “Measure what you treasure”.

Problem 4. Lack of articulation on what the MEL system needs to measure

Good MEL practice recommends that the project leadership and teams be clear about the changes (Impact, Outcomes and Outputs) the project will bring in the lives of target communities. The MEL system needs to be built to capture these changes.

Don’t be surprised, however, if you find that your project teams have not taken the time to articulate the changes they want to focus on. In most cases, conceptual clarity about targeted changes and the Theory of Change is a rarity. Most teams are too busy implementing the project to think about what they are trying to achieve. The Logframe or the Project Document is seen as the Bible – and is followed literally, rather than as a general guide. This cascades into a consequent lack of clarity on what the MEL system needs to track. As Yogi Berra said – “If you don’t know where you are going, you’ll end up someplace else.”

Problem 5. Seeing MEL Trainings as something that only MEL officers need to take

MEL trainings, on the rare occasions when they do happen, are generally targeted only at MEL staff. This is actually a manifestation of a lack of clarity on whose role is doing MEL. Good MEL practice recommends that MEL is the responsibility of EVERYONE, and the MEL officer is a facilitator and a resource person for technical support. This skewed understanding about MEL roles further reinforces the notion that MEL is only a donor requirement and not a learning tool.

Problem 6. Lack of hand-holding support for designing and implementing MEL systems 

For a new MEL officer, some (not all) organisations offer a one-time “orientation” in MEL. This is often not sufficient to equip them for designing and implementing a good MEL system. In most organisations where I worked as a MEL officer, I was just thrown into the deep sea of MEL and expected to “figure it out”.

Instead, what a MEL team requires is a continuous hand-holding support to design and implement a light but effective MEL system. The ability to design such a system comes from experience, which a junior MEL officer normally does not have. She/ he needs handholding support and mentoring, which is very rarely provided.

So what can you do to resolve the above issues? 

Solution 1. Ensure that the “L” in MEL is bigger than the M & E 

In most projects which have MEL systems, time and resources are spent on Monitoring, Evaluation and Learning – in that order. Instead, the ideal emphasis should be primarily on Learning, then Evaluation and least of all on Monitoring. (figure below)

But for learning to happen, a few simple tricks seem essential:

  1. Emphasise the Learning role of MEL at all times: Use every forum available for this – e.g. during trainings, strategic discussions, or while designing MEL tools. Most of the times, learning is lost because MEL officers themselves do not ask the question – “So what can we learn from this?”.
  2. b. Incorporate regular learning events into project schedules: Ideally, an Annual Learning Workshop (as different from an Annual Progress Review Workshop) is the MINIMUM that must be done. Just like an evaluation calendar, an annual learning calendar is a great tool to ensure that time is set aside for learning events/ processes.
    c. Use learning events/ meetings to resolve urgent problems: Identify a handful of urgent project issues troubling project staff and conduct thematic research, meeting or workshops on these issues. If the learnings that emerge resolve these problems, it will convince project staff of the day-to-day usefulness of MEL.
    d. Support cross-project learning processes: Learning events that foster learnings across projects (or geographical units) provide the most rapid learnings. This is because the normal tendency for projects is to learn from scratch what other projects have already tried in a different space-time. (e.g. One of the projects I was monitoring was trying to learn how to design social institutions for irrigation management from scratch – not benefitting from already available massive data (and Ph D dissertations) available on similar successful experiments in the same country or other countries).
    e. Peer reviews for rapid learning: We also found that peer reviews (e.g. a team from a different province reviewing project successes and constraints) can be a powerful cross-learning process. Teams usually come away with Aha Conversations like – “Oh, this is how to resolve X issue!” or “Hey, in our area, we did Y to resolve this problem, and it worked! Can you do the same?”. Such processes must be carefully designed to facilitate close interaction with target communities (e.g. women farmers) but should also create space for reflection within project teams.
    e. Knowledge Products for visibility and motivation: MEL teams need to be prolific in creating Knowledge Products from learning events and processes. Sometimes, these products are the most visible signs of individual and team performance. Also, everyone loves to see their project, their farmers, their ideas being documented in colourful, interesting to read documents. Moreover, in the digital age, these products can be circulated easily in soft forms. And if these products can also be in the local language, local communities can also benefit.

Solution 2. Build the constituency for MEL through regular FUN trainings

One of the most useful tricks that I used for advocating the importance of MEL was to always carry with me a module for a three-hour fun session on MEL basics. This session included team games, videos and group work. I called it the “MEL is Fun” module. Often, I would use this as a refresher for all staff or as an orientation for newcomers. Most people did not mind spending three hours. Then, when everyone got excited about MEL, I would offer a three-day training. In most countries, it worked very well and I saw a distinct shift in attitude towards MEL! One of my colleagues said as much:

“Making MEL fun is not an easy task, but you did it! My Team and I have a lot of respect and we highly value your contribution in introducing practical (operational) MEL where our program staff turned your MEL training into action”. ~ Mandefro Nigussie, Deputy Regional Director, Oxfam America – Horn of Africa Regional Office, Addis Ababa

Solution 3. Standardise MEL processes and formats – and don’t change them too often

In one meeting, a project staff told me – “I hate MEL!”. Curious about what could evoke such a harsh emotional response, I asked her what was wrong. It turned out that her MEL officer kept changing the MEL formats almost on a monthly basis, and she had to put in huge efforts and time to re-collect data and fill in tables. Understandably, she hated MEL!

To avoid such hatred against MEL, the MEL Officers need to put adequate thought into designing formats for data collection and test these rigorously before rolling them out on an organisation-wide scale. After this, formats should be changed only once a year, if at all.

Mobile technologies have made things much easier now. For example, in the MIS systems that we design these days, we use Mobile-based systems to track data with one clear ground rule – no data that is entered once should ever need to be entered again. The MIS system should be intelligent enough to “remember” what was already collected. This includes basic information such as the names of target beneficiaries, the interventions made in the last reporting period, and so on. Additional data – on the rare occasions when required at all – is collected in short 2-minute data collection modules, which can be quickly collected during the next community visit. And always, the purpose any data being collected needs to be clear to project teams. This vastly reduces the psychological burden of project staff.

Solution 4. Make support for MEL tasks easily available 

In addition to all the confusing things that are loaded on to a new-comer to the organisation, a hastily-put-together “MEL orientation” is often part of a package. Mostly, this confuses rather than clarifies things to the scared new-comer.

I found that instead of this approach, taking people through a MEL wiki which put all the processes, formats and (what I called) “Minimum Things to do in MEL” became REALLY useful and liked. Most people were relieved that they could access prioritised and useful MEL stuff whenever they wanted. In this Wiki, I usually maintained templates, formats and guidance notes for all mandatory MEL processes (e.g. Evaluation Terms of Reference, a two-page Field Visit Report with fun elements added in, etc.). See the Wiki I designed for Oxfam GB Asia Region here.

Another very useful slide to make permanently available is the one outlining very clearly mandatory MEL tasks and – importantly – where the budget for these tasks would come from. Some organisations ensure that a 5 to 10 per cent budget for MEL is mandatorily built into project budgets. But not all organisations do this, and information on where to get money for MEL often ensures faster implementation of MEL systems.

Project teams (and MEL teams) also normally find useful another slide on their expected roles in MEL, and what kind of support they could (and could not) expect from the MEL team.

Solution 5. Ensure timely Baselines

Finally, MEL officers must ensure (or push for or cajole) all projects to collect Baseline data within the first quarter of the project launch. This data must, at the minimum, cover information related to the key change areas targeted by the project. This change is often visualised in the Logframe or the Theory of Change.

For livelihood projects, I was very impressed by the Baseline Advisory developed by the Government of India’s National Rural Livelihood Mission to all State Livelihood Missions. Take a look here.

In a nutshell, the above have been some of my learnings in trying to operationalise MEL.

Do you face similar problems? Are there other issues you would like to flag? I look forward to hearing from you, fellow MEL practitioners and travellers!

 

Image source: http://www.rcrc-resilience-southeastasia.org/wp-content/uploads/2016/09/IFRC-ME-Guide-August-2012.pdf

Comments 12

  1. Thanks, Pankaj for putting together the often complex MEL subject in such a way that any MEL person can prepare himself for the challenges to come and use your proposed solutions for operationalising the MEL system of any project.

    One thing that I may add is the importance of setting targets (both annual targets and end of project targets). This should be done once you have your baseline values for key indicators of the results you are measuring. Annual targets should be set for output and immediate outcome indicators so as to continuously monitor/track achievement of project outputs annually using the formats you developed for collecting the data frequently (usually quarterly) to make informed decisions regarding regular project management or implementation.

    For higher level outcome and impact indicators we need to have end of project targets but measure its progress during mid-term evaluations and through other assessments and special studies so as to test our theory of changes and its assumptions and do mid-course adjustments based on those findings instead of waiting for the end of project life.

    At the end of the project, we need to conduct the final evaluation for measuring the outcome and impact of the project compared to the baseline situation. It also provides us with the intended and untended, direct and indirect outcomes of the project for our future learning and use of the evaluation findings.

    1. Post
      Author
  2. Dear Pankaj

    Your blog is very interesting and reflects real situation of MEL system. I don’t have so much knowledge about MEL but as per my knowledge and experiences I would like to share some points.

    1. Both the project teams and the MEL Officers should have a good knowledge of the project and MEL. This will help them to understand the nature of work – MEL systems and Project work need to be looked at together, not separately. The MEL system should therefore be like a guide to achieve goals, but should not increase work load of project teams, otherwise it will be seen as the policeman. Trust among both MEL and project teams and clarity on the objectives is a must.

    2. “L” – Learning plays a vital role for both project teams and MEL officers to move ahead in parallel. Sharing our learnings will help to modify the MEL system as per requirements and will also provide value.

    1. Post
      Author
  3. Hi Pankaj

    Sorry, I wanted to add something. On re-reading your post, I realised you HAVE included even target setting. However, you have used very few words to present to us a lot of things. That shows your creativity and brilliance. Your two powerful quotes: “measure what you treasure” and “if you don’t know where you are going, you will end up someplace else” are really useful.

    You really know how you make “MEL fun” and I am the living testimony for how you did that while you were providing us MEL training and were coaching me on my MEL functions in Ethiopia.

    1. Post
      Author
  4. Dear Pankaj Sir,

    Thank you so much for writing this excellent blog-post and raising the very basic and yet complex problems around MEL. Being a fresh MEL professional (and for many others like me), it is like a step-by-step guide for diving into the M & E processes. The entire post is very interesting, but the following points really resonated with me:
    1. Inadequate attention to MEL during programme planning i.e. conducting baseline studies and inadequate human resource capacity to take up the MEL role.
    2. The focus should be on more “L” learning rather than “M” monitoring in every meeting/training. This will help in delivering against the project objectives, and in measuring and evaluating our performances, and (based on the evaluations) will help us in making the required adjustments to reach the project objective.
    3. Build the capacity of MEL staff to take up this challenging role with lots of training. The FUN Modules you mention can play a vital role to understand MEL for newcomers or for junior MEL associates.
    4. MEL training is not meant only for MEL people but for all program people
    5. The best part of your blog-post is the suggestion to conduct baselines in the first quarter after project launch and that it should focus on the changes the project wants to bring about.
    6. Paying adequate attention to what you want to change and design the MEL around these changes, especially establishing a linkage between the desired changes with the 17 SDGs. This link is normally missed by MEL associates and program teams.

    1. Post
      Author
  5. Dear Pankaj Sir

    Thanks for your post.

    I have experience of working with livelihoods and public health projects. In my experience ‘L’ (Learning) has never been the priority in these projects. What is more, ‘L’ is not even a point of discussion. Annual reports, quarterly reports, impact assessment reports, case studies – preparing these were all priorities but only for monitoring and documentation purpose, never for ‘L’.

    If these activities had been all done with the viewpoint of learning, many of the fire-fighting scenarios we dealt with on an almost daily basis would never have happened and the final results of our interventions would have been much much better than what we achieved.

    You have rightly said that ‘L’ is important in all aspects of project life. In most organisations, it is presumed that MEL is the job of one person in the MEL position. I would like to add that Knowledge Management also suffers a similar fate and is often seen as a one-person responsibility.

    Adding ‘L’ and the way your article inculcates the ‘L’ idea into a broader scenario will help create a learning culture rather than following MEL Terms of Reference of one personnel strictly and losing the ‘L’ completely.

    Thank you Pankaj Sir for introducing the ‘L’ concept in this vast scenario.

    1. Post
      Author
  6. Dear Pankaj,

    I appreciate the sharing of your very RICH experience in MEL.

    Our experience in Udyogini reveals that the challenge lies in articulating and analysing the information generated during various stages of programs to take key /critical project or business decisions. The inability to tap the power of MEL for project or business decision making adversely affects Udyogini’s other systems such as communication, resource mobilization, performance assessment etc.

    Hence, I feel it is imperative for any project team to design an information-decision matrix prior to setting up and executing a MEL system. This helps in:
    Prioritizing information to be collected (as you often say – ‘LESS is MORE’)
    Selecting tools and techniques for data collection

    The information-decision matrix based MEL system will truly help in improving the efficiency and the effectiveness of project/program interventions.

    We strongly believe that your technical support will help Udyogini to design and execute a MEL system to take project/business decisions in a timely manner so that we can bring a sustainable and positive change in the life of poor women and their families.

    Awaiting your deeper insights in designing, testing and executing such a MEL system.

    1. Post
      Author

      Thank you, Brij for your comments. I look forward to working with your team on building a responsive, decision-supporting MEL system using the power of mobile and internet technology.

Leave a Reply

Your email address will not be published. Required fields are marked *