INVITATION TO TENDER FOR BRICS External Evaluation 2017

Building Resilience in Chad and Sudan (BRICS) is part of a 3 year, global resilience programme known as BRACED (Building Resilience and Adaptation to Climate Extremes and Disasters). BRACED is one of the largest resilience programmes currently operational and is funded by UKAID. The focus of the programme has been in countries in Sub Saharan Africa that experience regular climate shocks, particular floods and drought. There is a significant emphasis within the programme on learning about what works to build vulnerable communities’ resilience to climate extremes across a variety of settings. BRACED has a global log frame which all funded projects (15 consortia across 13 countries) feed into. At the end of the programme, UKAID want to understand what has worked across the different contexts to build resilience with the aim of making informed funding decisions. This evaluation is part of that learning process.

BRICS has been operational from April 2015 to December 2017 in Chad and Sudan. A consortium of 3 organisations is led by Concern Worldwide with a Consortium Coordinating Unit (CCU) based in Concern’s London office which is responsible for coordinating the 4 partners:

• Concern Worldwide implementing BRICS activities in West Darfur, Sudan and Sila Region, eastern Chad.
• The World Agroforestry Institute (ICRAF) who are supporting on a range of conservation smart and agroforestry interventions with Concern Chad.
• Tufts University Feinstein International Centre (Tufts FIC) who is engaged in a research capacity to support the baseline, longitudinal, end line studies in Chad and Sudan. Tufts FIC are also supporting on specific activities linked to pastoralism and early warning.
• Al Massar in the Republic of Sudan, a local NGO that specialises in working with Pastoralist communities and is implementing BRICS activities in North, South and West Darfur.

The BRICS Theory of Change hypothesis assumes that by delivering an integrated package of interventions which build the adaptive, absorptive and anticipatory capacity of households, then their resilience to climate change will increase. This resilience is sustained at the macro and meso level through the strengthening of local institutions (community, local authority, technical departments) to anticipate and plan for increasing variability and severity of weather events. Aligned with this, evidence from longitudinal studies and monitoring and evaluation of interventions is being presented to national governments in Chad and Sudan to demonstrate examples of what works to build resilience.

The project activities fall into the work packages listed below which were implemented from January 2015 to December 2017. These packages were delivered where possible in an integrated way with the same households and same communities.

Climate Smart Agriculture: Conservation Agriculture (CA), Agroforestry, Farmer Managed Natural Regeneration (FMNR), vegetable gardens, Integrated Soil Fertility Management (ISFM), soil and water conservation interventions, seed system development, support to pastoralists and agro pastoralists through training Community Animal Health Workers (CAHWs), vaccination campaigns, setting up of migratory routes and improved pasture management.

Health and Nutrition; Mother Support Groups, training of Volunteer Community Health Workers (VCHWs), Community Managed Acute Malnutrition and material support to Health Centres, and operating nutrition centres.

WASH: provision and repair of existing boreholes, setting up or strengthening of Water User Committees (WUCs), training of pump mechanics, Community Led Total Sanitation (CLTS), Participation Hygiene and Sanitation Transformation (PHAST) and the reinforcement of the spare parts chain.

Disaster Risk Reduction and Early Warning; Disaster Risk Reduction Committees (DRR) and support to improve functionality of National Early Warning Systems, supply of weather information.

Equality; Life skills curriculum targeted at women and men, staff gender equality training and the incorporation of gender analysis at the community level looking at Access and Control, Time Use and Community Resource Mapping.

Advocacy; Work at policy level to ensure the policy environment supports resilience building.

For additional reading on the wider BRACED programme and how BRICS is a part of this please go to http://www.braced.org/

Purpose of the Consultancy The overarching aim of the Final Evaluation (FE) is to gather information about project progress and lesson learning. This should help us to understand “How, where, when and why do or don’t BRACED interventions work, and what can be learned”. The FE should undertake robust reflection and gathering of evidence on project success and failure in order to explore, test and revise assumptions held by the BRICS implementing partners.

UKAID outsourced the day to day management of BRACED to KPMG (Fund Manager) and to ODI (Knowledge Manager). The Knowledge Manager’s responsibility is to aggregate all the learning across all programmes and to help UKAID understand what is being learned about how to build resilience. To do this the KM have developed a methodological frame work which we have had to retrospectively fit with our activities/log frame. Coupled with this, the KM has also created some pre-set evaluation questions that have to be answered (listed below).

The evaluators should examine the outcomes BRICS has achieved to date, linking these to the “activity packages” implemented under the project (such as climate smart agriculture, access to clean water, sanitation marketing, gender equality and action research), and any synergies between these, as well as taking a broad view of the project as a whole. In order to understand what is working well (or in need of improvement), and why. The final evaluation needs to critically examine the mechanisms through which these activity packages are working. It also needs to investigate the way that the project context shapes these activities and outcomes and if what was done was the right approach.

Scope The FE should focus on project progress and lesson learning throughout the project (January 2015 – December 2017) in both Chad and Sudan (Dar Sila region in Chad and Darfur States in Sudan).

The FE should be planned and delivered against a standard set of headline Evaluation Questions mandated by the BRACED Knowledge Manager with sub-questions tailored to the BRICS Project theory of change.

Process-orientated and explanatory in nature.

Focus on outcomes, testing the project theory of change and how change was or was not actually brought about, and why so, and critically examining whether contextual and/or circumstantial adjustments were made on time to ensure optimum realisation of desired project outcomes.

Evaluation Questions Under the OECD DAC criteria (i.e. Impact, Relevance, Effectiveness, Efficiency, Sustainability) the following set of evaluation questions are also to be answered by the Final Evaluation (from the BRACED Knowledge Manager M&E Guidance Notes).

1. To what extent have particular interventions led to anticipated changes and results? This opening question, requires evaluators to reflect on the evidence of results delivered against the results / changes anticipated in the BRICS Theory of Change.
2. Specifically focusing on understanding ‘mechanisms’ (the causal forces or powers that explain why a change happens or not), how and why have particular intervention packages led to observed results and changes? Were there any unintended attributable results, why? Projects should focus on defining ‘mechanisms’ in order to understand what it is about the nature and design of an intervention that has enabled it to be effective or not. Sub-questions under this question should explore: a. What has the project learned about delivering these packages of interventions? b. What evidence is there that the interventions and the mechanisms that support them have the potential to deliver ‘amplified results’ and/or ‘transformational impact’? c. What impact (if any) did the project have on internal (and external) thinking about Resilience – did it help to crystallise this? d. How did the consortium model enable/constrain the delivery of the project Theory of Change? e. How the relationship between the different agencies worked?
3. Based on BRICS accumulated knowledge and understanding, what key resilience strengthening lessons can be learned and replicated? This final evaluation question requires evaluators to reflect on the intervention-level learning BRICS have accumulated over the life of the project and to distil this learning into a set of evidence-based lessons, defining which of these can potentially be replicated elsewhere. What evidence is there of knowledge and capacity transfer between partners and local stakeholders?

The BRICS Consortium Coordinating Unit will provide evaluators with a detailed Inception Report and project-specific Evaluation Matrix with pre-populated project data. This will guide evaluation team throughout the Final Evaluation.

Methodology Prior to the signing of the evaluation team’s contract a detail inception report, with detailed methodology and draft data collection tools will be developed by the Consortium M&E Coordinator. This is to meet the BRACED Knowledge manager mandatory review and sign off process and to begin the internal Sudanese Humanitarian Aid Commission (HAC) approval process. Once the evaluation team is contracted the evaluators will be responsible for reviewing the inception report, methodology and tools to ensure that it covers the overarching evaluation questions (listed above) and is feasible. The methodology proposed is largely guided by the KM endorsed Realist Evaluation approach (see here for more details).

We also anticipate that:

• The evaluation will be participatory and gender-sensitive in nature and use and generate data through a range of qualitative and quantitative methods. The evaluation team will likely combine the review of existing project routine results reporting data, including Annual Reports, Mid-term review, baseline and end line data (if available) with their specific primary data collection activities which will most likely be qualitative to get at ‘how’ and ‘why’ things happened the way they did.
• The evaluation team will need to test ‘assumptions’ based on the theory of change about how the project worked, refining these in the light of your experience in implementing the project. This will build on what BRICS learned in the MTR. The BRICS M&E Coordinator will have defined/identified key outcomes and their indicators in the inception report which this evaluation team will need to evaluate.
• Data collected by the evaluation team will need to be triangulated with the existing project routine results and reporting data, including; KPI4, the 3As, Areas of Change, and Evaluative Monitoring, and MTR findings. An overview of all of these and results to date will be included in the inception report.
• Primary data collection is likely to be primarily qualitative (through interviews, focus groups and participatory exercises and feedback mechanisms).
• The evaluation team should meet a broad range of project stakeholders, from project team members to project beneficiaries/participants as well as wider key informants, partners, and observers.
• The evaluation team should aim to engage not just ‘direct’ project stakeholders but also those stakeholders who have an ‘external’ perspective on the project – for example, the teams of other resilience strengthening projects operating within the same area.

  Data Analysis The evaluation team must clearly describe the way that you plan to arrive at a set of robust and evidence-based findings and conclusions.

Evidence should be consistently cross-referenced throughout the FE report, and claims made should be substantiated / validated through reference to the evidence to support these claims.

Transcripts should be taken, written up and included as an Annex to the final report.

Essential and Desirable Experience/Qualifications Evaluation teams will be selected based on the below criteria for each evaluator. The lead evaluator may have more senior experience but their role is more based on the ability to take on more responsibility and ensure the findings from both Chad and Sudan are well balanced in the final report.

Essential Experience:

• University degree in social sciences or a related area. A post-graduate degree in project management, and/or related fields would be an advantage.
• At least 5 years of proven relevant professional experience in an international development environment, multi sectorial programmes, resilience, measurement of resilience
• Substantial experience in conducting evaluations
• Knowledge of programmes aimed at building resilience to climate change, experience of the BRACED programme would be an advantage
• Excellent drafting and report writing skills (in English)
• Regional experience and good understanding of political, socio economic and DRR/resilience issues in the BRICS context.
• Availability to carry out most of the work in November 2017 to early February 2018.

Desirable experience:

• Experience in monitoring and evaluation of UKAID projects/programmes is desirable
• Familiarity with the activities and procedures of Concern Worldwide is an advantage
• Familiarity with the BRACED Knowledge Manager and Fund Manager set up is an advantage
• The working language will be English for the reporting however the evaluator going to Chad will need to be fluent in French.
• Arabic is desirable.

Lines of Communication

• Lead Evaluators should send applications to Concern Worldwide Logistics and Supplies team suppliesandlogisticsunit@concern.net
• If interested in further information please request the full tender documentation from Rory Crew (BRACED Grants and Compliance Accountant) via email at rory.crew@concern.net
• Any queries prior to submission of tender documentation should be sent Rory Crew (BRACED Grants and Compliance Accountant) via email at rory.crew@concern.net