1. Introduction

1.1    Purpose

This evaluation assesses the effectiveness of Australia's response to the food security and refugee crisis in Somalia, Ethiopia and Kenya in 2011, known as the Horn of Africa (HoA) crisis. The purpose of this evaluation is to identify how to improve the effectiveness of Australian humanitarian assistance to future slow-onset crises. The areas of enquiry were:

  • how Australian responded
  • how Australian assistance was delivered
  • how well the needs of affected people were met
  • achievements of the Australian response.

1.2    Methodology

Process

The evaluation aimed to be participatory while maintaining rigour and evidence standards. The first step was to develop an evaluation plan that outlined the methods and timeframe. This was circulated to internal stakeholders for comment and was subject to scrutiny by the Independent Evaluation Committee.

Interviews with stakeholders were conducted face to face in Canberra, Melbourne and Nairobi, or by telephone. These interviews were complemented by an intensive process of collecting, sifting and analysing reports, the academic and grey literature, and other relevant material.

The evaluation sought to triangulate findings through a variety of means. This included a review of evaluative material examining the performance of the Australian aid program and implementing partners, as well as general findings relating to areas of the response that Australia funded. The analytical process included an interrogation of financial data with a view to understanding sector and geographic coverage, and for collating and codifying results data. Interviews were written up and analysed for patterns and common themes. An initial feedback session was held in Nairobi and a validation workshop presented initial findings in Canberra to begin the process of developing recommendations.

Evidence

Interviews focused on staff in the Australian aid program, implementing partners and donor organisations (Table 1). The interviews were semi structured, loosely following the four areas of enquiry. They were a strong source of evidence. People were generally candid and frank, and appeared comfortable to share all forms of information verbally. Interviews provided a high degree of consistent information, with major themes emerging quickly. Workshops were held with staff from the Department of Foreign Affairs and Trade (DFAT) to develop recommendations, and focus group discussions were undertaken with beneficiary communities in northern Kenya.

Table 1   Number and type of interviews

Organisation

Number of people interviewed or consulted

Canberra and Melbourne

AusAID (now DFAT)

Africa Branch (5), Humanitarian Preparedness and Response Branch (8), other (5)

NGOs

Australian Council for International Development (1), Australian Red Cross (2), CARE (2), Oxfam (2), Save the Children (1), Plan (2), World Vision International (2)

Nairobi / Wajir

AusAID (now DFAT)

Nairobi Post (5 interviewed, another 5 included in briefing or feedback sessions)

NGOs

Arid Land Development Focus (2), CARE (2), Caritas (10), Oxfam (7), Plan (3), Save the Children (5), Wajir South Development Association (2), World Vision International (2)

UN

UN Children's Fund (2), UN Food and Agriculture Organization (2), UN High Commission for Refugees, Kenya (2), UN High Commission for Refugees, Somalia (2), UN Office for the Coordination of Humanitarian Affairs (1), UN World Food Programme Kenya (9), UN World Food Programme Somalia (2)

Red Cross

International Committee of the Red Cross (2), International Federation of the Red Cross (2)

Donors

Denmark (1), Department for International Development, United Kingdom (2), European Community Humanitarian Office (2), Netherlands (1), Office of Foreign Disaster Assistance, United States (2)

Beneficiaries

Three focus groups of beneficiaries from Oxfam (12), Save the Children (5) and UN World Food Programme (15)

Total

137 people interviewed or consulted

AusAID = Australian Agency for International Development; DFAT = Department of Foreign Affairs and Trade; NGO = non-government organisation; UN = United Nations


The secondary data analysis used two main sources of information—agency reporting and external material relating either to the HoA crisis or to the international aid operation. This latter category included evaluations and academic literature (Table 2).

A number of ways were used to review the secondary data. A trawl of the academic literature and general evaluative publications was undertaken to draw out key general lessons appropriate to countries, sectors or agencies, and to look for specific mentions of Australia or AusAID. The documentation provided by the Australian aid program and its partners was reviewed to inform the narrative and factual base for this evaluation, as well as for any pertinent lessons. Reports of, financial allocations and beneficiary numbers were reviewed to compile new analyses of results including where funds were spent and how many people were assisted.

To make evaluative judgements, the extensive humanitarian experience of the team and their knowledge of the HoA crisis were used, along with the evidence gathered in the course of this evaluation. For details of the strength of evidence in each of the areas of enquiry, see Table 3.

Table 2   Type of secondary data review

Type

Number of documents

Academic and grey literature

81

Review of relevant evaluations

32

Australian aid program and implementing partner reports

99

Total

212


Table 3 Strength of the evidence

Area of enquiry

Data source

Analysis

Strength of evidence

The Australian response

  • stakeholder interviews
  • literature
  • evaluation reports
  • internal reports
  • workshops

Internal paper trail clear; interviews consistent; good body of literature; timeliness, influencing and staffing issues clear

Strong

Delivery of Australian assistance

  • stakeholder interviews
  • internal reports
  • workshops

Internal paper trail clear; interviews consistent; timeliness, influencing and staffing issues clear

Strong

Responsiveness to the needs of affected people

  • interviews
  • beneficiary focus groups
  • literature
  • agency reports

Interviews, focus groups and documentation consistent; timeliness, influencing and staffing issues clear

Adequate

Achievements of the Australian response

  • interviews
  • beneficiary focus groups
  • literature
  • agency reports

Few real facts on outcomes or impact beyond the recent mortality study,1 value for money unclear due to data limitations and definitions

Weak


Limitations

The evaluation was mostly based on secondary data. The strength of the findings was dependent on the quality and quantity of data provided, which was less than ideal. There were significant gaps in data and information. Some of this is understandable given the context, because it is difficult to monitor and collect data in humanitarian and conflict situations. Deficiencies in reporting, however, also stem from poor practice in the humanitarian sector, which traditionally has not prioritised accountability (either to donors or beneficiaries) and has not invested in rigorous measurement of effectiveness. Some agencies were reluctant to generate additional information for this evaluation. The United Nations (UN) system proved the most problematic in terms of providing both budgetary analysis and numbers of people reached or assisted. This was primarily because the reporting that Australia requires from these agencies is minimal. Since UN agencies received the bulk of funds, evidence on results is weak.

 

References

1         London School of Hygiene and Tropical Medicine & John Hopkins University Bloomberg School of Public Health. Mortality among populations of southern and central Somalia affected by severe food insecurity and famine during 2010–2012, for UN Food and Agriculture Organization, Famine Early Warning System Network & Food Security and Nutrition Analysis Unit for Somalia, Rome and Washington, 2013.

Last Updated: 9 December 2014