There is increasing interest in how evaluation can contribute to the evidence base on what works (and what doesn't) in aid. ODE is uniquely placed to assess performance across the Australian aid program and bring international best practice to bear in identifying new and better ways of working.

ODE's evaluations are guided by relevant professional standards including the Australasian Evaluation Society's guidelines for the ethical conduct of evaluations [PDF 496kb], and the DAC quality standards for development evaluation [PDF 1.8mb].

ODE conducts evaluations and reviews to assess the effectiveness of Australian aid to country programs and priority sectors, identify areas of good practice and highlight important lessons.

This page houses completed and published ODE evaluations and reviews. Visit the Current work page for evaluations currently in progress.

• 2014 Quality Review of Aid Program Performance Reports
• A desk based review of donor experiences in Sector-wide approaches in the health sector
• Evaluation of the Australia-Vietnam country strategy 2010–15
• A window of opportunity: Australian aid and child undernutrition
• Research for better aid: an evaluation of DFAT's investments
• Working in decentralised service systems: challenges and choices for the Australian aid program
• Smart Economics: Evaluation of Australian aid support for women's economic empowerment
• Australia's response to the Horn of Africa Humanitarian crisis, 2011 Evaluation Report
• Two reviews of Australian aid operational evaluations
• Evaluation of Australian aid to Timor-Leste
• Evidence reviews
• 2013 Quality Review of Aid Program Performance Reports
• Lessons from Australian Aid report (2013)
• Australian Volunteers for International Development (AVID) Evaluation Report (2013)
• Thinking and Working Politically: An evaluation of policy dialogue in AusAID (2013)
• Building on Local Strengths: Evaluation of Australian law and justice assistance (2012)
• Responding to Crisis: Evaluation of the Australian aid program's contribution to the national HIV response in Papua New Guinea, 2006–2010
• From Seed to Scale-Up: Lessons learned from Australia's rural development assistance
• Working Beyond Government: Evaluation of AusAID's engagement with civil society in developing countries

2014 Quality Review of Aid Program Performance Reports

The 2014 Quality Review of APPRs found that the 2013–14 APPRs are well written, informative descriptions of program activities and progress towards objectives. They discuss successes and failures in a largely balanced and evidence-based manner and appear to be used by programs as performance management tools. A particular highlight in the APPRs was strong content on gender equality.

The quality review identified a few issues facing some APPRs. Firstly, many do not clearly explain program strategy–though the introduction of Aid Investment Plans (due later in 2015) should assist with this. Secondly a substantial number of program objectives, and/or indicators lack clarity. The review also compared the 2012–13 and 2013–14 APPRs and found there has been considerable movement in some programs' objectives over the two years, which needs to be considered if APPRs are to be used to measure changes in performance over time.

Both this and the previous year's quality review of APPRs demonstrates that the credibility of APPRs' assessments of program performance is enhanced when programs can draw on good quality performance assessment frameworks (PAFs). As such, the 2014 Quality Review recommends that all programs that produce APPRs should develop PAFs as part of their Aid Investment Plans.

The full 2014 Quality Review of APPRs can be found below:

2014 Quality Review of Aid Program Performance Reports (PDF 375kb)
2014 Quality Review of Aid Program Performance Reports (Word 219kb)

A desk based review of donor experiences in Sector-wide approaches in the health sector

A desk based review of donor experiences in Sector-wide approaches in the health sector' report presents the findings from a desk-based review of evidence on the effectiveness of donor support for the health sector through the use of sector-wide approaches (SWAps), sector budget support and government systems in six countries where the Australian aid program has used such approaches and/or systems (Bangladesh, Cambodia, Nepal, Papua New Guinea, Samoa, and Solomon Islands). The report is based on the findings of donors' (including Australia) experience in supporting sector-wide approaches in Asia and the Pacific. It does not evaluate the effectiveness of Australia's broader interventions in health system reform.

Read the review:

Sector wide approaches in the health sector: A desk-based review of donors' experience in Asia and the Pacific (PDF 1.29mb)
Sector wide approaches in the health sector: A desk-based review of donors' experience in Asia and the Pacific (Word 387kb)

Evaluation of the Australia-Vietnam country strategy 2010–15

ODE's evaluation of the Australia-Vietnam 2010 – 2015 country strategy assesses the effectiveness of the strategy and the results achieved through Australian aid up to 2013. As the current strategy period comes to an end the evaluation aims to help inform the development of a new Aid Investment Plan (a new process which replaces country strategies) to guide the targeting and delivery of Australia's ongoing aid program with Vietnam.

The evaluation found that the Australia-Vietnam country strategy 2010-15 was developed through a collaborative process which resulted in a robust framework for the Australia-Vietnam aid relationship. The implementation of the Australian aid program has also been strong, with good achievements in sectors such as rural water and sanitation, infrastructure and scholarships. However, there is room for improvement in areas such as monitoring the outcomes of Australia's aid to Vietnam and integrating gender issues and the private sector into the aid program.

The evaluation made four recommendations to strengthen future Australia-Vietnam Aid Investment Plans and the delivery of Australian aid to Vietnam. DFAT has accepted all four recommendations.

Read the evaluation report:

Evaluation of the Australia-Vietnam country strategy 2010–15 (PDF 0.99mb)
Evaluation of the Australia-Vietnam country strategy 2010–15 (Word 968kb)

A window of opportunity: Australian aid and child undernutrition

The Australian Government's 2014 development policy identifies early childhood nutrition as 'a critical driver of better development outcomes'. Undernutrition in children between conception and 2 years of age can result in irreversible stunting (short stature), poor cognitive development and poor health. This ultimately affects educational attainment and future productivity and earnings. Several countries in Australia's immediate neighbourhood have stunting rates comparable to those of Sub-Saharan Africa and South Asia. ODE's evaluation found that the Australian Government's investments generally constitute good practice. A number of recommendations were made to improve the targeting of child undernutrition (especially during the critical first 1000 days of a child's life) and the monitoring and reporting of Australia's efforts.

Read the summary brief:

ODE brief: A window of opportunity: Australian aid and child undernutrition (PDF 540kb)

Read the evaluation report:

A window of opportunity: Australian aid and child undernutrition (PDF 1.06mb)
A window of opportunity: Australian aid and child undernutrition (Word 1.33mb)

Research for better aid: an evaluation of DFAT's investments

ODE's Research for better aid evaluation assesses the degree to which the investment by the Australian aid program into research has been appropriate, effective and efficient, and provides recommendations for improving DFAT's future management of research investment. The evaluation highlights the need for robust knowledge management systems and a strong culture of research use to be embedded in the department. It also finds that there are clear benefits in looking to improve the investment in developing country research capacity, either directly or through partnerships with Australian and international researchers. The full evaluation report and management response can be found below.

Research for better aid: an evaluation of DFAT's investments (PDF 1.81mb)
Research for better aid: an evaluation of DFAT's investments (Word 6.66mb)

Working in decentralised service systems: challenges and choices for the Australian aid program

This is the full report of an ODE-commissioned evaluation (conducted December 2012–April 2014) which examined Australia's support for service systems in decentralised contexts. The evaluation focussed on the health, education and infrastructure (water, sanitation and roads) sectors. It examined programs and initiatives in Indonesia, Papua New Guinea, the Philippines, Solomon Islands, Vietnam and Bangladesh, with the first three countries covered in more depth through field work. The evaluation found Australian aid is beginning to respond to the challenges of supporting service delivery in decentralised contexts, but noted the results are mixed and there is room for improvement. The evaluation made four recommendations for improving country level analysis, program planning and design to better address decentralisation, and to strengthen DFAT's capacity to effectively manage and monitor service delivery programs in decentralised contexts.

Working in decentralised service systems: challenges and choices for the Australian aid program–Full report (PDF 3.02mb)
Working in decentralised service systems: challenges and choices for the Australian aid program–Full report (Word 3.84mb)

Working in decentralised service systems: Short report on implications for aid managers

This is the ODE summary of the findings and recommendations of the evaluation Working in decentralised service systems: challenges and choices for the Australian aid program (Office of Development Effectiveness, 2015). The short report synthesises advice for program managers from the full evaluation report, drawing on Australia's experience with decentralised delivery of health, education and infrastructure (water, sanitation and roads) in Bangladesh, Indonesia, Papua New Guinea, the Philippines, Solomon Islands and Vietnam. Three priorities are suggested for improving future approaches: i) work to improve service systems rather than deliver services where this is feasible; ii) choose implementation partners carefully, and consider unintended effects; iii) work towards consistent and coordinated investments, policies and systems.

Working in decentralised service systems: Short report on implications for aid managers (PDF 193kb)
Working in decentralised service systems: Short report on implications for aid managers (Word 198kb)

Smart Economics: Evaluation of Australian aid support for women's economic empowerment

ODE has released its evaluation of Australian aid support for women's economic empowerment. The evaluation assesses the effectiveness of Australia's policies and programming to promote women's economic development. It finds that whilst Australia's policies to promote women's economic empowerment are sound, implementation approaches are generally weak. The evaluation makes four key recommendations to improve the effectiveness of Australian economic programming.

Read the full evaluation report, management response, policy brief and related documents

Australia's response to the Horn of Africa Humanitarian crisis, 2011 Evaluation Report

ODE has released its evaluation of Australia's response to the Horn of Africa Humanitarian crisis, 2011. The evaluation aimed to assess the effectiveness of Australia's response and identify how to improve Australian humanitarian assistance to future slow-onset crises.
The evaluation found that Australia's response was in keeping with the magnitude of the crisis. Australia's leadership role and the speed and flexibility of Australia's response were notable strengths.
The full evaluation report and management response can be found below.

Australia's response to the Horn of Africa Humanitarian crisis, 2011 Evaluation Report (PDF 3.30mb)
Australia's response to the Horn of Africa Humanitarian crisis, 2011 Evaluation Report (Word 9.70mb)

Read the summary briefs:

ODE Brief: Horn of Africa humanitarian crisis, 2011 (PDF 1.31mb)
ODE Brief: Gender dimension, Horn of Africa humanitarian crisis, 2011 (PDF 1.08mb)

Two reviews of Australian aid operational evaluations

The Government is committed to improving the effectiveness and efficiency of Australian aid, and has introduced a new performance framework which will see funding at all levels of the aid program linked to progress against rigorous targets and performance benchmarks. Independent evaluations are another important way of ensuring a strong focus on results and value for money.

All significant Australian aid initiatives require independent evaluation at least once, under current DFAT policy. These independent operational evaluations are commissioned and managed by DFAT areas with aid management responsibilities. They are an essential part of Australia's aid performance management and reporting system. The 87 operational evaluations completed in 2012 are the subject of these two ODE publications.

Quality of Australian aid operational evaluations assesses the quality of the evaluations, and considers underlying factors influencing evaluation quality and utility.

The review confirms that the majority of operational evaluations (74 per cent) are credible, and provide robust evidence about the performance of the Australian aid program.

The review identifies several areas for further improvement. It provides practical guidance for the DFAT areas that commission operational evaluations, as well as specific recommendations to improve evaluation quality and utility.

The reviewed operational evaluations that have been published can be accessed via ODE's operational evaluations page.

Read the summary brief:

ODE Brief: Quality of Australian aid operational evaluations (PDF 1mb)
ODE Brief: Quality of Australian aid operational evaluations (Word 841kb)

Read the full report and management response:

Quality of Australian aid operational evaluations (PDF 1.63mb)
Quality of Australian aid operational evaluations (Word 568kb)

Learning from Australian aid operational evaluations synthesises the findings of the 64 evaluations assessed as credible and offering lessons of potential value to a wider audience.

This review addresses an identified gap in the dissemination of Australian aid evaluation findings since many of the original evaluation reports can be hard to locate or readily digest. The review's purpose is to inform and improve program design and management within the Australian aid program, and to provide lessons of value to the broader development community.

The report identifies nine key lessons for improving the effectiveness of the Australian aid program.

Read the full report:

Learning from Australian aid operational evaluations (PDF 1.27mb)
Learning from Australian aid operational evaluations (Word 468kb)

Evaluation of Australian aid to Timor-Leste

ODE's evaluation of Australian aid to Timor-Leste focused on the planning and implementation of aid to Timor-Leste from 2006 to 2013. It concluded that the country program has made steady but uneven progress in implementing several of the country strategy's key commitments, and that DFAT needs to build coherent, long-term and appropriately-resourced programs that address a limited set of clear and compelling development objectives.

ODE also found that in countries affected by or transitioning from conflict, shorter-term, more flexible planning approaches which focus on prioritising and sequencing aid efforts may be required. These should pay stronger attention to resourcing requirements than in contexts where business-as-usual approaches may be adequate.

The evaluation made five recommendations aimed at strengthening the effectiveness of assistance to Timor-Leste, and the effectiveness of DFAT's country program aid more broadly.
DFAT has accepted all the recommendations.

Read the evaluation report:

Evaluation of Australian aid to Timor-Leste report (PDF 5.59mb)
Evaluation of Australian aid to Timor-Leste report (Word 4.37mb)

Read the summary brief:

ODE Brief: Australian aid to Timor-Leste (PDF 1.12mb)

Evidence reviews (2014)

ODE has released three evidence reviews that will inform its current work on evaluating child nutrition, women's leadership and economic empowerment, and teacher quality. These reviews gather and assess evidence that can be used to inform decisions about the scope, focus, questions and methods of proposed evaluations.

Teacher quality: Evidence review (2014)

An evidence review was commissioned to gather and assess evidence and inform recommendations about appropriate research methods for an evaluation of Australian aid for teacher development.

The review focuses on:

• the meaning of teacher quality and its relationship to student outcomes
• factors that influence teacher quality
• international influence and engagement with teacher quality.

Teacher quality evidence review (PDF 1.01mb)
Teacher quality evidence review (Word 1.41mb)

Women's leadership: Evidence review

ODE has commissioned an evidence review of women's leadership. This topic is the second of the three evaluations focused on gender equality and women's empowerment.

The review focuses on:

• Definition of women's leadership: How does Australian aid define women's leadership (in policies, strategies or programs)? How does this compare to other donor approaches and international literature?
• Data availability and quality: Are there adequate data (in Australian aid and other donor contexts) to assess the effectiveness of women's leadership programs on gender equality outcomes and broader development outcomes? What is the quality of these data?
• Leadership and empowerment links: Is there evidence to link women's economic empowerment and women's leadership outcomes? What does existing evidence say and what is the quality of this evidence?

Women's leadership: Evidence review (PDF 1.32mb)
Women's leadership: Evidence review (Word 2.09mb)

Child nutrition

Addressing child nutrition: Evidence review (2014)

ODE has commissioned a review of the current evidence-based approaches to addressing child under-nutrition and how they may apply to the Australian aid program. The review focuses on:

• interventions to address child under-nutrition
• contemporary policy thinking and approaches to nutrition
• child nutrition data.

Addressing child nutrition: Evidence review (PDF 2.07mb)
Addressing child nutrition: Evidence review (Word 1.47mb)

2013 Quality Review of Aid Program Performance Reports

ODE assesses the quality of APPRs and other performance management instruments that DFAT uses to monitor and evaluate the aid program.

The 2013 ODE APPR Quality Review confirms a gradual, clear improvement in the quality of APPRs since 2007-08 when they were first introduced. It found that the reports generally provide frank, well-written explanations of performance and serve their accountability role well.

APPRs have the potential to provide considerable guidance to senior managers on what constrains and enables good performance in programs. They highlight three key drivers of good program performance:

1. Effective policy dialogue to enable stronger partnerships and improved country "ownership" of aid programs
2. Capacity-building focused on institutional support
3. Less fragmentation of programs.

To improve their clarity and credibility, ODE's Quality Review recommends that APPRs provide better explanations about the rationale underlying program objectives, broaden the use of credible evidence (especially that gathered from partners), and improve the utility of the reports as management tools. Whole of Government performance also needs to be better assessed in APPRs.

The full Quality Review can be found below:

• 2013 Quality Review of Aid Program Performance Reports (PDF 668kb)
  2013 Quality Review of Aid Program Performance Reports (Word 1.39mb)

Lessons from Australian Aid report

The Office of Development Effectiveness (ODE) has released its 2013 Lessons from Australian Aid report. The first in an annual series, this major report provides an independent perspective on the strengths and weaknesses of the Australian aid program.

Lessons from Australian Aid synthesises findings from six recent strategic evaluations conducted by ODE. It also analyses the findings from 2012-13 performance reporting undertaken by DFAT program managers, and quality-assures these products. The report contains useful lessons for Department of Foreign Affairs & Trade (DFAT) aid managers, grouped around three main themes:

• Well-informed policy dialogue with Australia's development partners helps create high-quality policy frameworks. As a result, Australian aid reaches more of the poor.
• Government isn't the only development partner–DFAT should keep building strong relationships with the private sector and civil society as well, to leverage high-quality outcomes.
• In low-capacity settings, building the capacity of institutions (rather than individuals) in ways that are tailored and paced to reflect local realities holds most potential for success.

While these lessons are not new to international development, they continue to be relevant to the Australian aid program. They also pose a challenge for all other major donors. Applying the report's lessons consistently across the aid program should improve the effectiveness and impact of Australia's development assistance.

The full evaluation report can be found below:

• Lessons from Australian Aid–2013 report on independent evaluation and quality assurance (PDF 527kb)
  Lessons from Australian Aid–2013 report on independent evaluation and quality assurance (Word 503kb)

Australian Volunteers for International Development (AVID) Evaluation Report

ODE has released its evaluation of the Australian Volunteers for International Development (AVID) program. The evaluation aimed to enhance the effectiveness of the AVID program and the contribution that volunteers make to development.

Launched in 2011, AVID was designed to unite a variety of volunteer programs into one major program. AVID is funded and managed by DFAT, and is delivered by three core partners: Australian Volunteers International, Australian Red Cross, and Austraining International.

Overall, the evaluation found that AVID is making an effective and highly-visible contribution to the Australian Government's development and public diplomacy objectives. Volunteers contribute to developing the capacity of their host organisations, develop people to people links and generate goodwill for domestic and foreign diplomacy.

However, the program's efficiency and effectiveness can be improved and the evaluation makes recommendations to this effect including:

• Further consolidating AVID into a single program through one volunteer stream
• Improved branding of the program
• Improved administrative efficiencies including through consolidation of the number of countries receiving volunteers
• Greater focus on developing the long-term capacity of host organisations
• Developing a simplified and effective performance monitoring system to promote continuous improvement of the AVID program.

Read the summary brief:

• AVID summary brief (PDF 536kb)
  AVID summary brief (Word 452kb)

The full evaluation report and management response can be found below:

• Evaluation of the Australian Volunteers for International Development (AVID) program (PDF 1.74mb)
  Evaluation of the Australian Volunteers for International Development (AVID) program (Word 3.08mb)

Thinking and Working Politically: An evaluation of policy dialogue in AusAID

The Office of Development Effectiveness (ODE) has released its evaluation, Thinking and Working Politically: An evaluation of policy dialogue in AusAID.

The need for AusAID to identify, define and practice good policy dialogue was highlighted in ODE's 2009 Annual Review of Development Effectiveness and the 2008 OECD Development Assistance Committee peer review of Australia's aid program.

The evaluation included a literature review of international best practice, case studies from Indonesia and Solomon Islands, interviews with AusAID staff and a web-based staff survey.

Broadly, the evaluation recommended that policy dialogue be better incorporated into the agency's practices by:

• promoting a common understanding and providing senior direction on policy dialogue
• embedding policy dialogue into aid management practices
• ensuring policy dialogue is properly resourced
• supporting the skills development of staff.

More about the evaluation of policy dialogue in AusAID

Building on Local Strengths: Evaluation of Australian law and justice assistance

Law and justice is an increasingly important area of international development assistance, but there is little consensus among OECD donors on how exactly law and justice contributes to wider development processes. Australia has emerged as a leader in this area, delivering law and justice assistance through a number of Australian Government agencies including DFAT, the Australian Federal Police and the Attorney-General's Department. Law and justice assistance comprises nearly 15 per cent of the bilateral aid program.

ODE's evaluation, Building on Local Strengths: Evaluation of Australian law and justice assistance, assessed the relevance and effectiveness of Australia's law and justice programs.

More about the evaluation of the Australian law and justice assistance

Responding to Crisis: Evaluation of the Australian aid program's contribution to the national HIV response in Papua New Guinea, 2006–2010

The Australian aid program has played a prominent role in responding to HIV in Papua New Guinea (PNG) since 1995. The Office of Development Effectiveness conducted a major independent evaluation into AusAID's contribution to the response. The evaluation focused on the current HIV program activities from 2006, led by the PNG-Australia HIV and AIDS Program, in the broader historical context.

Responding to Crisis: Evaluation of the Australia aid program's contribution to the national HIV response in Papua New Guinea finds that AusAID has been a major driver of the national HIV response in PNG. However, the relevance and effectiveness of AusAID's interventions has been mixed.

In light of the evolving nature of the epidemic and response, the evaluation recommendations that AusAID:

• moves to an integrated health approach, with HIV activities managed as part of broader sexual and reproductive health activities
• steps back from a dominant role in shaping and implementing the response and intensifies support for PNG champions of the HIV response
• retains the leadership of the in-house senior technical expert but contracts out grant management and capacity building functions.

AusAID has provided a management response to the evaluation.

From Seed to Scale-Up: Lessons learned from Australia's rural development assistance (2012)

ODE has conducted a review of Australia's rural development assistance, to identify ways of maximising the benefits of Australia's growing investment in rural development. From Seed to Scale-Up: Lessons learned from Australia's rural development assistance reviewed 23 recent Australian activities across six countries in the Asia-Pacific. These activities were implemented by AusAID and/or the Australian Centre for International Agricultural Research (ACIAR).

The review found that Australian aid has helped poor rural women and men access more value from new markets and make more effective use of scarce natural resources. Australian aid has also helped them to accumulate assets so that they can afford to send children to school, pay for health care and gain access to other essential services.

From Seed to Scale-Up has identified 12 principles that characterise efficient and effective aid investment in rural development and makes recommendations to increase the capacity of Australia and its development partners to deliver results in rural development A joint AusAID-ACIAR management response is included in the report.

Working Beyond Government: Evaluation of AusAID's engagement with civil society in developing countries (2012)

Civil society in developing countries can be a powerful agent for change, and is an important development partner in addition to partner governments and the private sector. ODE has conducted an evaluation of AusAID's engagement with civil society, assessing how well AusAID is helping civil society in developing countries contribute to the development of effective states.

Working Beyond Government: Evaluation of AusAID's engagement with civil society in developing countries recommends that Australia build on current strategic approaches for engaging with civil society, work with local systems and partners, and apply good practice in the design of individual programs. The evaluation also found that choosing appropriate intermediaries can help donors enhance sustainability, manage risk, reduce high transaction costs and take small activities to scale.

Read the Working Beyond Government evaluation

Philippines Country Program and Strategy Evaluation (2012)

Country strategy and program evaluations are an important way for ODE to evaluate the performance of the Australian aid program. In 2007, the Philippines country strategy was the first to be evaluated by ODE. This evaluation follows on from that first one and provides a mid-term assessment of the Australia–Philippines Development Assistance Strategy for 2007–11.

Full report

• Australian aid to the Philippines–Mid-term evaluation of the Australia–Philippines Development Assistance Strategy 2007–11 [PDF 1.2mb]
  Australian aid to the Philippines–Mid-term evaluation of the Australia–Philippines Development Assistance Strategy 2007–11 [Word 5.3mb]

Supporting documentation for this evaluation

• Gender Assessment of the Australian Philippines Aid Program [PDF 1.1mb]
  Gender Assessment of the Australian Philippines Aid Program [Word 1.2mb]