The conduct of impact evaluation is a key to solving many of the country's problems. This was stressed by Education Secretary Bro. Armin Luistro on December 16 at the Policy Forum on Impact Evaluation organized by state think tank Philippine Institute for Development Studies (PIDS), National Economic and Development Authority (NEDA), and International Initiative for Impact Evaluation (3ie). Luistro, who closed the event, urged representatives from various government agencies to adopt the practice of evaluating the impact of their programs and projects.
"Unless you are able to say that a particular program is successful or that it needs to be enhanced and improved, it is just another activity," he said.
Impact evaluation can help improve government services by providing a deeper understanding of their effectiveness and by assisting leaders and policymakers make informed decisions.
NEDA Deputy Director-General Emmanuel Esguerra who represented Planning Secretary Arsenio Balisacan stressed the importance of institutionalizing impact evaluation across the government bureaucracy. Related to this, Australian Aid Counsellor Robyn Biti said her office awarded three research grants that will help some government agencies assess the impact of their programs. The 3ie administers one of these grants, aiming to conduct an impact evaluation of the Department of Social Welfare and Development's Sustainable Livelihood Program. The other grants are for the impact evaluation of the programs of the Office of the President Adviser on the Peace Process, the Department of Labor and Employment, and the Supreme Court.
Impact evaluation provides evidence of effectiveness
The forum also featured presentations of impact evaluation cases in the Philippines presented by PIDS Visiting Research Fellow and former World Bank Lead Expert Vicente Paqueo.
Three major government programs have been subject to impact evaluation, namely the Pantawid Pamilya Program (4Ps), the Legal Minimum Wage Policy, and the Kapit-Bisig Laban sa Kahirapan-Comprehensive and Integrated Delivery of Social Services (KALAHI-CIDSS) program.
The 4Ps was designed to assist poor households through conditional cash transfers tied to education and health objectives. Grants are given on the condition that their children will attend school and visit health centers regularly.
Throughout the implementation of the program, critics panned the 4Ps saying it was not effective at all. They claimed that it promoted dependency and laziness among the household beneficiaries.
Paqueo argued, however, that impact evaluation studies conducted so far show the program is, in fact, effective. It has improved key education and health indicators. Criticisms that it encouraged laziness and dependency proved to be unfounded. The hours of labor that people put in work did not change. The number of hours that children put in work declined but this is because they were going to school instead.
Paqueo said that despite these results, it is important to emphasize that the 4Ps is not a panacea. Necessary adjustments still have to be made to improve the program to make it more effective and sustainable.
In another case study Paqueo cited, impact evaluation was able to demonstrate the risk and cost of adopting policies without appropriate empirical evidence, and the fact that what might work in other countries may not always have the same effect locally.
The objective of the legal minimum wage policy (LMW) was to raise the household minimum wage of low-wage earners. But after evaluating the policy's real impacts in the Philippine setting, it turned out that it was doing more harm than good.
By raising the LMW, employment and work hours were reduced especially in small firms in order to cope with increased operating expenses. The rise in wages significantly decreased average household income by 20 percent.
In short, the policy negatively affected the very people it wanted to help.
Paqueo said the case study underscored the value of evaluating government policies to determine their unintended consequences and to think of more effective alternative policies.
In the last case study presented by Paqueo, he again demonstrated the importance of impact evaluation in producing new knowledge from unintended consequences. The study on the impact of the KALAHI-CIDSS program in conflict areas has shed new light on the mechanisms that link aid and conflict. The results showed that KALAHI-CIDSS led to an increase in violent conflict contrary to the popular belief that development aid helped reduce conflict. A development program threatens to reduce local support for insurgents' cause, creating a backlash from insurgent groups to try and sabotage the program.
The future of impact evaluation in Philippine governance
Impact evaluation is crucial to improving the way the government facilitates development but its importance needs to be widely appreciated for successful take-up by leaders and policymakers.
PIDS has started the groundwork by undertaking an impact evaluation project in 2014. The project involves the conduct of process evaluation and impact assessment of key government programs as well as capacitating the monitoring and evaluation staff of the NEDA, the Department of Budget and Management, and other key agencies in conducting impact evaluation. The project consists of 23 impact evaluation studies and several technical trainings on how to conduct impact evaluation.
For PIDS President Gilberto Llanto, these efforts are not enough. "To sustain what we started, we need to bring the importance of impact evaluation to the consciousness of other officials."
The policy forum was conducted to further raise awareness and appreciation of the importance of impact evaluation. On January 4-15, 2016, a two-week training course on impact evaluation methods organized by PIDS and 3ie will be held at the new PIDS' office in Quezon City. Thirty participants from selected regional universities and government agencies are expected to attend. The course will teach participants commonly used econometric and statistical methods to evaluate the impacts of social and other programs in developing countries.
Institutionalizing the practice of impact evaluation in government requires promoting a culture of evaluation, according to 3ie Executive Director Emmanuel Jimenez. Successful take-up, he said, needs the presence of certain factors, such as credible research, buy-in from key champions, continued engagement between researchers and policymakers across all stages of conducting impact evaluation, and timely and effective communication of impact evaluation results.
The cost of conducting impact evaluation may be substantial but Luistro countered that making it a regular activity will help government do a better job of serving the Filipino people, and in compelling the different agencies to work more in sync with one another in the areas of research, implementation, and evaluation. ###
"Unless you are able to say that a particular program is successful or that it needs to be enhanced and improved, it is just another activity," he said.
Impact evaluation can help improve government services by providing a deeper understanding of their effectiveness and by assisting leaders and policymakers make informed decisions.
NEDA Deputy Director-General Emmanuel Esguerra who represented Planning Secretary Arsenio Balisacan stressed the importance of institutionalizing impact evaluation across the government bureaucracy. Related to this, Australian Aid Counsellor Robyn Biti said her office awarded three research grants that will help some government agencies assess the impact of their programs. The 3ie administers one of these grants, aiming to conduct an impact evaluation of the Department of Social Welfare and Development's Sustainable Livelihood Program. The other grants are for the impact evaluation of the programs of the Office of the President Adviser on the Peace Process, the Department of Labor and Employment, and the Supreme Court.
Impact evaluation provides evidence of effectiveness
The forum also featured presentations of impact evaluation cases in the Philippines presented by PIDS Visiting Research Fellow and former World Bank Lead Expert Vicente Paqueo.
Three major government programs have been subject to impact evaluation, namely the Pantawid Pamilya Program (4Ps), the Legal Minimum Wage Policy, and the Kapit-Bisig Laban sa Kahirapan-Comprehensive and Integrated Delivery of Social Services (KALAHI-CIDSS) program.
The 4Ps was designed to assist poor households through conditional cash transfers tied to education and health objectives. Grants are given on the condition that their children will attend school and visit health centers regularly.
Throughout the implementation of the program, critics panned the 4Ps saying it was not effective at all. They claimed that it promoted dependency and laziness among the household beneficiaries.
Paqueo argued, however, that impact evaluation studies conducted so far show the program is, in fact, effective. It has improved key education and health indicators. Criticisms that it encouraged laziness and dependency proved to be unfounded. The hours of labor that people put in work did not change. The number of hours that children put in work declined but this is because they were going to school instead.
Paqueo said that despite these results, it is important to emphasize that the 4Ps is not a panacea. Necessary adjustments still have to be made to improve the program to make it more effective and sustainable.
In another case study Paqueo cited, impact evaluation was able to demonstrate the risk and cost of adopting policies without appropriate empirical evidence, and the fact that what might work in other countries may not always have the same effect locally.
The objective of the legal minimum wage policy (LMW) was to raise the household minimum wage of low-wage earners. But after evaluating the policy's real impacts in the Philippine setting, it turned out that it was doing more harm than good.
By raising the LMW, employment and work hours were reduced especially in small firms in order to cope with increased operating expenses. The rise in wages significantly decreased average household income by 20 percent.
In short, the policy negatively affected the very people it wanted to help.
Paqueo said the case study underscored the value of evaluating government policies to determine their unintended consequences and to think of more effective alternative policies.
In the last case study presented by Paqueo, he again demonstrated the importance of impact evaluation in producing new knowledge from unintended consequences. The study on the impact of the KALAHI-CIDSS program in conflict areas has shed new light on the mechanisms that link aid and conflict. The results showed that KALAHI-CIDSS led to an increase in violent conflict contrary to the popular belief that development aid helped reduce conflict. A development program threatens to reduce local support for insurgents' cause, creating a backlash from insurgent groups to try and sabotage the program.
The future of impact evaluation in Philippine governance
Impact evaluation is crucial to improving the way the government facilitates development but its importance needs to be widely appreciated for successful take-up by leaders and policymakers.
PIDS has started the groundwork by undertaking an impact evaluation project in 2014. The project involves the conduct of process evaluation and impact assessment of key government programs as well as capacitating the monitoring and evaluation staff of the NEDA, the Department of Budget and Management, and other key agencies in conducting impact evaluation. The project consists of 23 impact evaluation studies and several technical trainings on how to conduct impact evaluation.
For PIDS President Gilberto Llanto, these efforts are not enough. "To sustain what we started, we need to bring the importance of impact evaluation to the consciousness of other officials."
The policy forum was conducted to further raise awareness and appreciation of the importance of impact evaluation. On January 4-15, 2016, a two-week training course on impact evaluation methods organized by PIDS and 3ie will be held at the new PIDS' office in Quezon City. Thirty participants from selected regional universities and government agencies are expected to attend. The course will teach participants commonly used econometric and statistical methods to evaluate the impacts of social and other programs in developing countries.
Institutionalizing the practice of impact evaluation in government requires promoting a culture of evaluation, according to 3ie Executive Director Emmanuel Jimenez. Successful take-up, he said, needs the presence of certain factors, such as credible research, buy-in from key champions, continued engagement between researchers and policymakers across all stages of conducting impact evaluation, and timely and effective communication of impact evaluation results.
The cost of conducting impact evaluation may be substantial but Luistro countered that making it a regular activity will help government do a better job of serving the Filipino people, and in compelling the different agencies to work more in sync with one another in the areas of research, implementation, and evaluation. ###
