Call for Evaluation Proposals
Plan for the Impact Evaluation of Program to Support Employment in Bolivia
The purpose of this call for proposals is to select a candidate(s) who, in collaboration with specialists in the Labor Markets and Social Security Unit of the IDB will be responsible for designing the impact evaluation of the labor market project “Program to Support Employment” in Bolivia (BO-L1051), the technical supervision of the data collection process and the econometric analysis.
2.1 The program BO-L1051 “Program to Support Employment” in Bolivia
The Inter-American Development Bank approved in 2010 the project BO-L1051 “Program to support employment” in Bolivia. The general objective of the program is to support the Ministry of Labor, Employment, and Social Security (MTEPS) in its efforts to expand the coverage and effectiveness of its active labor market policies, thereby helping to achieve better integration of workers into the labor market. This objective will be achieved through the following three components:
- Component 1. Expansion of the Employment Service (US$6.4 million)
- Component 2. Employment support for adults (Programa de Apoyo al Empleo de Adultos, PAE) (US$11.9 million)
- Component 3. Planning, analysis of employment policies, and evaluation (US$1 million)
A decision was made to opt for a program that could be easily implemented given Bolivia’s incipient institutional capacity to develop labor market policies for labor intermediation and training. These are areas in which the MTEPS has been acquiring experience thanks to the introduction of a youth training program, “My First Decent Job” (Mi Primer Empleo Digno, MPED), in 2009. The intention is to draw on that experience to consolidate a comprehensive training and employment program built around three objectives: (i) scale up intermediation between employers and job seekers through enhanced dissemination of information on labor supply and demand and access to higher quality job opportunities; (ii) develop and consolidate a new on-the-job-training (OJT) and placement program to support better labor insertion for adults; and (iii) strengthen planning, employment policy analysis, and evaluation.
The main aim of this project is to increase the probability of formal employment and workers’ earnings through the development of active employment policies. The project will support two types of active labor market policies. On the one hand, the project is going to devote a substantial portion of its funds to strengthening the labor market intermediation system by improving the quality of labor intermediation services provided at employment offices. On the other hand, the project will be devoted to the provision and administration of short term, OJT and placement program targeted to adults.
By improving the employment service in Bolivia, the project aims to reduce labor market frictions which contribute to unemployment and high under-employment. Thus, the project addresses a number of institutional weaknesses, including the lack of a good intermediation system that provides information to both sides of the labor market – information about vacancies to job-seekers and information about qualified applicants to employers. In addition, the expansion and strengthening of the employment service aims to ameliorate barriers confronting job seekers, including the lack of a good support system to guide them through the job search process and lack of referrals to vouch for their credentials and trustworthiness.
On its part, the adult training and placement program contemplates the provision of specific supplemental training to address the lack of skills among adults who may have already dropped out of the formal education system. Unemployment and under-employment is much higher in Bolivia among high-school drop-outs. Given that skill mismatches are likely to also contribute to unemployment and under-employment, training programs may help overcome this problem . Moreover, training, just like education, may provide a signal to employers hiring in good jobs and helps employees gain access to high quality jobs. The PAE is a training program for adults, which provides internships with employers for those job seekers that match the requirements of a vacancy . On the job training has been deemed to be effective in other Latin American countries (since employers are best informed about the types of skills they actually need), yet there is not robust evidence to support this . The program is expected to increase employment opportunities, but also to enhance the quality of the workforce and the productivity of participants (measured by earnings), and the impact evaluation should provide evidence of that.
The project includes an important monitoring and evaluation component. It includes an impact evaluation of the employment service’s referral and placement actions -in terms of its impact on job placement (probability of formal employment and earnings). It also includes an impact evaluation of the OJT and placement program PAE. The proposals received as a result from this call should address evaluation of each of these parts and be explicit about the methodology or methodologies that will be used for the evaluation of the employment service and the adult training program. At the same time, the proposal should address whether new data collection will be required or whether secondary sources of data will be used. For this purpose, it is useful to keep in mind that the program involves the development of an information system that will be managed by the employment service to track beneficiary and employer characteristics as well as outcome indicators such a placement and duration of both job search and retention of jobs once placed. In particular, as part of the strengthening of the employment service, job-seekers will be required to complete a questionnaire as a tool to guide applicants through the job search process and to help them locate appropriate vacancies. Assuming the information system will be functioning at the time of baseline data collection, the information system could serve as a mechanism to provide baseline information on job applicants; the proposal should suggest the type of information needed for baseline and follow-up data collections.
2.2 Institutions and functioning of the labor market in Bolivia
Institutional framework of the Bolivian labor market.At the time of project design, the MTEPS had a fledgling institutional structure consisting of an Employment Service with limited coverage and scant technical and operational capacity. Moreover, there was a lack of active labor market policies to help workers find their way to jobs. An exception to this is a pilot program to train unskilled youth 18 to 24 years of age and help them enter the labor market, called “Mi Primer Empleo Digno” [My First Decent Job] (MPED). The principal contribution for the MTEPS in the near term is to modernize and extend its employment offices as to provide services that, combined with other activities , will both attract a greater number of job-seekers as well as more vacancy-postings by employers. At the same time, the employment service can be used to consolidate active labor market policies, including MPED and PAE. The MTEPS currently operates a network of offices that promote employment services in key municipalities (municipios) throughout the country, offering assistance to search for and obtain labor market information (employment opportunities, training opportunities, among others). The Ministry is also developing a Web-based market information system accessible to the general public throughout the country. Although the implementation of the project has recently begun, the Employment Service still faces some of the operational challenges identified during the design of the project, such as: (i) lack of resources, staff, and instruments to generate in-person job placement services as part of the MPED program; (ii) limited information service to match job training to market demand; (iii) challenges of a technical, operational (decentralization of administrative, communications, and procedural control functions) and technological (data system development, maintenance, and updates) nature in the local and regional offices; and (iv) scarcity of work and logistical tools to match job supply and demand and provide labor training.
Labor market assessment. About 4.8 million of the country’s almost 10 million inhabitants (65% in urban areas) are employed, although with a very high informality rate: 73.9% nationwide (Household Survey 2007).  The working age and employed population is mostly concentrated in urban areas. According to 2007 data, Bolivia has 7.6 million people of working age , 61% of whom are employed. In urban areas there are 5 million working-age people, and 2.7 million people are employed (53%). Among the urban employed, 52% have not completed secondary education, 19% have completed secondary education, and 28% have completed post-secondary studies.
More than 60% of the urban employed are informal workers , a far higher proportion than in other countries of the region. Nationwide, nearly 90% of the employed have no pension coverage or other basic benefits.  Another related issue is underemployment, or the partial underutilization of labor, which affects 21% of the urban employed in Bolivia. Over the past two decades, unemployment has been close to 7% in Bolivia, growing from 7.2% in 1999 to 7.7% in 2007 in urban areas. Although this is comparable to the regional average (8%), it obscures significant differences between groups based on age, gender, educational level, etc. In urban areas, for example, the unemployment rate for young people aged 18 to 24 is 10.6% for high school graduates versus 18% for those who graduated from higher education. The percentages are 6.5% and 10.6%, respectively, for people aged 25 to 35 years.
2.3 Evaluation of employment services and training programs
The impact evaluation of a social program always imposes a methodological challenge, since it is impossible to observe the “counterfactual”, i.e. what would have happened to the program participants in the absence of the program? Therefore, it is necessary to define a valid comparison (control) group to approximate this counterfactual. The credibility of the evaluation relies on the quality of the comparison group. Finding a valid comparison group becomes more difficult when it is not possible to randomly assign people to obtaining / not obtaining the service provided by the program. This is the case of employment services in Bolivia. The employment service offers a variety of services to any person that enters the office, regardless of current employment status, education achievements or any other observable characteristics. Voluntary access to the services offered poses a significant challenge for the determination of a valid comparison group. However, there is room to use regression discontinuity design (RDD) with eligibility of applicants, or random encouragement.
On the other hand, there is a vast literature of impact evaluations of training programs, especially for youth. Gonzalez-Velosa, Ripani and Rosas-Shady (2012) provide a complete and systematic review of training programs and their impact evaluations. They start by reviewing and classifying training programs into two main groups: Type-I programs, which combine off-job (in a classroom) and on-the-job (internship in a firm) training, such as MPED in Bolivia, and Type-II programs, which offer only on-the-job training, such as PAE in Bolivia. This distinction is important in terms of impact evaluation design and the interpretations of the results. As Gonzalez-Velosa et al point out, There is a lack of consistency among evaluation results, which can be explained not only by differences in methodologies used (e.g. randomized versus ex-post evaluations), but also by differences in the type of services offered by each training program. They emphasize that current evaluations are unable to identify the mechanisms that explain the impacts. On the one hand, studies fail to isolate the impact of the different services offered. For example, there are programs that combine off-job (in class room) training with on-the-job training, other programs that also include an internship, or others that combine on-the-job training with internships and/or intermediation services. On the other hand, they do not allow concluding under which context or circumstances these types of services are more effective; for example, it is not clear which participants’ characteristics and/or local labor markets favor these types of programs. In addition, only in rare occasions these evaluations include cost-effectiveness analysis, which would be very useful for comparisons with other alternative programs.
3. Contents of the proposal
The proposal should detail the experience of the researcher with impact evaluation studies and a preliminary description of the methodology and implementation plan. The selection of the candidate will be based on these two main parts, which should include:
PART I: Experience of the researcher with impact evaluation studies, including name and curricula vitae (four pages maximum) of the researcher describing previous experience in evaluations and publications of impact evaluation studies.
- a. Researchers may present proposals individually or jointly with other researchers. For administrative purposes, the Bank will request that each researcher signs an individual contract.
- b. Fluency in Spanish is required. If more than one researcher is involved, it is sufficient that at least one of them is completely fluent in Spanish.
Part II: Preliminary description of the methodology and implementation plan including:
A: Methodological proposal (5 page maximum)
The proposal should be based on a review of current documentation including: loan proposal, annexes, monitoring and evaluation plan, information system report, and other documents provided by the IDB and/or available in the literature. The review should consider that the survey “Encuesta Trimestral de Empleo (ETE)” is no longer be available at time of program implementation, due to the decision of the government to use that funding to implement a monthly employment survey, which is not a panel.
The methodological proposal should describe the proposed ways to evaluate the effectiveness of the modernized employment offices in getting workers out of unemployment or into better jobs. For example, knowledge and information about the services offered by the new offices may vary and may be manipulated through random encouragement (see Diamond and Hainmueller (2007)). In this case, the proposal needs to give serious consideration to the control group that will be used for this purpose and how to collect data on this group, as these will be individuals who do not approach the employment offices in the first place. Focusing the evaluation on the two or three main (biggest) employment offices in the country might be an option. The proposal may also want to consider the possibility that modernized offices offer basic services similar to what they used to offer to some job-seekers and new assessments and improved placement services to other job-seekers for example on the basis of age. Blundell et al. (2010) exploit both the random assignment across locations as well as the differential offer of job search assistance to some age groups but not others. The proposal may want to offer a number of alternatives and discuss the pros and cons of each of these in determining the effectiveness of the soon-to-be improved and modernized employment offices. It might be better to focus the evaluation to the offices located in the most populated areas such as La Paz, Santa Cruz, or Cochabamba.
In addition, the methodological proposal needs to detail the best way to evaluate the adult training program “PAE” that will be administered through the employment offices. Following the discussion in section 1.3 above, training programs in Latin America have been evaluated using various methods depending on the availability of data. Most notably, the best evaluations of training programs in low- and middle-income countries have been conducting as randomized trials in Colombia (Attanasio, Kugler and Meghir, 2011) and in the Dominican Republic (Card et al, 2011, Ripani et al, 2011). However, since random assignment of individuals may not be a possibility in Bolivia, other alternatives may have to be considered. For instance, a study for Peru (Galdo and Chong, 2006) uses matching to determine the effectiveness of training programs of different qualities. While matching may be a good alternative in this context, for matching to work it is important to obtain information on pre-program labor histories and characteristics so it will be important for the proposal to detail what information will be used to implement the matching techniques. Finally, regression discontinuity designs have recently been used in the evaluation of vocational training (Malamud and Pop-Eleches, 2010). In this context, it may be desirable to exploit discontinuities in the scores provided by the placement officers in the employment offices to identify the causal effect of training. Just as in the evaluation of employment offices, here too the proposal may want to detail a couple of alternative non experimental methodologies and the pros and cons of various methodologies given the non-experimental nature of the evaluation, as well as the pros and cons of combining two or more estimation methods.
B: Sketch the design of implementation plan (5 page maximum)
The second part is a preliminary proposal for implementation. Based on the reviewed materials in Part I and the experience and academic background of the researcher, the proposal should include a preliminary proposal for implementation plan of the evaluation of the employment service and of the on-the-job training program PAE.
The implementation plan should include a summary of data availability (maximum 2 pages) and a description of supplementary survey instruments to be used. This could include a description of the primary data from employment service. In the caseof secondary sources (administrative data, household surveys and/or other establishment-level surveys), year(s) available should be included as well. In each case researchers should describe the main characteristics of the information available (types of employment questions, feasibility of linking data across multiple surveys if they exist, etc.).
It should also include a description of the best options for impact evaluation for: (i) the employment service, and (ii) the on-the-job training program for adults (PAE). This section should explain the identification strategy to be used given the modernization of the employment service being implemented as part of the loan and the implementation of the PAE. The proposal may also want to propose how to reconsider the implementation phase to allow the use of alternative impact evaluation techniques (4 pages). The proposals should have in mind the need to address for example, the sample size needed to capture with statistical power to capture potential heterogeneity of the results considering the issues summarized in section 1.3 above (and described in detail in Gonzalez-Velosa et al, 2012), the timing of each stage, etc..
4. Selection Criteria
The selection process will be based on a weighted average of two elements (i) the quality of preliminary proposal (60%); and (ii) the background and academic and field experience in impact evaluation of social programs of the researchers submitting each proposal (40%).
The strongest proposals will be selected to be presented to the Bank staff and Executing agency staff in Bolivia. The discussion could be face to face or through videoconferencing, depending on the location of the selected institution. The discussion will be held exclusively in Spanish without simultaneous translation
Once a candidate is selected, the IDB will contract this person with the objective of developing a more detailed proposal for the design and the implementation plan, as well as to accompany the implementation, until the final stage of the elaboration of a document describing the methodology and results of the evaluation. The researcher would not implement the data collection process himself/herself, but she/he would act as external advisor during the implementation process and it is expected that would analyze and coauthor papers with Bank staff involved in the evaluation .
5. Proposal Submission
- Researchers interested in submitting a proposal may request additional documentation by email to Verónica Alaimo (email@example.com) and Manuel Urquidi (firstname.lastname@example.org).
- Proposals are due September 17th, 2012.
- Proposals should be submitted by email to email@example.com and firstname.lastname@example.org in Word or PDF formats.
6. Coordination and Schedule
The IDB project team consists of Carmen Pages-Serra (SCL/LMK, Unit Chief), Veronica Alaimo (SCL/LMK), Manuel Urquidi (LMK/CBO), and Rodolfo Stucchi (SPD/SDV).
Research proposals should be submitted in English or Spanish.
7. Financial Aspects
Depending on the quality of the proposals and the profiles of the researchers, the Bank will pay:
- a lump sum transfer of up to US$25,000 for the design of the methodology and implementation plan,
- a lump sum transfer of up to US$20,000 for the advisory services during implementation.
Payment will be provided as follows:
Proposal design and implementation plan:
- 10 percent at the moment of signing the contract
- 30 percent within 30 days of presenting and approving a report containing “Part I: Revised methodological proposal”.
- 30 percent within 30 days of presenting and approving a report containing “Part II: Design of implementation plan”
- 30 percent within 30 days upon approval by the Bank of the final proposal.
Advising services during implementation:
- 20 percent after supervision of baseline data collection
- 10 percent after supervision of each follow up surveys (number which will be determined in the methodological design)
- The reminder within 30 days upon approval by the Bank of the final data set.
The Bank will pay for travel expenses (travel and accommodation) for up to 4 field trips to Bolivia for 1 person. The actual amount will depend of country of origin of the researcher. They Bank will pay a coach class ticket, and a fixed amount for accommodation and meal expenses.
Attanasio, Orazio, Adriana Kugler and Costas Meghir. 2011. “Subsidizing Vocational Training for Disadvantaged Youth in Colombia: Evidence from a Randomized Trial,” American Economic Journal: Applied Economics, 3(3): 188-220.
Blundell, Richard, Monica Costa Dias, Costas Meghir, and John Van Reenen. 2010. “Evaluating the Employment Impact of a Mandatory Job Search Program,” Journal of the European Economic Association, 2(4): 569-606.
Card, David, Pablo Ibarraran, Ferdinando Regalia, David Rosas, and Yuri Soares. 2011. “The Labor Market Impacts of Youth Training in the Dominican Republic,” Journal of Labor Economics, 29(2): 267-300.
Diamond, Alexis and Jens Hainmueller. 2007. “The Encouragement Design for Program Evaluation,” Harvard University, Mimeo.
Galdo, Jose and Alberto Chong. 2006. “Does the Quality of Training Matter? Evidence from Bidding Processes Data,” IZA Working Paper No. 2202.
González-Velosa, Carolina, Laura Ripani and David Rosas-Shady. 2012. "¿Cómo mejorar las oportunidades de inserción laboral de los jóvenes en América Latina?" IDB. Labor Markets Technical Notes. February 2012
Huneeus, Cristóbal, Cecilia de Mendoza, and Graciana Rucci. 2012. "El Estado del Arte de la Capacitación de los Trabajadores en América Latina y el Caribe", IDB Labor Markets Technical Notes. June 2012.
Ibarraran, Pablo, Laura Ripani, Bibiana Taboada and Juan .M. Villa. 2011. "Youth Training in the Dominican Republic: New Evidence from a Randomized Evaluation Design", Mimeo.
Malamud, Ofer and Kiki Pop-Eleches. 2010. “General Education vs. Vocational Training: Evidence from an Economy in Transition,” Review of Economic and Statistics, 92(1): 43-60.
 For more information about labor market statistics, please request the additional information package.
 While PAE offers training for job seekers that match the requirements of a current vacancy, MPED offers training for job seekers between the ages of 18 and 24 that require additional skills to match vacancy requirements.
 According to Huneeus, Mendoza, and Rucci (2012), it can be expected that those firms which implement training programs and benefit from trained employees have better information on future skill demands and are in a better position to design effective training plans with more predictable costs and returns.
 Strategies to develop partnerships with the private sector, improvement of the information services, among many others that can be found in the project document
 Source: Own calculations based on the Household Survey, MECOVI, 2007.
 Population aged 10 or older. Official definition.
 UDAPE (2007) Working Paper “Informe Especial. La Informalidad en el Mercado Laboral Urbano 1996-2006 [Special Report: Informality in the Urban Labor Market 1996-2006].” www.udape.gob.bo.
 According to 2007 data, Bolivia is one of the countries with the highest proportion of informal workers (measured as the percentage of salaried workers with no social security contributions) at 68.2% for the population aged 15 and over, a percentage that would rise if the self-employed were factored in.
 The final publication will be conditional on permission granted by the Government of Bolivia to publish the results of the evaluations.