The Million Dollar Question

The challenge for the microfinance industry and for any development program: how to know what effect it has had?

Today, we speak of effectiveness in development, and yesterday it was management for results. Regardless of the fashionable phrase to define the subject, the question remains the same: how to evaluate the impact a development program has had? What would Latin America be like if the Ford Foundation, the IDB and the donor community had never existed? And the bottom line: is there empirical evidence of the effectiveness of microfinance institutions in improving the living conditions of microentrepreneurs?

The question is key, urgent and valid. And although in recent years the number of impact assessment studies has increased, there is still no clear answer.

It is possible that for commercial banks—as for any industry in which private initiative is paramount and the market imposes rules—the question sounds rhetorical. What validates an activity is its balance sheet, its results in "hard currency." If a business has a profit, it is effective. And in a certain sense, this vision has permeated many international microfinance organizations that have stopped investing in costly impact studies: if customers apply for and pay back loans, it is because they are better off with them.

But when an industry arises because of external intervention, whether government or donor community action, a company's end of year balance sheet only shows that this company is sustainable, not effective. Speaking, for example, of microenterprise development laws, consultant Miguel Cabal says that "I couldn't say whether we are worse off or better off with development laws because there have been no real efforts to assess the impact." In the conclusion of his presentation at the Microenterprise Forum in Cartagena, Princeton professor, Dean Karlan, posed the million dollar question. "Is there empirical evidence of the effectiveness of microfinance institutions?" His answer: "The bottom line is that we still don't know."

Discussion of the Method

Impact assessment is understood as any process that seeks to determine whether an action undertaken achieves the desired result. And the first problem that is presented is that microfinance activities can have three types of desired results: economic, sociopolitical and individual. The economic effects range from the impact of a single microfinance institution in a specific community, to the impact on the GDP of a country that a developed microfinance industry with dozens of institutions that serve hundreds of thousands of customers in the economy of a geographic area or segment of the population.

There is abundant empirical evidence of the positive economic impact of microfinance institutions. Two studies serve as examples: one on the microfinance projects of Save the Children in Honduras in 1999, which showed that access to micro credit helped many small agricultural producers not to have to sell their products in advance below market price to obtain income during the months without harvest; and another on the credit with education program [crecer] in Bolivia, revealed that thanks to the program, 67% of the participants had increased their income. But the MFIs can also produce changes in the political or social status of a segment of the population, and access to credit and to the formal financial system also impacts the borrowers' self esteem. Assessing these types of effects requires different methodologies.

When the objective pursued is poverty reduction, it becomes especially important to analyze the impact of microfinance institutions. At Development Gateway, an Internet site specialized in the subject, there is a consensus that it is more convenient to use multiple methods instead of just one, and that it is necessary to combine qualitative and quantitative approaches. However, in many cases, this is too expensive for programs in which the total investment is relatively small.

Control Group

Another one of the problems that arises is that the impact assessment must be initiated before the program is carried out. According to Karlan, given that the impact is causal and "refers to how a program produces changes," in order to measure it, a tool must be designed and put in operation along with the initiative itself. "Impact studies cannot be carried out retrospectively," he concludes. In addition, if the impact assessment hopes to be rigorous, it must avoid biases that place its objectivity in doubt, creating fundamental questions about those [assessments] that experts are involved in. Bias in selection is one of the most obvious: microfinance institution customers, due to the very fact that they are applying for micro credit, have enterprising traits, which may contribute to their success. How does one select comparable individuals without credit in a control group?

Another obstacle specific to microfinance institutions is the limited management skills to carry out impact studies, and MFI personnel are not enthusiastic about introducing empirical variables in the analysis tools, as this translates into additional work and an extra complication for them.

|sidebar| Control and Pressure

Miguel Angel Navarro, financial manager of ODEF, Organización de Desarrollo Empresarial Femenino (Organization for Women's Enterprise Development) of Honduras, one of the microfinance companies that participate in the ImpAct project which Navarro coordinates, stresses the biggest problem with assessing impact: "There has to be a control group in order to see how it behaves with respect to the group involved in the program, and this is quite difficult."

The ImpAct project, financed by the Ford Foundation and several British universities, is an attempt to assess the effect of microfinance institutions on customers from various regions of the world. It began in Latin America in 2001 with 10 Honduran MFIs, one of which was ODEF. It involved using five tools to measure the effect of the microfinance companies' services. "Since we detected 45% defection among our customers, the first tool we used was a survey to measure why people were leaving," reports Navarro.

ODEF used new customers as an initial comparative group, but the methodology was refined and now, for each customer randomly selected to participate in the impact assessment, other customers are chosen for the control group who belong to the same demographic group, geographic location and economic activity.

The results? 33% of those who left abandoned the MFIs because they had problems with the service they received from the institution, but 21% had done so because the microfinance program available did not seem satisfactory. The reasons stated include the high staff turnover, slow approval of the credit, not being helped by the same officer each time, and inflexibility with regard to time periods.

The most outstanding result was that 66% of those surveyed said that they would return to the microfinance company if the program improved. "This made us develop new programs and win back customers who had left," recalls Navarro. The reincorporation of former customers was enormous, "and much cheaper than going out to look for new customers."

To Be or Not to Be

The ODEF is clearly advantageous for improving the business of participating microfinance institutions and also increasing access to credit. But is it really impact assessment? Karlan maintains that it is definitely not, but Navarro thinks it is insofar as "it lets us know if microfinance institutions are fulfilling their objectives, and if they are not doing so, make the changes necessary."

However, the ODEF activity is more easily understood as a market study or customer satisfaction survey for improving a product, than as an authentic impact assessment. But its usefulness as a tool to improve customer service and make their demands and needs more transparent cannot be denied.

And when trying to instill more transparency in the market, the work performed by raters such as Microrate or the information platform The Mix also benefits customers and oils the wheels of an industry that has emerged under the suspicion of intervention, whether government or donor-related. However, neither credit rating or credit scoring, nor The Mix indicators are impact assessments.

For Karlan, the practical experiences that have been carried out in assessing impact generally have serious methodological deficiencies that deprive them of their validity. Although he clarifies: "There are two different types of impact: the one Miguel [Navarro of ODEF] refers to and the one I am talking about. The tools that ODEF is using are useful for a qualitative process. However, if you want to make an impact, it is necessary to have a valid comparison group. And the leaders of the microfinance institutions and donors are going to have to begin to do that."

The experience of those who have tried to conduct some type of impact assessments is that when an expert suggests changes, no one wants to implement them, but when the impact assessment is carried out from the bottom up, everyone wants to participate in it.

"The methodology always has problems," concludes Navarro, "and the way to improve a tool is to use it."