Assessing Impact

9 questions, 5 steps

Philanthropists often make progress simply by asking the right questions.

Here are nine questions that form a simple outline of how an assessment plan can develop naturally—and effectively— as part of an overall giving strategy. Reading through them in sequence is worthwhile as it gives a sense of the steps to be taken and how to focus one’s effort.

1. What problem are you trying to solve?
2. How do you think change will happen?
3. How long will it take?
4. How much money will it take?
5. What will success look like?
6. What will be the signs of progress on the way?
7. How do the groups directly addressing these problems answer these questions?
8. Who else is working on this, and what assessment tools are they using?
9. How much time and money are you willing to invest in assessment?

Many of these questions will be intuitive to any seasoned investor or practitioner. They can also be fairly described as common sense. But since philanthropy is about providing resources to address society’s most difficult problems, working with grantees requires a shared purpose for success. Having even rough answers to these questions will help identify what your gifts can, and cannot, realistically achieve.

The following steps can guide you in the development of answers and an appropriate assessment strategy.

Step 1: Understanding what assessment can do

Donors use assessment for a variety of reasons. Here are three of the most important ones:

Accountability

At the most fundamental level, donors want their grantees to be accountable. If you donated money to hire trainers, did the funds get used in the way they were intended? If the agency committed to training 100 women as health aides in 18 months, how many were trained in that time?

Decision Making

Many donors assess their grants as a way to make decisions about future giving: they want to understand which nonprofits should continue to get support, and which should be abandoned. Or, they may use assessment to make decisions about how to help their grantees: did this grant flounder because the nonprofit lacks good reporting systems? Maybe a grant for new software and training will allow the nonprofit to achieve its potential. Donors may also use their assessments of grantees as a way to assess themselves: have we made good choices, and how can we improve our decision-making?

“Proof of Concept”

Solid evidence that a program is making sustainable change is cause for celebration, but it also provides information that can leverage further development. Often evidence of success offers a funder and the nonprofit a chance to attract other funders and expand the initiative’s reach. A successful program can also become a model for other programs.

Step 2: Measuring effectiveness: what approach is the best fit

Three very different approaches dominate philanthropic assessment: evaluation, ROI and systems thinking. As can be seen below, the choices sometimes reflect different kinds of philanthropy. A small investment of time can quickly indicate which kind of measurement might go with which program.

However, it’s vital that donors ground their expectations in reality. The quality of assessment information will be influenced not only by the method adopted, but by how it’s implemented and how much one is willing to spend. Measuring and understanding the impact of a program, however valuable, is often complicated, requiring creativity as well as analytical rigor.

Like venture capitalists who support new companies because they have leadership and strategy, but not yet profits, philanthropists need to make decisions on the best available data. They can also help nonprofits measure their works in ways that provide actionable, timely information.

Evaluation

Evaluation is the most established method of assessing results in the nonprofit sector, and has a strong academic social science tradition behind it. Widely used in the government sector as well, it usually focuses on the grant or program level. Used well, evaluation helps the donor, the nonprofit and other stakeholders understand whether or not an intervention has worked and why. In some cases, however, evaluation simply becomes an impressionistic exercise that describes without analyzing. Sometimes, an evaluation is designed to make the funder feel successful regardless of what happened. This is worse than no evaluation at all. Types of evaluation include:

Goals vs. Outcomes

The most straightforward approach, this method simply compares a set of defined goals with actual outcomes. The goals need to be precise and measurable: the number of teachers trained; the number of students who got eye examinations; the number of performances; the number of acres conserved. It is essentially an assessment of activity. This typically is the least expensive to implement.

Process Evaluation

This method allows the funder to check in on a program as it unfolds, rather than simply at the end. It attempts to identify not just what has (or has not) happened, but the processes by which that result came about. It can help a funder and nonprofit understand external factors that are presenting challenges. It’s especially useful with new, untested programs for which no implementation “blueprint” exists. It can also lay the groundwork for mid-course corrections. In addition, it can be used as a decision trigger for whether a nonprofit is meeting the benchmarks for future payments.

Experimental Design

This rigorous method of evaluation seeks to document results by measuring the target population against a comparable group who did not participate in the program, and sometimes compares the effect on participants before and after the program. For example, health programs may compare participants who were carefully matched on most characteristics but who take part in different (or no) programs to reduce weight and increase exercise. Charter schools often try to compare their students’ test results with those of other students attending a traditional public school. This methodology requires considerable sophistication, resources and a large enough research base for findings to be statistically valid.

Cluster Evaluation

This methodology tries to compare results across a series of similar programs. It may combine the results as a way of assessing the type of program —for example, behavioral modification therapy for anger management. Or it may compare different types of programs that had the same goals: for example, comparing results of multiple initiatives using behavioral modification therapy vs. initiatives that used medication for anger management.

Other types of evaluation may focus on diagnosing how a nonprofit developed and delivered its programs, providing opportunities for the nonprofit (and funders) to identify gaps in capacity and skills.

Return on Investment (ROI)

This approach to measuring impact is a hallmark of both venture philanthropy and the social enterprise movement. It seeks to understand the relationship between dollars invested and results in as quantifiable a manner as possible.

The ROI approach can help a donor choose among an array of options (assuming the data exists) and can help spur replication. It’s very appealing to those for whom measurement is a critical goal and inspiration. And it creates discipline and common standards that can be shared by many donors and many nonprofits in a field.

While there is justifiable enthusiasm for this “return on investment” approach, one disadvantage is that it may not always use a long enough time horizon. Development experts sadly note that Africa is now dotted with wells, each built for a few thousand dollars to provide clean water to hundreds of villagers—but without funds to ensure that the wells can be maintained, they quickly stop working.

A second challenge is that for most complex problems, we know too little to be able to create the equations and formulas that provide reliable and actionable answers, so ROI calculations often rely on series of assumptions. Indeed, Acumen Fund, which was highlighted in the example, uses an ROI approach as just one element in a much broader process for evaluating investments.

Systems Thinking

Systems thinking begins at the broadest level of a complex problem and attempts to identify all the major underlying factors. It then assesses which interventions offer the greatest potential for changing the whole system.

The Harlem Children’s Zone exemplifies this approach. Geoffrey Canada is convinced that the fate of children in poor neighborhoods will change only if their entire environment changes. It seeks to end the generational cycle of poverty by addressing the needs of the community at all levels— home, school and neighborhood.

Another example is the work that produced the environmental report “Design to Win” by the Climate Action Project. Based on a comprehensive review of climate change data, as well as academic and practitioner research around the world, the report identified a handful of key levers for addressing climate change, including halting deforestation, eliminating coal plants, and building public support for governments to sign and adhere to international climate accords.

Step 3: Deciding on a level of assessment

Sustainable change is the goal of many donors. But in order to accurately assess what change their programs are achieving, they must first decide what level in the process of change they want to examine. Let’s take a grant to build a school in India as an example. Among the things to assess are:

1. Is the construction completed on budget and on time?
2. Does the school have the right facilities for the target population?
3. Does it have qualified teachers?
4. Are students attending?
5. Are students making academic progress?
6. Are students going on to higher education?
7. Are students getting better jobs than they would have otherwise?
8. Are their lives better?
9. Is their community better overall as a result?

Each of these questions deals with the issue of impact at a different level —and generally implies a different time frame. As a donor prioritizes goals, so too can the assessment level be defined.

Levels of assessment for funders

Assessing giving on a number of levels invites a deeper understanding of the relationships at the heart of a philanthropic endeavor. As the chart above shows, every grant can be assessed in a variety of ways. One can account for the details of expenditure as well as evaluate far larger community and societal priorities. Such an overview, when carried out at the planning stages of philanthropy, can give the donor a chance to see where resources can be leveraged, partnerships formed and new efficiencies catalyzed. In this sense, planning for assessment is a vital part of designing a giving strategy.

Step 4: Understand challenges in assessing impact

Donors sometimes struggle to get straightforward and clear answers to questions they have about the results of their giving. Most people want to know if they made a good choice in supporting a particular project. They want to know if they should consider changing their approach or remain steadfast in their giving. No one wants to mess with success, but, on the other hand, no one wants to fund failure either.

The matter has become even more complex in recent years as ambitious donors, innovative nonprofits and influential experts have sought to define how philanthropy is making a difference in solving fundamental social problems.

This evolution in theory and practice encourages donors to think about how their funds can be most effectively used to make sustainable change happen. But it can also lead to unrealistic expectations from well-intentioned but anxious donors who demand to know how their $50,000 gift to Doctors without Borders three months ago has changed longterm health outcomes for women in Haiti.

Many people say nonprofits don’t have a bottom line the way a business does. This analogy is faulty, however. Nonprofits do quite literally have a bottom line: they too have financial results and tax returns. In addition, no sophisticated person will choose to invest in a company based only its bottom line, which is a backwards-looking figure that offers limited indication of whether a company will provide a good future return. Venture capital investors, in fact, are frequently putting money into new firms with little record of past success. Like good philanthropists, they make judgments based in part on prior performance, but also on a well-researched viewpoint about what innovations may succeed, and whether or not the leadership of the enterprise is well equipped to forge a successful path.

The real difference between “investing” in a nonprofit or in a for-profit is that when we invest in a company, we only look for results at the company level. In the nonprofit sector, we are often looking at how the nonprofit can deliver programs that will bring about sustainable change in the community or even the world.

Clearly, no grant of $50,000 (or even $1,000,000) will, alone, solve women’s health problems in Haiti. In fact, it’s likely that funding health organizations alone may not end disease in Haiti: the inability to access clean water and sanitation, poor nutrition, benefits, inadequate transportation, poverty and ill-treatment of women are complex, interwoven factors that result in poor health for women. Grants can, however, provide important relief as the society rebuilds or works to make services more available.

Another factor that makes assessment difficult is the huge range in time horizons among philanthropic endeavors. Only for some vaccinations do the treatment and cure happen simultaneously. Medical research can take decades. Most social challenges—poverty, lack of human rights, ethnic conflict—have time horizons that in some cases may have no real endpoint. But donors can hope to see progress and certainly benefit to individual communities and organizations.

Does this mean that donors should walk away from thinking about impact at the system level? Not at all. Without considering the broader context, donors run the risk of funding “successful” programs that make no real difference in addressing the problem they are seeking to solve.

Step 5: Build a partnership

Donors need clarity about why they seek assessment and what level they will target. But they also need to be clear-eyed about what can reasonably be answered, and by whom. The broader the scope of the assessment, the harder— and more expensive—it is to produce useful information.

Recently a donor wanted an after-school program to undertake a 10-year longitudinal study of its participants. Not only would the cost of that research be greater than the entire budget of the nonprofit, but no one at the nonprofit was qualified even to engage a research firm.

Donors and grantees should, ideally, reach a respectful agreement before a grant is made about what kind of assessment can be provided, by whom, and at whose cost. What is the grantee responsible for? What will the donor do? What might a third party be engaged to do? Who pays? Are there existing reporting methods that will work for the donor? Could the donor partner with other funders on a common approach?

Many nonprofits today are suffering unduly from donors’ enthusiasm for reporting and assessment. Each donor, they feel, has a different set of reporting requirements—and the burden can be overwhelming. By recognizing that sometimes nonprofits already have good information on what programs work and why, donors can save money and effort.

A note about timing

One of the most valuable times to assess a grant is before it’s made. But donors—whether individuals, families or foundations—can consider assessment at any time. As a form of due diligence with grantee partners, it is a natural part of thoughtful, effective philanthropy.