Back to Home

Measuring Outcomes

Demonstrating the difference ministries make.
By Paul T. Penley

There is a shift among philanthropists today. Donors want to see progress. They want to see gifts become results. People know the problems are bad. They are looking to be inspired by working solutions. If a nonprofit can demonstrate the difference it’s making, it can rally a groundswell of supporters searching for measurable outcomes.

Between 2006 and 2012, One Acre Fund’s budget jumped from $225,000 to $19 million. How? There is no slick website or big fundraising department. It’s simply a program that works. While dozens of organizations provide emergency food to starving people in the Horn of Africa, One Acre Fund (OAF) is building a permanent solution to East African famines. OAF triples the harvest of farmers in East Africa, doubles their income, and does it all while covering 82 percent of operating costs from program fees. OAF started with 38 farmers in 2006 and now serves 130,000 in 2012. I remember these statistics because they report outcomes well. They have a real solution to a deadly problem. They track and report the right number and type of outcomes. It gives grant makers confidence about the difference that OAF programs make.

Our Programs Can’t Be Measured

As the director of research and evaluation for the philanthropic advisory firm Excellence in Giving, I meet with hundreds of nonprofits on behalf of the family foundations we serve. We discuss strategy, financial sustainability, and impact. I ask about the quality of personnel and the effectiveness of the organizational structure. We review past performance and future plans. Inevitably I ask a direct question about program impact: What outcomes do you measure to determine if your work is making a lasting difference?

Time and again I get a two-part answer to the outcomes measurement question. First, the ministry leader tells me how much the organization believes in measuring outcomes. I’m glad to hear it. Then there is a pause. And the infamous contrastive conjunction "but” begins the next statement. "… but our programs are different; we can’t measure the real impact of what we do.” Ministry leaders issue this statement with great sincerity. They believe in the importance of tracking the difference they make, but see themselves as the exception. Measuring outcomes just can’t be done for their programs.

I understand that identifying the right three to five outcomes to measure is difficult. But my experience tells me it can be done. It doesn’t matter if you are a training organization, a counseling group, a development organization, campus program, or medical mission. Every ministry can work back from its vision and mission to a simple set of outcomes to track. It may seem elusive at first and require some outside assistance, but it can be done. And it’s good stewardship to verify that donations are supporting effective programs.

Outcomes Measurement Case Study: Haggai Institute

For years the Haggai Institute has promised its supporters that every person trained trains another 100 people in the following two years. That sets a very specific expectation, inviting a commendable level of accountability. However, when asked if program graduates did in fact train 100 people, the response was anecdotal. No measurement system was in place to track the numbers. The anecdotes highlighted positive results, but no one could tell if they were the exception or the rule.

One Haggai Institute supporter decided the unknown was unacceptable. He informed Haggai Institute that he needed to see demonstrable impact before giving again at significant levels. That’s exactly what Chief Operating Officer Dr. Beverly Upton wanted to hear. Dr. Upton had already put together the concept for a Measures of Effectiveness initiative. In her words:

"I envisioned a measurement process for Haggai Institute that could accomplish multiple objectives, going beyond just a simple collection of results. That measurement system would allow us to support and encourage alumni, assess their impact on world evangelism, and improve our training process. It would be a full-orbed solution that would also inform donors about our results and facilitate organizational capacity building.”

Dr. Upton worked as an organizational development consultant before joining Haggai Institute’s executive leadership team. She knows the value of tracking progress against goals.

Dr. Upton surveyed the landscape of outcome measurement consultants and ultimately chose our firm. Excellence in Giving had evaluated hundreds of nonprofits, understood the complexity of data collection, and knew how to measure key outcomes both to improve programs internally and to communicate impact externally to donors.

Since Excellence in Giving only works for donors, three Haggai Institute supporters — including the one mentioned above — agreed to cover the cost. We love it when donors fund fixes to organizational problems they find.

Outcomes Measurement Survey System

From September 2011 to July 2012, Excellence in Giving worked with Haggai Institute to build a survey system. The curriculum goals, training content, and interview insights shaped the content of 10 surveys designed to track long-term outcomes. The first survey collects baseline data about participants’ ministries before the training. The second survey captures the plans that participants make for their next two years of ministry. The remaining eight surveys track each graduate’s subsequent activity up to five years after graduation.

The survey system benefits every department at Haggai Institute. The training department evaluates the quality of the curriculum. Recruiters learn which demographic and personal attributes lead to future success. The communications and fundraising departments can report the results of each training session to supporters. The alumni department tracks involvement in regional alumni associations around the world. The system continuously generates specific reports for each department that drive improved program outcomes from better participant feedback. The reports don’t provide excessive information, only a few key outcomes that give actionable direction to department heads.

Benefits of Outcome Measurement

The 2012 Cygnus Donor Survey found that middle-age donors are demanding results from nonprofits in exchange for their gifts — and 44 percent of donors said they could have afforded to give more in 2011. If nonprofits have a clear case for the difference they are making, they could tap into these giving reserves.

Haggai Institute can now attract this additional funding by answering the outcomes question. They can report how trained leaders accelerate productivity in five simple categories. Because Haggai Institute did it right, they can tell supporters the exact growth percentage in ministry activities from before training to after training. That’s no small accomplishment for an international training organization approaching 50 years of service.

For programs that train leaders, it is difficult to determine whether the recruiters pick great leaders or the trainers make great leaders. Haggai Institute does pick good leaders, but the data proves they make them even better.

The Difference Between Outcomes and Outputs

One basic mistake that ministries make when measuring outcomes is actually reporting outputs. Outputs represent annual program activities. Outcomes comprise the lasting improvements from those activities. Reporting that 120,000 volunteer hours were invested in making 160,000 warm meals and providing 92,000 nights in a clean bed does not say anything about outcomes. It tells donors how busy a nonprofit was last year, but it doesn’t give evidence of problems being solved.

Reporting annual activities is not enough. Serving 100 children, having 2,000 people complete your curriculum, or exceeding 100,000 attendees at conferences doesn’t ensure lasting impact. Outcomes must be tracked to show the sustained improvements to people’s lives.

For rehabilitation programs, outcomes are the number of people who can feed themselves and pay rent for their own apartments with warm beds six months after completing the shelter program. In education, outcomes are graduation rates, college acceptance, and career success. Outcomes mean the investment is paying off years later.

How to Help Ministries Measure Outcomes

Matt Forti of The Bridgespan Group advises funders to help nonprofits get "access to measurement experts or other professionals who could advise grantees on what and how to measure.” If you support or run a ministry and wonder if programs can be measured, enlist some help. Funders who are truly committed to a cause will likely sponsor outcomes measurement projects for a ministry. The value of improving programs from feedback and demonstrating the lasting difference will inspire them.

After seeing the Measures of Effectiveness survey system in action, Dr. Upton from Haggai Institute remarked: "I hope our process will inspire other ministries to engage in identifying and measuring outcomes.” I hope so too. On the one hand, donors do want gifts to become results. On the other hand, there is only so much money to go around. Ministries need to make sure funds invested in their programs make a lasting difference. Donors need to know they made a wise decision with their gifts. Measuring outcomes makes both possible.