Feature Article

How do you know if your organization is making a difference? It's a question you probably ask yourself all the time. Your staff, donors, funders, and stakeholders also want to know the answer.

Program evaluation is a key component of operating and sustaining effective nonprofit organizations. This article outlines the benefits of program evaluation and how evaluation results can be used for fundraising. We present information from interviews with three Bay Area nonprofits: DataCenter, Health Initiatives for Youth (HIFY), and Shelter Network of San Mateo.

Evaluation is the way to get systematic information about the results of your programs and how you are contributing to individual and community change. Investments in evaluation pay off. Evaluation can help you improve the work you are doing as well as prove your value to others.

WHY EVALUATE FOR FUNDRAISING?

Funders and donors are drawn to effective programs and organizations. They want to know how you measure success. The current trend of foundations wanting "results-based accountability" and venture philanthropists using the model of "social return on investment" are indicative of the attention funders are giving evaluation.

According to Sara Kimberlin, development director of Shelter Network of San Mateo, "Both private and public grantmakers are placing more emphasis on programs that can demonstrate 'measurable outcomes' in order to quantify the community benefit resulting from their grants. Many individual donors are also looking for documented program results before they support an organization."

If your organization receives foundation grants, evaluating your programs may already be a condition of funding. However, developing and implementing an evaluation strategy of your own can help you get ahead of funders' requests. If your organization approaches evaluation proactively, you can take control of your efforts and results.

Moreover, evaluations can spur fundraising. Creating a workable system for collecting information and tracking evaluation findings can help you avoid recreating the wheel for each proposal or program report; when you need them, important statistics and client stories will be readily available.

Carol M. Cantwell, co-director of the DataCenter, offers this advice to other nonprofits interested in using evaluation in their fundraising efforts: "Don't wait for funders to require you to do program evaluations. Evaluations are a great tool for documenting your successes and soliciting constructive criticism. They can also be a great way to include donors, funders, and members in providing feedback, which helps build closer relationships."

In addition to providing proof to current funders that they should continue to invest in your organization, evaluation findings may lead your organization to seek out new funding sources. If your evaluations show that your programs are serving populations you didn't originally expect to reach or meeting community needs in unanticipated ways, you may want to reconsider the program objectives and package your programs differently for varied audiences and potential supporters. For example, if you find that a program that originally focused on helping people find employment also helps clients secure affordable housing, you might look for funders who are interested in funding both employment- and housingrelated programs.

COMMUNICATING EVALUATION RESULTS

Evaluation can help you produce compelling information both for specific fundraising campaigns and for a long-term strategy of building community support. The information may include statistics related to program outcomes (quantitative data) and stories about a success with a client or organizing effort (qualitative data).

Using program evaluation as part of a successful fundraising strategy involves incorporating the results, or findings, of your evaluation into your communications. Tell your donors, members, and funders what you've learned about the good work you do! You can use a number of vehicles, described below by audience, for communicating your evaluation methods and findings to current and potential funders.

Foundations and Government Funders Include information about your methods of evaluating and important findings about your programs into grant proposals, ongoing reports, and presentations. At Health Initiatives for Youth (HIFY), Executive Director Sharon Dolan explains, "We use the impressive statistics from our evaluations in our applications [to funders] to demonstrate that our work is effective and has an impact. Foundations love this, and many government applications require it."

Individuals Use media campaigns, annual reports, brochures, your website, and one-on-one meetings as opportunities to communicate evaluation results to individual donors.

The DataCenter uses information in this way. Cantwell says, "Because we ask groups... how they've used our research and what the results of the organizing campaigns were, we often find out about really wonderful victories that we can share with [individual] donors to help illustrate the importance and real-world impacts of our work."

Sharon Dolan of HIFY explains, "When we send out updates to donors, we usually throw in a great statistic or two, or a quote from a participant... Foundations and individuals are moved by success stories, quotes, and anecdotal evidence."

TIPS FOR USING EVALUATION

As you start to plan your evaluation, anticipate how you want to use the results, avoid funder-driven evaluation, measure what is measurable, and prepare for the worst-case scenario.

Plan to Use the Results Consider how you will be communicating your evaluation results to current and potential donors and funders. What is the most useful format: a presentation, a report, charts and graphs, or some other format? Think about these end results before you begin.

As HIFY's Sharon Dolan cautions, "Decide how you will use the results before you set up your evaluation system. If you don't do this, it's likely the reports will sit on your shelf waiting to be used."

As Much As Possible, Avoid Funder-Driven Evaluation When evaluation is a condition of foundation or government funding, funders sometimes provide a specific list of outcomes to be reported. When an organization measures only what the funder is interested in, they may miss the questions and information the organization itself may have about a program's success or challenges.

If your questions and the funder's are way out of synch, you may need to educate the funder about the questions you want to answer and how you want to find those answers so you can reach some agreements about the form and content of the evaluation.

As Dolan explains, "We've had to resist pressure from funders — especially government funders — not to use our forms in favor of theirs. Sometimes we succeed and sometimes not. I know they have the best intentions, but the requests are often impractical and have a one-size-fitsnobody nature to them."

If your organization has created an intentional and coherent evaluation plan and design, having that plan may make it easier to advocate with some foundation or government funders for using it. On the other hand, if you know that a funder requires use of a particular form or process for collecting information about funded services, consider how to build these elements into your ongoing evaluation efforts.

Measure What Is Measurable Particularly if your organization is devoted to longterm and/or large-scale social change, measuring impacts attributable to specific programs may be difficult and costly.

Sharon Dolan describes this challenge for HIFY, saying, "[Funders would] like us to prove impact beyond what we're able to do without a long, expensive scientific study. We can't prove that we're lowering HIV rates among youth, for example, though we can prove that as a result of our workshops young people say they're more likely to get tested for HIV, more likely to use condoms, and less likely to have sex while high."

Organizations dedicated to community organizing or doing public relations campaigns to influence people's behavior — such as using billboards about the risks of smoking, or flyers about preventing HIV — may find it particularly challenging to prove that their programs are effective.

Some outcomes, however, may be finite and measurable: How many toxic sites were cleaned up? By what margin did the local proposition you opposed fail? How many people were likely to see your billboards? If you are doing campaign work, gathering quantitative (numerical, statistical) information by, for example, counting the number of outreach contacts you made or voters you registered, can provide you with helpful data. The challenge for you in evaluating this type of work is estimating how much change can be attributed to your organization's activities.

You can also collect qualitative (descriptive, nonstatistical) information from interviews with community members or a focus group of local decision makers to get useful information about what worked and what could have been strengthened.

Prepare for the Worst-Case Scenario One challenge of program evaluation is resisting the urge to focus only on "positive" results. Any evaluation may end up highlighting the need for program or organizational changes. Evaluation findings may produce recommendations related to changing program administration, rethinking a program's interventions and activities, and even revising program objectives. These are all useful for maximum program success as you proceed. To be prepared to make the most of all your evaluation findings, consider two questions before you begin an evaluation:

1) How can the organization prepare its staff to receive even "negative" evaluation findings? 2) How can you develop a relationship with funders that will promote consideration of all findings, including findings that point out challenges, without jeopardizing future funding?

EVALUATION WITH A LIMITED BUDGET

The biggest line items for program evaluation include dedicating staff time; building technological capacity to collect, store, and report on data; and paying for an external evaluator, if appropriate. There are some choices you can make to manage these expenses.

Internal or External Evaluation? Many organizations use a combination of internal and external program evaluation methods. Internal evaluations rely on staff to direct and carry out the evaluation design and implementation. External evaluations engage consultants or consulting groups, usually in combination with staff participation.

Internal evaluation by program staff provides opportunities and challenges. You can create an evaluation built on staff enthusiasm and expertise in their program areas, while increasing their knowledge what about evaluation is and their capacity to carry it out. Using staff time to complete program evaluation may also be less expensive than hiring an external evaluation consultant.

On the other hand, there are challenges with an internal evaluation: staff may be reluctant to devote time to evaluation activities if they don't see them as directly related to program; there may not be in-house time or expertise to train staff in evaluation frameworks and methods; and funders may not accept the "objectivity" of internal evaluation findings.

Doing evaluation internally is particularly appropriate, however, where there is frequent and ongoing data collection. At the frequent workshops and trainings for youth service providers that HIFY staff facilitate, for example, staff give anonymous evaluation forms to participants and then collect them and enter the data.

The DataCenter, too, launched their evaluation work in-house in order to help staff develop an understanding of evaluation methodology and the importance of recordkeeping. As Cantwell describes, "We felt it would strengthen our internal documentation systems once people became familiar with how the information they entered in our projects database is actually used in the evaluation process. Because we designed the evaluations ourselves, we can ask the questions that are most relevant to us."

There also are some clear benefits to using an external evaluator. Even if not a required component of reporting, some foundations may view more favorably those proposals that incorporate external program evaluation. For example, the W.K. Kellogg Foundation's Grantseeker's Guide describes proposals that "stand out from a crowd" as those that are "willing to let an impartial evaluator assess the project."

Using an external evaluator also makes sense when the scope of the evaluation is big or complex. Kimberlin explains that Shelter Network has "used outside evaluation consultants to complete more intensive and comprehensive program evaluations that require more time than our staff have available or more specialized expertise."

Even if you hire an external evaluator, it's important to stay involved. You and your staff have the best sense of the kind of information that will be useful to your organization. While you won't be responsible for carrying out the day-to-day work of the evaluation, you can't completely hand over the project to the consultant. You should expect to be involved in reviewing their evaluation plan, providing feedback on data collection instruments, and shaping what is included in their final products.

It's not unusual for an organization to rely on a combination of external and internal expertise. For example, an external evaluator could help organizational staff during the early stages of an evaluation to develop an evaluation framework, create or review selected data collection tools, and assist with developing database specifications. Or, an external evaluator might be hired to train staff in basic data collection methods and analysis. In these scenarios, the external evaluator provides a guiding or teaching role and the data collection and analysis are left to internal evaluators. In others, the external evaluator might carry out one piece of a larger evaluation plan, particularly one that requires specific expertise or involves confidential data collection that is not appropriate for staff to do.

Is It Worth Investing in Technological Capacity? Begin designing your evaluation by assessing your organization's ability to capture, store, and analyze information. Do you have spreadsheet and/or database software applications that could be used for entering and analyzing statistical data? Are staff knowledgeable about creating spreadsheets and databases? Are staff or volunteers available for data entry? Will investing in technological capacity benefit programs in other ways?

Shelter Network, for example, received a grant that required in-depth tracking and analysis of program data for annual grant reports. "In developing the new evaluation system [for this grant], we focused on creating a system that could be integrated into our existing data collection efforts," said Sara Kimberlin. "Doing so required securing additional matching funds beyond the original grant. As a result, however, we now have the capacity to track significantly more information about our program effectiveness, which has been useful for internal evaluation as well as for producing reports to the original funder."

Doing an evaluation in-house could require investing in database software and savvy, so you need to decide if this is a good use of organizational resources. If you are going to engage in ongoing evaluation, this investment makes a lot of sense. It makes less sense if you are conducting a one-time evaluation.

Focus on What Matters and Keep it Simple Ultimately, containing the cost of an evaluation involves making choices about the scope of information you gather. Rather than evaluating all programs, you may choose just one to look at closely. Rather than seeking evidence of community-wide impacts of your work, you may narrow your quest to program-focused outcomes. Which questions about your programs are essential to answer immediately and which can wait?

Fundraising for Evaluation You may want to build funding for evaluation directly into grant proposals. Research which foundations encourage and provide funding for evaluation as an integral component of a program plan. Some foundations consider program evaluation a capacity-building project; your organization may want to seek a capacity-building grant to develop the infrastructure for evaluation.

According to Shelter Network, "There are fewer foundations now than there were a few years ago that are willing to award grants specifically for program evaluation. With the more challenging fundraising climate of the past few years, we have found it helpful to emphasize in our requests to funders not only the value of the program information we gain from evaluation activities but also the ways in which improved evaluation systems can help us build our capacity to fundraise more effectively and improve our staff efficiency."

WHERE TO START

The first step in evaluating any program or organization is articulating the purpose of your work. What are you trying to achieve? Then work backwards and think about what information would indicate that you are or are not making a difference. Evaluation requires collecting the evidence you need in order to document and analyze how well you are achieving your objectives.

There are a number of resources that can guide you through your evaluation planning process (see Resources). Though the content and context of your work will affect the evaluation design that works for you, here are a few first steps to get you started (see the sidebar on Evaluation Terms for explanations of terms used here).

1. Find out how other organizations working in your issue area have evaluated their programs. A simple Internet search can get you started, but you might also want to ask funders or colleagues in your field.
2. Review and clarify the objectives of your program. What are you trying to achieve?
3. List and describe the specific activities or interventions you are carrying out to achieve each objective.
4. With your program's objectives and activities in mind, develop a set of research questions detailing what you want to learn about your program (see sidebar "Developing Research Questions").
5. Based on your questions, define program outcomes (the desired changes you are working toward in the clients or community you serve) and indicators (how you will know if these changes are occurring).
6. Consider your organization's or program's ability to collect, store, and analyze information. With this capacity in mind, develop a list of possible methods for collecting information about your program (for example, tools such as new forms, surveys, interviews), or a plan for increasing your evaluation capacity.

What Does It Cost? One ballpark estimate of how much to allocate for program evaluation is three to five percent of a program or organizational budget. Of course, this varies based on the complexity of the evaluation, which is determined by the type of information that will be collected, who will collect it, and the type of analysis and reporting that will be done.

How Can We Pay for It? Many funders allow grantees to incorporate evaluation activities into requests for program support. Shelter Network has found larger funders to support developing their evaluation systems. They then cover the costs of continuing and maintaining evaluation with general program funds. DataCenter's evaluation costs are in staff time, which is included in their budget as part of personnel expenses.

Who Should Do It? If staff are going to carry out the evaluation, select key staff with a background in research or evaluation for an evaluation team or committee. To find an external evaluator, ask other organizations you work with whom they would recommend. (Also, see sidebar, "Resources.") Always ask for references and sample work so that you can assess if they are the right fit for you.

How Long Does It Take? How long evaluation takes depends on your research questions and the scope of the evaluation plan. It is possible to get meaningful information from a single focus group, which can take from 8 to 20 hours to plan, implement, and report. A three-month, six-month, one-year, or three-year evaluation will provide you with different levels of information. Multi-year evaluations provide long-term findings, which can inspire greater confidence and information about changes over time.

———
RACHEL LANZEROTTI, MSW, IS AN INDEPENDENT EVALUATION CONSULTANT. CONTACT HER AT WWW.RLANZEROTTI.NET. LAURA LANZEROTTI IS ASSOCIATE PRINCIPAL OF LAFRANCE ASSOCIATES. CONTACT HER AT LAURA@LFAGROUP.COM. GFJ