How often should a program be evaluated?

We heard from an organization recently that wanted to know how often they should conduct an evaluation of their program. Of course it depends on a number of factors including budget but most importantly, what kinds of questions they have about their program. The group runs a number of summer camps around the country for inner city youth, an area we've worked in. They serve about 1500, mostly pre-teen kids. Here's what we said:

In terms of how often a program should be evaluated I would make the following recommendation. Once a solid evaluation design has been developed-- including instrumentation (surveys, interview guides, etc), and a data collection and analysis plan-- a program like yours might consider undertaking an annual evaluation. If the same survey can be repurposed each year, and the survey can be done online, the cost of such an approach is relatively low and will yield information useful to program planers and those within your organization who need to beat the bushes for funding. Going through the process every year will help you focus on results and ensure that basic information about program participants is complied and archived for future use (such as longer term follow-up with participants). Just as important this annual pulse-taking will help you generate ideas for less frequent but more in-depth evaluation work. Depending on budgets and the kinds of questions that arise in the routine annual evaluations, you may decide to do a more comprehensive study of your program say every five years or perhaps at an interval that corresponds to your strategic planning cycle.

Maybe a simpler way of putting this is the following, do an annual assessment to ensure that the program is staying on track-- that the staff and participants are happy and benefiting, at least in the short term, from the services they receive. Then do a more wide ranging evaluation when you have meaningful questions about how the program is working and about the kinds of longer term outcomes you are having. Think of the second evaluation as more strategic in nature.

What do you think?