Some Program Evaluation Reading and Listening

Two things of note we've come across recently: A book called Developing a Learning Culture in Nonprofit Organizations by Stephen Gill and a story by Emily Hanford on the impacts of pre-school recently aired by American Radio Works. We'll be blogging about these two pieces in the future but wanted to give them each a plug right away.

How are they related? The American Radio Works piece in fascinating to the extent that it discusses the unintended outcomes of a social program- pre-k schools for disadvantaged kids. How often do we think about assessing unintended outcomes (both good and bad)? How often do we build evaluation designs that allow us to capture them? The Hanford piece shows the importance of being able to look beyond what we are looking for. The Gill book talks about how, by promoting a learning culture, organizations can facilitate this.

What do both books have to do with program evaluation? They both lead us to the conclusion that evaluators ought to move away from the business of evaluating programs (which they have certainly begun to do over the last decade) and move towards supporting organizations in their efforts to learn from their work.

More in the coming weeks.

Jott was Nice but Reqall is Nicer

A couple of posts ago we introduced Jott. Jott is now owned by Nuance, which gave us Dragon Naturally Speaking, the speech recognition tool. Unfortunately, Jott has significantly increased its price so we've turned to Reqall which does much the same thing for less. Both Jott and Reqall are speech-to-text (or regular text-to-text) reminder systems. Reqall is available for the iPhone but I drink a different brand of kool-aide so I'll describe how it works for the BlackBerry.

You can send a to-do to Reqall from the Reqall for BlackBerry application as a text message, dictation, or via a phone call. Sending as a text message is pretty straightforward. To send by phone, select the 'Send By Phone' option, again, from the Reqall for BlackBerry application. Reqall dials and when the call connects it recognizes your phone number. You speak your to-do and, if you like, you give it a due date:

"Hello this is Reqall. Do you want to add, share or recall?"

"Add"

The system beeps

"Remind George about the Pensky file. Tomorrow, 2 PM."

Tomorrow at 2PM, you get a text message reading 'Remind George about the Pensky file.'

You can also choose to record the message on your BlackBerry rather as well (useful, if you're somewhere that doesn't have cell phone service). As soon as service is available, the application will connect and upload the audio for processing.

We use Reqall primarily as a way to capture to-do items as they come up. Reqall has a number of other features that are worth exploring. For example, if you smartphone has GPS capability you can associate particular locations with particular tasks. Let's say you make regular site visits as part of your job. You could create to-dos for eachsite and when the GPS sensed your proximity (you’d need to designate the site), Reqall would send you the appropriate reminders. You can also share to-dos. Instead of leaving yourself a to-do, you could send it directly to any of the contacts you've entered into Reqall.

In addition to getting SMS reminders of your to-dos you can also view them from your browser. That's what we tend to do around here. We use the BlackBerry to create them and the browser to manage them. As with many such applications, Reqall plays well others. You can use it with Evernote, Google Calendar and Outlook. In a mash-up contest however, Jott is the clear winner. Let us know what you think. You can find out more by viewing David Pogue's video tour which you'll find here.

We'll be posting about other tools in the coming weeks so stay tuned for more.

Internal Versus Enternal Evaluation

Should your organization outsource evaluation? Again, we favor a blended solution as discussed in the last post. Consider working with a consultant to set up an evaluation system for your organization but perform routine evaluation work on your own. If you decide to conduct a more comprehensive evaluation such as the one described in our previous post, call in that consultant to manage the project. The Kellogg Foundation has a worthwhile discussion on the topic which you'll find here. You'll find an even better one from Bernice Taylor Associates here.

There are two caveats. The first is that while a simple internal evaluation is entirely doable, few organizations are usually able to actually allocate the staff time to get one done. In most cases therefore it may pay to outsource the work. Most organizations want to spend on time with their participants not entering survey data or running analyses using Excel.

That said, distinguishing internal versus external evaluations may not really be the right way to think about the question of ‘who does the evaluation’. A good evaluator will work closely with program staff to make an evaluation a success so in this way there are both internal and external staff on the evaluation team. We have a lot to say on this subject and take it very seriously. Evaluation utilization is often founded on a collaborative approach that at times blurs the lines between an internal versus external study. To our thinking it’s more a question of accountability… who is responsible for project deliverables and this is usually a question of budget and organizational capacity. You may want to take a look at a white paper we have on this subject. You’ll find it here.

How often should a program be evaluated?

We heard from an organization recently that wanted to know how often they should conduct an evaluation of their program. Of course it depends on a number of factors including budget but most importantly, what kinds of questions they have about their program. The group runs a number of summer camps around the country for inner city youth, an area we've worked in. They serve about 1500, mostly pre-teen kids. Here's what we said:

In terms of how often a program should be evaluated I would make the following recommendation. Once a solid evaluation design has been developed-- including instrumentation (surveys, interview guides, etc), and a data collection and analysis plan-- a program like yours might consider undertaking an annual evaluation. If the same survey can be repurposed each year, and the survey can be done online, the cost of such an approach is relatively low and will yield information useful to program planers and those within your organization who need to beat the bushes for funding. Going through the process every year will help you focus on results and ensure that basic information about program participants is complied and archived for future use (such as longer term follow-up with participants). Just as important this annual pulse-taking will help you generate ideas for less frequent but more in-depth evaluation work. Depending on budgets and the kinds of questions that arise in the routine annual evaluations, you may decide to do a more comprehensive study of your program say every five years or perhaps at an interval that corresponds to your strategic planning cycle.

Maybe a simpler way of putting this is the following, do an annual assessment to ensure that the program is staying on track-- that the staff and participants are happy and benefiting, at least in the short term, from the services they receive. Then do a more wide ranging evaluation when you have meaningful questions about how the program is working and about the kinds of longer term outcomes you are having. Think of the second evaluation as more strategic in nature.

What do you think?

We Love Jott

Too many gadgets, too little time. Nonetheless we spend a good deal of time checking out new productivity tools that have zero learning curve, zero (or next to zero) setup curve and that are actually useful. Jott gets the nomination on all three counts. Their slogan "Talk to Jott, get simple back" is for real. So what is Jott. Jott is something for your cell phone.

You're walking down the street and you remember that you need to call one of your managers to discuss the Pensky grant. You're sitting in traffic, hear some interesting new music on the radio and want to check out the artist when you get back home. You're laying on the beach and suddenly remember that you realize you've forgotten to reschedule an important meeting. You call Jott and a friendly female voice says voice says:

"Who do you want to Jott?"
You say "Myself."
Jott says "Jott self."
You say "Remember to call Janice about the Pensky grant."
Jott says "Got it. Want a reminder?"

Jott recognizes your speech and converts it into text which it: 1) Emails to you, and 2) places in your online Jott to-do list. If you requested a reminder, Jott asks you the day and time and then sends a text message to your phone at the appointed time.

It really works, is really useful, and, as of this writing is dollar free and ad free. There are more features, for example you can send Jotts to others (either email to text messages) or to groups and you can categorize your Jotts as well. For example, you could create a folder for "The Pensky Project" (who knows maybe George Castanza could have kept his job if he'd had Jott) and then tell Jott to put a particular to-do in that folder for future reference.

You can get Jott here. It is probably the best productivity tool I've found all year. If you try it, let us know what you think.

Informed Consent

We just discovered a cure to the informed consent blues. You'll find it here. It's an informed consent builder from the University of Wisconsin. University culture, and the fact that such forms are usually developed with lots of input from the legal department, means that the form that gets created my be overkill but it certainly does cover all the bases and its easy to modify the form in a word processing program to suit your project's particular needs. Rather than struggle with wording a consent, we edited one the system created based on answers to a few simple questions. Give it a try if you need to obtain informed consent.

Play to Win: The Nonprofit Guide to Competitive Strategy

The title says it all in David La Piana and Michaela Hayes' Play to Win: The Nonprofit Guide to Competitive Strategy. Whether they recognize it or not, these authors write, nonprofit organizations compete with one another for clients, staff, funds and media attention despite the collaborative and often anti-competitive ethos that pervades nonprofit culture. The sooner they recognize and address this dimension of their work the better off they and the communities they serve will be.

The book begins with background on why nonprofits often fail to embrace the competitive dimensions of their work and instead choose to focus attention on forming collaborative relationships. The authors roll out the usual suspects here including an overall orientation towards inclusiveness and sharing and an antithesis to values normally associated with the marketplace whose spillover 'bads'― inequality, poverty, lack of opportunity― they seek to mitigate. La Piana and Hayes point out as well that collaboration is often something foisted on nonprofit organizations by the foundations and governmental entities that fund them.

Want to read the full review? The folks over at Nonprofit Central are now hosting our reviews on their site. Point your browser here to view it.