AidSource

(The Humanitarian Social Network)

Best NGOs to work for? some methodological issues...

Hi all, I wrote a blog post last week about the ridiculous "Top 100 Best NGOs" list that some outfit called the Global Journal publised. It was a silly ranking list endeavor and their methodology was terrible/nonexistent. More here. (And definitely check out the comment from the list's editor...)

But that's not why I'm writing here. Criticizing Global Journal's list made me wonder whether there might be some categories where you actually can make a principled, evidence-based, and methodologically sound comparison between organizations. It certainly couldn't be a comparison of something overarching like "impact" (because how would you ever compare, say, World Vision and Search for Common Ground? they do such different things) but maybe on something more narrow, you could make a comparison.

So I thought: what does every NGO do? They all hire people. Could we compare how NGOs are as employers? Maybe a set of metrics around compensation (both absolute levels and internal equity), advancement opportunity, turnover rates, staff development, etc? This could be useful to job seekers, and maybe provide some encouragement for NGOs to improve their practices.

I see three obstacles to doing this:

Data collection. Most NGOs wouldn't provide data on themselves. Could we collect anonymous surveys directly from employees themselves, using Aid Source, blogs and other networks to reach them? Would that create anything close to a representative sample?
Internal heterogeneity. I.e. does it all depend on which office/department of the organization you work in, especially for the big ones?
Which orgs to include, given the blurred line between NGOs, contractors, social enterprises, etc.

What do you think? Honestly, after laying this out, I suspect that this endeavor wouldn't work. But maybe I'm wrong. I thought some folks on here might have opinions on it.

7 members like this

Replies to This Discussion

Permalink Reply by Cynan Houghton on February 2, 2012 at 5:14am

Hi all, let me propose an alternative to both to the general e-pinion approach emerging below, and Dave's initial suggestion about trying to collect from agencies a massive bunch of enterprise data (turnover rates, salary scales).

How about a simpler exercise: an all comers, all-agency staff satisfaction survey?

These are the kinds of quite useful statements it would be possible to say afterwards

“We had 300 people from NGO-X respond. On average they had a satisfaction rating of 8/10. The most common positives they reported were good career development opportunities and a Jacuzzi in every guesthouse. Satisfaction fell off sharply for people there for more than 5 years though – that was only 5/10."

“In contrast, the 200 people from NGO-Y reported an average satisfaction of 4/10. The most common negatives they reported were a lack of strategy and quality in their programming, and not being allowed into NGO-X’s Jacuzzis.”

"The 50 people from UN Agency P reported 10/10 satisfaction. The most common positive response was working 4 days a week. The most common negative response was... working 4 days a week."

And so on. Here’s the USP that could set it apart. If you want to register an opinion about Agency X, you have to log your survey results and give an email address from Agency X. Without that, any yahoo (aid worker or general non-aidworker public) would be free to log responses. Of course that has benefits (verifiable data!) but also has limitations (people “between contracts” can’t report; people with a dire experience who left Agency X three years ago can’t report) so there may be a positive bias. Dave, you might have experience from the previous blog reader survey though that might tell you this concern is unwarranted. You also I guess have a pretty long list of email addresses from that survey you could send this one to...

Survey pretty simple, surely <10 mins for participants.

Section A – Who do you work for (and for how long), what kind of job, how many agencies have you previously worked for.
B – What gives you joy in your current agency (multiple choice + a free text response).
C – What gives you sad in your current agency. (multiple choice + a free text response).
D – Give us an overall rating of your satisfaction with your current agency. Go on!

Whoever does this (and gets n>2000) could probably sell the (anonymized) raw data to the HR departments of a dozen or more agencies, ho ho.

Permalink Reply by Shotgun Shack on February 2, 2012 at 5:49am

Hm, I think that might be 'easier' but it wouldn't be on-going or super dynamic or really 'available' to most people. How many of us had the time/capacity to dig into Dave's raw data on the blogger survey? I only was able to read his summaries, and it wasn't super searchable. I'm biased towards the idea of having a 'living register' (you know, similar to a 'living document') that regular folks can easily search for what they want/need at the drop of a hat. If ALNAP isn't interested or the right place for a grant of this type, there are other places. (I can't think of any off hand, but that's why this group is called "collective wisdom"). I like Ian's idea but would want to see some examples of what it looks like to see if it's relevant to what we want.... I think we could first off focus on 'the best places to work' based on a number of indicators. One of those could be 'feeling of'satisfaction that the organization's programs are having an impact' (or something better worded). It kind of gets at the point of quality of the organization's work but keeps us on track regarding what it's actually like to work for them.

Permalink Reply by Cynan Houghton on February 4, 2012 at 3:59am

Definitely easier to do, springboards off previous work, much more likely to see an output in the next 3-6 months. But this isn't either/or if there's people with appetite and time to spare.

Re Dave's previous stats - well I thought Dave did a bang up (and pretty detailed) job of digging into the stats across a bunch of blog posts, and given the subject matter (aid blog readership) didn't personally feel the need to dig further. I think given the length of this thread there'd be a lot more interest in delving into the results on this front.

Whichever approach, look forward to seeing what comes of this.

Permalink Reply by Andreailcujo on February 17, 2012 at 5:41am

I think this could be a good start and help to furthermore shape a rating application.

I believe comments are important: one might have a good experience with one NGO and be totally unsatisfied with another. I believe our job is really "contextualized". Maybe you jus get your HoM close to a burnout, you're just arrived and your life is impossible ...

BTW I believe it is a great idea, go along with it. HR departments need to be shaken-up (to be read as: interns shouldn't be responsible for selection of aid workers ... ups I said that!).

Permalink Reply by Jeffrey Frankens on February 3, 2012 at 2:25am

I think the "Amazon Review Style" is a fantastic idea. I'll reach out to my friends who do web development/coding and see if they have any ideas. Ideally if no one knows how to build the system from code completely a Wordpress plugin would be best. But you'd want to make sure you get it right the first time because you wouldn't want to lose any reviews if we move to a new system.

Permalink Reply by Tom Paulson on February 3, 2012 at 11:33am

On Humanosphere, I posted Dave's excellent critique to help me defend myself against the critics of my earlier (somewhat smart-ass) post entitled PATH 6th best NGO; Gates Fdn doesn't even make the list.

Thanks Dave!