Those Pesky Unmeasurable Variables

I read recently that we see diminishing creativity in society because we are too busy. We never take time to let our minds empty to allow those ‘ah ha!’ moments in. This is probably true. But I had a ‘seriously, it took me that long to realize that?’ moment this morning. Like, full on, I’d-like-to-go-back-and-rewrite-an-entire-evaluation kind of moment. And it all hinged on the confluence of events. The first took place over my morning coffee, reading a recent Guardian Development Pros article ‘Development is Not a Science and Cannot be Measured.’ Indeed, it is not (as I’ve argued here before). Despite all the tools and academic and policy maker claims to the contrary, there are too many variables in development for any of us to explicitly state ‘A therefore B’. And we know this (those of us who do the work day in and day out and try to meld development policy with practicality and bare bones reality). The second took place at the gym a short while later as I was reading ‘Aid on the Edge of Chaos’ by Ben Ramalingam (this book in itself providing me numerous ‘ah ha’ moments recently). We jump through hoops to make development somehow measurable, sacrificing ambition and creativity along the way (for more on that, read chapter 5 of the book). That is, until we lose our cool and tell off HQ for being ridiculous (me) or let it all out in an editorial in the Guardian (Michael Kleinman, see above).

Somehow, with the concepts of measuring change and impact and jumping through hoops to make people happy swarming in my head, all while aiming to ‘turn off my brain’ at the gym, I realized that one of my crowning ‘measurement’ glories was in fact only a partial victory.

Way back in 2008, when the provincial government in Aceh, Indonesia was properly getting back on its feet after years of conflict, a devastating tsunami and recent elections, I led the design of the monitoring framework for a programme to support the new Governor to ‘make government work’. Although colleagues and government were keen to measure the programme through the number of policies developed, staff trained and systems put in place, I advocated (strongly and loudly - sorry DJ) for something more robust, more meaningful. I wanted to see if the policies and trainings and systems would make a difference in the end game: the actual delivery of services. The provincial government had an appallingly poor record of delivering its annual budget and programmes. How about measuring change on that front and then back tracing the factors to see if our programme contributed? It sounded boring to many, and not immediately achievable - like that ‘low hanging fruit’ everyone is hell bent on when it’s the long game we should be worried about - but it remained.

In 2007 the provincial government delivered just over 60% of its annual budget. By the end of 2009, it was 80%. In 2010, it was just over 90%. And boy did I crow in glory. Yes, there were other capacity building programmes taking place, but our programme had indeed contributed significantly to this astounding change, and I took my indicator success for a ride all over town (metaphorically speaking).

But today, as I sweated my way through 10km on the bike, it dawned on me that we didn’t measure or indeed account for what may have been the deciding factor or variable in the change: leadership. We talk about it but we don’t have tools to measure it, development not being a science and all. Logframes tend to preclude us from doing so, so we unknowingly limited our measurement ambitions. And although hindsight may be 20/20, even back in 2009 and 2010 it was obvious to everyone that the leadership of the Governor and his highly capable department heads was the reason change was happening. You can draft all of the policies in the world and train millions of civil servants, but unless there is leadership to take the results of these activities and move forward, you are nowhere. And if you want evidence of this, look no further than the self-serving Governor who was elected in 2012 and the dramatic reversal of governance and government fortunes that Aceh experienced in the ensuring five years. (Happily, the Governor from 2007-2012 was re-elected in 2017 so Aceh’s future is brightening).

This is a lesson well learned for me: although development is not a science we must still measure it and we must try our best to responsibly capture and account for the variables which are not in our control but nonetheless determine the success or failure of a project or programme within our project monitoring, not just as a part of our risk management strategy. Am I potentially creating more work when I’d ideally like to see things simplified? Yes, probably. But accountability matters more than simplicity at the end of the day.