It’s been some months since GDS published our social media guidance for Civil Servants, and there’s been lots of interest across the civil service (and beyond) in the idea of social in government. We’ve had queries from other countries too about our approach, and I recently Skyped into a conference that the Dutch Government were hosting to talk how they approach social media and what they might learn from the UK.
There, and in our work across government, it’s clear there’s still a lot of work to do to represent the benefits of social media. Metrics and measurement are a particularly thorny issue, as colleagues are asked to make business cases for the use of social media.
Measurement and perception
Let’s begin with that last sentence. Making a business case to use these channels is like making a business case to read the newspaper; it suggests that there’s a decision to be made about whether to engage or not.
I’d argue that the tipping point for that decision is long gone. We have to be actively engaged in those networks to talk with and to the public.
Measurement then becomes tricky. If it’s something you have to do, how do you choose what to measure? What are you trying to prove or what audience you are trying to influence?
That’s not a problem unique to social media. In one of my previous roles, as Director of Communications in local government, our media monitoring companies would send us reports comparing the amount of coverage we received in local and national news with the equivalent amount we would have spent had we actually purchased that space as advertising. The equation therefore was if size of spread > cost of purchasing that space = success.
That, of course, is tosh. Most local politicians don’t care if you’ve gotten a lot of coverage if that coverage is negative.
(This was equally true if you actually managed to achieve positive stories. I remember one particular occasion where we had two positive front page stories in the local press - the holy grail of coverage - and a small negative paragraph buried deep on page 12 about CPZ’s (controlled parking zones). Guess which one generated the call to me from the Leader of the council?)
Which is all to say that its’ always been about perception. The metrics that mattered to me then were the number of phone calls I received from the Elected Members in any given week. No calls = good job.
Understanding sentiment
Of course, we do use several monitoring and measuring tools (like the Twitter API or Sproutsocial for example) across our various social media channels in GDS. We need to understand what is engaging people, what conversations are happening that we might usefully participate in, and to see what content is of sufficient interest that people ReTweet it, or comment on it.
All of that plays an important part in keeping us relevant to the people interested in our work at GDS, so we can work with them to deliver better services.
But reporting on that interaction is never going to be an exact science. We engage in conversation with those who are interested in what we do in government. We try to explain what and why we are doing something in a certain way. We need to accept that there will always be a multiplicity of views. Where there is negativity we try to understand what is driving that and accept that while we can engage we cannot always persuade. Very little of that can be captured in a graph that shows the number of tweets we’ve sent in a given week.
Conversations build trust
The act of engaging in the social sphere shifts the dynamic in other ways. Even where there is disagreement about a particular government policy or decision, being visible and taking part in dialogue builds trust in our ability to at least have a robust conversation.
In my experience, (trolling aside) most people have a strong sense of fair play. Though they may never agree with you they do respect you for stepping out of the shadows and making yourself visible to them. That’s really important in building trust with our users, something we hold dear at GDS.
I made this point to my Dutch colleagues, but at the end of our recent session they asked me a question: “You say that engaging in social media encourages trust in government. Could you tell us what percentage growth in trust your metrics have shown so far”?
I rest my case. There’s still a lot of work to be done in showing people how engaging with social can yield substantial benefits, even though they can’t always result in traditional metrics.
We are not yet in a space where reporting can become standardised (if it ever can). Not everyone is on the same page, and that’s something we need to acknowledge in a future iteration of our social media guidance.
39 comments
Comment by John Elliot posted on
Great post. You mentioned SproutSocial, Though I use Sendible it's a great tool for posting and schedule updates and reviewing posts. Its a UK company too! http://sendible.com
Comment by markbraggins posted on
I was late reading your post Emer, but I really enjoyed it and it prompted me to blog: 'Getting the measure of social media' http://wp.me/p2CQNt-Sb
Comment by Getting the measure of social mediamarkbraggins posted on
[...] November, Emer Coleman asked: Social media – Must we measure? In a fascinating post, Emer discusses how tricky it is to present the benefits of social media in [...]
Comment by LINK LOVE: 16 blog posts that have inspired me in 2012 « The Dan Slee Blog posted on
[...] DIGITAL STATS: Emer Coleman of the Government Digital Service wrote this cracking piece on the measurement of social media and what we should be looking out for. For anyone looking to get a handle on the changing landscape it’s essential. You can read it here. [...]
Comment by Curtis Worthington (@CurtUK) posted on
Although it can be hard to actually measure on paper/metrics, social engagement is a definite must in today's world and is only going to grow and grow. Being active on social media platforms definitely helps boost conversions for my clients, it's also a great way to support your customers (o2 do a pretty good job of this on twitter). Great post Emer Coleman
Comment by Claire Cyprien posted on
I'd be interested to hear if anyone who has commented above has been succesful in evaluating whether SM has helped drive traffic to online content, therefore keeping citizens in online channels and helping contribute to the Digital by Default agenda. So this is less about engaging content/conversations and more about using SM as a signposting/marketing channel
Although this isnt two way engagement, I wonder if there is any evidence, for or against, whether there is value in getting messages out through SM channels as a low cost/no cost option.
p.s. I am actually in favour of two way/engagement .... but just wondered if there is any evidence about the above.
Comment by Mark Weyland posted on
Measuring the progress made by a company through its social media engagements is really a tough job. This blog by Emer is certainly a good one. I believe every social media professional should read this blog and get to know the areas they need to understand in order to get an answer as to how successful they are in social media marketing strategies.
I would also like to say that simply starting a social media strategy by a company is not good enough. You also need to be active on such social media sites, so that you can create that sense of trust in your customers' minds. Interact with your customers regularly so that they feel that they are indeed important.
A must read I must say. Well said Emer.
Comment by marcschmid posted on
Digital communications should be treated the same as any other form of communications and measured accordingly. As a marketing campaign will be measured against the campaign objectives any use of social media should be measured so as to evidence its effectiveness. There are many occasions where digital won't work so metrics showing that will help adapt any future approach. With budgets squeezed it is more important than ever to evidence the impact our work has on our organisation's corporate objectives. The benefit of using social media is that it is often easier to measure than more complicated behaviour change approaches which rely on improvements to long term outcomes.
Comment by watherton80 posted on
Employing social media marketing strategies can be a great way to keep in contact with your customer base, and develop relationships. Customers that friend you on Facebook, or follow you on Twitter do so willingly. They want to hear from you. You can therefore notify them of new upcoming products, sales and discounts, or even just information concerning your business and industry without the fear of being too intrusive.
Comment by Jessica Orquina (@JAOrquina) posted on
Really good article. I agree, social media is changing how we do business. It's new, it's different. Rather than pushing information to your audience, you now must engage. I also agree that social media is one communications channel among many others. (It's also the new kid on the block and a lot of people are not comfortable with it yet.)
So, how to measure. I think that the place to begin here is to remember that this is changing how we do business. So, trying to merely adapt old ways of measuring success is not sufficient. What is success in engagement? Starting with that question, hopefully someday we'll get an answer.
Comment by melissacw posted on
>I rest my case. There’s still a lot of work to be done in showing people how engaging with social can yield substantial benefits, even though they can’t always result in traditional metrics.
Hear, hear! let no-one be deterred from engaged with new media; it is the only way forward
Comment by All Wales public services internet and social media survey 2012 | www.participationcymru.org.uk posted on
[...] of evaluation – the UK Government Digital Service team has just written a really interesting blog on this. I’ve also found Helen Reynolds’ blogs on how we evaluate social media an inspiration [...]
Comment by Arolwg o’r modd mae gwasanaethau cyhoeddus Cymru yn defnyddio’r we a chyfryngau cymdeithasol yn 2012 | www.participationcymru.org.uk posted on
[...] eithaf amserol o ran gwerthuso – mae Tîm Gwasanaeth Digidol Llywodraeth y DU wedi ysgrifennu blog diddorol iawn ar hyn. Mae blog Helen Reynolds hefyd wedi bod yn ysbrydoliaeth o ran gwerthuso [...]
Comment by Digital Measure Tape posted on
Thank you for providing such information. This is very generous of you providing such vital information which is very informative.
Comment by Barry A. Martin (@hypenoticbam) posted on
Nice. 2 thoughts:
1. I've used the the same newspaper analogy, but swapped in the telephone, over the years. It's understandable that people who aren't already engaged would have trouble understanding that it's a platform.
2. We try to help people shift their thoughts about measurement from ROI (which can be a much longer and more circuitous tail these days) to return on relationship. For example, if you can turn a detractor into a supporter, they make for fervent and often credible advocates. If you can capture how influential your community members, on what subjects, regions and platforms.
Comment by emercoleman (@emercoleman) posted on
Barry - see post to Robin above. We are in agreement.
Comment by tomsprints posted on
Good thought-provoker, Emer.
I have a view that much of the ROI on social media often accrues in places you'd not expect or might have problems measuring. I actually don't think that's completely unique to social media, either. The particular issue for me is that there's a lot of value in social media that lies outside the "cause and effect" equation. Post hoc metrics can be assessed using several known tools, but these are seldom help when trying to produce a business case.
Simple example of assessing the reach of messages on Twitter. Who can be certain of retweets? I'd not believe any business case that promised any specified level of penetration. Some might get more, some less, but who knows? In turn, levels of engagement are likely to be heavily dependent on levels and direction of reach.
Not suggesting the answer is "do nothing", of course, just that predictions and measurement of any balance between inputs and outputs shoul be treated with great caution when social media is involved.
Tom
Comment by emercoleman (@emercoleman) posted on
Thanks Tom this chimes with the response to Robin above regarding social capital. I believe also that regardless of that which we can measure it's also a choice between the default being open or closed. In my view we are now in a world where the default is open and therefore, as you say, "do nothing" cannot be the answer.
Comment by robinrileyuk posted on
Hi Emer,
A thought-provoking post on this oft-ignored subject. Four points from me:
(1) No one would dispute that the very poor measures of success you cite (and which I'm sure many of us, sadly, would recognise) are, well, very poor. But that's not necessarily an argument against measurement - merely an argument for *better* measurement.
(2) Social media can contribute value in many ways beyond engagement (which is, arguably, merely the first step in getting value from social media). For example, it can be used to complement customer contact channels (taking pressure off phone lines) or replace expensive marketing. Many of these things are highly measurable - and being able to compare 'dollar-value' between social media and other channels is vital if we are to make investment decisions in one over another.
(3) It's entirely possible to measure the contribution that social media makes to the more nebulous audience-outcomes, such as trust. One approach would be to survey your audience on how much they trust your organisation, and also on how they encounter / interact with your organisation. A bit of 2nd order stats will then cross-correlate the answers from one to the other - showing the extent to which dealing with you via social media is positively (or possibly, negatively) correlated with trust.
(4) Of course engagement has value, and should be a part of our daily work... but it takes time, and civil service time is not free. As a taxpayer I want government to measure the effectiveness of how it
spends every penny of my money. It's an important part of accountability. Hence measurement, now, for everything you do, please!
In summary - just because social media evaluation is difficult, doesn't mean we shouldn't try. If we *did* have a simple, reliable way of measuring the true value (in all senses) from social media... would you argue against it?
Comment by emercoleman (@emercoleman) posted on
Robin I don't disagree that social media has a role to play in the ways you suggest and certainly they can augment customer contact channels. The point of my post was to suggest that some of the outcomes of engagement cannot be measured in a linear/rational sense such as the value of building social capital. I'd give as another example a politician engaging with something like the Occupy London movement but then disengaging when they realise that this group will never cast a vote in a ballot box for them. This is to understand engagement as a chip that is cashed in for gain at election time. But while this kind of group might never engage in elections that is not to say they would not remain a committed group that could be called on for opinion in the social sphere on another issue or campaign (not just to vote). We cannot just use social media to support our own ends and we can never know what contribution we make to social capital that remains outside of our sights but nonetheless builds trust in government. As a civil servant I think that is a core part of our work.
Comment by Helen Hardy posted on
Thanks Emer and Robin for really interesting points. While we can always get too bound up in the measurables, I think I'm with Robin on saying it's always worth trying. In particular, it strikes me that although the decision on whether to engage via social media may be a given, decisions on how/when/where to do so are still very much up for grabs.
One thing that might bear watching, for example, is how much we end up repeating ourselves - which should be visible to us even if it's a qualitative judgement rather than a 'hard' one. That qualitative measure could then point to very measurable changes either to an underlying policy, piece of content etc which is not achieving its goals, and/or to how we spend our time with social media more productively.
There isn't enough time in the world, let alone in our working lives, to engage with every user on every channel, so we do need the measures, however approximate, to help us make those choices.
Comment by Participation Cymru posted on
Really interesting stuff, thanks for this! We're part of a Public Engagement Working Group that has recently run an All Wales Public Service Organisations internet and social media survey, and I've been tasked with looking at evaluation! I'm currently compiling the report.
You've hit the nail on the head with this article - that element of trust seems to be central to everything - trusting staff to use social media effectively and trusting that the public will respond at all!
I spoke to someone from Monmouthshire Council not long ago and they pointed out that's it's very interesting that we have to evaluate how we use social media to communicate but we're never asked to evaluate how we use a phone!
Also there seems to be an assumption by some organisations that being on social media platforms makes them sitting ducks for criticism, but that criticism will be there whether they're online or not, except when they're online they get the chance to respond and build bridges.
Comment by emercoleman (@emercoleman) posted on
Thanks for posting and for your comments. I think another issue with monitoring and evaluating is that the tools we have for this in the social sphere are not mature. So we reduce the colour, life and debate on platforms like Twitter to numbers. I try and use tools like Storify when presenting to senior people in Whitehall so they can actually see someones picture and their actual comments which restores some of the humanity back into a spreadsheet mentality. In the end social platforms often speak to emotional intelligence rather than hard facts. Which goes back to my point about the importance of perception.
Comment by Participation Cymru posted on
Thanks for that - I think that's a really good point about the qualitative side of it. I haven't mentioned Storify in my section of the report yet, but I agree that it's a really good way of capturing conversations. Cheers!
Comment by Martin Cantor posted on
Good stuff Emer, but one argument to deploy to the metricians (metricophiles?) is that of the counterfactual. What's the effect of not engaging. What's being said about us in spaces we're not even in?.......
Comment by emercoleman (@emercoleman) posted on
Martin I recently made that point to a senior leader in government. Increasingly it will appear strange (certainly to the digital natives) when someone has no visibility online. We also see examples of government departments or agencies trying to get people to use "official hashtags" and thereby missing important parts of the conversation - because the meaningful conversation is being driven elsewhere with user generated tags.
Comment by Mike Thacker posted on
Understood - we can't measure the fuzzy stuff, but what about specific service transactions? A TwitPic of a broken bike stand or damaged bridge with location might be treated as a service request via a social media channel. Should we not be measuring how much that that happens and the ways in which we support social media channels for actual services as well as influencing policy and trust as a whole?
Comment by emercoleman (@emercoleman) posted on
Mike I totally take that point. I have seen lots of good examples of the use of social media for this, including some good apps like I Love Lewisham. I think local authorities need to understand that service reporting in this way will become the norm and a challenge to existing CRM systems that are not built to factor in social reporting.
Comment by John Fox (@x333xxx) posted on
Unfortunately many local authorities still regard social media (and in particular Twitter) as a one way street. They churn out the council press releases or news of local events, but they need to do much more in terms of interacting with correspondents. Ownership of social media channels needs to be addressed too. The customer service function is generally the best area, though many corporate communications team will defend their right to tweet press releases and therefore control of the principal Twitter ID, which I suspect often leads to a bit of a stalemate (I certainly know of one council where this is the case). But channel ownership isn't enough - it needs to be adequately resourced, with staff who understand the medium and who can engage with customers and pass requests for information or service to the right part of the organisation, that in turn recognises the value of social media and its potential for cost effective service delivery and customer satisfaction rates.
Comment by Jessica Orquina (@JAOrquina) posted on
I agree! It's important to use Twitter for 2-way conversations!
Comment by Ranjit Sidhu (@rssidhu) posted on
Hi Emer, I have read this with interest, one area I have worked a lot with recently is universities and how to make the masses of data out there relevant for them
My take is that the measuring and valuing of online data is too often driven by what data can be produced and given to organisations, rather then the more difficult and perhaps less flattering information that fits into the organisations' goals and roles. Vague propositions on the effectiveness of new technologies are produced automatically with little and most probably no understanding of the organisation or even the sector, therefore measuring Coke-Cola and GDS in the same way.
Further, the ability to say "we don't know" or "this is our best guess" seems foreign to online measurement which takes it reams of data as a sign that it must "know", which means it often over extends on the perception of what it can deliver, again leading to necessary vagueness. This humility, is important and needed.
My opinion is it is essential to redefine the organisation as central and understand:
1. Social Media is simply one channel among other communication channels and therefore:
2. Where possible, create comparative measurements across the channels to define success or not, in the best way we can. This may mean leaving out data, simply saying we don't know certain areas or creating cross channel measurements that need forecasting, but it does start to build a structure for comparison and a firm place to stand on which to move forward.
What may be harder to get is definable goals, however once they have been agreed for separate projects, departments or campaigns then, in my opinion, the data needs to be manipulated to fit against them, rather the other way round.
So, to answer your question, in my opinion; yes, measure, but with clearly defined goals for the organisation using only the data that can educate and with the humility to say for some points we can have to guess.
Thanks for the article, very interesting.
Comment by emercoleman (@emercoleman) posted on
Ranjit I once heard Charlie Leadbetter address a group of local authority officers and he called for more "don't know managers". It speaks to your point about best guess too. I do think that much of social media measurement has to be best guess as sentiment is all subjective in the end. What matters most, and to whom, differs. What social does add is the "softer", human voice that compliments all of the quantitative work that government does. It helps us understand what might really matter to people rather than what we, as government, think matters to people. But these as I say will remain subjective opinions but the more collective voices we can hear the more likely we are to reach a more informed consensus.
Comment by Ranjit Sidhu (@rssidhu) posted on
Emer, completely agree and believe one of the most interesting elements of Social Media is both its qualitative and quantitative elements:
The qualitative element of Social Media you talk about to provide the enriching colour of the personal story; a micro view often missed by an organisation.
But, also the difficult quantitative macro perspective of understanding Social Media's worth in comparison to other channels. What I would argue is that a sector/organisational bespoke comparative analysis with other channels at its disposal means this "best guess" analysis can then be used to justify (or not) spend or help start defining deliverables for new social media projects. Not perfect, but a start to build on?
Thanks Ranjit
Comment by x333xxx posted on
In any field, but especially when one needs to prove take-up, a body of evidence is what really matters when one has a need to convince a doubter of the merit of doing something differently. You are generally able to slip into a conversation "... but don't take my word for it, look at ... " and you then illustrate with hard data how a particular channel is being used by real people rather than theorists.
I coined a phrase about 12 years ago in order to get people to think differently about website content: "Is it useful? Is it usable? Is it used?". I was delighted when it got picked up by the former Office of the e-Envoy! It's a useful mantra to now employ in the realm of social media. It's all very well having a Twitter ID or a Facebook page, but what evidence do you have that citizens find it useful and use it?
Reading this observation back to myself I'm thinking that this is common sense, that everyone will say "Yeah, I know that! Why don't you tell me something new?", yet it is amazing how often the simplest form method of illustrating or proving the value of doing something can be passed over in favour of writing some convoluted tome of a strategy or paper.
Comment by emercoleman (@emercoleman) posted on
I agree with your comments. Our mantra here in GDS is always "what is the user need"? that's the question we ask all the time in relation to service transformation and it applies equally well in relation to social media.
Comment by Ben Giddings posted on
Good article Emer.
Measuring is one of the trickiest areas of the whole Social Media piece (and arguably the digital one). I like what you say here: "We need to understand what is engaging people, what conversations are happening that we might usefully participate in, and to see what content is of sufficient interest that people ReTweet it, or comment on it" - absolutely a key area of focus. I would add that we could even be more proactive and identify specific people who we want to be engaging with and working out how best to connect with them.
The trouble is, digital is calling for a major change in the way people in government communicate, and some are nervous about changing their habits, which they know work well, and adopting new and unfamiliar tools. Because the web is seen as being more measurable than traditional media and traditional diplomacy techniques, there is an expectation that data can be summoned up to show how well digital works, thus reassuring them that it is safe or "profitable" to make the move.
Perhaps it is our job as digital communicators is to change the nature of the argument. Much of the data that circulates focuses on the mechanisms and the tools, whereas we should be focusing attention on the content of the message and the quality (for want of a better word) of the audience.
At the moment, the argument for stats on social media is rather like measuring the success of a speech by seeing how loud the microphone is. What we need to do is work on getting the right people into the room (whether it be physical or virtual) and then talking to them in the right way - just like we do in the real world.
Comment by emercoleman (@emercoleman) posted on
Ben Thanks for your comments I particularly like your microphone comparison and your reference to the real world even more. I have a good friend who always says what matters is are we relevant and what's our standing in the conversation. I think that by government engaging in this way we achieve both more relevance and ultimately more standing in the conversation.
Comment by Tema Frank (@temafrank) posted on
Love that analogy!
Comment by Jessica Orquina (@JAOrquina) posted on
Great analogy!