39 comments

  1. Ben Giddings

    Good article Emer.

    Measuring is one of the trickiest areas of the whole Social Media piece (and arguably the digital one). I like what you say here: “We need to understand what is engaging people, what conversations are happening that we might usefully participate in, and to see what content is of sufficient interest that people ReTweet it, or comment on it” – absolutely a key area of focus. I would add that we could even be more proactive and identify specific people who we want to be engaging with and working out how best to connect with them.

    The trouble is, digital is calling for a major change in the way people in government communicate, and some are nervous about changing their habits, which they know work well, and adopting new and unfamiliar tools. Because the web is seen as being more measurable than traditional media and traditional diplomacy techniques, there is an expectation that data can be summoned up to show how well digital works, thus reassuring them that it is safe or “profitable” to make the move.

    Perhaps it is our job as digital communicators is to change the nature of the argument. Much of the data that circulates focuses on the mechanisms and the tools, whereas we should be focusing attention on the content of the message and the quality (for want of a better word) of the audience.

    At the moment, the argument for stats on social media is rather like measuring the success of a speech by seeing how loud the microphone is. What we need to do is work on getting the right people into the room (whether it be physical or virtual) and then talking to them in the right way – just like we do in the real world.

    Reply
    • emercoleman (@emercoleman)

      Ben Thanks for your comments I particularly like your microphone comparison and your reference to the real world even more. I have a good friend who always says what matters is are we relevant and what’s our standing in the conversation. I think that by government engaging in this way we achieve both more relevance and ultimately more standing in the conversation.

      Reply
    • Tema Frank (@temafrank)

      Love that analogy!

      Reply
    • Jessica Orquina (@JAOrquina)

      Great analogy!

      Reply
  2. x333xxx

    In any field, but especially when one needs to prove take-up, a body of evidence is what really matters when one has a need to convince a doubter of the merit of doing something differently. You are generally able to slip into a conversation “… but don’t take my word for it, look at … ” and you then illustrate with hard data how a particular channel is being used by real people rather than theorists.

    I coined a phrase about 12 years ago in order to get people to think differently about website content: “Is it useful? Is it usable? Is it used?”. I was delighted when it got picked up by the former Office of the e-Envoy! It’s a useful mantra to now employ in the realm of social media. It’s all very well having a Twitter ID or a Facebook page, but what evidence do you have that citizens find it useful and use it?

    Reading this observation back to myself I’m thinking that this is common sense, that everyone will say “Yeah, I know that! Why don’t you tell me something new?”, yet it is amazing how often the simplest form method of illustrating or proving the value of doing something can be passed over in favour of writing some convoluted tome of a strategy or paper.

    Reply
    • emercoleman (@emercoleman)

      I agree with your comments. Our mantra here in GDS is always “what is the user need”? that’s the question we ask all the time in relation to service transformation and it applies equally well in relation to social media.

      Reply
  3. Ranjit Sidhu (@rssidhu)

    Hi Emer, I have read this with interest, one area I have worked a lot with recently is universities and how to make the masses of data out there relevant for them

    My take is that the measuring and valuing of online data is too often driven by what data can be produced and given to organisations, rather then the more difficult and perhaps less flattering information that fits into the organisations’ goals and roles. Vague propositions on the effectiveness of new technologies are produced automatically with little and most probably no understanding of the organisation or even the sector, therefore measuring Coke-Cola and GDS in the same way.

    Further, the ability to say “we don’t know” or “this is our best guess” seems foreign to online measurement which takes it reams of data as a sign that it must “know”, which means it often over extends on the perception of what it can deliver, again leading to necessary vagueness. This humility, is important and needed.

    My opinion is it is essential to redefine the organisation as central and understand:
    1. Social Media is simply one channel among other communication channels and therefore:
    2. Where possible, create comparative measurements across the channels to define success or not, in the best way we can. This may mean leaving out data, simply saying we don’t know certain areas or creating cross channel measurements that need forecasting, but it does start to build a structure for comparison and a firm place to stand on which to move forward.

    What may be harder to get is definable goals, however once they have been agreed for separate projects, departments or campaigns then, in my opinion, the data needs to be manipulated to fit against them, rather the other way round.

    So, to answer your question, in my opinion; yes, measure, but with clearly defined goals for the organisation using only the data that can educate and with the humility to say for some points we can have to guess.

    Thanks for the article, very interesting.

    Reply
    • emercoleman (@emercoleman)

      Ranjit I once heard Charlie Leadbetter address a group of local authority officers and he called for more “don’t know managers”. It speaks to your point about best guess too. I do think that much of social media measurement has to be best guess as sentiment is all subjective in the end. What matters most, and to whom, differs. What social does add is the “softer”, human voice that compliments all of the quantitative work that government does. It helps us understand what might really matter to people rather than what we, as government, think matters to people. But these as I say will remain subjective opinions but the more collective voices we can hear the more likely we are to reach a more informed consensus.

      Reply
      • Ranjit Sidhu (@rssidhu)

        Emer, completely agree and believe one of the most interesting elements of Social Media is both its qualitative and quantitative elements:
        The qualitative element of Social Media you talk about to provide the enriching colour of the personal story; a micro view often missed by an organisation.
        But, also the difficult quantitative macro perspective of understanding Social Media’s worth in comparison to other channels. What I would argue is that a sector/organisational bespoke comparative analysis with other channels at its disposal means this “best guess” analysis can then be used to justify (or not) spend or help start defining deliverables for new social media projects. Not perfect, but a start to build on?

        Thanks Ranjit

        Reply
  4. Mike Thacker

    Understood – we can’t measure the fuzzy stuff, but what about specific service transactions? A TwitPic of a broken bike stand or damaged bridge with location might be treated as a service request via a social media channel. Should we not be measuring how much that that happens and the ways in which we support social media channels for actual services as well as influencing policy and trust as a whole?

    Reply
    • emercoleman (@emercoleman)

      Mike I totally take that point. I have seen lots of good examples of the use of social media for this, including some good apps like I Love Lewisham. I think local authorities need to understand that service reporting in this way will become the norm and a challenge to existing CRM systems that are not built to factor in social reporting.

      Reply
      • John Fox (@x333xxx)

        Unfortunately many local authorities still regard social media (and in particular Twitter) as a one way street. They churn out the council press releases or news of local events, but they need to do much more in terms of interacting with correspondents. Ownership of social media channels needs to be addressed too. The customer service function is generally the best area, though many corporate communications team will defend their right to tweet press releases and therefore control of the principal Twitter ID, which I suspect often leads to a bit of a stalemate (I certainly know of one council where this is the case). But channel ownership isn’t enough – it needs to be adequately resourced, with staff who understand the medium and who can engage with customers and pass requests for information or service to the right part of the organisation, that in turn recognises the value of social media and its potential for cost effective service delivery and customer satisfaction rates.

        Reply
  5. Martin Cantor

    Good stuff Emer, but one argument to deploy to the metricians (metricophiles?) is that of the counterfactual. What’s the effect of not engaging. What’s being said about us in spaces we’re not even in?…….

    Reply
    • emercoleman (@emercoleman)

      Martin I recently made that point to a senior leader in government. Increasingly it will appear strange (certainly to the digital natives) when someone has no visibility online. We also see examples of government departments or agencies trying to get people to use “official hashtags” and thereby missing important parts of the conversation – because the meaningful conversation is being driven elsewhere with user generated tags.

      Reply
  6. Participation Cymru

    Really interesting stuff, thanks for this! We’re part of a Public Engagement Working Group that has recently run an All Wales Public Service Organisations internet and social media survey, and I’ve been tasked with looking at evaluation! I’m currently compiling the report.

    You’ve hit the nail on the head with this article – that element of trust seems to be central to everything – trusting staff to use social media effectively and trusting that the public will respond at all!

    I spoke to someone from Monmouthshire Council not long ago and they pointed out that’s it’s very interesting that we have to evaluate how we use social media to communicate but we’re never asked to evaluate how we use a phone!

    Also there seems to be an assumption by some organisations that being on social media platforms makes them sitting ducks for criticism, but that criticism will be there whether they’re online or not, except when they’re online they get the chance to respond and build bridges.

    Reply
    • emercoleman (@emercoleman)

      Thanks for posting and for your comments. I think another issue with monitoring and evaluating is that the tools we have for this in the social sphere are not mature. So we reduce the colour, life and debate on platforms like Twitter to numbers. I try and use tools like Storify when presenting to senior people in Whitehall so they can actually see someones picture and their actual comments which restores some of the humanity back into a spreadsheet mentality. In the end social platforms often speak to emotional intelligence rather than hard facts. Which goes back to my point about the importance of perception.

      Reply
      • Participation Cymru

        Thanks for that – I think that’s a really good point about the qualitative side of it. I haven’t mentioned Storify in my section of the report yet, but I agree that it’s a really good way of capturing conversations. Cheers!

        Reply
  7. robinrileyuk

    Hi Emer,
    A thought-provoking post on this oft-ignored subject. Four points from me:

    (1) No one would dispute that the very poor measures of success you cite (and which I’m sure many of us, sadly, would recognise) are, well, very poor. But that’s not necessarily an argument against measurement – merely an argument for *better* measurement.

    (2) Social media can contribute value in many ways beyond engagement (which is, arguably, merely the first step in getting value from social media). For example, it can be used to complement customer contact channels (taking pressure off phone lines) or replace expensive marketing. Many of these things are highly measurable – and being able to compare ‘dollar-value’ between social media and other channels is vital if we are to make investment decisions in one over another.

    (3) It’s entirely possible to measure the contribution that social media makes to the more nebulous audience-outcomes, such as trust. One approach would be to survey your audience on how much they trust your organisation, and also on how they encounter / interact with your organisation. A bit of 2nd order stats will then cross-correlate the answers from one to the other – showing the extent to which dealing with you via social media is positively (or possibly, negatively) correlated with trust.

    (4) Of course engagement has value, and should be a part of our daily work… but it takes time, and civil service time is not free. As a taxpayer I want government to measure the effectiveness of how it
    spends every penny of my money. It’s an important part of accountability. Hence measurement, now, for everything you do, please!

    In summary – just because social media evaluation is difficult, doesn’t mean we shouldn’t try. If we *did* have a simple, reliable way of measuring the true value (in all senses) from social media… would you argue against it?

    Reply
    • emercoleman (@emercoleman)

      Robin I don’t disagree that social media has a role to play in the ways you suggest and certainly they can augment customer contact channels. The point of my post was to suggest that some of the outcomes of engagement cannot be measured in a linear/rational sense such as the value of building social capital. I’d give as another example a politician engaging with something like the Occupy London movement but then disengaging when they realise that this group will never cast a vote in a ballot box for them. This is to understand engagement as a chip that is cashed in for gain at election time. But while this kind of group might never engage in elections that is not to say they would not remain a committed group that could be called on for opinion in the social sphere on another issue or campaign (not just to vote). We cannot just use social media to support our own ends and we can never know what contribution we make to social capital that remains outside of our sights but nonetheless builds trust in government. As a civil servant I think that is a core part of our work.

      Reply
      • Helen Hardy

        Thanks Emer and Robin for really interesting points. While we can always get too bound up in the measurables, I think I’m with Robin on saying it’s always worth trying. In particular, it strikes me that although the decision on whether to engage via social media may be a given, decisions on how/when/where to do so are still very much up for grabs.
        One thing that might bear watching, for example, is how much we end up repeating ourselves – which should be visible to us even if it’s a qualitative judgement rather than a ‘hard’ one. That qualitative measure could then point to very measurable changes either to an underlying policy, piece of content etc which is not achieving its goals, and/or to how we spend our time with social media more productively.
        There isn’t enough time in the world, let alone in our working lives, to engage with every user on every channel, so we do need the measures, however approximate, to help us make those choices.

        Reply
  8. tomsprints

    Good thought-provoker, Emer.

    I have a view that much of the ROI on social media often accrues in places you’d not expect or might have problems measuring. I actually don’t think that’s completely unique to social media, either. The particular issue for me is that there’s a lot of value in social media that lies outside the “cause and effect” equation. Post hoc metrics can be assessed using several known tools, but these are seldom help when trying to produce a business case.

    Simple example of assessing the reach of messages on Twitter. Who can be certain of retweets? I’d not believe any business case that promised any specified level of penetration. Some might get more, some less, but who knows? In turn, levels of engagement are likely to be heavily dependent on levels and direction of reach.

    Not suggesting the answer is “do nothing”, of course, just that predictions and measurement of any balance between inputs and outputs shoul be treated with great caution when social media is involved.

    Tom

    Reply
    • emercoleman (@emercoleman)

      Thanks Tom this chimes with the response to Robin above regarding social capital. I believe also that regardless of that which we can measure it’s also a choice between the default being open or closed. In my view we are now in a world where the default is open and therefore, as you say, “do nothing” cannot be the answer.

      Reply
  9. Barry A. Martin (@hypenoticbam)

    Nice. 2 thoughts:
    1. I’ve used the the same newspaper analogy, but swapped in the telephone, over the years. It’s understandable that people who aren’t already engaged would have trouble understanding that it’s a platform.

    2. We try to help people shift their thoughts about measurement from ROI (which can be a much longer and more circuitous tail these days) to return on relationship. For example, if you can turn a detractor into a supporter, they make for fervent and often credible advocates. If you can capture how influential your community members, on what subjects, regions and platforms.

    Reply
  10. Digital Measure Tape

    Thank you for providing such information. This is very generous of you providing such vital information which is very informative.

    Reply
  11. Arolwg o’r modd mae gwasanaethau cyhoeddus Cymru yn defnyddio’r we a chyfryngau cymdeithasol yn 2012 | www.participationcymru.org.uk

    [...] eithaf amserol o ran gwerthuso – mae Tîm Gwasanaeth Digidol Llywodraeth y DU wedi ysgrifennu blog diddorol iawn ar hyn. Mae blog Helen Reynolds hefyd wedi bod yn ysbrydoliaeth o ran gwerthuso [...]

    Reply
  12. All Wales public services internet and social media survey 2012 | www.participationcymru.org.uk

    [...] of evaluation – the UK Government Digital Service team has just written a really interesting blog on this. I’ve also found Helen Reynolds’ blogs on how we evaluate social media an inspiration [...]

    Reply
  13. melissacw

    >I rest my case. There’s still a lot of work to be done in showing people how engaging with social can yield substantial benefits, even though they can’t always result in traditional metrics.

    Hear, hear! let no-one be deterred from engaged with new media; it is the only way forward

    Reply
  14. Jessica Orquina (@JAOrquina)

    Really good article. I agree, social media is changing how we do business. It’s new, it’s different. Rather than pushing information to your audience, you now must engage. I also agree that social media is one communications channel among many others. (It’s also the new kid on the block and a lot of people are not comfortable with it yet.)

    So, how to measure. I think that the place to begin here is to remember that this is changing how we do business. So, trying to merely adapt old ways of measuring success is not sufficient. What is success in engagement? Starting with that question, hopefully someday we’ll get an answer.

    Reply
  15. watherton80

    Employing social media marketing strategies can be a great way to keep in contact with your customer base, and develop relationships. Customers that friend you on Facebook, or follow you on Twitter do so willingly. They want to hear from you. You can therefore notify them of new upcoming products, sales and discounts, or even just information concerning your business and industry without the fear of being too intrusive.

    Reply
  16. marcschmid

    Digital communications should be treated the same as any other form of communications and measured accordingly. As a marketing campaign will be measured against the campaign objectives any use of social media should be measured so as to evidence its effectiveness. There are many occasions where digital won’t work so metrics showing that will help adapt any future approach. With budgets squeezed it is more important than ever to evidence the impact our work has on our organisation’s corporate objectives. The benefit of using social media is that it is often easier to measure than more complicated behaviour change approaches which rely on improvements to long term outcomes.

    Reply
  17. Mark Weyland

    Measuring the progress made by a company through its social media engagements is really a tough job. This blog by Emer is certainly a good one. I believe every social media professional should read this blog and get to know the areas they need to understand in order to get an answer as to how successful they are in social media marketing strategies.

    I would also like to say that simply starting a social media strategy by a company is not good enough. You also need to be active on such social media sites, so that you can create that sense of trust in your customers’ minds. Interact with your customers regularly so that they feel that they are indeed important.

    A must read I must say. Well said Emer.

    Reply
  18. Claire Cyprien

    I’d be interested to hear if anyone who has commented above has been succesful in evaluating whether SM has helped drive traffic to online content, therefore keeping citizens in online channels and helping contribute to the Digital by Default agenda. So this is less about engaging content/conversations and more about using SM as a signposting/marketing channel

    Although this isnt two way engagement, I wonder if there is any evidence, for or against, whether there is value in getting messages out through SM channels as a low cost/no cost option.

    p.s. I am actually in favour of two way/engagement …. but just wondered if there is any evidence about the above.

    Reply
  19. Curtis Worthington (@CurtUK)

    Although it can be hard to actually measure on paper/metrics, social engagement is a definite must in today’s world and is only going to grow and grow. Being active on social media platforms definitely helps boost conversions for my clients, it’s also a great way to support your customers (o2 do a pretty good job of this on twitter). Great post Emer Coleman

    Reply
  20. LINK LOVE: 16 blog posts that have inspired me in 2012 « The Dan Slee Blog

    [...] DIGITAL STATS: Emer Coleman of the Government Digital Service wrote this cracking piece on the measurement of social media and what we should be looking out for. For anyone looking to get a handle on the changing landscape it’s essential. You can read it here. [...]

    Reply
  21. Getting the measure of social mediamarkbraggins

    [...] November, Emer Coleman asked: Social media – Must we measure? In a fascinating post, Emer discusses how tricky it is to present the benefits of social media in [...]

    Reply
  22. markbraggins

    I was late reading your post Emer, but I really enjoyed it and it prompted me to blog: ‘Getting the measure of social media’ http://wp.me/p2CQNt-Sb

    Reply
  23. John Elliot

    Great post. You mentioned SproutSocial, Though I use Sendible it’s a great tool for posting and schedule updates and reviewing posts. Its a UK company too! http://sendible.com

    Reply

Leave a comment