Open Source Content Management Systems like Drupal and WordPress have been gaining a lot of ground in recent years, with governments around the world adopting them. Drupal in particular has attracted a lot of fanfare as numerous high profile US government bodies -- most notably the House of Representatives and the White House -- have adopted it as their primary content management system. In the UK it's more often WordPress that gets the limelight as it's used for numerous central government sites, both in its traditional guise powering blogs, but also powering more complex sites like Number10.gov.uk and consultation projects like the Public Reading Stage prototype. And yet we've not chosen any off-the-shelf system to power the Single Domain's corporate platform (our Government Machine).
The phrase "not a CMS" has become a bit of a joke around the GovUK office (to the point where more than a few people were humming Once In A Lifetime), but it's a key part of our approach. The Single Domain will include several components that enable publishing on the web but they're part of a much broader ecosystem of tools wired together using APIs and designed to be constantly iterated to focus on user need. As we began to unpack what that means it became clear that we were going to need custom software.
We've got a very strong focus on opening up APIs. While both Drupal and WordPress can be used to offer APIs (recent revisions to Drupal and some of its key modules have helped significantly, as can be seen in the World Bank API project from DevelopmentSeed) adding the range of APIs we're aspiring to would require significant development work, and to make them perform as we'd like we'd need to work around the overheads introduced by WordPress and Drupal. By using development frameworks rather than content management systems we can more quickly optimise the code to serve those APIs.
In order to ensure that the Single Domain can be constantly improved we're going to be building in a large amount of instrumentation that will allow us to analyse how user needs are being served. At the top level some of that can come from traditional web analytics, and at the low end it'll come from server monitoring, but it's important that we have the flexibility to add instrumentation at every layer of the stack. Again, that's certainly possible with both Drupal and WordPress, but to do it effectively we'd be writing a considerable amount of custom code.
Perhaps most compelling for me is our focus on admin systems. There's a huge opportunity in the Single Domain project to identify the right workflow for any given part of the site and to make sure that admin systems are designed to support that. For a publishing tool, that means ensuring editors can quickly navigate the content based on their expectations of the workflow, but also embedding prompts and presenting back data that will draw them to content in need of attention (whether because of agreed schedules, changes in regulations, or quality metrics). Here too we could customise any open source content management system to do the job, but it's highly likely we'd either have to make significant changes to core code or develop a parallel admin system at which point much of the advantage of starting with the base system would be lost.
Having read all that it might be easy to accuse me of "not invented here" syndrome, but I don't think that's accurate. We're using open source up and down our stack, from the server operating system and key components, to development frameworks like Ruby on Rails, and a whole host of components. We're just choosing to start by assembling components rather than customising a package, and we'll be contributing code back to the wider community (whether in the form of new components or patches to existing ones) as we go along. You can see some of that effort by exploring our github profile.
And we're not ruling out using some of the open source content management systems where there's a clear place for them and we can let them play to their strengths. Where we have a need for blogging, or a specialist micro-site that fits a classic content management approach, we know where to find some excellent tools.
James Stewart is Tech Lead at GDS. You can follow @jystewart on Twitter, and read his personal blog.
25 comments
Comment by AlphaBeta.gov … | access-egov.info posted on
[...] one that proved wrong in the end as the engine was subsequently scrapped and replaced by Stellent. GDS look to be going custom againwhich is likely to prove interesting perhaps 2 or 3 years from [...]
Comment by uklotterynews posted on
It's quite humbling to know that my little blog I built in my spare time shares the same platform as Number10. It's amazing that one seemingly simple system can enable me to post the latest lottery results whilst also allowing the Prime Minister and whole government to disseminate information vital to the running of the coutry.
Comment by I’m not dead, I’m a dad « BASIC CRAFT posted on
[...] It’s mainly down to the cast which includes Mike Bracken (Executive Director of Digital), James Stewart (Tech lead at GDS) and Neil Williams (who’s been quiet on his own blog to blog here). What a [...]
Comment by Jon posted on
I'm personally, a little dismayed that the government is looking to do this all "in-house". From a little experience, the tenders/contracts issued are typically responsible for the approach and ultimate solution contracted. This alludes to the main point by benosteen I guess. Many suppliers would fall over them selves to offer solutions as they see appropriate, given their commercial experience. I'm not talking Fujitsu etc here, smaller SME contractors have lost out to this project. I'm all for open source, as the solution in itself provides transferable and flexible sites which can be carried at a lower cost between various sequential contractors. But, this is not the outcome of the build process you are employing. You are, in fact, building proprietary code using 'lower cost' tooling as far as i can see.
Admittedly, I know little about the actual tools you're using ( didn't catch any detail of this above), so it would be nice to have some discussion back why verbose scalable tools from Microsoft are not considered. I'm also assuming this is not considered by the way!
Its all in all quite an exciting project,and i wish you all the best of luck. It looks really cool, and well considered and you've got a great team there.
Comment by Michael Saunby (@msaunby) posted on
You're building a website! If what you want to do can't be done with existing technology then you're probably trying to do the wrong thing.
I know that sometimes new tech is needed, BADC built a complicated stack for the delivery of UKCP09 (climate predictions for the UK), and I'd be surprised if it ever gets reused. It worked, but was (IMHO) overkill.
Oh and back in 2007, in Rome, I had lunch with Eric Gundersen http://developmentseed.org/team/eric-gundersen/ - proper super-hero talent there, i.e. his powers are only ever used for good.
Here's my current thoughts on this - all that can be delivered using Drupal or similar, deliver in that way. If the remaining content has owners that can afford (or care enough) to have some more sophisticated system let them pay for it - both in cost, and the delay it will surely bring.
Comment by Comment on “What is that beautiful house?” by James Stewart on 03/10/2011 (http://digital.cabinetoffice.gov.uk/2011/10/03/beautiful-house/) « Random Hacks posted on
[...] A comment I placed on What is that beautiful house? by James Stewart on 03/10/2011 [...]
Comment by Rudi M posted on
I'll stay agnostic on the technical aspects of the Ruby v Drupal/Wordpress debate, but I have some questions about what happens to all those applications built, in good faith, using open source tools like Drupal which are delivering now. Are you expecting these to be rebuilt in Ruby? If so, who is going to fund this? As it is, we are barely managing to keep the hosting and maintenance of our existing platform, now only one year old, paid for. Any request for development money to rewrite the platform would be met with stony silence... at best.
Comment by Ben Toth (@bentoth) posted on
I have been around long enough to recall the day when the Department of Health website manager explained to me that an update to the DH website could take several days and involved sending floppy disks to the hosting agency. And later being told that it might take several months for DH to publish an RSS feed.
I can remember, as head of nhs.uk, wanting to establish http://www.nhs.uk on the DotP platform, to be told that the priority was to get Departments on board. Only to find that few departments got on board.
I'm thrilled to see the leap in technical sophistication in central government, as reflected in Single Domain. My concern is that the desire to build a platform for departments to use might replicate the DotP experience. Getting adoption of WordPress, whatever its limitations, was a big step forward and I hope the Single Domain project will build on that.
Comment by Where Is He Now? « a work on process posted on
[...] For those who want a little more detail I’ve written a couple of pieces for the GDS blog: one about our approach to APIs and another about our platform choices. [...]
Comment by Steve Donnegan posted on
I'm sure it will deliver on the promise
Comment by Steve Purkiss (@stevepurkiss) posted on
I've read and re-read your article but I'm still at a loss as to why you wouldn't use Drupal as a framework for this. Drupal is far more than just a CMS to use "where we have a need for blogging, or a specialist micro-site that fits a classic content management approach" - it's a rapid web application development platform providing modules and APIs for managing content, commerce and collaboration - all the things you say the site is to do.
You're going to be re-coding a lot of generic functionality which Drupal provides, then having to support and maintain it yourselves in the long run rather than joining a community of over 600,000 around the world who currently maintain and grow Drupal.
You speak about workflow - have you seen the Workbench suite of modules for Drupal which were developed in conjunction with institutions just like yourselves?
Millions of pounds worth of research and many years of tried & tested code and none of it you're going to use because you might have to write a little extra code to expose an API and do some performance improvement work?
There are so many good reasons to use a standard API for creating stuff like this - not least from a knowledge transfer point of view, but in terms of being able to focus your time and budget on the things you need rather than coding stuff which has already been done before (check out the 10,000-odd modules for Drupal). Sure - you might not like PHP, or you might not like Drupal - I'm sure there were a few who didn't like the choice of standard gauge when that came out too but it certainly helped make things work better.
Perhaps I'm missing something - perhaps Turner Broadcasting, ITV, and a host of other large users of Drupal who use it to serve up millions upon millions of requests per month are too but I just don't get it - but then the UK Gov are well known for their failed IT projects, so perhaps I'm just seeing the beginnings of yet another one. Please do feel free to prove me wrong, although I'd prefer if you just used Drupal 😉
Comment by Gareth Rushgrove (@garethr) posted on
As you say Drupal can definitely be used as a solid development framework, it's just that we're using Ruby and Rails in that place for the work at present. And in a similar way to Drupal having a great community, the Ruby community create and release some pretty great open source libraries - rubygems has 29,165 projects available - which we'll take advantage of where they meet our needs.
As an aside, the platform we're building (alongside the specific applications and services) isn't tied to any one language or framework. As James mentioned in the post we may very well use other systems like Drupal alongside what we're doing in quite neat ways.
Comment by westieukmark posted on
I totally agree with Steves points and hope the UK Gov rethink their actions.
Drupal can no longer merely be defined as a CMS and categorised with the likes of WordPress, doing so just demonstrates a lack of understanding of what Drupal is capable of. In my opinion, Drupal is far better described as a content management framework (CMF), in some ways it is more similar to Ruby on Rails with its extensive API's and developer community.
So on that point, it might seem likes its apples and oranges. However I really don't think that is the case. You have WordPress on the left and Ruby on Rails on the right. Drupal is so powerful because it sits right in the middle. While Drupal might not have all the benefits I believe it has the most overlap and total overall benefits between them.
So what prompted me to add this comment is:
"Perhaps most compelling for me is our focus on admin systems. There’s a huge opportunity in the Single Domain project to identify the right workflow for any given part of the site and to make sure that admin systems are designed to support that. For a publishing tool, that means ensuring editors can quickly navigate the content based on their expectations of the workflow, but also embedding prompts and presenting back data that will draw them to content in need of attention (whether because of agreed schedules, changes in regulations, or quality metrics). Here too we could customise any open source content management system to do the job, but it’s highly likely we’d either have to make significant changes to core code or develop a parallel admin system at which point much of the advantage of starting with the base system would be lost."
Please elaborate because I cant understand your points against Drupal. I can only go by personal experience and from reading other like case studies... but after reading that paragraph I felt overwhelming sense that the UK Gov doesn't understand Drupal - if that is your most "compelling" objection?
Yes Drupal comes with an administration interface / workflow both in core and contrib which is fairly stock and in some cases it wont provide what is required, but it is entirely customisable at a low expense to fit any specification. There are a great number of modules already available which are perfectly suited to prevent "parallel admin" but instead extend the initial offering.
I completely fail to see how you would avoid developing custom code / or reduce development expenses / timescales using Rails?
Comment by Jon (@publicsectorpm) posted on
Hmmm I'm still not certain why its necessary to bespoke your 'not a cms'. What's wrong with wordpress again? Which essential requirements for the Single Domain (its just a website, right??) cannot be achieved with WordPress?
How do the benefits of your approach (lets assume cost of a team of 5 coders on about £30K p/a for 12 months) outweigh the benefits of a wordpress install which can cost as little as £50 per month?
#justaskin
Jon
Comment by James Stewart posted on
There are two aspects to that.
The first thing is that it's not really fair to say it's "just a website". From the user perspective it may well appear as a single website, but there's a huge amount behind the scenes to provide the platform (top to bottom), to measure how well the various tools and pieces of content are serving user needs, and so on.
And it's not so much that there's anything wrong with wordpress as that wordpress isn't appropriate for all sites. You wouldn't try to build amazon in wordpress, or foursquare, or github and so on. If you want to publish pages then wordpress will cover a lot of use cases. If you want to provide a variety of tools beyond that content then you'll often end up having to either do a lot of wrangling of the plugin API (and/or core code) to embed them, or build other tools sitting alongside it.
Comment by Ross Gardler posted on
I'm afraid.defending yourself from "not.inventrd here" syndrome by saying "we're using an open source programming language and scaffold system" (my paraphrase) doesn't cut it for me. I can accept that your requirements might rule out all of the out-of-a-box CMS systematic, but have you evaluated the many components of CMS systems available.out there. You don't need to start from scratch (OK with a scaffold system). Dressing it up enthuse way doesn't change the fact you will (probably) be reinventing the wheel.
Comment by Houses and clouds | DavePress posted on
[...] What is that beautiful house? The phrase “not a CMS” has become a bit of a joke around the GovUK office (to the point where more than a few people were humming Once In A Lifetime), but it’s a key part of our approach. The Single Domain will include several components that enable publishing on the web but they’re part of a much broader ecosystem of tools wired together using APIs and designed to be constantly iterated to focus on user need. As we began to unpack what that means it became clear that we were going to need custom software. [...]
Comment by Gerrit Berkouwer posted on
Very interesting post. The comment of benosteen is good and long :-). Still I wanna add some lines.
I do like your approach: going for quality, choosing flexible elements to be able to manage all user needs. Always good if you start fresh, more or less. It's what we try in Dutch central government as well.
But how do you plan to manage all those choices? Did you make a choice for core technology? Or are all options possible? If the answer to the last question is yes, how would you manage things like security? The more different technologies you choose the more checks and balances you have to manage. Thats difficult. Unless your resources are unlimited...
There is also an advantage in choosing a certain 'base' technology stack, and stick to it, don't you think? After all, it's all 'just' technology. Good people can make good solutions, no matter what technical choice. As long as the technology is flexible enough, so you still can mix and match a certain number of different solutions. And choosing a certain steady 'core' saves you a lot of headaches in terms of finding the right staff, maintaining (secure) code and training all different kind of systems to different users.
So, I am very interested where your choices will take you. Let's hope it will not end into singing "My God, what have I done?" with your team :-).
Comment by James Stewart posted on
Gerrit - absolutely, it's vital to pick a solid core. Working on the alpha we used a mixture of python and ruby but for the beta we've been working primarily in ruby. There are a few key libraries that we're using across various apps, and we're not going to add anything to the stack simply for the sake of it -- there needs to be a clear need for each new piece that's added that can't be served by an existing piece.
That said, there is a lot of interesting thinking around about polyglot programmers and applications. Heroku's recent piece on the topic is well worth reading: http://blog.heroku.com/archives/2011/8/3/polyglot_platform/
Comment by benosteen posted on
Gerrit: I agree - there is a real cost to make all those decisions so I am particularly heartened to read James's response that each addition will be driven by necessity. (Just so long as you avoid the all-too-easy mistake of trying to make every problem look like a nail you can hammer in, as it were!)
James: The Heroku piece you link to does put forward a good point about programming languages being akin to "Silos" but I would extend it further than that. Frameworks are silos too, as is any moderately complex library.
To put that in context: I consider myself a decent 'agnostic' programmer but I would struggle to work on a project that used Apache Forrest for example.
(Also, while there is a trend for modern programmers to be polyglots, that does not mean that they are plentiful. In fact, discriminating between those who really are and those who simply believe they are is a tricky problem in of itself!)
Comment by benosteen posted on
After reading your points about the 'cost' of development, I would like to illustrate a number of the issues you will run into and the not-so-obvious gotchas you will have to manage which you have not mentioned:
Development phase:
- Milestone+Functionality management: You have chosen to develop a system that has many different, but interrelated (interdependant?) subsystems. This is manageable with a small and tight team, but gets a lot more difficult as soon as more stakeholders affect the direction of development (something you allude to with your paragraph on custom workflows and views on the content).
Typically, this means that the schedule of development is prone to slippage, and regardless to Agile or other methodology, there is a chance of friction between you and the people you have to report to as expected functionality and other projected milestones do not quite occur as you may have predicted.
- Scope creep: It is very hard to scope out a system that performs truly new functionality. I would very much like to see a CMS with more advanced administrative tooling as you describe but I would be naive if I thought I could guess the manner of user experience to present to even match expectation, let alone keep them from running to the hills. I would recommend extreme caution when suggesting the resources and time (NB not equivalent) required to achieve the system you plot out here.
- Bike-shedding. I cannot think of a single bespoke system I have been party to building and/or designing that has not been waylaid by this. The out-of-the-box experience of a CMS can be used as a very useful tool to define and contain these discussions - "If you want X part to change, then it will take Y time". It supplies a useful psychological constraint against demands which is not a strongly present when a system is being created "out of nothing".
- Ignoring existing workflows:
-- pre-existing internal workflows - if I can be frank, you will discover 'insane' procedures that already exist for the tasks you seek to replace with your system. People will expect your system to follow the pre-existing /cultural/ workflow, even if it makes no sense.
-- pre-existing 'external' workflows - people have become used to the idea that browsing the web is 'idempotent' - clicking around on blue links and using 'back' and 'forward' buttons should not change the pages they are viewing. Clicking 'back' 5 times should take me back to the page I saw 5 clicks ago, or pop up and warn me if not. There is a whole host of expected functionality and guidelines and it is worth pointing out that the best of these are not inventions, but they are catalogues, registries of behaviour. It is unwise to tinker with these expectations as people will not change.
There are more, but I want to move on to some important points
Maintenance phase:
The system is up and running, and the development team and relationship with your overseers is well in hand. Congratulations!
But...
- Hiring operational staff: This gets very difficult, and very quickly to, the more bespoke a system's environment is. In short, it will raise the cost of sourcing, training and keeping staff to carry out the mundane service administration. TL;DR can strike even within job listings - I suggest that you attempt to write a short and concise description of the skills and experience required by candidates and to keep that mentally updated when your system develops. If they need to know how to calm distributed RoR setups on Heroku, Redis or AMQP backend administration and node.js server environments then this will narrow the range of candidates and increase the role salary.
Once you have found your better-than-average staff (aided by your .gov project and branding), you will have to retain them. The more skilled the staff are, the more likely they will become bored by non-developmental tasks.
This is not generally a concern within private enterprises as the means to entice and compensate staff are more flexible than with a public sector payroll.
- In-house training: As the designer, builder and maintainer of your bespoke system, you will also have to arrange for trainers, and training documentation to teach people the unique manner in which your system works. This is not something that is best to leave to this phase, as it will take time to ramp up this side of things (and no, normal documentation efforts will not be enough.)
- Expectation management: The group of people who you are working with to spec out and scope this project are not likely to be representative of the majority of the systems users, unless you have gone out of your way to 'step over' the hierarchy and be in contact with a broad spectrum of users. If you are adopting a phased roll-out plan, you may find quite that expectations and feedback changes in tone dramatically.
Comment by James Stewart posted on
benosteen - you're quite right that managing our development schedule is a constant challenge, but a couple of months in we're making good progress towards our launch.
We're well aware that recruiting and keeping great developers is hard work! GDS is committed to building and keeping a strong development team and we're spending a lot of time working out how to ensure the right environment, good challenges, and appropriate training.
None of the tools we're using are so far out of the ordinary as to be a unique challenge to manage at the system administration level, and we've got a head start on that by focussing very heavily on configuration management tools in the infrastructure work we've done so far. We've got a fairly good sense of what we'll be needing as part of the operations team, but a bit more work to do before we're ready to put together job descriptions. We'll be talking a lot more about this over the coming months.
(fwiw, we're highly unlikely to be deploying on Heroku and as yet the only node.js used is to help us keep an eye on the builds from our Continuous Integration box. We may well be using Redis and a queueing system that implements AMQP but no decisions have been made yet)
Naturally, we're building a product that is of the web so we'll not be confounding user expectations with craziness like breaking the browser back button. But we're also not slaves to existing workflows for our admin systems. We're working hard with the people who'll be producing content (many of whom sit in the room as part of our team) to work out the right workflows rather than simply model the existing approaches in code.
Comment by benosteen posted on
Great to hear that you are "building a product that is of the web" - it goes without saying that adopting idempotent queries for your APIs goes a long way to helping you scale up.
I hope you have things well in hand and aren't surprised by subtle scope creep and schizophrenic stakeholders! The amount of time that can be lost through debating the WYSIWYG look and feel... sigh.
Good luck, and I hope that you get to make something genuinely interesting and useful.
(I should've added more context - I mentioned Heroku and node.js just because they are cool and trendy, not because I believed you were going to use them! :))
Comment by Gareth Rushgrove (@garethr) posted on
Some good points from @benosteen and hopefully James has helped address several of them. As one of the development team members obsessed with the operations side of things I'd like to come back to one thing in particular though, the phrase mundane service administration.
Personally I think system administration is where all the excitement is these days. Maybe that's just me. I've just come back from a systems administration conference in Portland, help organise the London DevOps meet ups and write the a weekly DevOps newsletter so maybe I'm biased.
The idea that rather than treating the operating of a system as simply a cost of doing business on the web, you can treat is as a strategic advantage. Look at what Amazon have done. Or Google. Or Facebook. All of them invested time, money and energy into operating and automating their respective services. That helps then do more with less people (and lower costs) than many other traditional organisations, at the same time as having more secure, stable and scalability sites and applications. We might not be at that scale but we can learn many of the lessons. Hopefully we'll be writing some blog posts on here soon on just that topic.
Comment by benosteen posted on
True, I was certainly more 'down' on the interestingness of a role in DevOps than I ought to have been. Lots of fun things to do with distributed computing, non-traditional backends, monitoring (interesting problem within itself) and data-mining to improve service efficiency and uptime amongst other tasks.