It Was Never About Innovation

This is the first in a series of articles about innovation and open computing. Because it’s a holiday time of year in the USA, I’ve decided that these next few articles will be a detour from the usual stuff you’ll find here.

Ever since a few of us got together to form the Open Cloud Initiative, I’ve looked at cloud computing with awe, but also mistrust. There are many good things that can come of cloud computing initiatives, but there’s also the opportunity (some might say inevitability) of abuse and exploitation.

Over the last few months, I’ve made a point of giving a talk at various conferences with the title of “It Was Never About Innovation.” The point being that Open Source software proved victorious in the data center, not because developers necessarily wanted to release more software under an open source license or because open source development models are necessarily more innovative. No, as I see it, open source led to more innovation and took over the data center because of the basic ground rules that were laid down from the beginning with the intent of creating an ecosystem that espoused the four freedoms as enumerated by Richard Stallman and the Free Software Foundation. It was those ground rules that leveled the playing field, forcing developers to treat end users as equal partners in the game of software development. You will note that it was open source that took over the data center, not freemium free-as-in-beer software. As I’ve grown fond of saying over the last few months, the hippies had it right the whole time. In this model, innovation wasn’t the end goal, it was just a very interesting by-product.

I posit that innovation is much like Douglas Adams’ description of flying in “The Hitchhiker’s Guide to the Galaxy,” which you’ll no doubt recall as the art of throwing yourself at the ground and missing. To attempt to fly would be to miss the point – and fail miserably. No, the trick is to distract yourself before you hit the ground so that flying becomes the end result. Innovation is very much like that. To attempt to be innovative is to perhaps miss the entire point of how the creative process happens. Technology office parks, anyone? Incubators? Are these supposed houses of innovation necessarily more innovative than the alternatives?

My point is that the innovation that’s taking place right now in multiple open source ecosystems is due to the positive feedback loop that was a direct consequence of implementing the four freedoms and mandating that all parties abide by them. It was the implementation of the four freedoms that created a system in which “freeloaders”, those who don’t pay anything for software, could be every bit as important as the developer or even paying customer. If developers didn’t necessarily want to participate in an open ecosystem, which forced participants into abiding by rules that weren’t necessarily in their direct self-interest, why, then, did they willingly participate?

That was the question I set out to answer way back in 2005, when I wrote “There is no Open Source Community.” The impetus for that paper was when I found myself unable to answer the essential question, “Why do developers willingly release open source software? Was it out of some sense of charity? Of providing for the greater good?” One of the most startling discoveries in my young career, back when I worked on SourceForge.net at VA Software, was that most developers who write open source software don’t really care about the concepts of “open source” or “free software”. At the time, we conducted a survey of developers on SourceForge.net and were surprised to discover that they really didn’t give two tosses about the four freedoms. They weren’t free software advocates, and yet they were still free software participants. But why?

What I discovered by working through thought experiments and measuring the results of my model versus reality was that developers didn’t write open source software because they wanted to – they wrote it because they had to. There is an economic incentive for developers to participate in open source communities due to three major trends:

1. The ubiquity of internet access

2. Possibly as a direct result of #1, the ubiquity of developers writing software, and

3. Also as a direct result of the previous bullet, the price of any given feature in software is asymptotic to zero over time.

When the forces of economics put constant downward price pressure on software, developers look for other ways to derive income. Given the choice between simply submitting to economic forces and releasing no-cost software in proprietary form, developers found open source models to be a much better deal. Some of us didn’t necessarily like the mechanics of those models, which included dual licensing and using copyleft as a means of collecting ransom, but it was a model in which developers could thrive. I wrote this back in 2005, all from the developer’s and software vendor’s point of view. For years, I struggled with how to incorporate the end user’s point of view. I simply didn’t find the problem that interesting to contemplate. Of course end users would take to open source software: who doesn’t want to get access to stuff free-of-charge?

Recently I started asking other questions as I contemplated the state of cloud computing and how it relates to open source economics. For example, if end users have the choice between something free-of-charge but proprietary versus open source software, how and why did open source win the day? If we believe that end users played a role in the model, and I think it’s clear that they did, why did they make this choice? Often I’m told that customers don’t care about open source or freedom, because they just want solutions that work. And yet, we have evidence that customers very much do care about those things. If they didn’t, why the overwhelming demand for software that’s open source? Also, the cost of acquiring software is minimal compared to the cost of maintaining software deployments, so why is the cost-free aspect of open source even a factor? One could argue that the speed and agility of acquiring something free-of-charge leads to ubiquity, but why prefer open source? After all, we’ve been told for many years that end users rarely use the direct access to software to make changes under the hood – they usually end up relying on the software developer for that, just as in the proprietary model. We simply have not understood very well why this process works.

What I’ve come to believe is that it’s all about agility and, yes, the four freedoms. In fact, those two things are very much related. Think about it: there is no such thing as a single vendor for everything in the modern data center. That’s impossible at the moment (and, one could argue, ever). Data centers have incorporated automation and orchestration of the different layers of software stacks that must interact seamlessly so that the operators can go home at night and not suffer through incessant pager alerts. These layers need to inter-operate at levels of complexity that make lesser operators fall asleep weeping, unsure if they could rebuild all of their services if called upon. One very telling anecdote regarding this phenomenon came from one such operators who casually commented to me that “if our data center went down and we were forced to do a complete reboot, we wouldn’t know how to do it.”

In this scenario, you have to rely on things that allow you to automate and orchestrate at will, on your timetable, not that of a vendor. Think of all the projects that Netflix, Twitter and Facebook have released that allow them to orchestrate massive amounts of software and data in ways that deliver services without single points of failure. There is no universe in which this could be done with proprietary software. This is where the four freedoms come in. By creating an ecosystem that mandated the four freedoms, end users, consequentially, were able to participate in this ecosystem for the first time as equal partners to the developers. This dramatically changed the dynamics of customer-vendor relationships. In the world of open source software, end users have the freedom to be as agile as they will ever need to be. They can report problems, patch software, release new versions if the other developers don’t step up, and work in a system that allows them to deliver solutions with much faster times to market. The other developers, perhaps pure software vendors or also developer-operators, also benefit by virtue of the fact that willing participants in the form of end users create better software, and thus begins the virtuous cycle.

To review:

Developers write version 0.1 of some set of tools → end users evaluate software and decide how valuable it is → if it’s usable, end users either add patches, file bug reports, send feedback or all of the above → developers incorporate feedback, accept or reject patch submissions, rewrite portions of the code → end users evaluate and either use or discard.

Lather, rinse, repeat. One couldn’t have set out to design a system that was as agile and innovative, which is precisely my point! This system works because the open source ecosystem is a level playing field, and there is an economic incentive for all parties. Developers don’t rule over end users, and end users, by definition, can’t rule the process because they don’t produce the code, but it was their adoption of open source solutions that forced the hands of developers. In this model, developers showed up to fill customer demand. It was this level playing field, which was a direct result of implementing the four freedoms, that led to all the innovation that happens today. And as we have all-too-often been reminded, humans don’t generally set out to create level playing fields, which is why groups of people could never have intended to create this feedback loop. All the various parties would have at various points become too greedy, protecting their own interests, thus the reason it had to begin with the four freedoms.

And look where we are now – in a world where much of the innovation is just as likely (if not more) to come from “users” as it is from venerable software vendors. This is the world we live in today, where a company like Yahoo developed some data analysis software internally and decided that it might be useful for the rest of the world to work with, thus beginning the Hadoop juggernaut. Rackspace, long known for its hosting and support, not software development, collaborates with NASA and unleashes what became the OpenStack ecosystem. In both cases, the role of traditional software vendors was reduced to, at first, fast followers, not lead innovators or developers. None of this would have been possible without stating, from the beginning, that every software program must honor these freedoms:

• The freedom to run the program, for any purpose (freedom 0).
• The freedom to study how the program works, and change it so it does your computing as you wish (freedom 1).
• The freedom to redistribute copies so you can help your neighbor (freedom 2).
• The freedom to distribute copies of your modified versions to others (freedom 3).

This is all fine and dandy, but what does this mean in a cloud computing context? You’ll have to wait for the next installment, coming soon!