While he was in Austin, I asked Damon Edwards to give us an overview of how DTO Solutions has been doing – including Run Deck – and the continuing evolution of DevOps.

If you want to see the luggage we talk about in the opening, here’s a quick picture.

Overview

Demo

Disclosure: SmartBear sponsored this video.

Back from Velocity, John and I catch up on things going on in the cloud and dev/ops space. We try to iron out what exactly the change and point of dev/ops is and how businesses could use it. The episode is capped of with a discussion about PaaSes, two possible types of them.

Download the episode directly right here, subscribe to the feed in iTunes or other podcatcher to have episodes downloaded automatically, or just click play below to listen to it right here:

Show Notes

• What is dev/ops? workshops – at Atlanta, Portland – who comes to these things? Mostly web startup guys, but some enterprise people.
• What’s with the Microsoft interest? They sponsored the Atlanta workshop.
• On Microsoft and cloud dev - Concero, the VisualStudio pickling tool. Opalis is another interesting asset.
• The rising need for orchestration in dev/ops? Which brings in DTO Solutions’ Run Deck - John gives an overview.
• How about OSLC? Anyone know about that?
• VisibleOps stuff, researching IT management in practice.
• What are the metrics for success and failure in dev/ops? Lots of MTTR, expecting failure.
• In a failure-driven world: never mind that premature optimization is the devil’s root canal business.
• If you have to move from Amazon to Rackspace, how long would it take?
• A painful problem statement: you can’t be agile with current IT Management. Also: it’s not how much it costs you, it’s how much you make.
• DRW pairing developers and traders at a desk. Making it a little bluer. “They can get things in front of customers really fast.”
• Adaptive Business Plans, Evolutionary Transactions – The only reason a business wants to be your friend is to get your money.
• Find the moribund businesses, revitalize them with more agile IT.
• How was Velocity?
• Some PaaS talk – “bring your own PaaS,” etc. And John likes “the private PaaS.” Or is it “build your own PaaS”? Like, using a Chef cookbook to use RabbitMQ. And then “purpose driven cloud” from John.
• John at Build a Cloud day in Chicago this weekend.

Disclosure: see the RedMonk client list for clients mentioned.

With the rise in popularity of Jenkins/Hudson, I’ve been wondering what happened with Cruise Control, the break-through project that helped bring continuous integration to programming. Charles Lowell of The Frontside tells us his theory.

In addition to viewing the video above, you can download it directly or subscribe to the make all podcast feed to get it automatically downloaded.

Transcript

As usual with these un-sponsored episodes, I haven’t spent time to clean up the transcript. If you see us saying something crazy, check the original audio first.

Michael Coté: So Charles, Jenkins very popularly used to be called Hudson.

Charles Lowell: Great!

Michael Coté: And yet there was CruiseControl. Why — I don’t want to say failed, but why — how did Jenkins, how was the space created that Jenkins took over from CruiseControl?

Charles Lowell: Well, let me start by answering the question with two words or talk two words but with one sentence, and then go on to expound for, I think, but you got your answer already. I think this thing just works when you like install it. You can download Jenkins and you’re up and running in about thirty seconds, whereas CruiseControl never was that. It was always a pain in the ass to get up and configure and blah, blah, blah and, I mean, I lost track of it.

But certainly while I was at ThoughtWorks, it started out as an ENDscript and a con job, right, and kind of snowballed from there and it was never kind of brought around from project to project and there was kind of good contributions from each place but it wasn’t ever — at least in my experience, a coherent project so much as an idea. And there were bunch of implementations on that idea.

And the thing is this, because it was for the people, it owned it, and in this case I’m thinking ThoughtWorks and ThoughtWorkers, it pretty much worked on –

Michael Coté: Quite a lot of extra –

Charles Lowell: I mean pretty much work but they were familiar with that. So setting it up on — at the beginning of each project wasn’t a lot of overhead in a grand scheme of things. It’s a couple three days or something, but you’ve done it a bunch and — so there’s no need to invest and package it so that it’s — so that it’s –

Michael Coté: So it wasn’t sort of like a product.

Charles Lowell: — that’s; yeah, it was never a product. I mean there was trend — it was, like you said, they were rumblings now making into a product, but I think that it was definitely more of an idea, and a very successful idea that was thought.

Michael Coté: Oh, yeah, yeah. That was — if I remember, it was, if I may use the word, it was kind of revolutionary in that sense, it was like oh, continuous integration.

Charles Lowell: Right.

Michael Coté: Huh.

Charles Lowell: Right. And so I guess that’s –

Michael Coté: And it just never evolved beyond that.

Charles Lowell: It never evolved into –

Michael Coté: — beyond a collection of scripts, as we would say, in the IT management world. There’s this distinction between, you’d get a bunch of products to monitor things, and it’s just a bunch of scripts. And then at some point, those scripts turn into a product.

Charles Lowell: Right.

Michael Coté: And I guess CruiseControl was sort of a downloadable thing that you could get up and running, but what you’re saying is it just didn’t get polished as well as later on, and then Hudson, then later we name Jenkins came in, and sort of polished this up.

Charles Lowell: And up and without having anything to build or anything knowing how to build it, you just –

Michael Coté: And there’s lots of other continuous integration tools out there, but that’s the one that you prefer.

Charles Lowell: It’s very — it’s pretty much to me just; can I use another buzzword “Zero Configuration”?

Michael Coté: Oh, yeah. Yeah.

Charles Lowell: You don’t have to configure a database.

Michael Coté: Did you know that?

Charles Lowell: — no dependencies; in the simplest case, there’s no dependencies, no configurations.

Michael Coté: Yeah. I think — did you know that Apple’s Bonjour thing was supposed to be called Zeroconf I think. But I think someone had a trademark on it so they couldn’t call it that.

Charles Lowell: Oh, is that what it is?

Michael Coté: — or something like that.

Charles Lowell: I thought Bonjour was Apple’s implementation of Zeroconf.

Michael Coté: Who knows.

Charles Lowell: Who knows.

Michael Coté: Bonjour!

Charles Lowell: Good evening!

Michael Coté: Well, thanks for the history lesson, Charles.

Charles Lowell: Yeah.

How can new technologies and practices like cloud, social, dev/ops, activity streams, model-driven automation, JavaScript injection, and rogue developers make IT better in staid organizations? That’s the question I tackled in my Knowledge11 talk, above. I wanted to go over some general and some pointers for how these new technologies can make the role of people more important, valuable, and efficient.

The them of people and IT is one of my favorite. For me, the most important thing new technology can do is make the role of people more important. Usually, people are the ones who figure out how to best make a company money and then make sure the execution takes place. Also, you know, I want to keep The Robots at bay (I for one, however, would welcome our new Robot Overlords, should they arise!).

Disclosure: ServiceNow is a client.

I’m often asked for tips on “scaling up Agile.” This usually means, “we got Agile working at our team level, but as we’ve tried to expand it to the rest of our GIANT COMPANY, we’ve encountered all sorts of problems. People don’t see the light! How can we make them BELIEVERS?!?!”…or something a little more level-headed than that.

At some point, adopting Agile development becomes an organizational change management exercise that has little do with Agile itself. Last week at the IBM Impact Unconference, I was part of the opening “unpanel” where someone had asked for help in adopting Agile in this context…or, at least, that’s the question I chose to answer. Here’s what I said:

1. You need an executive sponsor, get one – doing Agile at the team level is fine, and you can do a bottoms-up approach. You might even be able to expand out to QA teams (who are traditionally separate from development teams) and others, but to really spread to the rest of the organization, a high-level executive must bless and then force the effort on the organization. Here, the hierarchy of the organization works in your favor: if the big boss says to do it, people tend to do it. Executive sponsorship always gives you permission to screw-up as you learn. It reduces your personal risk (and that of your team). And, let’s face it, the first few iterations of an Agile team are rough: so much of Agile, initially, is about exposing the weaknesses and poor development practices organizations are doing (mainly: promising too much in each release, also, not fully working as a team around one product) – so there’s going to be some failure at first. The executive is your blame-game safety net here: a good Agile executive knows that the first few cycles of doing Agile is all about making people realize they’ve been doing it wrong by exposing the poor results of their practices…and then learning from that, not punishing them.
2. Learn how to report to Management, and do it – as “lower-level” technical people, you probably could give a rats ass about all that “reporting” management does up the chain. The fact of the matter is, all those damned spreadsheets and project charts are a huge part of their job. Management has to somehow measure and report on the progress beneath them so that they and their management can make decisions about what the keep doing, what to stop doing, new things to try, and plans to start making for good or bad futures. “Reports” are practically the only input management has to their job. In your organization, there’s probably a set of metrics (or KPIs) that your boss, your bosses boss, etc. are looking for. These metrics are used to gauge “how things are going,” but are also used by even higher management to judge how that boss is doing (and, thus, you). As someone trying to push Agile, you should learn what these metrics are and learn how to wangle in all the project management data you have into them. Treat that reporting as “story” or part of your iteration – track how much time you spend on it, the benefit it gives you, make it part of your iteration retrospectives and so on. To get all meta: reporting on the project is a key requirement of the project. (Thus, if it’s taking too long, you can track that and report to management and ask them how they’d like the weigh that time against other features.) Recently, the topic of “technical debt” has been an excellent, new metric to start tracking. Also, here, Israel Gat‘s The Concise Executive Guide to Agile provides loads of good input on “reporting.”
3. Isolate the resistors, and hope they drop-off – no matter how well things are going, there will be people who simply don’t want to go along with The New Thing. You’ve tried to reason with them and get them to see how great Agile is, but they still just don’t want to do it. When I was part of the Agile-at-Scale efforts at BMC (under Israel Gat), I was shocked that the most resistant people were fellow developers. They were so used to the way things had been and the fiefdoms they’d carved out that the fluid nature of Scrum caused them all sorts of problems. In general, you can get a sense if someone is a “team player,” or if they’re resistant. While the Machiavellian side of you would like to see these people fired, that’s often not as easy as you’d think in large organizations. In these cases, you should isolate these people (and their work) from the rest of the project. Essentially, you want to make sure The Stagnent Resistors don’t infect the your timeline, code quality, and moral. As I said at the Impact Unconference, it’s like what you do with a weird growth you find on a dog: tie a silk string around it, cutting off the circulation, and hope it just falls off eventually.

It’s getting a bit long in the tooth now, but my “smells of Agile” paper from several years back can be helpful for gauging how Agile your organization is: a sniff test, if you will.

Disclosure: IBM, who put on the Impact Unconference mentioned above, is a client.

As you may recall, I did short talk on this topic last month. The slides are above and you can see the replay as well, including two other talks from vendors on the topic. In my part of this ADT Supercast on requirements management, I quickly outline some of the new ways and tools to help with requirements management and also get into some new and better ways to do software development and delivery in general.

Disclosure: some companies mentioned in the presentation are RedMonk clients.

Developers are Insourcing

Developers have been in-sourcing tasks they’d previously jettisoned from their core functionality for a few years now and cloud computing has brought one more land-grab: operations. Agile tricked developers into caring about QA and testing, but also requirements and product management. Now, devops is “tricking” developers to care about operations. I put tricking in quotes because the developers actually want this – the good ones at least. The benefits, or at least goals are clear: delivering software that users like with a more frequent cadence. Those are the two aspects of software development that most interest me now: frequent functionality and using rapid feedback-loops to improve the user experience and overall usefulness of the software.

A Brief, Hand-wavy History

At the very beginning, development teams did everything, or at the very least were intimately involved: gathered requirements, writing the code, testing the code, running the code. IT being expensive, organizations sought to divide up those tasks into shared resources, often under different management chains. Developers, of course, also just wanted to write code: not “talk with customers about what they needed” (requirements and product management) or keep their applications humming along nicely in production (operations).

Throw in mainframe MIPs accounting and high costs, and you can see how separating out those rascally developers from expensive mainframe toys makes business sense, at least in a spreadsheet. (Side-note: with the metered pricing of cloud computing, thus far there’s no reason to think that in 10 or so years, cloud computing resources will be any less iron-gripped controlled than mainframe resources – that is, use and consumption of them will be slowed down in favor of controlling costs. We’ll see.)

And writing and running tests? What developer wants to do that?

Agile Starts to In-source: QA

One of the emergent principals of Agile (perhaps an anti-pattern) is that once the core team who wrote the software gives control of that software to another group, overall quality tends to go down. This isn’t across the board, but you tend to see that. As an individual, if you don’t somehow “own” the software, you won’t give it the same love and care that an “owner” will. This applies to QA and operations, usually not so much to product management.

Agile is always trying to get developers to do more process. To own more of the software life-cycle. It never says this outright, but cynical developers will spot it right away: “wait, my job is to write code, not write-up use cases and rank them. Let alone help take care of my software in production, in the hands of dirty users. I mean: are you going to pay me more?”

Somehow, Agile got developers to care about QA. It started small with unit testing by promising, after a huge amount of initial work, to make them more productive. Then functional tests came in, and now if you’ve got yourself wired up correctly, you can test whole use cases (or “stories” to use the term Agile hid the dread “use case” behind).

The bigger win was getting QA to be part of the development team. In reality, good QA people are often the foremost experts on the product – they’re the ones that spend hours, weeks poking and prodding it. Getting their input, starting at the beginning, is a great way to improve software. As with developers who are writing and running tests, the QA person ceases to be “just QA”: they’re one of the many owners of the software.

“I take the requirements from the customer to the developers.”

Seeking feedback and rich interaction with users hasn’t always been a strong-point of developers. They haven’t wanted to do product management (in this context: figuring out what users/customers want and then prioritizing which ones get into which release). “That’s someone else’s job” is what you’d hear. As with QA, that role has seeped a bit into developers hands as well. There’s simply not enough time in the day to write code, test, and also do product management, so it’s probably a good idea to have a whole person filling that role: but, having that person be the sole conduit between developers and users is not entirely helpful.

Thanks to the ability to run software as a service (SaaS) and mature cloud offering, a lot of the feedback teams need to do product management can now be automated. If you’re running your application as a SaaS, you can see what every single user is doing all the time. It’s like you have infinite one-way mirror usability tests going on. Public web apps (Amazon, Facebook, etc.) have known this for a long time, getting into advanced practices like A/B feature testing: let’s release two different ways of implementing this feature to sub-sets of user group and see which one results in more book sales, and then we’ll switch everyone over the better one. What you’re driving at is feeding aggregated user behavior data into the product management process.

There’s a quote I use all the time from Alterity’s Brian Sweat that I use all the time to summarize this point:

I can actually look at [a feature] and say, ‘nobody uses feature X.’ It’s not even being looked at. And it really helps us shape the future of the app which, on a desktop product, we don’t have a lot of data like that.

There’s two concepts running around here that I speak a lot about: frequent functionality and rapid feedback loops. Getting features in small chunks in production sooner and then collecting a huge amount of usage feedback from users. You’re first delighting your users by keeping your software up-to-date and more functional (a lesson learned from the consumer space where new frequent functionality and integrations with other services is key for long-term success) and also giving yourself the chance to see what works best (or worst!) with users. You can empirically improve the quality of your software. And here, by “quality” I don’t mean “bug free” (which is a less helpful definition in this context) but something more like “the ability to help users do their job better, faster, and more profitably.”

I go over these concepts a lot in talks now-a-days, most recently on requirements management and back at devoxx 2010.

Operations

Ask any developer if they’d like to take a pager and be woken up at 3am to reboot a server. You can guess what the answer is. A blank stare that says “no” in a string a four-letter words. These guys are coders, not pager monkeys.

And yet, developers are increasingly taking on that task, perhaps not seeing those bat-belt ops guys clip a pager onto the developer’s t-shirt. The speed at which cloud computing technologies and practices allows developers to get their core job done (spinning up virtual labs, getting access to resources without having to wait 6 weeks for the DBA to create a new column in a table, etc.) has sold developers on cloud computing. Throw in SaaSified parts of the development tool-chain like GitHub, and you start to see why developers like cloud-based technology so much: it speeds up their work, gives them more power (they don’t have to ask IT permission for a server, and then wait for it), and overall improves their ability to produce good software.

As developers using cloud computing technologies get closer and closer to production, you can see them starting to in-source even operations. At the bleeding edge, those using Platform-as-a-Services (PaaS) are almost forced into doing this. Indeed, as its name implies, much of what devops is wants to do it bring the operations function back to the core team.

This doesn’t suggest that you get rid of the operations people at all – the good ones at least. Rather, it means that as with QA and product management, their role moves from “keeping the lights green” to “delivering good, productive experiences.” Operations becomes one of the product owners, not just the “monkeys” who hook up wires to servers and increase disk-space.

As some point, IT became a cost-center, a provider of “services” to the business. That’s terrible for them. No one wants to be the manager of a “shared resource.” Just go ask your janitorial staff or the guy who keeps your office supply closet stocked how secure and well paid they are. Before the current crop of IT Management technologies finally ripened (Agile development, virtualization, open source, and now cloud), perhaps, taking this “services” approach was cost effective. But, now it means that IT has one way to show value to the rest of the company: budget cuts. For IT, the promise is to become top-line revenue: part of the way a company makes money, not hidden somewhere in the expenses.

(Hey, I’m a big IT Management guy, here, so realize that I’m being tongue-in-check with this whole “monkey” thing. If you want equal servings the other way: “Developers”? Those are the guys who write bugs, right?)

Automation

All of this in-sourcing relies on automating parts of what’s being brought back to the core development team: automating testing with the write testing frameworks and continuous integration tools, automating understanding how users are interacting with your software with cloud-driven feedback, and automating IT management in production.

Evaluating Offerings & Programs

To me, this understanding that automation is key is critical because it means what I’ve been seeing here and there is driven by actual, new technology: not just vapor-ware and Unicorn-meat. It means something else great: if someone comes peddling devops or some other whacky cloud-based way to improve your software delivery process, you can ask them to show you the tools – “where’s the beef?” and all that.

Avoiding Outsourcing

[I]f you worked for AT&T in my day, it was a great bureaucracy.  Who in the hell was really thinking about the shareholder or anything else?  And in a bureaucracy, you think the work is done when it goes out of your in-basket into somebody else’s in-basket.  But, of course, it isn’t.  It’s not done until AT&T delivers what it’s supposed to deliver.  So you get big, fat, dumb, unmotivated bureaucracies.

–Charles Munger

There’s a tremendous amount of “cultural change” (read: getting employees to do things differently and like it) needed, but the hope is that this trend of developers in-sourcing tasks means that cultural change will be possible. It doesn’t happen often, but many parts of the IT department (developers, QA, and a bit of ops) are actually looking for new ways of doing things. The best part is that the promise – that some folks have been realizing – is that IT can become part of the business, not just a cut-to-the-bone cost center that keeps email up and running and AntiVirus software updated.

Or, to put it all more simply: Conway’s Law.

Disclosure: GitHub is a client.

Next week, I’ll be giving a short talk online, as part of a “super-cast”, on what I see interesting and helpful going on in the Application Lifecycle Management space. Loosely, when I don’t want to kill myself with enterprise-y phrases, I think of this as all the junk an organization needs to help manage the process of getting software out the door (or over the pipes, as it were), hopefully good software that people like using.

Here’s the abstract of my talk:

It’s been a long time since technology and process innovations have allowed development teams to change and improve the way they deliver software to their users. Agile software development was the last major change: the further evolution of those and other practices, along with recent innovations in cloud computing and other areas, have given teams even better ways to manage the requirements and features delivered to users. This talk focuses on using those innovations to not just “track and capture requirements,” but to create and deliver better software for your users.

I’ll be talking about agile software development, feedback and Frequent Functionality (a good way to make your requirements gardening better), dev/ops, social programming (e.g., GitHub), SaaS toolchains and cloud development suites, tricking developers into micro-managing themselves and doing “dashboards,” and the usual governance and controls that always seem to tag along with ALM.

Registration is free, so if you’re interested, go over there and sign up. After my talk, there’s some interesting looking talms from MKS and Jama Software.

It was a great show: I met and talked with all sorts of interesting folks. One of the better developer conferences I’ve been to in a long time.

Here are the presentations for the two talks:

Pragmatic Cloud Computing, or, Dealing with Morlocks, or, Agile Infrastructure

What’s the point of dev/ops?

As I recall the talks were recorded, so they should be up sometime. (Also, my wife, son, and I took a little vacation around the dates of the conference, there’s some photos up if you’re interested.)