Its hardly the most original insight that cloud apps provide a lot of data about how users use the app – this data can then be used to improve the application. Tim O’Reilly tagged this thinking in Web 2.0 terms years ago. We have seen SaaS providers take advantage accordingly. Useage data? That’s some valuable metadata right there.

But I was still struck by something VMware CEO Paul Maritz said the other day at a London briefing event. You see- VMWare had a problem. It didn’t actually know how or what people were using its free ESX download for.

We had no idea what for, so we created a free hosted ESX SaaS service to try and gain better insight into what they are doing. Now 400 people a day look at it, and we’re getting our first upsell opportunities.

This really is a new way of doing business. It underpins our new RedMonk Analytics platform. You need to get your software hosted to better understand how people want to use it. That’s a key cloud benefit. In a SaaS environment when a customer calls something a “requirement” you’re in a much better position to push back – oh, you mean extending that function that nobody actually uses?

Cloud offers all kinds of new opportunities for tech firms. But also of course enterprise IT. As Maritz puts it:

“Managers are going to get a new kind of light they can shine into the hedgerows of IT”.

If you can’t measure it you can’t manage it (Maritz actually worked under Andy Grove, one of the claimed originators of that maxim). Well Cloud allows for measurement in a way that on prem just doesn’t.

Last week I went to EMC’s EMEA Analyst Summit at my favourite London conference venue- Kings Place in Kings Cross. As a software guy I tend to focus on SpringSource and VMware, rather than the storage mothership, but I have been around long enough to know EMC pretty well. I like to take a historical view on the IT business, because context is everything. Something that struck me really clearly is that in many respects nothing has changed in terms of narratives, strategy and remorseless execution. EMC is even still arguing that tape is dead.

It seems like every computing revolution drives storage volumes, which drives EMC’s bottom line. Back in the 1990s EMC talked about the tail wagging the dog – with storage becoming a bigger market than servers. But everything drives storage. Virtualisation drives storage (which helps explain both the rationalisation, and the huge success, of EMC’s VMware acquisition. The cloud drives storage. Big Data drives storage (obviously). Data Center consolidation drives storage. The Web drives storage.

And that bottom line. What a bottom line. EMC just turned in its fifth record quarter in a row. Arguably EMC sailed pretty close to the wind from a market disclosure perspective, given the theme of the event was Record Breaking… just a week before EMC reported yet another storming set of numbers. In Q4 2010, EMC’s consolidated revenue was $4.9 billion, an increase of 19 percent from the year-ago quarter. Compare and contrast with Cisco – another firm you’d expect would benefit from many of the same trends driving EMC growth- I mean networks and storage grow in lockstep, right? Apparently not – Cisco has had a tougher downturn – missing forecasts, and hitting “air pockets”.

At EMC though there is apparently no such turbulence to report, although of course questions remain about future growth- with EMC giving the standard answer for Elder Companies- future growth will come from the mid-market.

If one single thing screamed out at me during the two day summit it was the utter seriousness with which EMC views Amazon as a competition. This should be no surprise- EMC was mentioning Amazon in 10-k filings while other major enterprise vendors had never even heard of AWS. EMC executives repeatedly dinged what they called “the bookstore”. For example:

“Amazon is clearly being subsidised by the bookstore at some level.” – EMC CEO Joe Tucci.

Funnily enough Tucci then went on to explain how EMC’s Mozy backup business gave EMC a better volume story. Well if you do plan to compete with one of the best retailers on the planet you better have volume in place.

“We buy more disc than anyone else on the planet. Our cost structure is based on cost of storage”

Big Data

EMC is looking at workloads such as gene sequencing, geophysical exploration, media editing. It recently acquired GreenPlum, a massively parallel processing engine, built on commodity hardware – to accelerate its go to markets in Big Data. In terms of engagement models and vertical industry skills and experience EMC may lag IBM, but certainly not in terms of ambition. I would be very surprised if EMC didn’t quickly make a number of software and services acquisitions to bulk up there, adding to EMC Consulting. Because at least as far as the enterprise is concerned Big Data is all about domain knowledge. Thus for example, in rail transport Big Data can be used to lower operational maintenance costs – by tracking the roundness of wheels. Seriously- over time wheels go out of round, and become a little oblong, which lowers fuel efficiency, and increases stress on the track. IBM SVP Steve Mills totally owns this story when he tells it to customers and influencers. You see for IBM Big Data is a facet of a broader Smarter Planet play, while at EMC its a market in its own right.

A little surprising was that in talking to Elie Simon, GreenPlum’s EMEA general manager, it became clear he is sceptical about in-memory processing of Big Data. This seems a little short-sighted. Flash SSD, Nearline Disc, In memory, and yes even tape- are all going to be part of the Big Data processing landscape going forward. Check out SAP’s High Performance Analytic Appliance (HANA), for example. But Simon doesn’t lack aggression:

“I dont believe that analytics workloads belong on the enterprise data warehouse”

Where is the Developer?

EMC is all about selling to ops. But to succeed big in Big Data EMC will need to address not only more Line of Business folks, but also software developers. Luckily it has SpringSource in the arsenal for that. It also doesn’t have any direct relational database revenues to protect, so it can get busy with the noSQL wave. I wrote up VMware’s alternative database play here – VMware’s SpringSource Redis and Rabbit acquisitions: A Database Play is Emerging. If you’re more interested in a customer view check out this write up about how the Guardian is offloading workloads from Oracle to more modern data architectures – The Guardian: NoSQL EU. Don’t Melt The Database.

Because make no mistake – the cloud is all about developers (it certainly has been driven by them do far). With that mind its interesting that EMC CEO Joe Tucci said:

“There are two kinds of service providers – those that wont buy from us- google, microsoft, amazon, and then classic, evolving – telcos, hosters, outsourcers, pure plays that will, making EMC the platform of choice for service providers who share our vision.”

EMC’s goal is “to be the undisputed leader in enabling cloud computing in service providers and enterprises”

Records, Records, Records

Record quarters is impressive, but I was perhaps less impressed by the fact EMC claimed last week was the largest product launch in the storage business’s history: 41 new products. Why? Because it felt responsive – IBM Storage’s big claim of 2010 was that it had undertaken a “complete refresh of the portfolio”. Seems EMC didn’t want to be seen as less revolutionary. To be fair some of the new products look solid, and more on that next week. In summary EMC is going great guns, and is well positioned for future group. It needs to get closer to developers- but it has some of the necessary pieces in place to do just that. See SpringSource Code2Cloud, which is being adopted by the same customers choosing VCE.

But Big Data- that is a record any storage company wants to listen to. In case you’re wondering the photo is of a big ball of tape.

IBM this week clarified its plans to handhold enterprises into the cloud, working with Red Hat to bypass VMware with the announcement of Smart Business Development & Test on the IBM Cloud.

I have been talking for a while about what I call The VMware Pattern, in posts such as Amazon Web Services: an instance of weakness as strength.

Amazon is the new VMware. The adoption patterns are going to similar. Enterprise will see AWS as a test and development environment first, but over time production workloads will migrate there.

It makes a great deal of sense to encourage its customers to adopt the pattern. That is – start with test, and go from there. Don’t tell the customer to immediately migrate everything to, and run everything on, the cloud. Which would of course be insane. On the contrary recommend a low barrier to entry approach. Production is an end state where the customer finally just says: “remind me again why we aren’t using this flexible infrastructure as a production environment?” That’s the VMware Pattern. Which I may have to rename the AWS pattern…

In the meantime though, according to IBM Research, the average IT department devotes up to 50% of its technology infrastructure to development and test, with up to 90% of that infrastructure remaining idle most of the time.

Sounds like a job for virtualisation… or the cloud.

As Dave Rosenberg points out:

“Testing services are an excellent use-case for cloud services, and a number of start-ups including Sauce Labs and SOASTA have offerings that allow customers to test their applications without having to build a massive test infrastructure.”

HP Mercury has to find the VMware Pattern pretty galling. “What do you mean you don’t plan to buy any test servers”.

VMware may have owned the (x86) virtualisation wave, but with cloud everything is in play again.

For a cloud integration company like Appirio this stuff seems old hat at best. Thus for Balakrishna Narasimh aka appirio_nara:

at #cloudconnect hearing IBM. Feels like 3 years ago – all talk focused on dev and test environments in the cloud

But for the mainstream, for IBM’s customers, its very early days indeed. Cloud is far from mainstream. Public clouds are scary, and full of FUD. Development and test though is a toe in the water, with IBM holding the customers hand, and of course recommending a range of related products and services – step forward Rational Software Delivery Services for Cloud Computing v1.0.

Just to show IBM is keeping up with the cool kids, one early customer is Paypal, which according to IBM’s press release is using the offering as the basis for a collaborative environment for its own developers. IBM hosting a developer cloud for PayPal – that’s not bad for a “cloud laggard”. IBM partners for the launch include RightScale (an acknowledged cloud leader, and the aforementioned SOASTA).

For those skeptical of Red Hat’s role in the service, its certainly worth pointing out that Amazon Web Services runs on Red Hat – it makes sense to adopt the same infrastructure as the de facto market leader.

I am quietly impressed.

disclosure: IBM is a client. Amazon, HP, VMware and Red Hat are not.

RedMonk has always positioned itself on the side of openness and positive community interaction, which is why after the recent muck slinging and general unpleasantness in the Cloud standards space we’re pleased to see some sanity returning with a new initiative from the Free Software Foundation – the Free™ and Open™ Cloud Alliance™ (FOCA), an industry-wide trade marketing association supporting Free™ and Open™ Cloud Computing™ (FOCC). Perhaps most importantly, unlike the Open Cloud Manifesto announced yesterday, FOCC actually has an artefact on the table that we can all work with – namely the CloudLeft Public License (CPL); this won’t just be another talking shop.

Any initiative kicked off by Larry Ellison and Richard Stallman has to stand a good chance of success, which is why RedMonk is very proud to announce we’re founding members of of FOCA. We have been lobbying behind the scenes at Microsoft on behalf of the FSF with some success, and hope to see an announcement about Azure and the CPL in the very near future. It’s great to see Microsoft doing the right thing, especially so early in its product plans.

All in all I think we now have a basis to proceed with interoperable clouds.

Link to the press release here.

disclosure: Microsoft is a client.

“Perhaps ironically, the less Amazon do themselves, the more popular AWS will be. Instance-based clouds are portable… unlike the fabric offerings of Amazon competitors. What other firms see as a weakness (“not enough IP in AWS”) is actually a strength…”

That is what I said in an email to Alexis Richardson and the ESME team the other day, talking about Amazon Web Services and some comments CTO Werner Vogels made at SXSW about the AMQP messaging protocol.

It was my business partner Stephen O’Grady who made the simple Instance vs Fabric distinction, and I find it pretty useful in contextualising the cloud. Of course any instance cloud has some fabric qualities, and any fabric also has some instance to it.

When you look at the market though its pretty clear the fabric players are suffering from feature-itis.  No surprises there given cloud definitions are so flaky, but in the meantime Amazon is just getting on with it.

Compare and contrast with other players.

Take the Google cloud fabric, otherwise known as App Engine. Google seems to be puttering around, in some cases carrying out science experiments like porting Jaiku to AppEngine, before working out it doesn’t know what it actually wants to do with it (“Google will no longer actively develop the codebase”).

Or lets consider Microsoft Azure – the fabric fell down for 22 hours last weekend, but seemingly noone noticed. Azure is in limited beta, and so on… but my question for Microsoft is: why not just offer some Windows machine instances for developers to deploy to, a kind of simple Azure onramp? Note to self: ask Amazon how Windows in EC2 is going. Its especially surprising that Microsoft is delivering something that requires developers to learn a bunch of new methods because normally the firm is all about backwards compatibility. The beauty of instance simplicity is… no new skills.

The problem with fabric complexity is the promises being made: “your apps will scale linearly”.

What about the enterprisey types? Salesforce.com is clearly the most successful fabric play at this point, by some margin. Third parties like Coda have built entire financial applications, namely Coda2go, from the ground up to run on Force.com.

I have had some interesting chats with some senior technical leaders at companies like IBM and SAP in the last few months who have been dismissive of the Amazon Web Services cloud offerings. My favourite comment – “there isn’t much IP in there”.

Not much IP perhaps. But check out the success. I have seen all this before. A few times. There was a time IBM thought the Oracle database couldn’t threaten its own Mainframe DB2. IBM later ceded x86 virtualisation to VMWare partly because it thought there “isn’t much IP in there”. IBM invented virtualisation- no upstart was going to be able to do what IBM could. Maybe not- but perhaps virtualisation has some different roles to play.

Amazon is the new VMWare. The adoption patterns are going to similar. Enterprise will see AWS as a test and development environment first, but over time production workloads will migrate there.

IBM now supports AWS (check it out, it really On Demand!), joining Adobe and Oracle. None of these programs required code rewrites. That’s the genius of AWS. The Amazon blog talked to the Oracle offering like so:

What does this mean? Instead of budgeting for and acquiring hardware, setting it up, installing an operating system and several layers of complex packages, you can simply launch one of these AMIs on EC2 and be up and running in minutes. This is definitely no-fuss, no-muss application development and deployment.

Amazon isn’t the de facto standard cloud services provider because it is complex – it is the leader because the company understands simplicity at a deep level, and minimum progress to declare victory. Competitors should take note – by the time you have established a once and future Fabric infrastructure Amazon is going to have created a billion dollar market. And what then? It will start offering more and more compelling fabric calls… People will start relying on things like SimpleDB and Simple Queue Service. Will that mean less portability? Sure it will…

Brandon Watson from Microsoft alludes to the dynamics I am describing here.

I think the key takeaways would be that Amazon has built some very cool technology and they continue to innovate.  However, that must be tempered with some cost considerations (tied to growth) and the fact that the platform itself doesn’t solve any hard problems for you.  Google, on the other hand, has little in the way of cost concerns (they have a stated goal of supporting up to 5 million page views for free), but what you can do with the framework is pretty limiting in the context of the richness of applications now possible.  Lastly, Azure is a contender, but we have some things yet to prove, and of course, we are late to the game.

Italics mine. Doesn’t solve any hard problems… but instant server provisioning… is a hard problem that Amazon is solving. Lets worry about magic applications later.

disclosure: Adobe, IBM and Microsoft are clients.

I was reading The Wisdom of Clouds blog today and it struck me how fast we’re moving. Webside thinking on the cloud is ratcheting up quickly, hubs being linked by spokes, spokes to rims, rims to rubber meeting road.

I like the way James Urquhart ties his thinking about Microsoft’s Oslo domain specific language modeling technology story into a broader analysis of Microsoft’s Azure announcements from last week’s PDC.

Oslo goes beyond the Java/C# debates. It creates a platform in which complex systems can be described by a series of fairly simple models, assembled together as needed to meet the task at hand. Text tools for the low level stuff, visual tools for the high level assembly, etc. It really was an eye opening view of what Microsoft has been working on now for a few years.

Now take a look at Azure, and the “All-Enterprise” infrastructure that will be available there. Identity services (OpenID, no less), an Enterprise Service Bus, relational and unstructured database services–the list goes on. If you take the time to learn .NET, you can get an amazing experience where the development tools just flow into the deployment models, which just flow into the operational advantages, whether on-premises or in the cloud.

James worries that Amazon will crowd out (cloud out) other innovators and “mom and pop shops”- for example in the provisioning, monitoring and management spaces. Amazon as Walmart? Well, Amazon is already the premier retailer on the web isn’t it? You’d expect it to innovate and tightly manage the supply chain aspects of the cloud operation just as WalMart has in the world of goods warehouses, bags and bricks.

Clearly a lot is going to depend on standards for cloud computing: standards for on-ramps, standards for off-ramps, standards for virtualised data shipping containers. Walmart of course was able to assume a dominance because of supply chain standards rather than in spite of them. Standards and free markets lead to dominant suppliers, often two or three – its the rules of statistical physics at work. They can also create opportunities for entirely new players to emerge. Nick Thinks Tim is wrong. I think we’ll see what I call an hourglass economy, with huge players dominating the top, and small companies filling out the long tail. In the middle- consolidation. Hugh as so often lit a short fuse which fired a bunch of people’s synapses.

RedMonk’s very own Stephen O’Grady has been working the cloud a lot lately (well he is a developer advocate, after all) and has a nice Q&A to Microsoft’s launch last week – FORZA AZURE.

Q: For those readers that may not have read the news in detail, can you summarize the nature of the offering?
A: Certainly. Azure is a base platform with several available service components layered on top of it, that can be consumed in an ad hoc fashion. Specifically these service components are: Live Services (incl. “Mesh”), .NET Services (ACL, workflow, service bus, etc), SQL Services (a query engine), Office Sharepoint Services, and Microsoft Dynamic CRM Services. Last, there is a layer of directly available, discrete applications rather than building blocks (Sharepoint Online, Exchange Online, etc). These are built upon Azure, a unique and not generally available flavor of Windows that was designed for the custom datacenters that Microsoft has constructed.

The occasionally awkward naming aside, these services offer developers and ISVs the fundamental building blocks necessary for constructing applications.

Q: Ok, so there are a lot of cloud offerings: how does this compare to the current slate?
A: It is highly differentiated from Amazon’s EC2/S3 paradigm, which preserves in the cloud traditional notions of machine instances and so on. In terms of the development style it compels, it is most similar to Force.com or Google’s App Engine, both of which eschew the instance in favor of a fabric. Rather than plan for capacity using machine instance metrics, fabric cloud developers simply deploy to the platform, which – in return for the user surrendering substantial control – assumes the burden of scaling.

As was the case with Google App Engine, there will undoubtedly be much debate on whether this is a good or a bad thing for developers: while the removal of scaling concerns eases their responsibilities slightly, it also introduces a learned helplessness.

But irrespective of the debate, the model that Azure follows is one increasingly popular in the cloud.

Stephen points to the suboptimal tooling experiences of current cloud offerings.

Put bluntly, it could be said that today’s cloud platforms are succeeding in spite of their tooling, rather than because of them.

Against this backdrop, then, it’s nothing but logical that Microsoft would leverage its development strength and position – Visual Studio having set for years, in my opinion, the development tooling benchmark. Still, their execution in Azure – at least from the briefing and demo that I’ve received, is impressive

As I have previously argued – in business generally and technology specifically the best packager wins. Which brings us back to standards, de facto and de jure. RedMonk thrives on understanding patterns of technology standardisation and helping our clients to understand them. I don’t expect the cloud to be any different, and its becoming clearer who we’ll be learning most from. The social networks behind cloud computing are going to be an important element of the story ahead. Microsoft just threw down the gauntlet to be on of the two big ecosystems left standing as all this shakes out.

I didn’t want to blog too early about Azure because frankly I came away pretty impressed. Microsoft putting its considerable weight as an educator behind REST could be very very significant indeed for corporate developers, and is one reason I disagree with Stephen somewhat that “the platform [Azure] is a natural fit for Microsoft developers, and a somewhat unnatural one for everyone else.”

I see it as very likely we’ll see REST calls to Azure URIs from all manner of alternative environments. Note that Cardspace and Microsoft Live IDs will support OpenID for example. I don’t remember Dopplr moaning about the Live Identity API – on the contrary the implementation helped Matt overcome the Password Antipattern. You see Dopplr, its made of messages.

Microsoft look to be in pole position right now for enterprisey clouds, while Amazon rules the roost with web developers. Google and IBM – your move. Oracle says Cloud is Fashion and I expect Larry will be wearing the latest hemline in early 2009. SAP is in deep retrenchment mode, and as yet doesn’t have a volume, developer-led story for its own Business ByDesign platform. Zoho continues to make cloud-based apps that sing. I understand Dell is supplying Microsoft’s data center build out, which has to have ticked off HP, which has yet to to demonstrate thought leadership here.

But its Saturday. I should be spending time with my family.

To further the discussion why not come to CloudCamp in London on the 13th November.

disclosure: IBM, Microsoft, and SAP are all clients.