When Cloudera and Hortonworks, two of the biggest big data behemoths, first announced they were coming together in a $5.2 billion blockbuster merger in October, the questions were almost infinite. Yet the most important two seemed to be: why now? And what does the future hold?

Now, after the transaction officially closed earlier this month, the answers can be a little more candid. Perhaps surprisingly, the new company admits the move came about as a result of feeling the heat from its traditional competitor base as well as the public cloud giants.

Here’s what we know about the deal:

• Conversations between Cloudera and Hortonworks around M&A activity had started as far back as 2015 with various factors meaning last summer was ‘the right time for both businesses’ to merge
• The new company will be called Cloudera going forward – the common platform which results will be called Cloudera Data Platform aimed as a nod to the Hortonworks Data Platform suite
• The first technical kick-off of the joined entity took place earlier this month in Scottsdale, Arizona. Cloudera says ‘80%-85%’ of the most burning issues around overlapping product sets were able to be resolved there

It’s usually good if you know you’re under pressure to face up and admit it rather than endlessly be in denial. And it’s even better if, like Cloudera, you have an action plan to resolve the situation.

Chief marketing officer Mick Hollison (left) notes that, despite the arrows coming in from two different fronts, the challenges are broadly similar. The company’s software, as well as its strategy, is based around what it is calling the ‘enterprise data cloud’. The strategy is based around three strands; supporting every possible cloud implementation, from hybrid to public to multi-cloud; supporting a wide range of analytic capabilities; and going the extra mile on an open philosophy, from open storage, to compute, to integration.

Getting this mix right, Cloudera hopes, will satisfy even the largest and most demanding enterprise customers – and put them one step ahead of the competition in the process.

“If I look at the public cloud providers, they’re inherently never going to be multi-cloud,” Hollison tells CloudTech. “You’ll continue to see [them] dipping toes more into hybrid – [it’s] future state but something they plan to get into. Multi is something they’re not likely to get into.

“The second part public cloud vendors will struggle with a bit is that security and governance and common metadata layer,” Hollison adds. “That doesn’t exist for those vendors today. If you buy EMR from Amazon and you also buy RedShift from Amazon, you get a different security, governance and metadata stack with each of those offerings. We offer commonality at that layer.”

Looking at the more traditional big data companies – the word traditional being used loosely – Hollison again pulls no punches. “If I look across the way, with the more purpose-built data warehouse cloud [vendors], those companies and those offerings are very compelling for their one function that they offer. They’re not at a point where they’re building out their technology into a platform they can offer a set of shared services across.”

It’s worth noting here that Cloudera freely admits it hasn’t got all the pieces in place yet. Yet one element which all sides can agree on is that customer expectations have skyrocketed in recent years. “The expectations are seemingly infinite,” says Hollison. “The raw scale and quantity of data consumption by our largest customers is just orders of magnitude beyond what any of us could have ever imagined not terribly long ago.

“The other dimension is that customers have high demands around cloud,” Hollison adds. “Many of our large enterprises customers have a bit of a concern around being locked in to any one cloud vendor. Regardless of partnership, they don’t want to take the public cloud on as a new version of an IBM or Oracle lock-in.”

Part of this heightened sense of expectation is around the promise of artificial intelligence (AI) and machine learning (ML); a necessary strategic point for the vendors. A report from venture capital firm Work-Bench in August predicted that ‘all modern [business intelligence] vendors to either release an automated machine learning product or buy a startup’ by the end of 2019.

Cloudera was ahead of this curve buying data science platform Sense.io in 2016, with the technology acquired forming the backbone of the company’s Data Science Workbench product. “It’s a very logical step from my point of view,” Hollison says of fusing AI and ML with big data. “The term we’ve been using is to ‘industrialise AI’, to make it more like a factory.

“Most of the ML and AI that has been done in enterprises to date has been pretty bespoke. It hasn’t necessarily been done against well secured and governed data sets supported by IT,” adds Hollison. “It’s often been scraped onto a laptop by a data scientist, putting that data at risk. When you combine data security and management capabilities that Cloudera offers, with an easy to use workbench that allows them to continue to use the languages and frameworks that they like, it’s a pretty good combination that makes both the data scientists and IT happy that data is being used in an intelligent way.”

Going forward, Hollison promises a lot of hard work on integrating and hardening the sales operations, as well as more go-to-market pieces. Yet in other areas progress has been more seamless than one would expect. The companies noted that approximately two thirds of each other’s code base had commonality, while the engineering teams were easier to satiate.

“You might have thought we’d have more challenges on that front, but what I think people forget is a lot of these engineers have been working together for upwards of 10 years in the open source community,” he says. “Even though the big corporate push might have been very competitive, at a code-writing, engineering level, there’s a lot of mutual respect between the teams.”

While there’s still a lot to do, the building blocks are in place. “We knew we would be a much stronger, more formidable competitor together than we would be continuing to take shots at one another,” adds Hollison.

Interested in hearing industry leaders discuss subjects like this and sharing their experiences and use-cases? Attend the Cyber Security & Cloud Expo World Series with upcoming events in Silicon Valley, London and Amsterdam to learn more.