Google Cloud has announced the launch of Cloud Dataproc on Kubernetes, adding another string to the bow for the product which offers a managed cloud service for running Apache Spark and Hadoop clusters.
Google – which originally designed Kubernetes before handing it to the Cloud Native Computing Foundation (CNCF) – is promising ‘enterprise-grade support, management, and security to Apache Spark jobs running on Google Kubernetes Engine clusters’, in the words of a blog post confirming the launch.
Christopher Crosbie and James Malone, Google Cloud product managers, noted the need for Cloud Dataproc to utilise Kubernetes going forward. “This is the first step in a larger journey to a container-first world,” Crosbie and Malone wrote. “While Apache Spark is the first open source processing engine we will bring to Cloud Dataproc on Kubernetes, it won’t be the last.
“Kubernetes has flipped the big data and machine learning open source software world on its head, since it gives data scientists and data engineers a way to unify resource management, isolate jobs, and build resilient infrastructures across any environment,” they added. “This alpha announcement of bringing enterprise-grade support, management, and security to Apache Spark jobs on Kubernetes is the first of many as we aim to simplify infrastructure complexities for data scientists and data engineers around the world.”
To say Kubernetes is not a major priority for both vendors and customers would be something of a falsification. The recent VMworld jamboree in San Francisco two weeks ago saw the virtualisation giant launch a major attack on the product, with the primary launch being VMware Tanzu, a product portfolio which looked at enterprise-class building, running and management of software on Kubernetes.
As this publication put it when KubeCon and CloudNativeCon hit Barcelona back in May, it was a ‘milestone’ for the industry. Brian Grant and Jaice Singer DuMars certainly thought so; the Google Cloud pair’s blog post at the time agreed Kubernetes had ‘become core to the creation and operation of modern software, and thereby a key part of the global economy.’
The goal now is to get the most out of it, whether you’re an enterprise decision maker or developer alike. Writing for CloudTech last month Ali Golshan, co-founder and CTO at StackRox, noted the acceleration in user deployments. “Despite the fact that container security is a significant hurdle, containerisation is not slowing down,” Golshan wrote. “The advantages of leveraging containers and Kubernetes – allowing engineers and DevOps teams to move fast, deploy software efficiently, and operate at unprecedented scale – is clearly overcoming the anxiety of security concerns.”
Golshan also noted, through StackRox research, that Google still ranked third among the hyperscalers for container deployments in the public cloud but had gained significantly in the past six months.
“Enterprises are increasingly looking for products and services that support data processing across multiple locations and platforms,” said Matt Aslett, research vice president at 451 Research. “The launch of Cloud Dataproc on Kubernetes is significant in that it provides customers with a single control plane for deploying and managing Apache Spark jobs on Google Kubernetes Engine in both public cloud and on-premises environments.”
Interested in hearing industry leaders discuss subjects like this and sharing their experiences and use-cases? Attend the Cyber Security & Cloud Expo World Series with upcoming events in Silicon Valley, London and Amsterdam to learn more.