Google has launched a new cloud service called Cloud Composer to help organisations design, create, and manage consistent workflows within Google Cloud Platform (GCP).
The service, which is currently in beta, is designed to develop, schedule and monitor enterprise workflows across internal datacentres or multiple clouds.
It offers End-to-end GCP integration, and users can orchestrate their full GCP pipeline through Cloud Composer’s integration within Google Cloud Platform.
It also connects a user’s pipeline through a single orchestration service whether a workflow exists on-premises or in multiple clouds.
“We believe there should be an easy and reliable workflow solution at a platform level, like other cloud services,” said James Malone, Google product manager, in a blog post.
“With the aforementioned features and others outlined below, Cloud Composer delivers a single managed solution to create and manage workflows, regardless of where they live, and gives you the portability to take these critical pieces of infrastructure with you if you migrate your environment.”
Google said the service was a “starting point” and a number of features are planned for the future including additional Google Cloud regions, Airflow and Python version selection, and autoscaling.
Cloud Composer and Airflow have support for BigQuery, Cloud Dataflow, Cloud Dataproc, Cloud Datastore, Cloud Storage, and Cloud Pub/Sub. The Airflow GCP documentation includes specifics on how to use the operators for these products.
Malone said an earlier alpha program gave access to hundreds of users and feedback from the helped improve the product.
«At Blue Apron, our data workflows need to operate flawlessly to enable on-time delivery of perishable food. Cloud Composer helps us orchestrate our pipelines more reliably by allowing us to author, schedule, and monitor our workflows from one place using Python,” said Michael Collis, staff software engineer at Blue Apron.
There will be a consumption-based pricing structure for Cloud Composer that includes virtual CPU per hour, Gb per month and GB transferred per month as the cloud data orchestrator is based on several Google Cloud platform components.
Image: Shutterstock