For a significant number of companies, beginning their cloud migration journey is hard. In spite of the greater scalability, flexibility, optimisation, and lower costs for big data in the cloud, organisations struggle to mobilise their teams to begin their cloud journey. Once they have successfully migrated their workloads to the cloud, however, a lot of organisations think the journey is finished.
Unfortunately, there are a number of operational and visibility challenges that exist on-premises which don’t disappear once workloads have been migrated. While the benefits of cloud migration are clear, it is frequently oversimplified and considerations such as application dependencies and system version mapping are not given due thought. As a result, costs begin to overrun through over-provisioning or production is delayed through provisioning gaps.
Even post-migration, issues can persist. Modern businesses are powered by data applications that rely on a myriad of platforms which frequently creates issues in understanding, planning, optimising, and automating the performance of their data apps and infrastructure. These difficulties are compounded by the use of disparate technologies and siloed approaches to managing data applications and data infrastructure. With the majority of monitoring solutions frequently lacking end-to-end support for big data environments, full-stack compatibility, or requiring complex instrumentation, data teams require deep subject matter expertise to configure changes to applications or components. As a result, organisations can struggle to find teams skilled enough to deliver strong application performance, often resulting in poor user experience, inefficiencies and mounting costs as organisations buy more and more tools to resolve problems.
Quantifiably, organisations see a high Mean Time to Identify (MTTI) and Mean Time to Resolve (MTTR) issues due to difficulties in understanding dependencies and retaining focus in root cause analysis. Data collection and correlation can be time-consuming when trying to collect granular cluster and application-specific runtime information, as well as metrics on infrastructure across platforms, application and system log data, configuration parameters, and other relevant data.
Moreover, resources using native Hadoop APIs will only send data while an application is executing creating yet further complications. The lack of granularity and end to end visibility makes it impossible to remedy all of these problems, leaving businesses with little visibility of their data applications. Even once all this data has been collated, further difficulties arise in evaluating and interpreting it. Minor human errors such as a missed configuration parameter, incorrectly sized container, or a rogue stage of your Spark application, which can completely cripple a data cluster, may be entirely missed.
For enterprises that have only recently migrated, these myriad issues can leave cause for doubt in their choice. However, it should be noted that cloud adoption is not a finite process with a clear start and end date — it’s an ongoing lifecycle with four broad phases (planning, migration, operation, and optimisation). To ensure a painless and efficient cloud migration, each of these four phases needs to be given proportionate attention.
Broadly, In the planning phase, decisions need to be made surrounding which applications are most suited for the cloud, what resources do they require, which data sets need to be migrated and whether permanent, transient, autoscaling, or spot instances should be used. During the migration and operation stage, there is a need for continuous monitoring of performance and costs, and assessment of the critical dependencies and service mapping. Finally, once these workloads are in production on the cloud, it is time for data teams to begin considering how they optimise their applications and performance in order to guarantee SLAs.
A comprehensive approach to operational planning goes a long way in resolving the various challenges of managing big data technologies (both on-premise and in the cloud). With enough time and focus spent on each stage of the cloud adoption lifecycle, and adhering to best practice, the benefits of cloud migration can be realised faster. The main thing to remember for data-teams, is not to take the foot off the gas and keep momentum up once they’ve moved to the cloud.
Photo by Shannon Lam on Unsplash
Interested in hearing industry leaders discuss subjects like this and sharing their experiences and use-cases? Attend the Cyber Security & Cloud Expo World Series with upcoming events in Silicon Valley, London and Amsterdam to learn more.