(c)iStock.com/Erik Khalitov

The vast majority of companies who are moving to cloud applications also have a significant current investment in on-premise operational applications and on-premise capabilities around data warehousing, business intelligence and analytics. This means that most of them will be working with a hybrid cloud/on-premise data management environment for the foreseeable future.

Moving at ‘cloud speed’, and setting up a new application in a matter of hours, is a big advantage in terms of business agility, but one cost is that managing data becomes more complicated.

It seems that more often than not, a hybrid architecture isn’t planned – it just happens. The ‘typical’ use-case pattern starts with organisations integrating a cloud application, perhaps for CRM or HR, then, they add cloud database before finally adding a cloud data warehouse and/or analytics capability.

A lot of the time it’s the business that drives this pattern in an effort to solve a particular problem. Most frequently, IT is hugely involved in the effort, but the cloud analytics decision is made by the business side. As a result, IT inherits significant new data management complexity.

Implementing data governance

It can be difficult to retrofit governance into existing systems. Often, the focus is on the initial data migration to the new operational application or analytics, where a simple bulk data loader is employed in the interest of speed and agility. This has several downsides:

• There is no metadata. Using a data integration tool with metadata support, instead of a bulk loader, will give you an end-to-end view of your data lineage throughout your environment. This will be critical as the complexity grows and you need to make changes, quickly and without errors
• The opportunity to do a data clean-up while moving the data is lost, and you risk populating a new application with ‘bad data’
• You miss the opportunity to think about ongoing data security and data governance on a broader scale

Once the new applications have gone live, focus shifts to ensuring data consistency. Moving between cloud and on-premise systems and cloud-to-cloud brings new challenges, and leave fewer resources dedicated to overall data management.

If you don’t want to slow down the business initiatives that are driving the new applications, but still want to prevent that data complexity or chaos, it will pay to have a data management architecture and best practices in place before-hand.

The hybrid data management checklist

Data management in a large organisation is challenging. But this gets even more complex when it’s hybrid. The key is to plan ahead so that you’re not a roadblock to the business. Key considerations for a successful programme should question:

• Does your software provider offer out-of-the-box, high-performance connectivity to all of the on-premise and cloud sources?
• Does your software provider have compatibility across their on-premise and cloud data integration capabilities including shared skills, shared code (mappings), shared management tools?
• Do they support multiple integration patterns: batch, real-time, API integration?
• Can they smoothly enable you to grow your data management capabilities as you need them covering data quality, data governance, master data management, metadata management, security, B2B?
• Do they support wizards and template-drive development for ‘citizen integrators’?
• Do they have metadata management tools for data lineage and business meaning and context. This is critical for reducing errors, enabling self-service, and speeding changes to the environment
• Do you have a central point of operational management for data?
• Can you manage quality and governance of data across a hybrid environment?

At the end of the day, the business challenge is to deliver value faster than the competition. The IT challenge comes with meeting the speed and quality requirements of the business while enabling them to accelerate their business agility. It can be done, but it takes careful planning and a good, forward-looking data management architecture.