Clare Hopping

Joe Curtis

Bobby Hellard

The four “Vs” of data are well known – volume, velocity, variety and veracity. However, data warehousing infrastructure in many organisations is no longer equipped to handle these. The fifth elusive “V” – value – is even more evasive. Meeting these challenges at the scale of data that modern organisations have requires a new approach – and automation is the bedrock.

For CDOs, it’s all about finding methods of using data for value creation and revenue generation, which occupies 45% of their time. This means harnessing the growing beast that is data in a way that is practical, manageable and useful.  That’s where the data warehouse comes in, providing a centralised space for enterprise data that business users, including the CDO, can use to derive insights.

Creating a successful data warehouse is critical for CDOs to succeed in monetising data within their organisation. However, the traditional waterfall approach to data warehousing, first introduced in the 1970s, delivers only a fraction of the value that it could potentially offer. Instead, the approach needs to evolve to become more responsive as organisational needs change, addressing new data sources and adapting to business demand.

Practical steps for the successful implementation of an automated data warehouse

As IT departments are expected to do much more with much less, processes need to change. Data warehouses can no longer be created “artisinally” – IT teams need to focus on producing an adaptable decision support infrastructure. Here are five steps for CDOs to help their company achieve this:

Understand the desired outcomes: Before making any decisions as to the future of your data warehouse infrastructure, CDOs need to ensure they understand the specific challenges the business teams are facing where data could help. In essence, the data warehouse automation and modernisation program needs to be built around enabling decision-making that will lead to differentiation in the market place.

According to 39% of respondents a TDWI survey, re-alignment to business objects is the top reason for data warehouse modernisation, selected by. By enabling collaboration between business teams and IT teams, the CDO helps chart the course for how business goals and technology meet. In turn, this will lead to overall business transformation, accelerated through the new data warehouse’s approach to data-driven decisions.

Understand what you have already: Most organisations already have sophisticated data management tools deployed as part of their infrastructure – however these may not be working to the fullest of their abilities. Organisations already using SQL Server, Oracle, or Teradata, for example, have a range of data management and data movement tools, already within their IT real estate, which can be automated and leveraged more effectively as part of a data warehouse automation push.

However, in that inventorying process, CDOs should be ensuring they have considered the capacity requirements of their data warehouse. Data will continue growing exponentially, so while the data warehouse may be fit for purpose today, it’s important that the automation processes, storage requirements and general infrastructure are capable of handling this in the future too.

As part of this, data warehouse automation needs to integrate with the business as it is, rather than the business as the IT teams wish it might be. CDOs need to encourage their teams to understand the data that is available, and the automated analytics and evaluation processes which can be used to meet specific business priorities. The data warehouse automation strategy needs to be designed not just for an ideal set up of data, expertly managed and curated, but for the realistic “messiness” of the business data landscape.

Automate efficiently: Data warehouse automation, as with any other large-scale transformation project, requires resources – and these are often scarce due to strict budgets and competing priorities. This means that CDOs need to think hard about what actually should be automated in order to free up man-hours in the future. In particular, these should be systematic processes, where data warehouse automation can either eliminate the need for human involvement or dramatically accelerate the process.

Embrace change: CDOs should look at data warehouse modernisation and automation as an avenue of constant, on-going development. As business needs change and new data sources emerge, CDOs need to be able to re-strategise different parts of the infrastructure to match. Similarly, to minimise disruption and ease the transition for business users, CDOs should take a staged approach to the initial automation and modernisation process, with a set schedule of when different requirements will be met. Post-production change is inevitable due to evolving business needs, new technologies used and continuous improvement desired. Change needs to be planned for.

At the same time CDOs need to prepare for the human change that automation will create. In business teams, users can be re-deployed on analysing business intelligence and translating insight into business value. In the IT teams, automation provides new capacity to plan for the future – looking at new analytics tools, or planning for smarter, better ways to deliver on business priorities further down the line.

A data warehouse automation mentality

Data warehouse automation is not solely software you buy. It’s a philosophy and culture you implement. Tools and technologies form the bedrock of the processes, but a data warehouse strategy requires strong leadership, a transparent process, and an unrelenting focus on the business’s end goals in order to succeed.

Without robust data warehouse automation, businesses will struggle to capitalise on the potential of data and its associated technologies. As the strategic lead for data-driven transformation, and the change agents across both business and IT teams, the responsibility falls to CDOs. Professionals in this role need to understand, strategise, and execute on the way that large-scale data usage will influence future business decisions. The adaptability of the supporting data infrastructure can either be a CDO’s greatest weakness or greatest asset. Use the four steps above to ensure it is the latter, and to achieve the ultimate goal of any business investment – value.

The security of data – and in particular people’s personal data – has been a hot topic in recent months. The EU’s rollout of new GDPR regulations; the Cambridge Analytica scandal; or the seemingly weekly revelations of financial institutions or consumer service providers which have had their databases hacked, are all examples most of us will be aware of.

Less often discussed but just as important as the security of our data, is the security of the data centres that house it. And at first glance, identifying, reviewing and prioritising all the elements that a data centre must contain in terms of security would appear to be a very complex subject, depending on myriad variables including facility size, organisation type, service commitments, system complexity, customer requirements, the list goes on…

However, independent of the variables mentioned above, in my view data centre security can be boiled down to just two areas – physical security and operational security.  And while both of these clearly depend to a great extent on technology, the single most important element is the establishment of appropriate policies, processes and operating procedures – and critically, of course, actually following them.

Unfortunately, over the years I have seen many examples of security – both physical and operational – being seriously compromised through the lack of clear and well-defined security processes and procedures. And ironically, I have seen this most often in data centre facilities that had state-of-the art security equipment installed.

For example, implementing the latest and most sophisticated biometric access systems does not, by itself, ensure that supposedly secure areas are actually secure and that access is fully controlled. On the contrary, I have witnessed unauthorised and unsupervised personnel wander in and out of secure areas at will. The failure here not being due to any fault with the access control equipment itself but to appropriate security protocols not being implemented or maintained.

As for operational security, a standard requirement for any modern data centre is to have redundancy capabilities fully integrated in order to ensure continuous operation even if disaster strikes. And for many data centre operators’ customers, this is non-negotiable, given their dependence on the often mission-critical systems the data centres house.

However, just as with ensuring physical security, implementing systems for fully redundant facility operation is not simply a matter of installing more of the latest equipment. Ensuring data centre redundancy is a hugely complex undertaking. Initial design is clearly important, as is the correct installation and interlinking of redundant systems, whether for power, cooling, monitoring, or communications. But most important of all, once again, are the protocols and procedures that must be implemented and followed in order to ensure that redundant gear actually kicks in to action if and when it needs to.

Regardless of whether the data centre in question is hyperscale or a relatively small edge facility, having the right processes in place and the right people following them are typically what makes the difference between, on the one hand, a data centre’s security being fully maintained and on the other, a catastrophic failure.

So when securing even the most technical of environments, technology is only part of the answer. Without the disciplined application of associated policies and processes, success cannot be guaranteed. After all, the best tools in the tool box are of little value without the appropriate knowledge and experience to use them.

Hewlett Packard Enterprise (HPE) has seen the future – and it’s all about the intelligent edge.

The company has announced a $4 billion (£3.04bn) investment over four years in technologies and services to deliver personalised user experiences, seamless interactions, and artificial intelligence (AI) and machine learning to improve customer experiences and adapt in real time.

HPE cited Gartner figures which argue that by 2022 three quarters of enterprise-generated data will be created and processed outside of the traditional data centre or cloud, up significantly from 10% this year. It’s a race against time for companies to get proper processes and actionable insights from their data wherever it lies – and HPE feels as though it has the solutions to those problems.

“Data is the new intellectual property, and companies that can distil intelligence from their data – whether in a smart hospital or an autonomous car – will be the ones to lead,” said Antonio Neri, HPE president and CEO. “HPE has been at the forefront of developing technologies and services for the intelligent edge, and with this investment, we are accelerating our ability to drive this growing category for the future.”

Details are a little scant on where this money will go – however HPE did note that it will ‘invest in research and development to advance and innovate new products’ as well as ‘continue to invest in open standards and open source technologies, cultivate communities of software, AI and network engineers, and further develop its ecosystem through new and expanded partnerships.’

The $4bn HPE is putting aside for this investment is not quite the $5bn Microsoft announced back in April focusing on the Internet of Things. Microsoft also favours the term ‘intelligent edge’ when discussing the future of technology. In February, during the company’s Q2 financial report, CEO Satya Nadella told analysts that the ‘intelligent cloud and intelligent edge platform [was] fast becoming a reality.’

When we talk about the home of the future, we might think of technologies that will bring convenience to our lives. Refrigerators that know when we’ve run out of milk and order more for us, perhaps, or 3D printers that will make any shape of pasta you can imagine. But while this Jetsons-like vision might initially hold some appeal to the average person on the street, such innovations are likely to come and go in a flash.

In truth, we struggle to adapt to technologies that are designed to help us around the home, often reverting to old ways of doing things. Take Amazon’s Alexa for example. The vast majority of voice skills that the AI assistant is capable of are largely unused by consumers.

It’s time we shifted focus from gimmicks to changes that will offer genuine value to the household and the wider community. Because in the home of the future, the most interesting things will be happening behind the scenes.

Imagine a home that is also a data centre. While on the surface this might seem like a far-fetched idea, it would actually bring many benefits. A vast amount of computational power currently goes unused in homes, with computers, games consoles set-top boxes and smart televisions under-utilised and in many cases in standby mode for most of their life. This untapped power could be used to drastically reduce reliance on existing data centres.

And then there’s speed. The UK’s average broadband speeds of 46.2Mbps downstream and 6.2Mbps upstream might not sound much in comparison to a tier-one data centre, but by linking homes together in a decentralised network the potential is enormous.

Of course, any such plan would need to be incentivised – by using blockchain technology for example, to create a tokenised system of reward for contributing to this decentralised network.

Imagine having the cost of your broadband bill covered by being part of such a network, or buying a wireless speaker that pays for itself over time. Those that contribute the most earn the most, and by using a decentralised system the rewards would be distributed in a completely transparent and verifiable way, in contrast to traditional cloud platforms that centralise control and network revenues.

These networks would provide real, community-owned alternatives to the services provided by Amazon, Google and the like, without the massive environmental impact. Data centres are of course incredibly costly to run, mostly due to the incredible amount of energy required to keep the servers cool. While Microsoft has recently embarked on a curious experiment with Project Natick, sinking a data centre into the sea off the Orkney Islands in an attempt to boost energy efficiency, it’s hard to believe that this is a realistic option for the future.

Being surrounded by seawater might keep the temperature of the hardware under control without requiring the specialist cooling systems used in conventional server farms, but it also makes servicing a faulty node pretty much impossible, and a lot of energy has to go into making the thing in the first place. Surely it makes much more sense to maximise the potential of the devices we already have at our disposal, which would otherwise be idle for around three-quarters of their lifetime.

In the UK, we’re already starting to see solar panels built into many new homes, as well as installed in existing ones – with owners able to sell any excess electricity that they generate back to the National Grid. And you can certainly imagine a near-future where this shift paves the way for individual homes becoming nodes of vast, decentralised networks.

So, the concept of cloud might not be sinking without a trace – but it is certainly time for an upgrade. And this year, you can expect the first wave of decentralised, faster, cheaper networks to arrive – bringing a much needed working use case along for the ride.

Jonathan Parkyn

Web-based email has never been so popular yet there are plenty of headaches associated with having to be online to read your messages. Even the best cloud services out there struggle to replicate the ease of use that desktop-based clients bring, whether it be easily backing up emails, accessing attachments offline, or simply offering the same flexibility when it comes to capacity.

We’ve tested some of the most popular email clients to see which offer the most well-rounded experience for users, including software performance, feature set, and their ease of use.

eM Client

www.emclient.com

Price: Free

With a smart-looking, modern interface and plenty of advanced features, eM Client is easily the best email software for Windows PCs.

Calendar, contacts and tasks are all integrated and there’s even built-in support for chat (via Facebook, Google or Jabber). Setting up accounts is very straightforward – most popular email services are automatically recognised and configured without you having to faff around with SMTP server settings and suchlike. If you’re switching from another email program, eM Client will helpfully offer to import data from your old application, and if you’re using an Outlook.com or Gmail account, your calendar and contacts will be automatically synced, too.

The client can be switched to a stylish Dark theme

eM Client’s default interface should feel instantly familiar – it uses the tried-and-tested, three-column (folder list, message list, preview pane) layout. But the program doesn’t look old-fashioned and you can customise its layout to suit your tastes by clicking Menu, Tools, Settings, Appearance – we prefer the stylish Dark theme. It’s also possible to switch Conversation view off, if you prefer.

Notable features include a super-fast search, advanced filtering tools, templates, signatures, tags and the ability to categorise mail using colour-coding. There’s even a built-in translator, which uses Bing’s translation engine.

The latest version of eM Client (7.1) adds a number of useful new features, including an improved backup tool that can automatically back up your data in the background, and support for PGP encryption.

How it can be improved

The free version of eM Client only supports two mail accounts. If you need more than that, you’ll have to pay for the Pro version, which costs £36 (or £72 if you want lifetime upgrades to future versions). After the 30-day trial, you’ll need to apply to eM Client for a free license to keep using it for free, which seems like an unnecessary step. There’s currently no integration with Windows 10’s Action Centre – instead you’re alerted to new mail via eM Client’s own Notification area icon pop-up.

Verdict

Quibbles aside, eM Client easily beats the competition. It offers a great balance of simplicity and adaptability, while its familiarity makes it a great replacement for older tools, such as Outlook Express and Windows Live Mail.

Features: 5

Performance: 5

Ease of use: 5

Overall: 5

Mozilla Thunderbird

www.thunderbird.net

Price: Free

Thunderbird is a resolutely old-school email program that offers support for multiple POP and IMAP accounts, and provides easy set-up for popular services, such as Gmail and Outlook.com.

You can configure it so that it looks and works how you want it to, and there are loads of features, including powerful filtering tools, an RSS reader and instant messaging. Like Mozilla’s more famous web browser, Thunderbird’s abilities can be expanded further by installing add-ons – anything from alternative themes to mail merge tools, password managers and more – though many ‘legacy’ extensions are being phased out. The once-optional Lightning add-on is now integrated into the program, meaning that calendar and tasks features are now built in.

How it can be improved:

Thunderbird lacks native Exchange support, meaning some accounts (including Outlook.com ones) don’t get the full range of features. There’s no support for Windows 10’s Action Centre, either – so there are no native Windows 10 notifications.

Mozilla has made no secret of the fact that its struggling to justify Thunderbird’s ongoing development. Last year Mozilla found a way to keep Thunderbird alive by separating it from its core business and new features have been promised, but its future is far from guaranteed.

Verdict

With plenty of built-in features and many more available through add-ons, Thunderbird is highly versatile, though its ageing interface and lack of support for some newer standards are disappointing – and may never be fixed.

Features: 4

Performance: 4

Ease of use: 4

Overall: 4

Microsoft Mail app

www.microsoft.com

Windows 10’s built-in apps tend to come in for a bit of a knocking, but Mail is actually pretty good. It has a nice, clean interface, supports most account types (including POP and IMAP) and is refreshingly simple to set up and use.

Its close integration into the OS has a number of benefits, including a live tile in the Start menu and a cross-app relationship with the People (contacts) and Calendar apps. Microsoft keeps improving the Mail app, too. Last year it added a Focused Inbox feature for Gmail users, for example (click Settings, Reading to toggle this on or off). On touch-screen devices, the app’s intuitive Swipe Action controls are an added bonus.

How it can be improved:

Some of Mail’s tools are a little too simplistic – there’s no filtering, for example, and it only supports plain text signatures (though you can hack it by adding in your own HTML code). Also, since Mail is tied so closely to the OS, its reliability can be affected by Windows 10 updates. We’ve experienced problems like these first-hand and many user reviews on the app’s Windows Store page would suggest that we’re not alone.

Verdict

If simplicity is what you’re after, look no further – Mail’s already installed on your PC and is a piece of cake to set up. That said, you may find its lack of features frustrating.

Features: 3

Performance: 4

Ease of use: 5

Overall: 4

Best of the rest

Microsoft Outlook (www.microsoft.com)

If you subscribe to Office 365 (from £5.99 per month or £60 per year), you get Outlook with it, which is an email client, calendar tool, contacts manager and to-do list all in one. Outlook offers a lot of advanced tools, including a powerful Rules function, fast search and built-in archiving tools. But, these days, Outlook feels like overkill – there are simpler tools available for free.

Mailbird Lite (www.getmailbird.com)

Mailbird Lite feels a little like Microsoft’s Mail app on steroids. It looks great and has the ability to connect to popular apps and services, including WhatsApp, Slack and Facebook, as well as the usual email accounts. The free version of Mailbird restricts you to a single email account and lacks some of the full program’s better tools, such as attachment previewing and email snoozing. Adding these will cost you £19.50.

Postbox (www.postbox-inc.com)

Postbox’s interface is uncluttered and should feel familiar. RSS feeds and newsgroups are supported but, strangely, there’s no built-in calendar. Some of the program’s more innovative tools include automated responses and placeholders, which can save time if you find yourself frequently sending similar replies. The big drawback is that Postbox isn’t free. Beyond the 30-day free trial you’ll need to pay – currently it’s $40 (£29) for the full version.

Image: Shutterstock

Should you send your data to computers or bring your computing assets to the data?

This is a major question in IoT. A few years ago you might have said “everything goes to the cloud,” but sheer size and scope often makes a smart edge more inevitable. IDC estimates that 40% of IoT data will be captured, processed and stored pretty much where it was born. While Gartner estimates the amount of data outside the cloud or enterprise data centres will grow from 10% today to 55% by 2022.

So how do you figure out what goes where?

Who needs it?

IoT will generate asinine quantities of data across all industries. Manufacturers and utilities already track millions of data streams and generate terabytes a day. Machine data can come at blazingly fast speeds, with vibration systems churning out over 100,000 signals a second, delivered in a crazy number of formats.

Machines, however, aren’t good conversationalists. They often just provide status reports on temperature, pressure, speed, pH, etc. It’s like watching an EKG machine; companies want the data, and in many cases need to keep it by law, but only a few need to see the whole portfolio.

The best bet: look at the use case scenario first. Chances are, every workload will require both cloud and edge technologies, but the size of the edge might be larger than anticipated. 

How urgently do they need it?

We’ve all become accustomed to the Netflix wheel that tells you your movie is only 17% loaded. But imagine if your lights were stuck at 17% brightness when you came home. Utilities, manufacturers and other industrial companies operate in real-time – any amount of network latency can constitute an urgent problem.

Peak Reliability, for instance, manages the western U.S. grid. It serves 80 million people spread over 1.8 million square miles. It also has to monitor over 440,000 live data streams. During the great eclipse it was getting updates every ten seconds

Rule of thumb: if interruptions can’t be shrugged off, stay on the edge or a self-contained network.

Is anyone’s life on the line?

When IT managers think about security, they think firewalls and viruses. Engineers on factory floors and other “OT” employees—who will be some of the biggest consumers and users of IoT — think about security as fires, explosions and razor wire. The risk of a communications disruption on an offshore drilling rig, for example, far outweighs the cost benefits of putting all of the necessary computing assets on the platform itself. Take a risk-reward assessment.

What are the costs?

So if the data isn’t urgent, won’t impact safety, and more than a local group of engineers will need it, do you send it to the cloud? Depends on the cost. Too many companies have responded to cloud like a teenager in 2003 given their first smart phone. Everything seems okay, until the bill comes.

In the physical world, no one sends shipments from L.A. to San Francisco via New York, unless there is a good reason to go through New York. Distance means money. Sending data to the cloud that could just as effectively be stored or analyzed on the edge is the digital equivalent. Getting the right balance of edge and cloud is the key to managing the overall TCO.

How complex is the problem?

This is the most important and challenging factor. Are you examining a few data streams to solve an immediate problem such as optimizing a conveyor belt, or comparing thousands of lines across multiple facilities? Are you looking at a patient’s vital signs to determine a course of treatment, or studying millions of protein folds to develop a new drug?

Companies often use the cloud to crack a problem, and then repeat it locally at the edge. Projects resulting in millions in savings aren’t being produced by a magical algorithm in the cloud – instead, people look at a few data streams and figure it out on their own.

Another way to think about it: the cloud is R and the edge is the D in R&D.