The shift to the cloud is encouraging enterprises to rethink their options on storage. According to a June 2019 study from IHS Markit, 56% of organisations said they plan to increase investment in object storage, putting it ahead of unified storage at 51%, storage-area networks at 48% and network-attached storage at 36%. Most object storage is in the cloud, with popular examples including AWS S3, Azure Blob Storage and Google Cloud Platform (GCP) Cloud Storage.

But shifting to a new storage architecture at the same time as the cloud move is not entirely painless.

At the beginning of the decade, Moneysupermarket.com, the consumer online comparison and information site for financial services, was using a combination of SQL databases and SAS analytics environment. By 2014, it had moved to AWS for website hosting and data analytics, including use of S3 object storage and Vertica data warehouse. By May 2019, it moved its data and analytics to GCP using the BigQuery data warehouse and Cloud Storage object storage. The website itself remains on AWS.

Harvinder Atwal, Chief Data Officer at MoneySuperMarket, tells Cloud Pro: «One of the good things about the cloud is the initial learning curve is very shallow: it’s easy to start. But then you get to the point where it’s very much steeper and you need to understand some of the complexities involved.»

One example of those complexities is the introduction of object lifecycle policies. The idea is to define policies to manage objects throughout the time the organisation needs them. That might be to move them to cheap long-term storage such as AWS Glacier or to expire them all together. Getting these rules right from the outset can save costs.

«That’s one of the things that maybe we should put a little more effort into from the very beginning,» Atwal says.

Other advice for those moving to object storage in the cloud includes avoiding biting off more than the team can chew.

«I would not do the migration all in one go,» Atwal says. «I think the bigger project and the more money and resources it uses, the more likely it is to fail. I would encourage people to think of their use case and application and build a minimal viable product around that.»

It’s worth getting advice about the transition from independent third parties, which the cloud platform vendors can recommend. For example, Moneysupermarket.com used a consultancy called DataTonic with its transition to Google Cloud Platform.

Lastly, there can be a cultural change in store for the IT department, Atwal says. «The IT function can be very traditional in its thinking around how you use data. They think you must cleanse it, put it into a relational schema and only then can users access it. But with data today, the value in analytics comes from actually being able to use data for many sources and join them together, and IT has to learn to ditch its historic mindsets.»

Nasdaq, the tech stock market, began working with AWS in 2012. It stores market, trade and risk data on the platform using S3 and Glacier. It uploads raw data to Amazon S3 throughout the trading day, using a separate system running in the cloud, converts raw data into Parquet files and places them in their final S3 location. This way, the system is able to elastically scale to meet the demands of market fluctuations. It also uses Amazon Redshift Spectrum to query data to support billing and reporting, and Presto and Spark on Elastic MapReduce (EMR) or Athena for analytics and research.

«Migrating to Amazon S3 as the ‘source of truth’ means we’re able to scale data ingest as needed as well as scale the read side using separate query clusters for transparent billing to internal business units,» says Nate Sammons, assistant vice president and lead cloud architect at Nasdaq.

But getting the scale of analytics solutions right for the problem has been a challenge, he says. «We currently operate one of the largest Redshift clusters anywhere, but it’s soon to be retired in favour of smaller purpose-specific clusters. Some of the custom technologies we developed [in the early days] have since been retired as cloud services have matured. Had technologies like Amazon Redshift Spectrum existed when we started, we would have gone straight to Amazon S3 to start with, but that was not an option.»

The advantage of using S3, though, was that it made the organisation less concerned about individual machine outages or data centre failures, Sammons says. «If one of the Amazon Redshift Spectrum query clusters fail, we can just start another one in its place without losing data. We don’t have to do any cluster re-sizing and we don’t require any CPU activity on the query clusters to do data ingest.»

Rahul Gupta, IT transformation expert at PA Consulting, says those exploiting object storage in the cloud should know that apparent scalability and elasticity does not remove the need to do some basic housekeeping on data.

«A lot of people feel storage is cheap, so they build systems with vast amounts of data and think the impact on cost is not that great. They push the data into S3, or an equivalent, and then once it’s in there, they feel that they can impose structure on the data, which is not the right thing to do,» he says.

He says that by understanding data structure upfront and creating governance such as role-based access, organisations will not have to revisit the architecture once the data grows.

Just because so many organisations are moving storage to the cloud, does not mean they all get the same value from the transition. The considerable investment cloud infrastructure, storage and analytics application will offer the greatest returns to those who understand the storage lifecycle upfront, create some governance rules around access and understand data structure from the outset.

The NHS has migrated two of its national services to the cloud, hoping to cut costs while improving security and efficiency of services.

The NHS e-Referral Service (e-RS) and NHS 111 Directory of Services (DoS) are the first major NHS systems to make the migration under the government’s cloud-first policy, with both using AWS.

The aim is to cut costs at both services, but there are other benefits, says Neil Bennett, director of services at NHS Digital. «Costs are lowered, reducing pressure on the public purse, there is better security and reliability, as well as greater flexibility, performance, scalability and availability, to name a few,» he said.

The e-RS is a booking service that handles 18 million referrals annually, letting patients from more than 1,100 GP practices choose clinics, hospitals and dates and times for appointments. It’s now enabled for booking and managing such appointments via the internet, but that won’t be available until later this year when NHS Identity takes over authentication, the NHS said in a statement.

The DoS helps connect patients to the appropriate service for the health concerns, helping relieve pressure on urgent and emergency care. It handles 16 million searches annually.

Migrating such important services without disrupting patient care was key, explained Bennett: «This was a tremendous collaborative effort across many different teams here and with external partners, to migrate such large systems with a minimum of disruption to users, in a reasonably short timescale.» 

Alongside cutting costs and improving services for patients, the migration to cloud is a key part of the NHS’ sustainability strategy, said Ben Tongue, sustainability manager at NHS Digital.

«Large cloud operators like AWS provide significant energy and carbon savings against enterprise and legacy systems,» he said. «We are working with AWS to achieve full transparency on the energy use and carbon impact of the contract, so that we can continue to focus on ensuring that our storage systems are as energy efficient as possible, reducing carbon emissions and minimising environmental impact.»

Last year, the NHS unveiled a cloud framework to simplify procurement, hoping to help migrate more services as part of the government’s cloud-first policy. Patient records stored by EMIS are already making the shift to AWS, while Barts Health NHS Trust is moving its IT estate to the cloud via Capgemini, but research suggests many NHS trusts remain wary of the cloud.

Despite the growing popularity of cloud computing, organisations still struggle to control costs. Research from Flexera, a provider of IT management software, shows 84% of enterprises find optimising cloud costs a growing challenge. The 2019 survey of 786 technology professionals also found organisations underestimate their wastage, putting it at 27% of cloud spending whereas the real figure — according to Flexera — is around 35%.

Part of the problem is the shift from the old world to the new, as organisations lift more of their applications and infrastructure to the cloud, according to Adrian Bradley, cloud advisory lead at KPMG

«With on-premise contracting, you get a set of unit prices and service levels, you negotiate with a provider, and then that will be a one-off procurement exercise that lasts three to five years. You set the value within that initial negotiation, and everyone can go home and leave it to relatively junior people to execute the contract because the unit prices have protection,» he tells Cloud Pro.

In the cloud, however, decision-making can often be handed down to junior developers and infrastructure managers, but pricing can be very dynamic with complex discount arrangements. «The consequence is that the cost of cloud has actually been more than expected,» Bradley explains.

The challenge of controlling costs comes in two parts. Firstly, can organisations exploit the standard price structures of the big cloud vendors to help them get better value for money? And secondly, can organisations try to negotiate their own ‘special’ deals from cloud vendors, and get better value than the standard price structures offer?

On the first question, KPMG’s Bradley says mature organisations are finding ways to get more bang for their bucks.

In AWS, for example, prices are mostly baked into reserved instances, where users commit to a certain level of compute for a certain period, with incremental discounts in line with how much they commit.

«Mature users on cloud have become quite sophisticated in planning effectively around that,» he says.

Organisations can also get better deals depending on when they reserve computing power. «There’s an element of arbitrage, because it’s a bit like looking for holidays. If you book early, you get a good deal.»

But, in a similar vein, there are bargains to be had in last-minute deals in cloud computing spot markets via companies such as Spotinst and Cloudability, Bradley says.

«And just like a holiday, if you book late and you’re unfussy about where you go, then you can also get a great deal. That’s not something that’s part of the initial negotiation, and what you have to work out is how you can best make use of the economic models the cloud providers have created,» he says.

The second question addresses whether organisations can negotiate away from the standard price structure. It’s possible, but only for the world’s largest corporations such as multinational consumer goods firms and banks, Bradley says.

«If you hit that threshold of scale and you’re talking about really substantial workloads, then you can have a specific negotiation. That does get you a little bit further,» he says.

To lower prices further still, large businesses can propose creative deals with cloud providers. For example, BP, the oil and energy company, has agreed to supply AWS with 170 MW of renewable energy as part of its cloud computing contract.

But to negotiate, organisations have to prepare. Their chances of success depend as much on the measures they put in place internally as they do on their approach to suppliers.

Mike Jette, industry lead for telecoms, media and technology at procurement outsourcing firm GEP says: «In the early days, it was like the wild west. In a lot of organisations, tons of different people were buying cloud services in an uncoordinated fashion, trying to align with their strategic objectives with very little structured governance or procurement. It was just a lot of people trying to say, ‘hey look I moved to the cloud’.»

How organisations manage their cloud consumption is half the challenge in getting more value, he says. «You need to be thoughtful on the buy side, but the management side is really important to maintaining costs and getting value out of the service providers.»

This means understanding how much the organisation is consuming, and how that might vary, he says. «To get leverage [with suppliers] you have to have management and controls in place. The early adopters have gone through this exercise and they’ve taken 30%-plus of the cost out.»

If they go to market with enough volume, there is always room to negotiate, he says. «You need to have a sense of what the estate looks like and where it’s going to grow to, but there’s definitely an opportunity to negotiate. The cloud service providers like to talk about their market share: they’re in the business of buying volume now,» he says.

To get the best deals from suppliers, organisations need to understand and predict the volumes they will require. It can be a thankless task and even goes against some of the advantages of cloud computing, says Matt Yonkovit, chief experience officer at Percona, an independent consulting company that helps move open source databases to the cloud.

Although organisations can create guardrails to try to guide developers to certain platform providers and solutions, many still want the freedom to choose. Meanwhile, the cloud providers offer so many services — as many as 180 from AWS for example – each with a separate pricing structure, that estimates of demand are often inaccurate, he says.

While there are machine learning tools that can help, some organisations want to burst out workloads to support business demand: ecommerce companies supporting Christmas shopping, for example.

Just as important as forecasting demand is to ensure applications and databases are configured to the cloud environment and minimise consumption, he says. «People don’t understand the shared responsibility model, and that causes most of the extra spend. Understand the technology and optimising system can reduce costs.»

The big three cloud providers – which command more than half the market between — may have the upper hand in negotiating with customers. But buyers are strengthening their position by better understand and controlling their demand, exploiting spot markets, and better configuring their technology. Excelling in these areas will build value for cloud buyers.

Cloud communications platform Twilio has revealed it’s powering Be My Eyes, an app to help visually impaired and blind people make their way in the world.

Be My Eyes pairs cameras and video chat to help 178,000 visually impaired people get help from more than three million volunteers. For example, the camera can be pointed at a sign, document, or even food packaging, letting the remote volunteer read it out.

The chat function can also be used with specific companies that support the app, which include Microsoft, Google and Lloyds Banking, making it easier for visually impaired people to get specialist help using their services, be it for banking, shopping, or booking tickets.

«Relying on friends and family for everyday tasks can be taxing on relationships and prevent people with visual impairments from achieving true independence,» said Alexander Hauerslev Jensen, chief commercial officer of Be My Eyes.

The free app has been in use since 2015 across both iPhone and Android. Upon launch, it gained 10,000 volunteers and 1,000 users overnight, suggesting there could be high demand. And as the user base grew, so too did lags in connection time, making the app less useful for those who needed it most.

At the time, Be My Eyes was using multiple providers for video connectivity. Now, it’s switched to Twilio Programmable Video for more stability and higher quality connections, reducing connection times in half. Now, Be My Eyes aims to help everyone within a minute of a request, with 90% of connections made within 30 seconds.

«Be My Eyes is a Twilio-powered community support platform that solves a visually impaired person’s problem in a fraction of the time that it would take via audio,» said Jensen. «When you’re asking for help, a little bit of time can feel like an eternity. Every second we can shave off wait times means more trust, more engagement, and a stronger bond in our community. A 50% reduction in connection time can mean a world of difference for the user and the Twilio platform enables us to achieve this.»

The partnership comes under the remit of Twilio.org, the cloud company’s social enterprise division. «Twilio.org was established to help social impact organisations use the power of communications to create positive change on a global scale, and it’s inspiring to see Be My Eyes doing just that,» said Erin Reilly, chief social impact officer of Twilio.

«Be My Eyes is enabling people with visual impairments to live independent lives, no matter where they reside in the world,» Reilly said. «Their innovative use of Twilio enables Be My Eyes to make sure that their users get help when they need it.»

The Six Nations rugby tournament is expanding its partnership with AWS to add live in-game analytics during this season’s broadcast of the competition.

Fans will be placed «at the tactics board», according to AWS, with five new statistical features that analyse key segments of the game.

This follows on from the 2019 Championships, where AWS technology provided fans with insights into scrummages, play patterns, try origins and team trends with statistics generated by data gathered during the games. This is streamed and analysed by AWS platforms like SageMaker and then delivered as insights back to the live TV broadcasts.

«The introduction of the advanced statistics – powered by AWS – in the 2019 Guinness Six Nations Championship was just the start of how we are planning to change the game of rugby through advanced in-game analytics,» said Ben Morel, CEO of Six Nations Rugby.

«This year will see the introduction of even more engaging and informative stats that bring fans even closer to the action. With these innovations, together with AWS, we are seeking to significantly enhance the viewing experience for all rugby fans by providing them with unique data-led insights.»

These have been developed in partnership with data analytics firm Stats Perform and the Amazon Machine Learning Solutions Lab (as well as SageMaker). One of the first new features will be a heatmap of the pitch that illustrates where most of the action is taking place. This will also «visualise» where a team is turning over possession, according to AWS.

Tackling will also be analysed with machine learning models that will map out the location of success and highlight areas that should be exploited. Fans will also be able to «visit the 22» with insights into how successful a team is at entering the oppositions 22-metre area. This will be calculated for every phase of play, with the aim of gaining deeper insight into how long a team stays in the attacking area, how many opportunities it creates and how many it converts.

When it comes to conversions, kicks from the side of the pitch are extremely difficult and with AWS Kick predictor, machine learning will inform fans just how difficult they are. According to AWS, the calculation happens in real-time, during a break in play while the kicker sets himself up. It takes into account a number of real-time, in-game factors, such as the location of the kick, the period of the game it’s being taken, the current score and if the kicking team is playing home or away.

Additionally, it analyses other historical data, for example, the average success rate of the kicking player in the given field zone, during the Championship and during the player’s entire career.

This is very similar to the NFL Next Gen stats platform that predicts the likely success of passes from a quarterback. It also further expands AWS’ portfolio of sports partnership after its recent deal with the German Bundesliga and its deep ties with Formula 1.

Security firm FireEye has acquired Cloudvisory, aiming to bring cloud visibility tools to its offering.

The buyout was for an undisclosed sum and is FireEye’s seventh acquisition. Last year, the security firm bought Verodin for $250 million (£190.4 million) and in 2014 bought Mandiant for $1 billion.

«Customers need consistent visibility across their public and hybrid cloud environments, as well as containerised workloads,» said Grady Summers, Executive Vice President of Products and Customer Success at FireEye. «Cloudvisory delivers this visibility and allows FireEye to apply controls and best practices based on our frontline knowledge of how attackers operate.»

Cloudvisory’s system was created to provide visibility into network traffic, spot and fix misconfigurations, and keep an eye out for compliance issues. On the security front, it claims to detect, block and quarantine attacks.

Summers notes that’s key, as most companies see security as a leading concern with using the cloud. «Security is top of mind for almost all organisations as they migrate critical workloads to the cloud,» Summers adds. «With the addition of the Cloudvisory technology, FireEye is able to offer a comprehensive, intelligence-led solution to secure today’s hybrid, multi-platform environments.»

The acquisition will see Cloudvisory’s capabilities added to FireEye’s Helix, letting customers see all of their cloud environments from one single dashboard, expanding monitoring and compliance.

“Joining FireEye offers Cloudvisory a unique opportunity to combine our innovative approach to cloud visibility and FireEye’s unrivaled insights into the threat landscape,” said Lisun Kung, Cloudvisory co-founder and Chief Executive Officer prior to the acquisition. “We’re excited by the potential to quickly scale and help more organisations secure their cloud and container workloads.”

Alongside the acquisition, FireEye’s Mandiant division also unveiled a pair of new cloud-focused services.

The first is Cloud Security Assessments, available for Office 365, Azure, AWS and Google Cloud. These will look for common misconfigurations and other issues with cloud that attackers use to slip past security measures. «Through tactical coaching and comprehensive recommendations, organisations achieve increased risk visibility and enhanced functional capabilities,» the company said.

The second new service is Cyber Defense Operations, which offers hands-on support and training to help in-house detection and response take a step up. The process begins with an evaluation to highlight goals and capabilities, such as threat hunting.

Then Mandiant personnel will offer training, analysis and other support within the client’s environment. «Through this process, areas for maturation are identified and pursued, helping to identify and resolve visibility gaps and procedural issues,» the company says.

«Our Cloud Security Assessments and Cyber Defense Operations consulting services are two new offerings to help clients protect their key assets before, during and after an incident,” said Jurgen Kutscher, EVP of Service Delivery at FireEye.