Exploring the evolution of Kubernetes to manage diverse IT workloads

Kubernetes started in 2014. For next two years, the adoption of Kubernetes as container orchestration engine was slow but steady, as compared to its counterparts – Amazon ECS, Apache Mesos, Docker Swarm, GCE et al. After 2016, Kubernetes started creeping into many IT systems that  have wide variety of container workloads and demand higher performance for scheduling, scaling and automation.

This is so as to enable a cloud native approach having a microservices architecture in application deployments. Leading tech giants (AWS, Alibaba, Microsoft Azure, Red Hat) have started new solutions based on Kubernetes and in 2018, they are consolidating to build a de-facto Kubernetes solution which can cover every use case that handles dynamic hyperscale workloads.

Two very recent acquisitions depict how Kubernetes has created a huge impact in IT ecosystem. One is IBM’s Red Hat and VMware’s Heptio acquisition. IBM did not shown the direct interest to target container orchestrations but had eyes on Red Hat’s Kubernetes Based Openshift.

At VMworld Europe 2018, the acquisition of Kubernetes solution firm Heptio by VMware triggered a lot of heat. This acquisition is said to have a significant impact on the data centre ecosystem where Red Hat (IBM) and Google are among the top players. Heptio’s solution will be co-integrated with VMware’s Pivotal Container Services (PKS) to make this as a de-facto Kubernetes standard which will cover maximum data centre use cases from private, multi-cloud and public cloud.

Heptio was formed by ex-Google engineers Joe Beda and Craig McLuckie back in 2016. In its 2 years Heptio captured the eyeballs of industry giants with its offerings and contribution to cloud native technologies based on Kubernetes. Also, Heptio had raised $33.5 million through two funding rounds.

So, the question is why and on which kind of use cases Kubernetes is being used or being tested to use.

Enabling automation and agility in networking with Kubernetes

Leading communication service providers (CSPs) are demonstrating 5G in selected cities. 5G networks will support a wide range of use cases with a lowest possible latency and high bandwidth network. CSPs will need to deploy network services at edge of the network where data is generated by number of digitally connected devices.

To deploy services at the edge of the network and have a control on each point of the network, CSPs will need automated orchestration on each part. What's more, as software containers are being adopted by CSPs to deploy virtual network functions, CSPs will be leveraging cloud native approach by employing microservices based network functions and real time operations by employing CI/CD methodologies. In this scenario, Kubernetes emerged as an enterprise level container management and orchestration tool. Kubernetes brings a number of advantages in this environment.

Jason Hunt wrote in a LinkedIn post that “Kubernetes allows service providers to provision, manage, and scale applications across a cluster. It also allows them to abstract away the infrastructure resources needed by applications. In ONAP’s experience, running on top of Kubernetes, rather than virtual machines, can reduce installation time from hours or weeks to just 20 minutes.” He added that CSPs were utilising mixing of public and private clouds for running network workloads. Kubernetes works well for all types of clouds to handle workloads of any scale.

Other example of Kubernetes utilisation in telecom is the recent release of Nokia CloudBand software for NFV. With this release of CBIS 19, there is support for edge network deployments along with support for containerised workloads and integration of Kubernetes for container management along with OpenStack which will handle virtual machine as well. In the last few years, usage of containers has being discussed within NFV architecture. But this release is one of the first representations of employing containers and container management for handling network functions in NFV infrastructure.

Kubernetes and AI/machine learning

KubeFlow – Managing machine learning stacks: Moving further on managing containers, Kubernetes has evolved to the extent that it is used to manage complex workloads for machine learning applications.

Machine learning applications or systems contain several software components, tools and libraries from different vendors which are all integrated together to process information and generate output. Connecting and deploying all the components and tools require manual efforts which are tedious and takes a fair amount of time. Also, for most of the cases the hardest part is that the machine leaning models are immobile, and require re-architecture while transferring from the development environment to a highly scalable cloud cluster.

To address this concern, Kubernetes introduced open framework KubeFlow which has all machine learning stacks pre-integrated into Kubernetes which will instantiate any project easily, quickly and extensively.

KubeFlow Architecture for ML Stacks

Image source: https://www.kubeflow.org/blog/why_kubeflow/ 

Kubernetes for eCommerce retailer JD.com: Besides the launch of KubeFlow, one interesting application of Kubernetes for AI is JD.com, a Chinese eCommerce retailer, which is managing the world’s largest Kubernetes clusters with more than 20,000 bare metal services in several clusters across data centres in multiple regions.

In an interview with CNCF, Liu Haifeng, chief architect at JD.com, was asked about how Kubernetes is helping JD for AI or big data analytics. He disclosed: “JDOS, our customised and optimised Kubernetes supports a wide range of workloads and applications, including big data and AI. JDOS provides a unified platform for managing both physical servers and virtual machines, including containerised GPUs and delivering big data and deep learning frameworks such as Flink, Spark, Storm, and Tensor Flow as services. By co-scheduling online services and big data and AI computing tasks, we significantly improve resource utilisation and reduce IT costs.”

JD.com is declared as winner in the top end user award by CNCF for its contribution to the cloud native ecosystem.

Managing hardware resources using Kubernetes

Kubernetes can also be used to manage hardware resources like graphics processing units (GPUs) for public cloud deployments. In one of the presentations at KubeCon China this year, Hui Luo, a software engineer at VMware demonstrated how Kubernetes can be used to handle machine learning workloads in private cloud as well.

Summary

As enterprises have started embracing open source technologies in considerable manner to reduce costs, it has been observed that Kubernetes has been evolved from just a container orchestration framework to handling even more complex workloads of different types.

Even though most of the software industry has leaned towards cloud-native, dividing monolithic applications in small services which can scale, managed independently and communicate among themselves through APIs, Kubernetes has become a de facto standard to completely take care of all services residing in containers. A similar mechanism of Kubernetes has been adopted to handle NFV, machine learning, and hardware resources workloads.

Download our eBook to know more about the Kubernetes technology and industry/market insights.

The post Evolvement of Kubernetes to Manage Diverse IT Workloads appeared first on Calsoft Inc. Blog.

https://www.cybersecuritycloudexpo.com/wp-content/uploads/2018/09/cyber-security-world-series-1.pngInterested in hearing industry leaders discuss subjects like this and sharing their experiences and use-cases? Attend the Cyber Security & Cloud Expo World Series with upcoming events in Silicon Valley, London and Amsterdam to learn more.

Samsung is building a data centre in Korea’s DMZ to challenge Azure and AWS


Clare Hopping

30 Nov, 2018

Samsung has announced it’s building a data centre just a mile from the South Korean border with North Korea on an old civilian shooting range called Chuncheon.

The demilitarised zone (DMZ) is pretty close to the site, with military personnel patrolling around 50km away, over the border. But Samsung doesn’t seem to think that’s too concerning and even hopes in the years to come, it’ll be able to open a data centre over the border in North Korea.

The company will use the facility to develop its own cloud computing hub, going head to head with more established rivals Amazon, Google and Microsoft.

The reason Samsung has set its sights on such a dangerous place to build its biggest data centre? The cool air from the mountains surrounding the building will cool the servers, saving on power.

The area is generally a few degrees cooler than capital city Seoul and this, the engineers think, will cut the amount of power needed to keep servers running at their optimum by more than 80%.

“The era of cloud is coming, so if we don’t do it, we’re going to be weeded out of business,” said Kim Ho, vice president of Samsung SDS Co., the division that’s been given the responsibility of developing its cloud business.

Although Samsung already owns data centres around the world, they aren’t designed for customer use – they just house the company’s own data.

“This is not a bad strategy for Samsung, indeed perhaps the only one that they can try right now given the maturity of the market,” David Linthicum, chief cloud strategy officer at Deloitte Consulting told Bloomberg.

“However, they are really in the managed services provider world, with hundreds of companies currently playing in that space. While Samsung should be able to scale better, they need to do some things that are more creative and innovative to stand out.”

Box isn’t losing as much money as expected, and builds customer base


Clare Hopping

30 Nov, 2018

Box has revealed smaller than expected losses and managed to substantially grow its user base, despite rather bleak analyst estimates.

The company posted its quarterly results this week, revealing it now has more than 90,000 paying customers – up from 87,000 last quarter and this has had a significant impact on its profits.

The company posted revenues of $155.9 million for the quarter, up 21% year-on-year, with billings for the period at $155.6 million, an increase of 10% compared to the previous quarter. Nevertheless, Box posted a loss of $39.5 million, namely because if is still focussing on growth rather than consolidation, but its losses were less than the same quarter 12 months ago. 

“Our solution selling strategy continues to gain momentum with strong attach rates for add-on products and large deal growth in the third quarter,” Aaron Levie, co-founder and CEO of Box said.

“With more than 90,000 customers, including BBVA Compass, National Bank of Canada, and Shiseido Company, Box continues to expand its role as a strategic technology partner to power digital transformation for enterprises.”

He added that the cloud company is meeting business demand for a single, open platform for cloud content management, coupled with enterprise-grade security and “powerful workflow capabilities.”

Box is predicting its revenues for the full fiscal year will be between $608.2 million to $609.2 million, much higher than analyst predictions of $607.5 million.

“With more than 40% growth in deals worth more than $100K and our attach rate for add-on products increasing to over 80% of these deals, we are capturing our market opportunity while driving continued leverage for long-term growth,” Dylan Smith, co-founder and CFO of Box added.

AWS re:Invent: A blockchain service for the right market at the right time


Bobby Hellard

30 Nov, 2018

The announcement of blockchain services at AWS re:Invent came as a surprise, but the cloud giant believes it has the right product for the right market at the right time.

Amazon QLDB, a cryptographically verifiable ledger and the Amazon Managed Blockchain, which is a fully managed blockchain service, were unveiled by CEO Andy Jassy during his keynote speech. The company hadn’t shown much interest in the technology before and Jassy said it was because “it hadn’t seen any examples in production that couldn’t be solved by a database”.

Looking at blockchain itself, which is formed from a perpetual list of records, called blocks, linked using cryptography and contains timestamps and transaction data, there are not yet masses of innovative use cases and its original purpose, underpinning cryptocurrency, is not the best example of it as Bitcoin continues to sink.

Then there’s the empty hype, such as the Hdac advert that played out during this year’s world cup, which offered smart home technology powered by the magic of blockchain, without much explanation of how it actually worked.

But according to Philip Moyer, the global director of financial services at AWS, the Amazon QLDB and the Amazon Managed Blockchain service are not empty products following tech trends or just blockchain for the sake of it, they’re what AWS customers have asked for.

“A lot of people would say we are late to the game,” he said. “But we actually think we are finding the right product for the right market at the right time.

“Over 90% of our roadmap, is driven by what customers ask us to do. That’s a really important aspect, we don’t just build science projects, we’re really building the things the customers ask us for.”

As was evident at re:Invent, AWS works with many financial organisations, such as DTDC, insurance firm Guardian Life and Australian National Bank, which announced a long-term cloud partnership with AWS. For Moyer, these large financial organisations supported its work with blockchain.

“They were really excited,” he said. “They offered loads of support, as did Guardian for those announcements for blockchain-as-a-service and also for QLDB.”

“When you deal with very high-value transactions like the financial industry does, having the veracity of that transaction occurred and being able to have traceability of it, especially if you’re a large scale, highly distributed bank around the world, if somebody puts in a credit into your bank account in one place and someone makes a transaction in another place at the precise same time, to be able to resolve those things, QLDB is a really exciting advancement for the financial industry.”

Liverpool FC focuses on user-centric UC in strategic partnership with Mitel


Maggie Holland

26 Nov, 2018

Liverpool FC has signed a multi-year deal with comms firm Mitel as it continues on its quest to boost the customer experience and satisfaction levels.

The idea behind the union is to make use of the cloud and communications platform to ensure fans can interact with the club using whatever medium they choose, whenever suits them best. That also includes a self-service element should fans so wish.

By focusing on Omnichannel, LFC hopes to be able to offer fans a better experience regardless of how they choose to interact with the club. The key reason, LFC CEO Peter Moore says, being because fans are the heart and soul of what makes the club successful.

He stressed the LFC is focused on three key areas: engaging fans, driving growth and investing in technology.

“We couldn’t be happier that you are part of our family,” Moore told a room full of assembled Mitel executives, partners and press.

“We can’t do this [the fan-centric vision] without technology partners. We can do it through technology and with unified communications. I know you’re [Mitel] going to make our lives and our families’ lives better.”

Moore claimed the club has a whopping 771 million fans around the world. However, with a home stadium capacity of just shy of 55,000, most of them will never get to see their team play at Anfield.

“We are powered by a global pulse and that is both a challenge and an opportunity for me… We are by far the most watched team in the world,” he said.

“How do we manage that emotional conflict between the lad from Bootle and the lad from Bangkok who both believe they deserve a ticket?”

The partnership, the financial terms of which have not been disclosed, will also have wider benefits that help enhance efficiency for those working in Anfield stadium itself as well as LFC’s head office, training grounds and retail facilities.

Employees will have the necessary tools at their disposal to work anywhere on any device to communicate with their peers and customers alike.

Moore is not shy about claiming it is the most-watched football club in the world and while the stats themselves may be contested the sentiment cannot.

“We are the number one tourist attraction in the city. The Beatles are big… So, one of the challenges I have as CEO is how do I take this incredibly unique football club [to the next level]?” He said.

“Football means more to the people in this city than it does anywhere else in the world. We take enormous pride in the fact that we have received – for the third year in a row – the Visit Football awards for the best stadium to visit… For those who work for the club, it means more than just a job. It’s not a 9-5. It is something very special.”

The partnership will involve the integration of Salesforce CRM to provide one single view of the fan so their comms history with the club is visible to agents regardless of the channel utilised. This, LFC hopes, will position it to offer a much more tailored fan experience.

“We have both evolved and both changed. You have to evolve, you have to grow, you have to change and you have to be prepared to bet on the future,” said Graham Bevington, Mitel’s executive vice president of business development.

“Customer experience is at the forefront of everything Liverpool does. And it’s at the forefront of everything we think about.”

Moore added: “I couldn’t be more excited about the partnership as to what you bring to what we need.”

Everton FC ‘lucky’ to have SureCloud’s data protection suite in place for GDPR kickoff


Keumars Afifi-Sabet

29 Nov, 2018

With each passing season, the footballing industry seems increasingly detached from the realities most businesses face. This is underlined by extortionate sums exchanged between clubs, players, and supporters on a daily basis; not to mention a counterintuitive penchant for amassing mountains of debt to drive footballing success.

But the General Data Protection Regulation (GDPR) has affected every organisation large and small in the same way, with the sporting world no exception. Just as with startups, massive football clubs must comply with demands to bring data practices in line with modern standards – from appointing a Data Protection Officer (DPO), to training staff.

For Everton FC, this process entailed leaving it to as late as January to get things started; putting faith into the all-in-one, modular GDPR suite developed by SureCloud. Maintaining a database of 32,000 season ticket holders, 60,000 registered fans, 360 employees, players and agents as well as third-party suppliers, through Excel spreadsheets, is a laborious task, with or without GDPR. But a changing landscape spurred the Premier League stalwart into re-examining how it managed data and processed GDPR’s additional demands.

Everton was still using a series of spreadsheets to manage its data within the football club, community outreach programme, and pre-school, as soon as January 2018. This is when the club hired Ian Garratt as its DPO to single-handedly oversee the transition to SureCloud. But the platform wasn’t initially up to the standards expected, Garratt tells IT Pro, and needed a significant amount of custom tailoring to suit the club’s data protection needs.

“I hadn’t worked with a full management system before. I’d looked at OneTrust which is an equivalent, very template-based, and then what I’d worked on was spreadsheets, Excel and ones that we’d built in-house, at my old employer.

“So I went into SureCloud with a long list of tailoring. Most of them were only quite minor but there was quite a few.”

Although compliant by 25 May, implementation took so long that Everton considered hanging onto its spreadsheet-based system as the deadline fast-approached. It would’ve posed a massive headache given how slow searching through spreadsheets would have been, not to mention handling internal and external queries taking a great deal longer compared with SureCloud’s touted greater functionality.

“By the time we started the discussions it was probably late January, early February,” Garratt continues. “Knowing we had to get all of the data mapping done, and in place before May, we were considering whether or not we had to do that spreadsheet-based, and import it into SureCloud afterwards, just because of the timing.

“But we were lucky in that they got it all done for us.”

Bringing the human touch for higher-quality data

Before joining Everton Garratt was information governance manager with the Southport and Ormskirk Hospital NHS Trust in Wales. Using spreadsheets in this post meant he could slot straight into the role with Everton, but would have to quickly adapt to the platform.

Fresh to the club, and sole member of the data management team, he had to gain a wider understanding of what data each department held, and their internal processes. He devised an approach to overcome these challenges all at once, sending questionnaires to each department, and inputting the answers into SureCloud himself. But the key, Garratt says, lied in working through them with people one-on-one, to personally guide them through what needed to be sent back.

Instead of giving everybody within the organisation their own SureCloud login, Garratt decided to limit access to the club’s data to three individuals: himself, the director of risk, and head of IT. They also decided against setting up email reminders and alerts, despite the fact this approach takes longer. But, why? 

“I think just from my experience you get better quality input if you actually sit down with people and do it with them, rather than sending an email alert and asking them to update something themselves when they’re not specialists in the area,” he said. 

A matter of when, not if

During implementation, Garratt oversaw the migration of data from on-prem infrastructure to the cloud. But assurances over security and the decision to go with SureCloud in the first place rested with the club and were a matter for before he joined.

“Football clubs are getting targeted more and more often. Certainly, from a backup point of view, I feel happier with it being hosted rather than living on a server,” Garratt says.

“The risk is always there. Cyber security is now on our risk register, and I think always will be. I’d expect it to be on every company’s register nowadays. The other threat I suppose is malicious staff.”

“If we did have an incident,” he explains: “We should straight away be able to see what the data types are, what the fields are, the volume, what systems there are, and what associated systems. So we’d be able to get a really good idea of the scale of the incident, and we’d be able to get that very quickly.”

And what about minor incidents, such as supporters’ email addresses inadvertently leaking due to a lapse in staff concentration, as struck West Ham FC in August?

“If that happened with us, any mass marketing should go up to our marketing department, and they’ve got a system that sends them all as individual emails – all personalised – so you don’t need to do it as BCC.

“If we had a lot of emails like that going out – and it’s largely to Hotmail or Gmail sort-of accounts, we’ve got systems that would flag them, quarantine them, then either myself or someone from the IT department would be able to review them… I imagine West Ham has probably got the same sort of system, and it just, for whatever reason, didn’t go through that system.”

Revisiting supplier contracts proves the biggest GDPR hurdle

The most difficult part of Everton’s wider compliance journey involved re-examining the several existing contracts with the club’s many suppliers. Although just a handful of suppliers have access to personal data held by the club, reaching out to renegotiate a GDPR-compliant addendum proved the toughest aspect for Garratt.

“The data mapping is what took the most time, but that’s because there was a lot of it. But getting contracts in place with suppliers with the GDPR-standard terms has been the hardest bit of the gameplay.

“They would’ve had general data protection and confidentiality terms, but GDPR stipulated a wider scope for what the contracts had to include – even things like assistance with impact assessments, acceptance of audits by us and by the ICO, and breach reporting.”

By using SureCloud, Garratt says, the club was able to list all their third parties, and a subsection of those who were charged with handling the club’s data, as well as whether they were based in an EU country, or a non-EU country with or without data adequacy.

But it was no substitute for the hard graft the club’s had to put in to ensure GDPR-compliant terms were included in each contract individually, with each supplier providing their own template, and seeking to consult with their own legal teams respectively.

AWS Re: Invent: AWS adds more programme languages to Lambda


Bobby Hellard

29 Nov, 2018

AWS is giving developers the choice to integrate their prefered programming languages into Lambda Runtime API and Lambda Layers.

These two new AWS Lambda features enable developers to build custom runtimes and share and manage common code between functions.

Making the announcements on stage in Las Vegas at the cloud giant’s re:Invent conference, CTO Werner Vogels told the crowd: “You asked for it, so we’ve given it to you.”

It turns out, what they wanted was more options with Lambda, more flexibility to use the code they are au fait with. AWS Lambda is an event-driven serverless computing platform the company launched in 2014. It was designed to simplify the building of smaller, on-demand applications that are responsive to events and new information.

Up until now, the platform only supported some programming languages, such as Node.js, Python, Java, Go and NET Core, which had previously limited developers with other language preferences.

The Runtime API for AWS Lambda defines a standardised HTTP-based specification which codifies how Lambda and a function’s runtime communicate. It enables users to build custom runtimes that integrate with Lambda to execute functions in response to events. With the Runtime API, AWS said that developers can use binaries or shell scripts, and their own choice of programming languages and language versions within the Lambda tools.

“We decided to change course and give you the ability to start bringing your own language to Lamda,” said Vogels. “We are launching today, custom runtimes for Landa, where you can bring your own execution environment.

“Now there is no limitation anymore for what kind of language you can use to do serverless development in.”

Lambda functions in a serverless application typically share common dependencies such as SDKs, frameworks, and now runtimes. With layers, AWS said users can centrally manage common components across multiple functions enabling better code reuse.

This announcement swiftly followed news that Ruby, the Japanese object-orientated, general purpose programme language has been made available on AWS Lambda functions.

How IT services are adapting to ongoing digital transformation

As CIOs and CTOs shift their focus to digital transformation projects and the launch of new digital business offerings, demand for traditional IT services has evolved. Worldwide revenues for IT services and business services totaled $506 billion in the first half of 2018 (1H18) – that's an increase of 4 percent year over year, according to the latest market study by International Data Corporation (IDC).

During 1H18, it was a mixed picture for tier-one global outsourcers and systems integrators headquartered in developed countries. Most remained flat or declined slightly. But this was partially offset by stronger performances by two large global vendors, who returned to double-digit growth.

IT services market development

Indian IT services firms still outpace the U.S. and European counterparts, but their growth slowed from a year ago, continuing their 2H17 deceleration. While most large Indian vendors continued to grow at rates in the low single digits to high teens, it was offset by a few vendors' sharp slowdowns.

Project-oriented revenues grew by 5.2 percent in 1H18 to $191 billion, followed by 3.6 percent growth for managed services and 2.7 percent for support services. The above-the-market growth in project-oriented markets was mostly led by business consulting and application development markets with growth rates of 7.5 percent and 6.5 percent, respectively.

Most major management consulting firms still posted strong earnings in 2018, although growth rates cooled slightly: business consultants still extract more value in digital transformation. However, the market is now being driven by enterprise buyers who are executing their digital growth agenda.

In outsourcing, revenues grew 3.6 percent to $238 million in 1H18. Application-related managed services revenues (hosted and on-premise application management) outpaced infrastructure and business process outsourcing.

On the infrastructure side, while hosting infrastructure services revenue accelerated to 7.2 percent growth in 1H18, mostly due to cloud adoption, IT Outsourcing (ITO) – still almost twice as large a market and mostly big buyers and vendors – declined by 1.5 percent, largely chipped away by cloud cannibalization across all regions.

On a geographic basis, the United States grew by 4.3 percent, slightly higher than the market rate, while Western Europe grew only by 2.6 percent. IDC expects Western European services revenues to be stable but structurally weaker than North America. IDC forecasts the region to grow below 3 percent annually in the coming years.

In emerging markets, Latin America, Asia-Pacific (excluding Japan) (APeJ), and Central & Eastern Europe led in growth. In Latin America, most major economies are turning the corner despite problems in Argentina and Venezuela.

In APeJ, Australia saw its growth scaled back slightly to 3.8 percent in 1H18, from 4.3 percent in 1H17. The largest market, China, trimmed its growth rate to just 7.2 percent, down from the 8 percent to 9 percent during the last two or three years.

Outlook for IT services in emerging markets

So far in 2018, the weaker growth in China and Australia was partially offset by faster growth from other emerging markets in APeJ. IDC expects this trend to continue. Governments will fund large digital transformation initiatives and a better investment outlook will also drive IT spending.

"Steady growth in the IT services market is being driven by continued demand for digital solutions across the regions," said Lisa Nagamine, research manager at IDC. "But during 2018, as well as most of 2017, it is really the Americas and cloud-related services that are having the largest impact on revenue worldwide."

https://www.cybersecuritycloudexpo.com/wp-content/uploads/2018/09/cyber-security-world-series-1.pngInterested in hearing industry leaders discuss subjects like this and sharing their experiences and use-cases? Attend the Cyber Security & Cloud Expo World Series with upcoming events in Silicon Valley, London and Amsterdam to learn more.

AWS re:Invent: Andy Jassy announces ML Marketplace, Blockchain and more


Bobby Hellard

29 Nov, 2018

At AWS re:Invent on Wednesday, Andy Jassy said that he had “a few things to share”. But, over the course of his two-hour keynote, the CEO announced a barrage of new services and capabilities from blockchain to machine learning.

The boss of the world’s biggest cloud computing company has had a busy few days at the annual event in Las Vegas. From making announcements to meeting many of the developers and partners that have flocked to Sin City, Jassy has put himself about and offered plenty of information on everything he’s revealed.

And, there was a ridiculous amount of them…

Machine Learning Marketplace

Available now, the AWS Marketplace for Machine Learning includes over 150 algorithms and models, with more, said to be coming every day, that can be deployed directly to Amazon SageMaker. Its a giant algorithm hub for developers to find and offer machine learning models for the benefit of all.

For Gavin Jackson, the managing director of AWS UK and Ireland, this was the biggest news of the day and also a very good example of an underlying theme of this year’s re:Invent. It’s about catering to both those who can and those who can’t.

“The big announcement, I thought, was the Machine Learning Marketplace,” said Jackson. “Because while SageMaker is a good use of existing training models that you can just plug straight into your application, customers who are building their own training models and algorithms for applications can just look at a much wider s

et of use cases that are available in the marketplace and then just plug them in so they don’t have to build them for themselves.

“At the same time, those that do have data scientists and are building their own algorithms and training models can plug them into the marketplace and monetise it. It’s kind of a marketplace for those that can and those that can’t and everybody wins in the end. It just accelerates the progress of machine learning artificial intelligence over time.”

Blockchain

Unexpectedly, the CEO announced two new services to help companies manage business transactions for blockchain, starting with Amazon Managed Blockchain. Jassy said that this new service makes it easy to create and manage scalable blockchain networks using the popular, open source Ethereum and Hyperledger Fabric frameworks.

It’s run from an AWS Management Console, where customers can set up a blockchain network that can span multiple AWS accounts and scale to support thousands of applications and millions of transactions

The second blockchain offering, Amazon QLDB, is a transparent, immutable, and cryptographically verifiable ledger for applications that need a central, trusted authority to provide a permanent and complete record of transactions, such as supply chain, financial, manufacturing, insurance, and HR. This option is for customers who want to build applications where multiple parties can execute transactions without the need for a trusted, central authority.

According to Jassy, the company was asked why they had not shown any previous interest in blockchain, despite many of its customers and partners using the technology.

“We just hadn’t seen that many blockchain examples in production or that couldn’t easily be solved by database,” said Jassy. “People just assumed that meant we didn’t think blockchain was important or that we wouldn’t build a blockchain service. We just didn’t understand what the real customer need was.”

Data

Also announced during the keynote were new services for automating data applications and detailed guidance to help customers build faster on AWS services.

The AWS Control Tower is a cloud interface that allows users to govern multiple AWS workloads, particularly for companies migrating to the cloud. Jassy said it offers pre-packaged governance rules for security, operations, and compliance, which customers can apply enterprise-wide or to groups of accounts to enforce policies or detect violations.

Jumping on the data lake bandwagon, the company is now offering AWS Lake Formation, which will run on Amazon S3. Data lakes are storage systems that source data from multiple locations and stores it in files for technologies like machine learning. The AWS version is said to automate and optimise data, reducing the data management burden for customers.

Hybrid

There was some noise before the event that AWS would address hybrid cloud systems and it has confirmed AWS Outposts, which is a fully managed and configurable compute and storage racks service built with AWS-designed hardware. It’s a service that allows customers to run on-premise computing and storage functions while connecting to other AWS services in the cloud.

These outposts come in two variants; first, an extension of the VMware Cloud on AWS service that runs on AWS Outposts and second, AWS Outposts that allow customers to run on-premise ccomputing and storage that uses the same native AWS APIs used in the AWS cloud

AWS looks to redefine hybrid cloud at re:Invent 2018 – plus make big moves in blockchain

Andy Jassy, CEO of Amazon Web Services, noted during his keynote at re:Invent today that on a recent business trip, a senior AWS executive found themselves sat next to an exec from an unnamed competitor. They pulled up a presentation deck which – Jassy paraphrasing – noted its product strategy was to ‘look at what AWS launches, and then move as fast as possible to get something vaguely in that area.’

So with that story in mind, customers and partners, media and analysts, and perhaps a few cloud vendors as well, sat down in Las Vegas to absorb AWS’ updates. Database, blockchain and machine learning all ended up getting a significant airing – but the last announcement, on hybrid cloud, stole the show.

Compute and storage

Jassy started out by putting down the various numbers which asserted AWS’ dominance in the cloud market, both in terms of market share and absolute growth. Regular readers of this publication will be more than aware of these numbers; Synergy Research, for instance, said at last count that AWS led across all geographies and was ‘in a league of its own.’ It was here that Jassy threw in his obligatory Oracle dig. “There are some providers who don’t have enough revenue to show up here,” he said, who only appear when they try and grab attention for themselves. You can guess the rest.

Yet with so much to get through, competitor-bashing was relatively brief. The first segment underlined the breadth of the AWS portfolio. Take containers as one example. Customers can use ECS for a container service most tightly integrated with AWS, EKS to use Kubernetes in a managed service, or Fargate for a more ad hoc approach.

With that in mind, a new storage class was unveiled. Glacier Deep Archive is aimed as being the lowest cost storage available in the cloud, at less than one tenth of a cent per gigabyte per month – or $1 per terabyte per month. Naturally, it is aimed at the coldest possible usage; enterprises who are managing data on ancient tape. “You have to be out of your mind to manage your own tape moving forward, and this will be here for you in 2019,” said Jassy.

One of the key themes throughout the presentation focused around how developers and technology builders needed the right tools to get the job done. Indeed, practically every announcement AWS made was in response to specific customer pain points. In terms of file systems, Amazon already had Linux workloads covered with its FSx storage product, but added Windows Server and Lustre to it. While the former is an evident attempt to lure Windows users, the latter is of particular interest, focusing on high performance computing (HPC). Both products have the HIPAA, ISO and PCI-DSS security standards out of the box.

The second generation of builders

To some extent, the second stanza contradicted the first. With AWS’ huge depth in its products, complexity can of course be seen as an issue. Indeed, the cloud management space, with vendors such as CloudHealthTech – acquired by VMware, more on whom later – is testament to that. Jassy noted how a second type of builder was emerging; mostly found in enterprise organisations who wanted more of a guiding hand in how to set up products. AWS Control Tower and AWS Security Hub – the former enabling customers to set up a landing zone or environment easily, the latter centrally managing security and compliance across an AWS environment – were launched with that in mind.

One of the more eye-opening statistics was that more than 10,000 data lakes were being built on top of AWS S3. As Jassy noted, the data lake may be the in-vogue concept for 2018. “People realise there is significant value in moving all that disparate data and making it much easier by consolidating it into a data lake to enable you to run analytics and machine learning,” he said. “But if you’ve tried to build a data lake, it’s hard.”

AWS Lake Formation, therefore, was launched in order for organisations to take their data out of silos in days, rather than months, with Amazon offering to do the heavy lifting, from cleaning to partitioning, to indexing and cataloguing. “This is a step level change in how easy it’s going to be for all of you to have data lakes,” said Jassy.

“It’s obvious what The Beatles were singing about,” Jassy joked as the strains of Blackbird came to a close. “Database freedom!” Indeed, they ‘were only waiting for this moment to be free’, and this was where some of the harshest criticism came in – reserved for the legacy, relational database players. “People are sick of it, and now they have choice,” he said.

Again, the breadth of the portfolio was noted with three customer examples. For simpler iterations there is DynamoDB; Lyft uses it to coordinate passenger information and GPS coordinates. Airbnb uses ElastiCache for its single sign on (SSO) to be firing with microsecond latency. Nike used Neptune to build an app whereby athletes, their followers, and all their interests correlated. But if you have tables that fluctuate due to seasonality, or spikiness, then it’s a matter of guesswork knowing how to scale. The snappily-named Dynamo DB Read/Write Capacity On Demand aims to take care of that.

Perhaps the biggest cheer of the keynote came when Amazon Timestream was announced. The database can process trillions of events per day at one tenth of the cost of relational databases, and is focused on IoT and edge computing.

Blockchain, machine learning, and old friends

When it came to blockchain and machine learning, both saw leaps forward. Amazon Quantum Ledger Database (QLDB) – cited by Amazon CTO Werner Vogels as one of his favourite announcements – aims to solve the problem of providing a ledger with a trusted entity without having to surf through complicated functionality provided by blockchain frameworks. The second, Amazon Managed Blockchain, does what it says on the tin, supporting both Hyperledger Fabric and Ethereum. The company has certainly come a long way from this time last year when it said it wasn’t especially interested in the technology.

As far as machine learning went, Amazon SageMaker Ground Truth, which aims to help label data more accurately, and AWS Inferentia, a high-performance machine learning inference chip, stood out. Yet in one of the few nods to previous business, Ross Brawn, managing director of motor sports at Formula 1, took to the stage to expand on AWS’ partnership with the sporting giant first announced in July.

ML had been promised as a cornerstone at the time, and Brawn duly delivered. ‘F1 Insights Powered By AWS’ had been launched to some extent this season, providing more data as well as predictions on what may happen. This is being extended next season by further integrating telemetry data to predict performance and race strategy, as well as using HPC to simulate environments where slipstreams don’t knock out as much equilibrium of the preceding car, leading to closer racing. “These are insights the teams have always had – but we’re going to bring them out to the fans to show them what’s happening,” said Brawn.  

Mindful perhaps that looking back rather than forward may have turned into a habit, the next segment was also the last. Pat Gelsinger, CEO of VMware, went on stage – much as Jassy had done during VMworld keynotes – to help launch AWS Outposts. In some way, the best – or perhaps most shocking – had been left till last. The company claims to deliver a ‘truly consistent hybrid experience’ by bringing AWS services, infrastructure and operating models to ‘virtually any’ on-premises facility. This can be achieved either as AWS-native, or running VMware Cloud on AWS.

“The breadth and scale of the AWS platform now, combined with the sheer velocity of new feature releases means that few firms on the planet are moving faster,” said Nick McQuire, VP enterprise at CCS Insight. “It bodes ominously for Microsoft and Google in the high stakes cloud wars.”

Picture credits: AWS/Screenshot

https://www.cybersecuritycloudexpo.com/wp-content/uploads/2018/09/cyber-security-world-series-1.pngInterested in hearing industry leaders discuss subjects like this and sharing their experiences and use-cases? Attend the Cyber Security & Cloud Expo World Series with upcoming events in Silicon Valley, London and Amsterdam to learn more.

The cloud news categorized.