Amazon Web Services (AWS) has announced the availability of C5 instances, aimed at more compute-intensive workloads for the EC2 cloud.

The C5 instances – three from the sharp end in Amazon’s compute class, behind G2, P2 and F1 – were introduced as the newest iteration back in November last year at the company’s Re:Invent show. The C5 promises 3.0 GHz Intel Xeon Scalable processors and double the vCPU and memory capacity – up to 72 vCPUs and 144 gibibytes of memory – when compared with previous C4 instances.

Applications the C5 instances are better equipped to handle include batch processing, distributed analytics, high performance computing (HPC), ad serving, video encoding, and multiplayer gaming. The instances will be available in three regions; US East (N. Virginia), US West (Oregon), and EU (Ireland), with support for additional regions in the pipeline.

Alongside this, AWS dropped a few customer names into the mix. One customer is particularly well-known – having been analysed by this publication on several occasions – and is arguably the poster child for AWS itself. Netflix said it saw up to a 140% performance improvement in industry standard CPU benchmarks compared with C4.

For the high performance computing side, Alces Flight offers researchers on demand HPC clusters, or ‘self-service supercomputers’ in minutes. The company, a member of the AWS Marketplace, said C5 had a ‘direct benefit’ for its user base ‘on both price and performance dimensions.’

The press materials also made mention of a new hypervisor which AWS is rolling out for C5 instances to ‘allow applications to use practically all of the compute and memory resources of a server, delivering reduced cost and even better performance.’

According to this page, accessed by CloudTech earlier today (screenshot), and first spotted by The Register, the new hypervisor for Amazon EC2 “is built on core Linux Kernel-based Virtual Machine (KVM) technology, but does not include general purpose operating system components.”

KVM’s best known user in this sphere is Google. In January this year, the search giant issued a blog post advocating seven methods they use to security harden the KVM hypervisor. As Ariel Maislos, CEO of Stratoscale, pointed out in this publication last year, AWS has long been partnered with Xen for its hypervisor needs.

The FAQ page added that all new instance types will ‘eventually’ use the new EC2 hypervisor, but for now some new instance types will use Xen ‘depending on the requirements of the platform.’ Yet, as The Register reports, references to KVM have been disappearing from the company’s pages.

The cloud-first enterprise is gaining in prominence, through multi-cloud strategies, losing data centres, and adopting cloud-native infrastructures.

That is the key finding from a new report released by hybrid cloud IT operations management provider OpsRamp. The study, which polled IT leaders in companies with 500 or more employees, also found that public cloud services are grabbing a bigger share of IT budgets, and that security – again – is the primary reason for reticent companies sitting it out.

While the areas the paper covers have been commonly reported in this publication, it is interesting to see the trends all in one place. More than half of respondents say they have been using public cloud for more than three years, with 7% saying they have done for more than seven. Despite this, only 29% of those polled said their level of cloud adoption was ‘mature’, compared with 50% for ‘developing’ and 21% for ‘emerging’.

When it came to benefits of cloud-native infrastructure, scalability and flexibility, cited by 62% of those polled, came out on top, ahead of reduced capital investments (47%) and consumption-based pricing models (47%). More than half (53%) added that 30% to 50% of IT budgets would be cloud-based in the near future, while an additional 27% said more than half their IT budget would be.

“The survey results are consistent with what we’re hearing from customers and partners,” said Varma Kunaparaju, OpsRamp co-founder and CEO. “Cloud is becoming a bigger part of their IT portfolio, they’re likely to use more than one cloud platform, and oversight and management of cloud services is paramount.

“We expect these trends to accelerate over time, as the cost, scalability and flexibility advantages of cloud services become even more obvious.”

As far back as 2015, this publication wrote that ‘multi-cloud was the new holy grail of cloud computing’. The survey results showed that this was essentially a reality today; three quarters of respondents said they expect to work with different cloud providers for their business needs, with Microsoft Azure – not for the first time – the most popular, ahead of Amazon Web Services and Google.

You can read the full report here.

Companies are harnessing data in ways we once associated with science fiction. Analysts have access to a plethora of visualization and reporting tools, but considering the vast amount of data businesses collect and limitations of CPUs, end users are forced to design their structures and systems with limitations. Until now. As the cloud toolkit to analyze data has evolved, GPUs have stepped in to massively parallel SQL, visualization and machine learning.

read more

Revenues from the global public cloud services market stand at $63.2 billion (£48.2bn) with the market growing 28.6% year over year in the first half of 2017, according to IDC.

The largest segment, by some distance, remains software as a service (SaaS), totalling $43.4bn comprising 68.7% of the market share. This is down from the first half of 2016, where SaaS saw 71.8% market share, yet the segment still grew 22.9% year over year.

The fastest growing segment, whilst remaining the smallest, was platform as a service (PaaS), growing 50.2% year over year and hitting $8.6bn, at 13.6% of overall market share. IDC attributes the continued rise down to the ‘rapid’ adoption of container technology, giving developers additional tools to accelerate application development and deployment.

Infrastructure as a service (IaaS) comprised 17.8% of market share in 1H17 totalling $11.2bn in revenues and at a yearly growth of 38.1%.

In terms of regional growth areas, Asia Pacific saw the highest regional growth in the most recent figures at 38.9%, according to IDC, with a major contributing factor being strong public cloud spending in China, which was at 55.6% year over year growth.

“Businesses now think ‘cloud first’ when it comes to their IT strategy and software footprint, since the benefits of cloud are clear and have been broadly demonstrated in most industries,” said Eric Newmark, program vice president for IDC’s SaaS, enterprise applications and industry cloud research practices. “Many companies have picked the low-hanging fruit, in terms of apps that could be easily moved to the cloud, and are now evaluating the migration of their next set of larger strategic systems to a SaaS model.

“These projects, coupled with companies’ efforts to embrace digital transformation, will continue to fuel strong SaaS growth.”

As this publication reported earlier this week, IDC recently published its Worldwide CIO Agenda 2018 predictions, which offer the 10 most important shifts taking place in IT organisations over the next three years. Among the more interesting predictions include the forecast that the majority of CIOs who ‘cross the digital divide’ by 2020 will reap the rewards of their efforts to become digital business leaders for their enterprises.

Big data analysis can provide continuous business intelligence to managers, enabling sophisticated monitoring of current activities, and helping them to make smarter, faster decisions based on fact data and hidden trends and patterns. But while managers may think it’s cool, rushing to implement “a big data analytics” program can get you into trouble.  There are many architectures, tools, and algorithms to sort through, and you’ll need to manage stakeholder expectations. Software engineers deal with these kinds of problems regularly, so it’s helpful, to take their approach in building data analytics solutions.

Management questions

First it is very important to manage customer expectations. Software engineers would seek to discover the true intent for the analytics program, by asking the following questions: What is the real problem that is being solved? Is analytics really the answer? Is it a technology problem or a political problem in disguise? Assuming the problem can be solved by analytics, there may be constraints that need to be addressed. For example, is the required data scattered across databases and in many places in the organization? Are the legacy systems up to the task? Are there data governance issues related to ownership, and what are the privacy, security and trust issues?  What risks is the manager willing to take to relax any constraints?  Buy-in from all stakeholders in the organization is also going to be needed — without this you may face political difficulties from uncooperative colleagues.

Finally, you need to ask: what is the budget for this project? While many of the tools enabling data analytics are open source, commercial tools may be needed for some aspects. There could be significant costs to purchasing or leasing hardware, hosting (depending on the architectural model) and providing release time to set up the system and for training and support.

Software architecture

Software engineers seek generalizable solutions and compatibility across the enterprise and industry. In seeking an appropriate, and reusable architecture for data analytics, the focus is on efficient, cost-effective utilization and sharing of resources. Architectural decisions include whether to query in batch mode or real-time, to host on site or use third party provisioning, or to use some combination of these.

Hadoop is frequently a first choice as a data analytics platform, but there are many alternatives. While “large” organizations may have the hardware infrastructure to collect, process and analyze massive amounts of data, smaller organizations may not. Whatever architecture is chosen, problems may arise when the database lacks in situ analytics, the analytics are too slow, or can’t scale in terms of data load time. Some platforms may choke on the vast amounts of data that are frequently updated from live feeds from social media, Websites, mobile applications and even sensors in cyberphysical systems.

Tools

Data analytics can involve vast amounts of data possibly petabytes worth. This data will likely come from different platforms, data stores and sources and data quality can vary greatly. Data can come from multiple internal and external sources including email text, sales records, Web server logs, internet clickstream data, mobile phone data and even sensor data from devices connected to the Internet of Things. This data will be varyingly structured, unstructured, or semi-structured, and much of it will be redundant and inconsistent. Special tools, then, will be needed to clean, compress, format and visualize this data before, during and after analysis.

There are hundreds of tools of various intent to choose from; many are open source. Choosing the right tool is an important software engineering problem. Considerations include: compatibility with the operating environment, support provided (if any) and programming language needed to interface to the tool. Building data analytics solutions will likely involve more than one programming language, typically C, Python, Java, SQL and others. Is your development team prepared for this challenge?

These tools will also need to be configured to the data set and analytical problem. But seamless integration of these tools into a one button solution for managers isn’t always easy. Finally, you need to consider how reconfigurable the solution will be for different kinds of related problems. A single purpose analytics solution isn’t going to be cost effective.  

Analytics

Finding the right machine learning algorithm to apply to data sets in search of patterns and relationships for situational analysis and for predictive analytics is a significant challenge.  There are numerous desktop data miners, deep learning libraries and cognitive toolkits, to choose from, but deep learning using multilayer neural networks is computationally expensive. Failure to consider performance and throughput at full scale can lead to customer dissatisfaction.

Fortunately, software engineers and related professionals are working together to solve these kinds of problems. For more information visit the IEEE Big Data Initiative and NIST Big Data Working Group.

Bert Loomis was a visionary. This general session will highlight how Bert Loomis and people like him inspire us to build great things with small inventions. In their general session at 19th Cloud Expo, Harold Hannon, Architect at IBM Bluemix, and Michael O’Neill, Strategic Business Development at Nvidia, discussed the accelerating pace of AI development and how IBM Cloud and NVIDIA are partnering to bring AI capabilities to «every day,» on-demand. They also reviewed two «free infrastructure» programs available to startups and innovators.

read more