Companies are harnessing data in ways we once associated with science fiction. Analysts have access to a plethora of visualization and reporting tools, but considering the vast amount of data businesses collect and limitations of CPUs, end users are forced to design their structures and systems with limitations. Until now. As the cloud toolkit to analyze data has evolved, GPUs have stepped in to massively parallel SQL, visualization and machine learning.

read more

Revenues from the global public cloud services market stand at $63.2 billion (£48.2bn) with the market growing 28.6% year over year in the first half of 2017, according to IDC.

The largest segment, by some distance, remains software as a service (SaaS), totalling $43.4bn comprising 68.7% of the market share. This is down from the first half of 2016, where SaaS saw 71.8% market share, yet the segment still grew 22.9% year over year.

The fastest growing segment, whilst remaining the smallest, was platform as a service (PaaS), growing 50.2% year over year and hitting $8.6bn, at 13.6% of overall market share. IDC attributes the continued rise down to the ‘rapid’ adoption of container technology, giving developers additional tools to accelerate application development and deployment.

Infrastructure as a service (IaaS) comprised 17.8% of market share in 1H17 totalling $11.2bn in revenues and at a yearly growth of 38.1%.

In terms of regional growth areas, Asia Pacific saw the highest regional growth in the most recent figures at 38.9%, according to IDC, with a major contributing factor being strong public cloud spending in China, which was at 55.6% year over year growth.

“Businesses now think ‘cloud first’ when it comes to their IT strategy and software footprint, since the benefits of cloud are clear and have been broadly demonstrated in most industries,” said Eric Newmark, program vice president for IDC’s SaaS, enterprise applications and industry cloud research practices. “Many companies have picked the low-hanging fruit, in terms of apps that could be easily moved to the cloud, and are now evaluating the migration of their next set of larger strategic systems to a SaaS model.

“These projects, coupled with companies’ efforts to embrace digital transformation, will continue to fuel strong SaaS growth.”

As this publication reported earlier this week, IDC recently published its Worldwide CIO Agenda 2018 predictions, which offer the 10 most important shifts taking place in IT organisations over the next three years. Among the more interesting predictions include the forecast that the majority of CIOs who ‘cross the digital divide’ by 2020 will reap the rewards of their efforts to become digital business leaders for their enterprises.

Big data analysis can provide continuous business intelligence to managers, enabling sophisticated monitoring of current activities, and helping them to make smarter, faster decisions based on fact data and hidden trends and patterns. But while managers may think it’s cool, rushing to implement “a big data analytics” program can get you into trouble.  There are many architectures, tools, and algorithms to sort through, and you’ll need to manage stakeholder expectations. Software engineers deal with these kinds of problems regularly, so it’s helpful, to take their approach in building data analytics solutions.

Management questions

First it is very important to manage customer expectations. Software engineers would seek to discover the true intent for the analytics program, by asking the following questions: What is the real problem that is being solved? Is analytics really the answer? Is it a technology problem or a political problem in disguise? Assuming the problem can be solved by analytics, there may be constraints that need to be addressed. For example, is the required data scattered across databases and in many places in the organization? Are the legacy systems up to the task? Are there data governance issues related to ownership, and what are the privacy, security and trust issues?  What risks is the manager willing to take to relax any constraints?  Buy-in from all stakeholders in the organization is also going to be needed — without this you may face political difficulties from uncooperative colleagues.

Finally, you need to ask: what is the budget for this project? While many of the tools enabling data analytics are open source, commercial tools may be needed for some aspects. There could be significant costs to purchasing or leasing hardware, hosting (depending on the architectural model) and providing release time to set up the system and for training and support.

Software architecture

Software engineers seek generalizable solutions and compatibility across the enterprise and industry. In seeking an appropriate, and reusable architecture for data analytics, the focus is on efficient, cost-effective utilization and sharing of resources. Architectural decisions include whether to query in batch mode or real-time, to host on site or use third party provisioning, or to use some combination of these.

Hadoop is frequently a first choice as a data analytics platform, but there are many alternatives. While “large” organizations may have the hardware infrastructure to collect, process and analyze massive amounts of data, smaller organizations may not. Whatever architecture is chosen, problems may arise when the database lacks in situ analytics, the analytics are too slow, or can’t scale in terms of data load time. Some platforms may choke on the vast amounts of data that are frequently updated from live feeds from social media, Websites, mobile applications and even sensors in cyberphysical systems.

Tools

Data analytics can involve vast amounts of data possibly petabytes worth. This data will likely come from different platforms, data stores and sources and data quality can vary greatly. Data can come from multiple internal and external sources including email text, sales records, Web server logs, internet clickstream data, mobile phone data and even sensor data from devices connected to the Internet of Things. This data will be varyingly structured, unstructured, or semi-structured, and much of it will be redundant and inconsistent. Special tools, then, will be needed to clean, compress, format and visualize this data before, during and after analysis.

There are hundreds of tools of various intent to choose from; many are open source. Choosing the right tool is an important software engineering problem. Considerations include: compatibility with the operating environment, support provided (if any) and programming language needed to interface to the tool. Building data analytics solutions will likely involve more than one programming language, typically C, Python, Java, SQL and others. Is your development team prepared for this challenge?

These tools will also need to be configured to the data set and analytical problem. But seamless integration of these tools into a one button solution for managers isn’t always easy. Finally, you need to consider how reconfigurable the solution will be for different kinds of related problems. A single purpose analytics solution isn’t going to be cost effective.  

Analytics

Finding the right machine learning algorithm to apply to data sets in search of patterns and relationships for situational analysis and for predictive analytics is a significant challenge.  There are numerous desktop data miners, deep learning libraries and cognitive toolkits, to choose from, but deep learning using multilayer neural networks is computationally expensive. Failure to consider performance and throughput at full scale can lead to customer dissatisfaction.

Fortunately, software engineers and related professionals are working together to solve these kinds of problems. For more information visit the IEEE Big Data Initiative and NIST Big Data Working Group.

Bert Loomis was a visionary. This general session will highlight how Bert Loomis and people like him inspire us to build great things with small inventions. In their general session at 19th Cloud Expo, Harold Hannon, Architect at IBM Bluemix, and Michael O’Neill, Strategic Business Development at Nvidia, discussed the accelerating pace of AI development and how IBM Cloud and NVIDIA are partnering to bring AI capabilities to «every day,» on-demand. They also reviewed two «free infrastructure» programs available to startups and innovators.

read more

Become the disruptor of your industry's status quo, or face the consequences. To succeed, CEOs need to know the answer to a key question in 2018 and beyond. Does our CIO and/or CTO have what it takes to contribute to a meaningful and substantive digital transformation agenda?

As digitally-fuelled disruptors reshape industries, the clear mandate for every enterprise is to re-imagine itself to compete in the increasingly digital economy that's platform-powered and ecosystem-enabled.

To help CIOs and CTOs through this period of multiplied innovation, International Data Corporation (IDC) published its Worldwide CIO Agenda 2018 Predictions.

Top 10 strategic IT planning changes

They outline IDC's vision for the ten most important shifts that will happen in IT organizations over the next 36 months, and will guide IT executives in the formation of their three-year strategic IT plan.

According to the IDC assessment, lines are being drawn that separate industry laggards from "digital-native enterprises" that can harness the power of technology to accelerate their business development.

In the context of continuous emergence, IDC asserts today's business technology environments must adapt at an accelerated pace to the scale, scope, and speed of progressive digital transformation.

"For CIOs and senior IT executives, the challenge is to think and operate like a digital-native enterprise in the face of the emergence of platforms, innovation accelerators, machine learning, augmented skills, micro-personalization, new partnerships, and new relationships," said Serge Findling, vice president at IDC.

The IDC predictions for the Worldwide CIO Agenda are:

• By 2018, 75 percent of CIOs will put experiential engagement, data monetisation, or digital business at scale at the top of their agenda.
• Through 2019, dragged down by conflicting digital transformation imperatives, ineffective technology innovation, cloud infrastructure transition, and underfunded end of life core systems, 75% of CIOs and their enterprises will fail to meet all of their digital objectives.
• By 2020, 60% of the CIOs who have crossed the digital divide will prevail in C-suite turmoil and competition to become digital business leaders for their enterprises.
• By 2019, 60% of CIOs will complete infrastructure and application replatforming using cloud, mobile, and DevOps, clearing the deck for accelerated enterprise digital transformation.
• By 2019, 60% of IT organisations will deploy DX platforms supporting new customer- and ecosystem-facing business models.
• By 2019, 75% of CIOs will refocus cybersecurity around authentication and trust to manage business risks, initiating the retirement of systems that cannot ensure data protection.
• By 2020, 40% of CIOs will leverage vision- and mission-driven leadership to inspire and empower their organisations to create digital transformation capabilities.
• Recognising the failure of existing IT governance and the need for a shared digital transformation vision; by 2020, 40% of CIOs will adopt new digital governance models to accelerate innovation and speed.
• By 2018, 70% of CIOs will take agility to the next level, gearing up to a product model using design thinking and DevOps.
• By 2020, 60% of CIOs will implement an IT business model and culture that shifts focus from IT projects to digitally-oriented products.

"As the new digital economy emerges from disruption, CIOs are seeing their last opportunity to cross the digital divide and earn their right to play in the next phase," concluded Findling.

Meet the cloud’s newest strategic partners: Salesforce and Google. The two companies announced a deal at Dreamforce late yesterday, with Google Cloud becoming Salesforce’s preferred public cloud provider alongside new product integrations.

There were four primary integrations announced. Salesforce Lightning for Gmail enables users to surface Salesforce CRM data in Gmail, as well as customer interactions from Gmail directly within Salesforce. Lightning for Sheets allows the Google spreadsheet tool to be embedded anywhere in Salesforce, while Quip – the word processing and productivity app bought by Salesforce last year – will be able to integrated with Google Drive and Calendar. Salesforce will also integrate with Hangouts.

“Our partnership with Google represents the best of both worlds for our customers,” said Salesforce CEO Marc Benioff in a statement, “There has never been an easier way for companies to run their entire business in the cloud – from productivity apps, email and analytics, to sales, service and marketing apps, this partnership will help make our customers smarter and more productive.”

“Our teams are working very closely to develop new integrations that will connect Salesforce CRM with G Suite to offer the only cloud-native collaboration platform of its kind,” wrote Nan Boden, Google Cloud head of global technology partners in a blog post.

“We hope this partnership enables more companies to take advantage of the cloud and that the combined solutions will provide an unmatched experience for customers,” added Boden.  

Salesforce naturally has plenty of other strategic partners in place. In March, the company signed up with IBM in what was described as a ‘landmark global strategic partnership’ focusing around artificial intelligence (AI), an area which Benioff is certainly tuned into. At IBM’s InterConnect event in Las Vegas that month, Benioff joined IBM CEO Ginni Rometty on stage to discuss their shared initiatives.

The company signed a deal with Amazon Web Services (AWS) last year stating the Seattle giant as its preferred public cloud infrastructure provider. According to Business Insider, Salesforce will still work with AWS in some capacity but Google has ‘taken some of its territory.’

For Google’s part, this represents another win in a year laden with them. According to financial results back in July, the company tripled the number of its big cloud deals – ranked as $0.5 million or above – year over year. Among its more recent customers, as announced at Google’s Next conference in San Francisco in March, include Verizon, Colgate-Palmolive, and eBay – three companies listed in the most recent Fortune 500.