DevOps is under attack because developers don’t want to mess with infrastructure. They will happily own their code into production, but want to use platforms instead of raw automation. That’s changing the landscape that we understand as DevOps with both architecture concepts (CloudNative) and process redefinition (SRE).
Rob Hirschfeld’s recent work in Kubernetes operations has led to the conclusion that containers and related platforms have changed the way we should be thinking about DevOps and controlling infrastructure. The rise of Site Reliability Engineering (SRE) is part of that redefinition of operations vs development roles in organizations.
Monthly Archives: March 2018
Dropbox IPO highlights: The company’s journey to going public and its future direction
Dropbox’s IPO filing, confirmed late last week, makes for very interesting reading. For a start, it confirmed the rumours swirling around last month, as well as ensure the company won’t be making the next Forbes Cloud 100 list, having been ranked the second biggest private cloud firm two years running.
Yet more importantly, it showcases the company’s recent direction of travel – and where it is heading. Under the ‘recent initiative’ section of the S-1 filing, the storage provider outlined its initiatives to ‘improve the efficiency of the infrastructure that supports [its] platform’.
“These efforts include an initiative that focused on migrating the vast majority of user data stored on the infrastructure of third-party service providers to our own lower cost, custom-built infrastructure in co-location facilities that we directly lease and operate,” the company wrote. The process, called ‘infrastructure optimisation’, which is still ongoing in places, saw a reduction in cost of revenue, an increase in gross margins, and an improvement in free cash flow.
Speaking of margins, Dropbox posted revenues of $1.1 billion in 2017, a 31% increase from the year before, and seeing a gross profit of $737 million. Yet the ‘infrastructure optimisation’ process the company has undertaken is moving away from Amazon Web Services (AWS) to its own solution. This would be known as ‘Magic Pocket’. In March of that year, Akhil Gupta, Dropbox vice president of engineering, explained that the process of building its own dedicated storage infrastructure had taken two and a half years and had resulted in more than 90% of users’ data on the custom-built product.
This, as Shira Ovide wrote for Bloomberg, was what made Dropbox a viable IPO company today. “Without exaggeration, the shift away from cloud computing is one of the biggest reasons Dropbox is able to go public now,” Ovide wrote.
The filing also provided industry watchers with various insights into how the company saw its product going forward. The company’s mission, as unveiled last year, is to “unleash the world’s creative energy by designing a more enlightened way of working”, while the letter from co-founders Drew Houston and Arash Ferdowsi noted the importance of machine learning in improving the search and visibility of workspaces. “Over time, machine intelligence will allow Dropbox to better understand both you and your team,” the letter explained.
Houston and Ferdowsi also added a touch of braggadocio to proceedings, citing its journey to a billion-dollar revenue run rate, the fastest SaaS company to do so. “While we’re at scale, we can still move quickly,” the letter read. “We have a lot less baggage than the incumbents. The legacy office suites have had a good run, but they were designed for a world where the most important thing you did was print something out. There’s a reason why BlackBerry didn’t come up with the iPhone… sometimes it’s better to start fresh.”
For quarterly revenues, the quarter ending December 31 2017 saw revenue of $305m, compared with $238m this time last year at an increase of 28%.
Microsoft reveals new AI and cloud-powered health initiatives
2 Mar, 2018
Microsoft has unveiled a range of initiatives to boost its presence in the healthcare and science verticals, with their foundations in AI and the cloud.
As an extension of the company’s Healthcare NExT scheme, which aims to boost innovation in the healthcare sector, this latest announcement will see the company’s groundwork come into fruition.
“The explosion of data, incredible advances in computational biology, genomics and medical imaging have created vast amounts of data well beyond the ability of humans to comprehend,” Peter Lee, corporate vice president of Microsoft’s AI and Research division said.
“Clinicians and care teams are yearning to swivel their chairs from the computer and pay more attention to the patient, yet still they spend two-thirds of their time interacting with burdensome IT systems. And healthcare organizations everywhere still struggle with the lack of operational and regulatory clarity in managing and analyzing the datasets that they are generating every day.”
Microsoft Genomics is the first launch. It offers researchers and data scientists a cloud platform on which they can process genomics using their data-rich workloads. The company has partnered with St. Jude Childrens Hospital to pilot the technology for research into childhood diseases.
Microsoft Azure Security and Compliance Blueprint: HIPAA/HITRUST – Health Data & AI is an end-to-end app development engine built to help healthcare organisations move to the cloud, with security and compliance at the centre. The company has also launched Microsoft 365 Huddle Solution Templates to help health teams collaborate more effectively as part of Microsoft Teams.
AI Network for Eyecare has been expanded and become AI Network for Healthcare, growing to include cardiology thanks to a partnership with Apollo Hospitals in India, while Microsoft’s Project Empower MD is being developed in partnership with UPMC to create an AI-powered system to help with patient diagnosis via learning from physician/patient conversations.
“Our mission at Microsoft is to empower every person and organization to achieve more, and with that in mind, our ambition is that innovators will be able to use AI and the cloud to unlock biological insight and break data from silos for a truly personal understanding of human health and in turn, enable better access to care, lower costs and improved outcomes,” Lee added.
New figures show staggering capex spending of hyperscale cloud providers
Hyperscale cloud and data centre operators spent $22 billion on capital expenditure (capex) in the final quarter of 2017 making the total for the year almost $75bn, according to the latest figures from Synergy Research.
Google, Microsoft, Amazon, Apple and Facebook comprise the top five spenders of the 24 major cloud and internet service firms analysed in Q417, according to the data, with growth ‘particularly strong’ at Amazon and Facebook. Alibaba, IBM, Oracle, SAP and Tencent made up the top 10.
A large amount of this spend has gone on building and expanding data centre operations. In December 2016, when the total number of hyperscale data centres passed 300, Synergy predicted this number to surpass 400 by 2018 – a forecast which came in ahead of schedule.
Synergy added capex at Alibaba more than doubled last year – as this publication has regularly reported, the company has driven cloud ambitions with recent focus on European expansion – while growth of Oracle and SAP, also no slouches in this department, was above average.
“Over the last four years we have seen many companies try and fail to compete with the leading cloud providers. The capex analysis emphasises the biggest reason why those cloud providers are so difficult to challenge,” said John Dinsdale, a chief analyst and research director at Synergy Research.
“Can you afford to pump at least a billion dollars a quarter into your data centre capex budget? If you can’t, then your ability to meaningfully compete with the market leaders is severely limited,” Dinsdale added.
“Of course factors other than capex are at play, but the basic financial table stakes are enormous.”
Oracle’s Chatbot, #AI and Mobility with Suhas Uliyar | @ExpoDX #ChatBots #IoT
I am excited to share an interview I recorded with Oracle’s digital, AI, chatbot and mobility expert Suhas Uliyar. In this interview we discuss the meaning of ambient human interfaces, the technology stack that enables chatbots, the power of interfaces that you don’t have to learn, and we learn that algorithms haven’t change that much in 25 years. I learned a lot and hope you will to! Enjoy!
Microsoft still won’t hand over private email data to the US
1 Mar, 2018
Microsoft still won’t hand over private email data to the US that sits at the focus of a long-running court case with the US government.
The case, which has been ongoing since 2013, revolves around information related to a US criminal case that’s held in Microsoft’s Dublin data centre. Despite the fact the data resides in the Republic of Ireland, the US Department of Justice (DoJ) has demanded the company hand it over, as it’s an American entity. The suspects in the case are also believed to be American.
In January 2017, a US appeals court rejected a government appeal to rethink its denial of the attempts to get Microsoft to hand over the Outlook.com email, which has something to do with a drug trafficking prosecution.
The EU intervened in the case in December to ensure the US Department of Justice (DoJ) understands European data laws. The EU claimed that because the data would have to be moved from Europe to the US, it would come under EU data protection laws.
At the time, the European Commission submitted an amicus brief – information filed by non-litigants to give the court a better understanding of a matter – to help it make a decision about whether the data should be handed over or not.
This week, however, liberal justices Sonia Sotomayor and Ruth Bader Ginsburg wondered why Congress isn’t regulating “this brave new world” of cloud storage, rather than expecting the nation’s top court to interpret the legality of a warrant obtained under the dusty old 1986 Stored Communications Act.
“[In 1986,] no one ever heard of clouds. This kind of storage didn’t exist… If Congress takes a look at this, realising that much time and… innovation has occurred since 1986, it can write a statute that takes account of various interests,” said Justice Ginsburg in an oral argument filling.
“Wouldn’t it be wiser to say, ‘Let’s leave things as they are. If Congress wants to regulate this ‘brave new world’ let them do it’?”
Microsoft’s lawyer, E. Joshua Rosenkranz, retorted: “If people want to break the law and put their emails outside the reach of the US government, they simply wouldn’t use Microsoft.”
The cloud of yesterday, today, and tomorrow – and the influence of emerging technologies
Cloud computing in the form we understand today started more than 10 years ago, with the launch of Amazon Web Services (AWS). This was the first commercially viable option for businesses to store data in the cloud rather than on-premise, and acted as a shared service for anyone connecting to the platform.
Early stage cloud computing was certainly more technical than it is now; however, no more than managing a data centre – something that IT departments at the time were well used to. Anyone who was able to successfully start up virtual engines was able to set up a cloud environment.
Cloud started off by offering much more basic services – network, storage and compute. Over the last 10 years, we have seen the volume of data stored in the cloud grow exponentially with businesses frequently dealing with petabytes of data, compared to the gigabytes of yesteryear. With data exploding around us, cloud services have had to learn to be much more efficient.
The cloud of today
As data volumes grew and users grew, cloud providers began to offer more cloud-based value-added services. Analytics and machine learning capabilities that are housed in the cloud are frequently offered alongside other business objective oriented services. As a result, the range of people purchasing cloud services has expanded and is no longer confined to the IT department.
This is because one of the great advantages of cloud and SaaS services is how easy they are to set up and how flexible and scalable they are. Business users without extensive IT knowledge became able to utilise services previously only accessible to the IT department. Of course with this came a rise in “shadow IT” and while being able to access cloud based business tools were very helpful for achieving business objectives and accessing new services that could not run on legacy IT systems, it did open up businesses to greater data privacy risks.
Today, the IT and business functions increasingly collaborate to implement the most up-to-date, data-driven technologies to solve real business needs, making “shadow IT” less of a problem. By working together, IT and business are able to offer smarter responses to business needs. The business function is better able to define the requirements of their cloud solutions, while the IT department has more flexibility to implement and test different technologies to find the best solution.
An important upside of this is that it allows the IT department to keep a better overview of where data is stored with third party services like Salesforce. To ensure this is implemented throughout the business, the IT department needs to take on an educational role within the organisation – teaching business users about the implications of new data privacy laws and safe and compliant ways to use data within the business. Considering the rapidly approaching GDPR deadline this is good news for organisations as the regulation will mandate much stricter approaches to data protection.
This strategy is being led from the top down, with new data protection officers and C-suite positions like the CDO (chief digital officer) and CTO (chief technology officer) straddling the IT and business function. This is helping redefine the IT department as a department for technological creativity and innovation, rather than simply focusing on solution implementation.
The cloud of tomorrow
With GDPR coming into effect in May 2018, the cloud will need to evolve and adapt. Increasingly, there will be more security services attached to the cloud, as well as greater oversight around what data is stored and where it is. This will be essential for changes like the right to be forgotten, where a person can ask a business to delete all personal data relating to them.
For businesses, this will require significant re-architecting in the cloud environment to increase the ability to analyse and discover data across the storage landscape. The cloud is ready to implement these changes, but it will require a change of mindset within organisations. Cloud usage cannot just be determined on the basis of features and costs, but also on ensuring that the data is stored in a controlled, managed, compliant environment.
In the next five to 10 years, as the volume of data continues to expand exponentially, there will be an increasing need to align this data in terms of format and quality. Organisations will need to be able to pull together multiple data-streams across multiple cloud environments into combined, high-quality insights. This is where an open-source, vendor neutral management layer will become crucial to help organisations bridge the gap between their vast data reserves and the insights offered by machine learning and AI technologies. All of this will contribute to a future where businesses can use data stored in the cloud to provide predictive analytics for the business – such as predicting load requirements for peak shopping days, or market fluctuations to prepare investors.
Google’s Hangouts Chat is now generally available through G Suite
1 Mar, 2018
Hangouts Chat, Google’s answer to highly popular collaboration tool Slack, has been taken out of its beta phase and made generally available to all users through the G Suite platform.
Both Hangouts Chat and its video streaming platform, Hangouts Meet, were revealed at Google Cloud Next conference in March last year, but while the latter was released immediately to the public, its chat app has so far only been available as a private test version.
The newly released service works in essentially the same way as Slack or Microsoft Teams, offering a means for users to engage in group or private chats, only within the G Suite environment. As you might expect, the app works closely alongside Google’s host of other services, including the option to launch video chats through Hangouts Meet, share content through Drive, and collaborate through the Docs app. It also supports up to 28 languages and can hold up to 8,000 members in each chat room.
Much like Slack, Google has also opened up the platform for those wanting to develop bots, with RingCentral, Salesforce, Kayak, Trello, and Xero, to name a few, having already released AI plugins for the service.
The problem for Google is that the collaboration market has become a fiercely contested arena, and those organisations that have already adopted existing services could be reluctant to make the switch given the disruption that may cause.
Much like Microsoft’s Teams app and its integrations with Office 365, Google will be relying on the fact Hangouts Chat is built into the G Suite platform and will be able to take advantage of all the tools on offer, including its built-in security, all available through a single subscription.
In February the company announced it was upgrading the hardware it offers as part of its Hangouts Meet service, including a new camera that allows users to accommodate rooms of up to 20 people, and a better quality mic system that can be daisy-chained together to speakers in different rooms.
Hangouts Chat will be slowly rolled out to G Suite customers over the next seven days. Alongside the rollout, Google announced it’s bringing Drive’s Quick Access feature to Docs, an AI tool that recommends files based on activity and information found in your other documents.
The company also said that the Calendar app will soon get a similar AI-powered recommendation tool to help book meeting rooms based on your building, floor you work on, and booking history, a feature that’s expected to arrive in the next few months.
Main image: Shutterstock, body image courtesy of Google
A roundup of machine learning forecasts and market estimates for 2018
- Machine learning patents grew at a 34% compound annual growth rate (CAGR) between 2013 and 2017, the third-fastest growing category of all patents granted
- International Data Corporation (IDC) forecasts that spending on AI and ML will grow from $12bn in 2017 to $57.6bn by 2021
- Deloitte Global predicts the number of machine learning pilots and implementations will double in 2018 compared to 2017, and double again by 2020
These and many other fascinating insights are from the latest series of machine learning market forecasts, market estimates, and projections.
Machine learning’s potential impact across many of the world’s most data-prolific industries continues to fuel venture capital investment, private equity (PE) funding, mergers, and acquisitions all focused on winning the race of Intellectual Property (IP) and patents in this field. One of the fastest growing areas of machine learning IP is the development of custom chipsets. Deloitte Global is predicting up to 800K machine learning chips will be in use across global data centers this year.
Enterprises are increasing their research, investment, and piloting of machine learning programs in 2018. And while the methodologies all vary across the many sources of forecasts, market estimates, and projections, all reflect how machine learning is improving the acuity and insights of companies on how to grow faster and more profitably. Key takeaways from the collection of machine learning market forecasts, market estimates and projections include the following:
Within the business intelligence (BI) & analytics market, data science platforms that support machine learning are predicted to grow at a 13% CAGR through 2021
Data science platforms will outperform the broader BI & analytics market, which is predicted to grow at an 8% CAGR in the same period. Data Science platforms will grow in value from $3B in 2017 to $4.8B in 2021. Source: An Investors’ Guide to Artificial Intelligence, J.P. Morgan. November 27, 2017 (110 pp., PDF, no opt-in).
Machine learning patents grew at a 34% CAGR between 2013 and 2017, the third-fastest growing category of all patents granted
IBM, Microsoft, Google, LinkedIn, Facebook, Intel, and Fujitsu were the seven biggest ML patent producers in 2017. Source: IFI Claims Patent Services (Patent Analytics) 8 Fastest Growing Technologies SlideShare Presentation.
61% of organizations most frequently picked machine learning / artificial intelligence as their company’s most significant data initiative for next year
Of those respondent organizations indicating they actively use Machine Learning (ML) and Artificial Intelligence (AI), 58% percent indicated they ran models in production. Source: 2018 Outlook: Machine Learning and Artificial Intelligence, A Survey of 1,600+ Data Professionals (14 pp., PDF, no opt-in).
Tech market leaders including Amazon, Apple, Google, Tesla, and Microsoft are leading their industry sectors by a wide margin in machine learning (ML) and AI investment
Each is designing ML into future-generation products and using ML and AI to improve customer experiences and improve the efficiency of selling channels. Source: Will You Embrace AI Fast Enough? AT Kearney, January 2018.
SAS, IBM, and SAP lead the predictive analytics and machine learning market based on 23 evaluation criteria applied to 14 vendors by Forrester in 2017
Forrester predicts the Predictive Analytics & Machine Learning (PAML) market will grow at a 21% CAGR through 2021 as evidenced by the increase in client inquiries and purchasing activity they are seeing with clients. Source: Data Science Association, Predictive Analytics & Machine Learning Vendors, 2017 and The Forrester Wave
: Predictive Analytics And Machine Learning Solutions, Q1 2017 courtesy of SAP.
Deloitte Global predicts the number of machine learning pilots and implementations will double in 2018 compared to 2017, and double again by 2020
Factors driving the increasing pace of ML pilots include more pervasive support of Application Program Interfaces (APIs), automating data science tasks, reducing the need for training data, accelerating training and greater insight into explaining results. Source: Deloitte Global Predictions 2018 Infographics.
60% of organizations at varying stages of machine learning adoption, with nearly half (45%) saying the technology has led to more extensive data analysis & insights
35% can complete faster data analysis and increased the speed of insight, delivering greater acuity to their organizations. 35% are also finding that machine learning is enhancing their R&D capabilities for next-generation products. Source: Google & MIT Technology Review study: Machine Learning: The New Proving Ground for Competitive Advantage (10 pp., PDF, no opt-in).
McKinsey estimates that total annual external investment in AI was between $8B to $12B in 2016, with machine learning attracting nearly 60% of that investment
McKinsey estimates that total annual external investment in AI was between $8B to $12B in 2016, with machine learning attracting nearly 60% of that investment. Robotics and speech recognition are two of the most popular investment areas. Investors are most favoring machine learning startups due to quickness code-based start-ups have at scaling up to include new features fast. Software-based machine learning startups are preferred over their more cost-intensive machine-based robotics counterparts that often don’t have their software counterparts do. As a result of these factors and more, Corporate M&A is soaring in this area. The following graphic illustrates the distribution of external investments by category from the study. Source: McKinsey Global Institute Study, Artificial Intelligence, The Next Digital Frontier (80 pp., PDF, free, no opt-in).
Deloitte Global is predicting machine learning chips used in data centres will grow from a 100k to 200k run rate in 2016 to 800k this year
At least 25% of these will be Field Programmable Gate Arrays (FPGA) and Application Specific Integrated Circuits (ASICs). Deloitte found the Total Available Market (TAM) for Machine Learning (ML) Accelerator technologies could potentially reach $26bn by 2020. Source: Deloitte Global Predictions 2018.
Amazon is relying on machine learning to improve customer experiences in key areas of their business including product recommendations, substitute product prediction, fraud detection, meta-data validation and knowledge acquisition
For additional details, please see the presentation, Machine Learning At Amazon, Amazon Web Services (47 pp., PDF no opt-in).
Other statistics
- International Data Corporation (IDC) forecasts that spending on AI and ML will grow from $12B in 2017 to $57.6B by 2021. Source: Machine learning: things are getting intense. Deloitte (6 pp., PDF. No opt-in)
- The global machine learning market is expected to grow from $1.41B in 2017 to $8.81B by 2022, attaining a 44.1% compound annual growth rate (CAGR). Factors contributing to the market’s global rapid growth include new technologies that excel at data aggregation, integration and analysis, and more scalable cloud platforms. Source: Machine Learning Market – Global Forecast to 2022 – Market Overview & Industry Trends.
- Worldwide revenues for cognitive and artificial intelligence (AI) systems will increase from $12.5B in 2017 to more than $46B in 2020. Source: Worldwide Spending on Cognitive and Artificial Intelligence Systems Forecast to Reach $12.5 Billion This Year, According to New IDC Spending Guide.
Sources of market data on machine learning:
2018 Outlook: Machine Learning and Artificial Intelligence, A Survey of 1,600+ Data Professionals. MEMSQL. (14 pp., PDF, no opt-in)
Advice for applying Machine Learning, Andrew Ng, Stanford University. (30 pp., PDF, no opt-in)
An Executive’s Guide to Machine Learning, McKinsey Quarterly. June 2015
An Investors’ Guide to Artificial Intelligence, J.P. Morgan. November 27, 2017 (110 pp., PDF, no opt-in)
Artificial intelligence and machine learning in financial services Market developments and financial stability implications, Financial Stability Board. (45 pp., PDF, no opt-in)
Big Data and AI Strategies Machine Learning and Alternative Data Approach to Investing, J.P. Morgan. (280 pp., PDF. No opt-in).
Google & MIT Technology Review study: Machine Learning: The New Proving Ground for Competitive Advantage (10 pp., PDF, no opt-in).
Hitting the accelerator: the next generation of machine-learning chips, Deloitte. (6 pp., PDF, no opt-in).
How Do Machines Learn? Algorithms are the Key to Machine Learning. Booz Allen Hamilton. (Infographic)
IBM Predicts Demand For Data Scientists Will Soar 28% By 2020, Forbes. May 13, 2017
Machine Learning At Amazon, Amazon Web Services (47 pp., PDF no opt-in).
Machine Learning Evolution (infographic). PwC. April 17, 2017 Machine learning: things are getting intense. Deloitte (6 pp., PDF. No opt-in)
Machine Learning: The Power and Promise Of Computers That Learn By Example. The Royal Society’s Machine Learning Project (128 pp., PDF, no opt-in)
McKinsey Global Institute Study, Artificial Intelligence, The Next Digital Frontier (80 pp., PDF, free, no opt-in)
McKinsey’s State Of Machine Learning And AI, 2017, Forbes, July 9, 2017
Predictions 2017: Artificial Intelligence Will Drive The Insights Revolution. Forrester, November 2, 2016 (9 pp., PDF, no opt-in)
Risks And Rewards: Scenarios around the economic impact of machine learning, The Economist Intelligence Unit. (80 pp., PDF, no opt-in)
Smartening up with Artificial Intelligence (AI) – What’s in it for Germany and its Industrial Sector? Digital/McKinsey & Company. (52 pp., PDF, no opt-in)
So What Is Machine Learning Anyway? Business Insider. Nov. 23, 2017
The 10 Most Innovative Companies In AI/Machine Learning 2017, Wired
The Business Impact and Use Cases for Artificial Intelligence. Gartner (28 pp., PDF, no opt-in)
The Build-Or-Buy Dilemma In AI, Boston Consulting Group. January 4, 2018.
The Next Generation of Medicine: Artificial Intelligence and Machine Learning, TM Capital (25 pp., PDF, free, opt-in)
The Roadmap to Enterprise AI, Rage Networks Brief based on Gartner research. (17 pp., PDF, no opt-in)
Will You Embrace AI Fast Enough? AT Kearney. January 2018