Category Archives: Guest Post

Data center infrastructure management, EMEA, Eurozone, Guest Post, Hosting, London

Euro Data Centre Viewpoint: 2013 to be a Year of Growth, Uncertainty

Guest Post by Roger Keenan, Managing Director of City Lifeline

The data centre industry forms part of the global economy and, as such; it is subject to the same macro-economic trends as every other industry.  For 2013, those continue to be dominated by uncertainty and fear.  The gorilla of course, is the on-going problem in the Eurozone.  This time last year, many commentators predicted that this would come to a head in 2012, with either the central monetary authorities accepting fiscal union and central control across the Eurozone, or the Eurozone starting to break up.  In the event, neither happened and the situation remains unresolved and will continue to drive uncertainty in 2013.

One major uncertainty has been resolved with a convincing win for Barack Obama in the US presidential elections and the removal of the possibility of a lurch to the right.  However, the “fiscal cliff” remains and will cause a massive contraction in the US economy, and hence the world economy, if it goes ahead at the end of 2012.  For the UK, predictions are that interest rates will stay low for the next two to three years as the banks continue to rebuild their strengths at the expense of everyone else.

So the macro-economic environment within which the data centre industry operates is likely to stay uncertain and fearful in 2013.  Companies have massive cash reserves, but they choose to continue to build them rather than invest.  Decision making cycles in 2013 are likely to be as they are now – slow.  Companies will not invest in new project unless they have the confidence that their customers will buy, and their customers think the same and so the cycle goes round.

At a more specific industry level, the on-going trend towards commoditisation of infrastructure is likely to continue.  Whereas data centres five years ago were specific and unique, new entrants to the market have made data centre capacity more available than it was and driven up technical standards.  Older facilities have upgraded to match new builds, which ultimately benefits the industry and its customers.  The new builds and rebuilds are of varying quality and veracity, with some being excellent, however, others are claiming tier levels and other standards which are simply not true or claiming to be in central London whilst actually being somewhere else – perhaps following the example of London Southend Airport?  Even in a more commoditised market, quality, connectivity, accessibility and service still stand out and well-run established data centres will always be first choice for informed customers.

The next part of the consolidation process is probably networks; new entrants are coming into a market where prices continue to fall at a dizzying rate.  There is no end of small new entrants to the marketplace, some of which will succeed and some of which will fall by the wayside.  At the larger end, consolidation continues.  In City Lifeline’s central London data centre alone, Abovenet has become Zayo (and consequently moved from the very top of everyone’s list to the very bottom, possibly not causing joy in Abovenet’s marketing department), Cable and Wireless/Thus has become part of Vodafone, PacketExchange has become part of GT-T and Global Crossing has become part of Level 3.

Data Centre Infrastucture Management (DCIM) systems may establish themselves more in 2013.  DCIM was predicted to have a massive impact, with Gartner stating publicly in 2010 that penetration would be 60% by 2014.  In the event, penetration at the end of 2012 is only 1%.  DCIM is hard and laborious to implement but it offers serious benefits to larger organisations in terms of the management of their physical assets, power, space and cooling and can quickly repay its investment in answering the basic question “how many servers can I have for the capacity I am paying for”.  DCIM deserves more success than it has had to date, and perhaps 2013 will be the year it takes off.

Power densities will continue to increase in 2013.  Five years ago, many racks drew 2KW (8 amps).  Now 8 amp racks are becoming unusual and 16 amps racks are the norm.  Five years ago 7KW racks (about 30 amps) were unusual, now they are common, and 20KW racks are starting to appear.  The trend to higher and higher performance and power densities will continue.

The data centre industry continues to grow, driven by the move to Cloud.  By the end of 2013, an estimated 23% of all data centre space will be in commercial colocation operations.  The leading market segments are likely to be Telecoms and Media, with 24%, Healthcare and Education, with 21% and Public Sector, also with 21%.  In-house data centre capacity is likely to continue to decrease and the commercial colocation market to grow, even in spite of the uncertain macro-economic environment.

Roger Keenan, Managing Director of City Lifeline

 Roger Keenan joined City Lifeline, a leading carrier neutral colocation data centre in Central London, as managing director in 2005.  His main responsibilities are to oversee the management of all business and marketing strategies and profitability. Prior to City Lifeline, Roger was general manager at Trafficmaster plc, where he fully established Trafficmaster’s German operations and successfully managed the $30 million acquisition of Teletrac Inc in California, becoming its first post-acquisition Chief Executive.

Cloudify, Guest Post, High Tech Security, Larry Ellison, OpenStack, Steve Wozniak, Wozniak

Woz is Worried About “Everything Going to the Cloud” — the Real Issue is Giving Up Control

Guest Post By Nati Shalom, CTO and Founder of GigaSpaces

In a recent article, Steve Wozniak, who co-founded Apple with the late Steve Jobs, predicted “horrible problems” in the coming years as cloud-based computing takes hold. 

“I really worry about everything going to the cloud,”.. “I think it’s going to be horrendous. I think there are going to be a lot of horrible problems in the  next five years. ….“…with the cloud, you don’t own anything. You already signed it away.”

When I first read the title I thought, Wozniak sounds like Larry Ellison two years ago, when he pitched the Cloud is hype, before he made a 180-degree turn to acknowledge Oracle wished to be a cloud vendor too. 

Reading it more carefully, I realized the framing of the topic is instead just misleading. Wozniak actually touches on something that I hear more often, as the cloud hype cycle is moves from a Peak of Inflated Expectations into through the Trough of Disillusionment.

Wozniak echos an important lesson, that IMO, is major part of the reason many of the companies that moved to cloud have experienced lots of outages during the past months. I addressed several of these aspects in in a recent blog post: Lessons from the Heroku/Amazon Outage.

When we move our operations to the cloud, we often assume that we’re out-sourcing our data center operation completely, including our disaster recovery procedures. The truth is that when we move to the cloud we’re only outsourcing the infrastructure, not our operations, and the responsibility of how to use this infrastructure remain ours.

Choosing better tradeoffs between producivity and control

For companies today, the main reason we chose to move to the cloud in the first place was to gain better agility and productivity. But in starting this cloud journey, we found that we had to give up some measure of control to achieve the agility and productivity.

The good news is that as the industry mature there are more choices that provides better tradeoffs between producivity and control:

  • Open source cloud such as OpenStack and CloudStack
  • Private cloud offering
  • DevOps and automation tools such as Chef and Puppet
  • OpenSource PaaS such as Cloudify, OpenShift and CloudFoundry
  • DevOps and PaaS combined such Cloudify

As businesses look at cloud strategy today, there isn’t a need to give up control over productivity. With technologies like Cloudify, businesses can get the best out of both worlds.

References


EMEA, G-Cloud, Government, Guest Post, IT service management

Examining the G-Cloud Initiative – How the UK Public Sector is moving to the Cloud

Guest Post by Ben Jones

Ben Jones is a tech writer, interested in how technology helps businesses. He’s been assisting businesses in setting up cloud based IT services around the south of England.

There’s a cloud on the horizon of Whitehall. But this isn’t a prediction of stormy times ahead. No, this is the G-Cloud, and it’s being heralded by some as government’s biggest ever IT breakthrough.

In years gone by, the government has been accused of paying too much for IT contracts, many of which were won by a small number of suppliers. But now, the G-Cloud initiative aims to change this. The online system called, CloudStore, is part of the government’s plans to slash IT costs by £200million per year. So how is this going to be achieved? Well, the target is to move half of the government’s IT spending to cloud computing services and the CloudStore, also dubbed the government’s app store, is the key.

It was first announced as a government strategy almost 18 months ago in March 2011 with specific aim of making IT services for the public sector easier and cheaper. This means ditching the expensive bespoke IT services with lengthy, expensive contracts. Instead this initiative aims to replace these with more choice both in suppliers and, as a result prices. It’s a radical change in the historic approach by both the government and the public sector. Furthermore, cloud computing has the potential to be a global governmental strategy, with the American government already having its own version in place. And a look at the figures gives a clear indication why, with some governmental departments reporting a drop in the cost of IT services by as much as 90 per cent. And following the first CloudStore catalogue launch in mid-February, some 5000 pages were viewed in the first two hours, and in the first ten weeks, contracts worth £500,000 were signed. In this first procurement, around 257 suppliers offering approximately 1700 services were signed to the first G-Cloud CloudStore.

It’s the government’s attempt to bring competitiveness to its suppliers, encouraging a wider selection and promoting flexibility in procurements thus allowing more choice to the public sector. And what’s interesting is the mix of both small and medium sized businesses with over half of the suppliers signed to the first CloudStore being SMEs. This includes the likes of web hosting company Memset whose managing director Kate Craig-Wood backed the G-Cloud Services, who says they offered value for money for the taxpayer.

This new initiative heralds a new era for the British government and the wider public sector. And it’s hoped the new IT system will put paid to the Government’s history of ill-advised and mismanaged IT projects. That’s not to say there haven’t been any concerns over the G-Cloud Initiative. Some key concerns have related to how it’s going to be rolled out to public sector workers across the UK with some employees having fears over security as well as a lack of understanding. However, these haven’t stopped the second round of procurement for the G-Cloud in May 2012 with the total procurement value now available there soaring to £100 million. And in this time, the framework will run for 12 months and not the six as per the first iteration. This year-long contract will then become the standard, although it has been reported that this could be extended to 24 months in certain cases.


Andrew Grove, Eveready, Guest Post, Information technology consulting, Information technology management, Innovation, Knowledge Management, Pixar, Steve Jobs

The ‘Curse of Knowledge’: An IT Consultant’s Tips to Avoid Stifling Innovation

Guest Post by Bob Deasy, CIO, Lead I.T. Consulting

Bob Deasy

Bob Deasy is CIO of Lead I.T. Consulting, which provides focused IT solutions and business strategy consulting in the Portland area.

The phrase “curse of knowledge” first appeared a 1989 paper titled “The Curse of Knowledge in Economic Settings: An Experimental Analysis,” which introduced the concept that “better informed agents are unable to ignore private information even when it is in their best interests to do so; more information is not always better.” While most of us assume that experts are the best people to turn to for new ideas, the truth is that experts are often less able to innovate than greenhorns. For instance, if your IT consultant thinks along the exact same lines as you, it’s difficult to find new ways of doing things.

Although this concept is counterintuitive at first, it makes sense upon consideration of the knowledge-building process. Every field has its own lingo and agreed-upon principles. These guidelines help organize and canonize information that would otherwise be difficult to remember. To gain entry into upper academic echelons and posh corner offices, a person must learn how to follow the appropriate industry rules. IT consultants, for instance, often have a set of IT management rules, such as the ITIL guidelines, practically engrained on their brains, so they may not see areas that are best served by alternative approaches.

The more you know, the harder it is to get out of the box of agreed-upon industry rules that you’ve built around yourself. The mind of an expert can easily settle into a certain pattern or rut, simply because “that’s the way we’ve always done it.” When entire technology consulting firms are operating from the same handbook, it’s difficult to achieve true innovation. Intel co-founder Andrew S. Grove put it this way: “When everybody knows that something is so, it means that nobody knows nothin’.” As we get to know a topic better, it is harder for us to see it in creative, new ways. Understanding the “rules” of knowledge limits our ability to bend or break them.

Sophisticated but ultimately useless software is one example of how the curse of knowledge thwarts IT innovation. Engineers, in their insulated community, can’t help but design software for other engineers. Too often, the product of their efforts is packaged well and marketed widely but ultimately impractical or downright useless for the average company.

Brothers Chip and Dan Heath explore how to evade the curse of knowledge in their book Made to Stick: Why Some Ideas Stick and Others Come Unstuck. Below, we explore a few of these suggestions through an IT management perspective. IT consultants and mangers can cultivate innovation by following these tips:

1. Build a team with a variety of skills.

Steve Jobs took this approach to heart when he created the Pixar building, which was designed to force accountants, animators and all other niche experts to interact in the building’s sole set of bathrooms. As Jobs said, “Technology alone is not enough – it’s technology married with liberal arts, married with the humanities, that yields us the results that make our hearts sing.” When an IT consultant, CFIO or other IT management guru is forced to work with complete novices, new ways of thinking naturally open up. Total beginners will likely have unique knowledge in other areas that can be applied to IT management in unique, groundbreaking ways.

2. Avoid jargon; seek teaching opportunities.

Explaining the basics can help experts think about their understanding in a new light, fostering innovation. In her book Innovation Killer, Cynthia Barton Rabe tells of a colleague at Eveready who came to the flashlight business with no preconceived notions of what did and did not work. At that time, all Eveready flashlights were red and utilitarian. Drawing from her years of experience in marketing and packaging at Ralston Purina, this flashlight newbie overhauled the Eveready line to include pink, green and baby blue torches – colors that would be more likely to attract female shoppers. Thus, the floundering flashlight business was revived.

Rabe concludes that such “zero gravity thinkers,” as she calls them, fuel innovation by asking very basic questions that force experts to step back into a beginner’s mindset. Because going back to the basics can seem like backtracking to those who are very familiar with specialized knowledge, it’s not unusual for frustration to run high when zero gravity thinkers are on the scene. However, if a team can work through this irritation, innovation soon follows.

3. Hire “Renaissance thinker” consultants.

Ms. Rabe concedes that outside parties, such as IT consultants, can serve as zero gravity thinkers, assuming they have a broad range of knowledge. If your IT consultant’s only employment has been through technology consulting firms, he or she will not be as likely to innovate. In contrast, an IT consultant who came to the field as a second career will be able to see wholly new approaches.


Guest Post, Memcached, NoSQL, Redis, Relational database management system

Taking In-Memory NoSQL to the Next Level

Guest Post by Ofer Bengal, Co-Founder & CEO, Garantia Data

 

Ofer Bengal

Ofer Bengal has founded and led companies in data communications, telecommunications, Internet, homeland security and medical devices.

Today Garantia Data is launching the first in-memory NoSQL cloud that promises to change the way people use Memcached and Redis. I think this is a great opportunity to examine the state of these RAM-based data stores and to suggest a new, highly-efficient way of operating them in the cloud.

Challenges of Cloud Computing

Memcached and Redis are being increasingly adopted by today’s web-applications, and are being used to scale-out their data-tier and significantly improve application performance (in many cases improvement is x10 over standard RDBMS implementation). However, cloud computing has created new challenges in the way scaling and application availability should be handled and using Memcached and Redis in their simple form may not be enough to cope with these challenges.

Memcached

It’s no secret Memcached does wonders for websites that need to quickly serve up dynamic content to a rapidly growing number of users. Facebook, Twitter, Amazon and YouTube, are heavily relying on Memcached to help them scale out; Facebook handles millions of queries per second with Memcached.
But Memcached is not just for giants. Any website concerned with response time and user based growth should consider Memcached for boosting its database performance. That’s why over 70% of all web companies, the majority of which are hosted on public and private clouds, currently use Memcached.

Local Memcached is the simplest and fastest caching method because you cache the data in the same memory as the application code. Need to render a drop-down list faster? Read the list from the database once, and cache it in a Memcached HashMap. Need to avoid the performance-sapping disk trashing of an SQL call to repeatedly render a user’s personalized Web page? Cache the user profile and the rendered page fragments in the user session.

Although local caching is fine for web applications that run on one or two application servers, it simply isn’t good enough when the data is too big to fit in the application server memory space, or when the cached data is updated and shared by users across multiple application servers and user requests. In such cases user sessions, are not bound to a particular application server. Using local caching under these conditions may end up providing a low hit-ratio and poor application performance.

Distributed Memcached tends to improve local caching by enabling multiple application servers to share the same cache cluster. Although the Memcached client and server codes are rather simple to deploy and use, Distributed Memcached suffers from several inherent deficiencies:

  • Lack of high-availability – When a Memcached server goes down the application’s performance suffers as all data queries are now addressed to the RDBMS, which is providing a much slower response time. When the problem is fixed, it could take between a few hours to several days until the recovered server becomes “hot” with updated objects and fully effective again. In more severe case, where session data is stored in Memcached without persistent storage, losing a Memcached server may cause forced logout of users or flush of their shopping carts (in ecommerce sites).
  • Failure hassle – The operator needs to set all clients for the replacement server and wait for it to “warm-up”. Operators sometimes add temporary slave servers to their RDBMS, for offloading their Master server, until their Memcached recovers.
  • Scaling hassle – When the application dataset grows beyond the current Memcached resource capacity, the operator needs to scale out by adding more servers to the Memcached tier. However, it is not always clear when exactly this point has been reached and many operators scale out in a rush only after noticing degradation in their application’s performance.
  • Scaling impact on performance – Scaling out (or in) Memcached typically causes partial or entire loss of the cached dataset, resulting, again, in degradation of the application’s performance.
  • Manpower – Operating Memcached efficiently requires manpower to monitor, optimize and scale when required. In many web companies these tasks are carried out by expensive developers or devops.

Amazon has tried to simplify the use of Memcached by offering ElastiCache, a cloud-based value-added service, where the user does not have to install Memcached servers but rather rent VMs (instances) pre-loaded with Memcached (at a cost higher than plain instances). However, ElastiCache has not offered a solution for any of the Memcached deficiencies mentioned above. Furthermore, ElastiCache scales-out by adding a complete EC2 instance to the user’s cluster, which is a waste of $$ for users who only require one or two more GBs of Memcached. With this model ElastiCache misses on delivery of the true promise of cloud computing – “consume and pay only for what you really need” (same as for electricity, water and gas).

Redis

Redis an open source, key-value, in-memory, NoSQL database began ramping-up in 2009 and is now used by Instagram, Pinterest, Digg, Github, flickr, Craigslist and many others and has an active open source community, sponsored by VMware.

Redis can be used as an enhanced caching system alongside RDBMS, or as a standalone database.
Redis provides a complete new set of data-types built specifically for serving modern web applications in an ultra-fast and more efficient way. It solves some of the Memcached deficiencies, especially when it comes to high availability, by providing replication capabilities and persistent storage. However, it still suffers from the following drawbacks:

  • Failure hassle – There is no auto-fail-over mechanism; when a server goes down, the operator still needs to activate a replica or build a replica from persistent storage.
  • Scalability – Redis is still limited to a single master server and although cluster management capability is being developed, it probably won’t be simple to implement and manage and will not support all Redis commands, making it incompatible with existing deployments.
  • Operations – Building a robust Redis system requires strong domain expertise in Redis replication and data persistence nuances and building a Redis cluster will be rather complex.
DB Caching Evolution

The Evolution of Caching

A new cloud service that will change the way people use Memcached and Redis

Imagine connecting to an infinite pool of RAM memory and drawing as much Memcached or Redis memory you need at any given time, without ever worrying about scalability, high-availability, performance, data security and operational issues; and all this, with the click of a button (ok, a few buttons). Now imagine paying only for GBs used rather than for full VMs and at a rate similar to what you pay your cloud vendor for plain instances. Welcome to the Garantia Data In-Memory NoSQL Cloud!  

By In-Memory NoSQL Cloud I refer to an online, cloud-based, in-memory NoSQL data-store service that offloads the burden of operating, monitoring, handling failures and scaling Memcached or Redis from the application operator’s shoulders. Here are my top 6 favorite features of such service, now offered by Garantia Data:

  • Simplicity – Operators will no longer need to configure and maintain nodes and clusters. The standard Memcached/Redis clients are set for the service DNS and from this moment on, all operational issues are automatically taken care of by the service.
  • Infinite scalability – The service provides an infinite pool of memory with true auto-scaling (out or in) to the precise size of the user’s dataset. Operators don’t need to monitor eviction rates or performance degradation in order to trigger scale-out; the system constantly monitors those and adjusts the user’s memory size to meet performance thresholds.
  • High availability – Built-in automatic failover makes sure data is guaranteed under all circumstances. Local persistence storage of the user’s entire dataset is provided by default, whereas in-memory replication can be configured at a mouse click. In addition, there is no data loss whatsoever when scaling out or in.
  • Improved application performance – Response time is optimized through consistent monitoring and scaling of the user’s memory. Several techniques that efficiently evict unused and expired objects are employed to significantly improve the hit-ratio.
  • Data security – For those operators who are concerned with hosting their dataset in a shared service environment, Garantia Data has full encryption of the entire dataset as a key element of its service.
  • Cost savings – Garantia Data frees developers from handling data integrity, scaling, high availability and Memcached/Redis version compliance issues. Additional savings are achieved by paying only for GBs consumed rather than for complete VMs (instances). The service follows the true spirit of cloud computing enabling memory consumption to be paid for much like electricity, water or gas, so you “only pay for what you really consume”.

We have recently concluded a closed beta trial with 20 participating companies where all these features were extensively tested and verified – and it worked fine! So this is not a concept anymore, it’s real and it’s going to change the way people use Memcached and Redis! Am I excited today? Absolutely!


Apache Tomcat, Cloudify, Guest Post, MySQL, Rackspace

Bursting into the Clouds – Experimenting with Cloud Bursting

Guest Post by Dotan Horovits, Senior Solutions Architect at GigaSpaces

Dotan Horovits is one of the primary architects at GigaSpaces

Dotan Horovits is one of the primary architects at GigaSpaces

Who needs Cloud Bursting?

We see many organizations examining Cloud as replacement for their existing in-house IT. But we see interest in cloud even among organizations that have no plan of replacing their traditional data center.

One prominent use case is Cloud Bursting:

Cloud bursting is an application deployment model in which an application runs in a private cloud or data center and bursts into a public cloud when the demand for computing capacity spikes. The advantage of such a hybrid cloud deployment is that an organization only pays for extra compute resources when they are needed.
[Definition from SearchCloudComputing]

Cloud Bursting appears to be a prominent use case in cloud on-boarding projects. In a recent post, Nati Shalom summarizes nicely the economical rationale for cloud bursting and discusses theoretical approaches for architecture. In this post I’d like to examine the architectural challenges more closely and explore possible designs for Cloud Bursting.

Examining Cloud Bursting Architecture

Overflowing compute to the cloud is addressed by workload migration: when we need more compute power we just spin up more VMs in the cloud (the secondary site) and install instances of the application. The challenge in workload migration is around how to build a consistent environment in the secondary site as in the primary site, so the system can overflow transparently. This is usually addressed by DevOps tools such as Chef, Puppet, CFEngine and Cloudify, which capture the setup and are able to bootstrap the application stack on different environments. In my example I used Cloudify to provide consistent installation between EC2 and RackSpace clouds.

The Cloud Bursting problem becomes more interesting when data is concerned. In his post Nati mentions two approaches for handling data during cloud bursting:

1. The primary site approach – Use the private cloud as the primary data site, and then point all the burst activity to that site.
2. Federated site approach – This approach is similar to the way Content Distribution Networks (CDN) work today. With this approach we maintain a replica of the data available at each site and keep their replicas in sync.

The primary site approach incurs heavy penalty in latency, as each computation needs to make the round trip to the primary site to get the data for the computation. Such architecture is not applicable to online flows.

The federated site approach uses data synchronization to bring the data to the compute, which saves the above latency and enables online flows. But if we want to support “hot” bursting to the cloud, we have to replicate the data between the sites in an ongoing streaming fashion, so that the data is available on the cloud as soon as the peak occurs and we can spin up compute instances and immediately start to redirect load. Let’s see how it’s done.

Cloud Bursting – Examining the Federated Site Approach

Let’s put up our sleeves and start experimenting hands-on with the federated site approach for Cloud Bursting architecture. As reference application let’s take Spring’s PetClinic Sample Application and run it on an Apache Tomcat web container. The application will persist its data locally to a MySQL relational database.

The primary site, representing our private data center, will run the above stack and serve the PetClinic online service. The secondary site, representing the public cloud, will only have a MySQL database, and we will replicate data between the primary and secondary sites to keep data synchronized. As soon as the load on the primary site increases beyond a certain threshold, we will spin up a machine with an instance of Tomcat and the PetClinic application, and update the load balancer to offload some of the traffic to the secondary site.

On my experiment I used Amazon EC2 and RackSpace IaaS providers to simulate the two distinct environments of the primary and secondary sites, but any on-demand environments will do.

REPLICATING RDBMS DATA OVER WAN

How do we replicate data between the MySQL database instances over WAN? On this experiment we’ll use the following pattern:

1.     Monitor data mutating SQL statements on source site. Turn on the MySQL query log, and write a listener (“Feeder”) to intercept data mutating SQL statements, then write them to GigaSpaces In-Memory Data Grid.

2.     Replicate data mutating SQL statements over WAN. I used GigaSpaces WAN Replication to replicate the SQL statements  between the data grids of the primary and secondary sites in a real-time and transactional manner.

3.     Execute data mutating SQL statements on target site. Write a listener (“Processor”) to intercept incoming SQL statements on the data grid and execute them on the local MySQL DB.

 

 

 

 

 

 

 

 

 

 

 

 

To support bi-directional data replication we simply deploy both the Feeder and the Processor on each site.

AUTO-BOOTSTRAP SECONDARY SITE

When peak load occurs, we need to react immediately, and perform a series of operations to activate the secondary site:

1.     spin up compute nodes (VMs)

2.     download and install Tomcat web server

3.     download and install the PetClinic application

4.     configure the load balancer with the new node

5.     when peak load is over – perform the reverse flow to tear down the secondary site

We need to automate this bootstrap process to support real-time response to peak-load events. How do we do this automation? I used GigaSpacesCloudify open-source product as the automation tool for setting up and for taking down the secondary site, utilizing the out-of-the-box connectors for EC2 and RackSpace. Cloudify also provides self-healing  in case of VM or process failure, and can later help in scaling the application (in case of clustered applications).

Implementation Details

The result of the above experimentation is available on GitHub. It contains:

§  DB scripts for setting up the logging, schema and demo data for the PetClinic application

§  PetClinic application (.war) file

§  WAN replication gateway module

§  Cloudify recipe for automating the PetClinic deployment

See the documentation on GitHub for detailed instructions on how to configure the above with your specific deployment details.

Conclusion

Cloud Bursting is a common use case for cloud on-boarding, which requires good architecture patterns. In this post I tried to suggest some patterns and experiment with a simple demo, sharing it with the community to get feedback and raise discussion on these cloud architectures.

More information can be seen at an upcoming GigaSpaces webinar on Transactional Cross-Site Data Replication on June 20th (register at: http://bit.ly/IM0w9F)


GigaSpaces, Guest Post, No-Ops, PaaS, Platform as a service

Does PaaS Really Mean No-Ops?

Guest post by Yaron Parasol, Director of Product Management, GigaSpaces

Yaron Parasol is GigaSpaces Director of Product Management

I’d like to start with a brief overview of the evolution of the cloud – and why I think a new approach to PaaS solutions is needed – and the best scenarios for this to come into play.

First there was IaaS. Cloud was created with the notion of IT agility and cost-reduction. You need servers? No problem! Forget about red tape, forget about sys admins. You create an account and in few clicks you select the hardware profile and OS image you need, and voila, your server is out there, ready for you to use. No hassles, immediate gratitude.

Well, this is true as long as the images you get form your cloud provider match your needs. If you have custom needs, you will have to create and maintain your own image. So, you need the sys admin’s knowledge. However, we’re also seeing a change in methodology here – sys admins no longer need to install the servers once they’re up. Instead, they provide their expertise using approved and maintained images on the cloud. Application developers can choose the right image for them and from that point on, create virtual machines in the amount and hardware size they need for their applications.

Now let’s switch to PaaS. The idea of no-ops is the guideline for many of the existing PaaS offerings. Their use cases and features are all about developers. As Michael Cote put it:

“The point of PaaS is to make a developer’s life even easier: you don’t need to manage your cloud deployments at the the lower level of IaaS, or even wire together your Puppet/Chef scripts. The promise of PaaS is similar to that of Java application servers: just write your applications (your business logic) and do magic deployment into the platform, where everything else is taken care of.”

Developers need to deploy applications to the Cloud. They don’t want to care about the OS but they also don’t want to care about platforms, load balancers etc. They want to focus on what they know – writing code.

This is definitely a very productive approach for some developers and some applications. Reality shows that a big portion of Cloud users don’t find these solutions a good fit for their purposes. These users continue to deploy and manage their applications on infrastructure clouds, as if they were running on premise leveraging the good old Ops folks. Others have taken a more agile approach, using configuration management and automation tools such as Chef.

These users chose not to use PaaS because they need flexibility and control. PaaS doesn’t seem to answer a lot of the current IT challenges – see for example here and here.

Existing applications with a variety of platforms, some using extremely complex topologies (like Hadoop, or a sharded MongoDB setup) are some of the reasons why PaaS won’t cut it for many users.

They would like to install their chosen versions of their preferred platforms, use their OS image, with their configured security groups, tune them in a manner that fits their applications and deploy their applications with the topology they choose.

Chef, like other DevOps tools, go a long way here and helps to achieve the flexibility while re-establishing a new agile relationship between Dev and Ops. Ops bring in their knowledge and skillset, but they document it and maintain it as code in a more structured and configurable way. This in turn gives the application guys the agility they need, putting a complex application to work in a single click and eliminating the platform black box experience which they dislike so much.

Application vs. Separate Platforms

However, DevOps tools still fall short when it comes to managing applications. They are not aware of the application’s inner dependencies. They don’t know how to monitor the application, scale it or even run a complex multi-tier recovery process. Most of these tools can’t even provision an entire application on the cloud.

So what if you could extend the DevOps experience to apply to the entire application lifecycle?

What if you could use Chef and the likes for installation but not stop there – automate things like failover and recovery, and even monitoring and scaling? You will still have all the Ops wisdom tailored to each of your applications and be able to automate any of your existing applications without re-architecting them.

This is exactly our take on PaaS, a DevOps style process that can describe any application’s lifecycle on any runtime environment, providing full automation without taking away the control.  And this is exactly what we set out to do with our Open Source PaaS platform – Cloudify, borrowing the idea of recipes but extending it to be application-centric and not infrastructure-centric.

The recipe describes the application dependencies and lifecycle events externally without any code or architecture change.

lifecycle{
init "mongod_install.groovy"
start "mongod_start.groovy"
postStart "mongod_poststart.groovy"
}

view rawmongod.groovyThis Gist brought to you by GitHub.

See how to create your own recipes here.

Mapping events like installation, start, post-start and stop to scripts or Chef cookbooks, exposing groovy andREST interfaces for context sharing and dynamic configuration and even provide a way to describe monitoring techniques, scaling rules and process “liveness” detection.

So think about it this way: while most PaaS services come with a catalog of predefined application blueprints, allowing the user to control only the application code, this new kind of PaaS stack, allows the user to define the blueprint, that is – any blueprint!

So, the recipes combine the Ops expertise with the power of automation for the developers. They completely remove the lock-in risk from the application-stack perspective.

You can read more the recipes, download Cloudify and experience it yourself or even join the community and influence the roadmap at cloudifysource.org. Or you can come see more when we present at HP Discover next week in Las Vegas.


Application Tools, Constellation Research, Consumerization, Guest Post, iPad, Tools

The Consumerization of IT: No Longer Just About Phones or Tablets

Pete Khanna of TrackVia

Pete Khanna is the CEO of TrackVia, a cloud-based application platform that allows business users to build and deploy their own enterprise-grade SaaS applications.

Guest Post by Pete Khanna, CEO, Trackvia

What CIOs Need to Know – And Consider Doing – About the Emergence of Build-it-Yourself Enterprise Applications

It used to be that when a department or even a single employee wanted a new business application, they submitted an IT ticket and patiently waited for an answer. IT would evaluate the request, and decide whether to buy something, build something or simply do nothing. Often, this process took weeks, months and sometimes even years.

Emboldened by the “Consumerization of IT” trend and empowered by new cloud-based build-it-yourself application platforms, more front-line workers are bypassing IT all together, using their corporate credit cards to buy, build and deploy their own enterprise-grade SaaS applications within hours or days.

Experts agree this trend is likely to continue – and even accelerate. Analyst research firm Gartner predicts that “citizen developers” will build 25 percent of all new business applications by 2014.

Early case studies showed employees were building and deploying highly specialized or one-of-a-kind ‘rogue’ applications. These were the type of applications where an off-the-shelf software or SaaS solution either didn’t do the job or simply didn’t exist. However, more and more often, workers are using these build-it-yourself applications to deploy highly customized versions of common department applications, ranging from CRM SaaS solutions and inventory applications to HR systems and even customer service software.

The benefit of building your own department solution is that it’s often as much as 80 percent more affordable than off-the-shelf software or SaaS solutions. More importantly, these build-it-yourself solutions can be designed, customized and tailored by the users to meet the exact needs and requirements of the business. So instead of departments changing their processes or workflow to match that of the solution, users can change the solution to match their own processes. Simply put, features can be added, removed, changed or tweaked with a few clicks of the mouse. This leads to both greater efficiency and satisfaction by users.

All of this poses a growing challenge to IT professionals. Namely, how does IT manage the implementation of these so-called rogue applications without impeding employee productivity? The biggest concern from IT professionals is that they don’t want to be in constant clean-up mode, fixing or supporting solutions that end-users built themselves. A close second is that they want to be assured that the application is secure, ensuring that sensitive company information isn’t being compromised.

Having worked with thousands of businesses – from small cash-strapped start-ups to Fortune 100 companies – to implement their own custom applications, we’ve complied this list of best practices.

Step 1: Surrender to Win

The first and perhaps most important (and difficult) thing to do is decide whether or not to fight or embrace the “Consumerization of IT” movement. Even in highly regulated industries like banking and healthcare, where they use airtight firewalls, employees are finding ways around the lockdown mode of their internal IT organization. Doctors are bringing iPads into the examination room with their own apps installed. Marketers within banking organizations are using social media tools to distribute information. USB ports can be used to transfer information from unsecured laptops to company computers. The simple question becomes whether or not you want to use your limited IT resources and time policing employees or educating and empowering them. Getting alignment on this critical question at the highest levels of the organization is key.

Step 2: Offer Something Secure, Scalable and Supportable

Rather than hope you’re employees find and pick a secure and reliable build-it-yourself solution, many companies get out in front of their users and identify a single platform that meets both IT’s requirements and end-users’ needs.

This is where cloud-based application platforms are showing the most favorable response from business users and IT managers alike. Most organizations have larger, more complex enterprise-level applications that on some level need some customization. By adopting and implementing a secure, cloud-based application platform, CIOs proactively meet their own requirements while still providing end-users with a solution they can use to meet their own unique individual or department-level needs.

Implementing a single platform solution also helps streamline ongoing management and support, while making it faster and easier for employees to learn. For example, if all in-house applications are built atop a single platform, employees don’t have to learn to use multiple solutions. It also means IT doesn’t have to support multiple solutions. Everyone wins.

Step 3: Consider the Reliability

Nothing gets CEOs cackling faster than an email about a server or a critical system going down and employees checking out for the rest of the day. It’s also no fun for CIOs or IT Directors who have to deal with the hundreds of emails from employees asking, “Is the server back up? When will the CRM system be back up? What’s going on?”

A recent Constellation Research study shows most SaaS vendors report internal reliability ranging from 97 percent to 99.1 percent. However, it’s not uncommon to expect or demand that your cloud vendor partner demonstrate four-nine reliability. Most vendors will bake this guarantee into a standard SLA agreement. And as always, check the vendor’s track record and ask for customer references.

The Road Ahead

Like water, businesses and workers will always find the path of least resistance when it comes to working faster, more efficiently or effectively. The role of IT has always been to help clear technical obstacles for users, and protect them along the way. That hasn’t changed.

What has changed, however, is that IT has the opportunity to get out in front of the cloud computing and the “Consumerization of IT” trends, while playing a more proactive and strategic role in the overall organization’s future, versus trying to play catch up and doing damage control.

And that’s something I think we’d all agree is a good thing.


Cloud Sherpas, Google, Google Apps, Guest Post, Microsoft, Microsoft Office, Microsoft Office 365, Office 365

Cloud Competition: Microsoft Cuts Prices, Adds Platform

Guest Post by Sharon Shapiro of Cloud Sherpas

Although it only entered the cloud computing market in July 2011, Microsoft has already made a name for itself with Office 365, its hybrid cloud solution available to businesses.  But despite its initial climb, Microsoft has yet to catch up to Google when it comes to cloud computing.

Google has worked as a cloud service provider, with its fully-based cloud solution Google Apps, since 2006.  Google offers customers four different cloud computing platforms (business, education, government, and non-profit) and a 99.9% uptime guarantee, including service and updates, that it regularly exceeds.  In contrast, Office 365 is currently available only for businesses and has had issues meeting its promised 99.9% uptime, which does not include service and updates.  Google Apps pricing is also much lower than Office 365 pricing, with the highest priced Google Apps platform (business) operating at $50/user/year and the lowest Office 365 plan operating at $72/user/year.  But a lot of that is about to change as Microsoft has announced news that it hopes will bridge the gap.

First, Microsoft recently announced that it will be cutting the price of Office 365 by up to 20 percent.  Microsoft says this is because it now costs less to run the hybrid cloud platform than it did when Microsoft first introduced Office 365 last July.

Second, Microsoft says it will soon add a new plan – Office 365 for Education – which will widen the customer base.  The release of Office 365 is clearly an attempt to compete with the success of Google Apps’ free education platform.

Although lowering its prices and adding a plan for educational institutions are steps toward competing with Google Apps, Microsoft still has a big gap to close.  Google Apps will still boast more platforms (government and non-profit, in addition to the free version that many people use in their personal lives) and five more years of experience in the cloud.  With this greater experience, comes enhanced cloud service, as Google Apps is a fully cloud-based solution that offers its users complete universal access and proven reliability.  In contrast, it is important to remember that Office 365 is still a hybrid cloud solution that requires on premises servers and that, when it comes to document creating capabilities, works best in conjunction with Microsoft Office installed on a desktop.  Both of these necessities limit the mobile access that Office 365 users can enjoy.   Additionally, Office 365 has already been plagued with a number of outages that have resulted in significant amounts of downtime for users across the world.

These key differences between Office 365 and Google Apps may be part of the reason why the governments of major cities, like Pittsburgh and Los Angeles, trust Google Apps for their communication needs.

While Microsoft’s price cut and addition of an education plan will definitely improve the Office 365 cloud suite, these changes will certainly not put Microsoft and Google on an even playing field, as Google still boasts more authority, reliability, and a wider range of services.


DropBox, Guest Post, Icloud, SpiderOak, Storage, SugarSync, Windows Live SkyDrive

Industry Dynamics of Online Storage and the Decade Ahead

Guest Post by Eric Greenwood

Eric Greenwood is a technophile whose interests have led him to study all things related to the cloud computing movement from online storage to software as a service. Get more tips and advice on his Cloud Computing blog.

Online, or cloud storage, is a massively growing industry, already poised to change the way we use our computers. By 2016, consumers are predicted to be spending as much $16 billion on cloud storage annually.

Big names are flying into the cloud. Oracle and Hewlett Packard are rushing to sell cloud computing tools; Amazon’s cloud services division has earned an estimated $750 million from cloud services in 2011 – and predictions are for earnings of $2.5 billion by 2014 from all cloud services including their Simple Storage Service. Amazon CEO Jeff Bezos suggests potential for Amazon Web Services could surge to match that of its retail earnings, which last year topped $25 billion. Rackspace’s cloud servicing is also surging.

While currently only approximately 10% of global spending IT goes to cloud computing, the shift to cloud storage is a growing trend and market.

Popular cloud storage service Dropbox already has over fifty million users, and $250 million in venture capital; and Google Drive’s new online storage is poised to rival them. Like Dropbox’s chief competitor, Sugar Sync, Drive offers 5 GB of free storage, over doubling the free storage amount provided by Dropbox.

Storage competitors are also likely to follow Dropbox’s option of gallery pages that allow users who follow a link to see photos, presentations and videos without downloading each individual file. Dropbox is valued at approximately $4 billion, currently. The company’s CEO recently turned down a reported nine-figure offer from Apple. Apple of course maintains its own online storage system, iCloud, free to all users of iOs5. iCloud’s seamless interface with Apple products keeps this cloud storage service somewhat above the competitive fray.

Dropbox was recently voted “startup of the year,” and is reportedly the fifth most valuable web start-up, globally. But along with iCloud, SugarSync, and Google’s new drive, competition is fierce from other online storage startups ranging from Box.net, now known simply as Box, to Microsoft’s massive SkyDrive, Carbonite, which offers solid data backup services, and SpiderOak, which offers data encryption. Each of these cloud storage companies have greatly benefitted from the decline in pricing for online storage. Clearwell Systems research estimates that the storage cost for 1 gigabyte of information that cost $20 in 2000 is now approximately ten cents. HTML 5 has also greatly accelerated the growth of cloud storage companies. The cost and technology trends that have made cloud computing expand will only accelerate over the next ten years.

Dropbox’s popular rival SugarSync is an outgrowth of Sharpcast, the 2006 creator behind Sharpcast Photos, utilized for synching photo images to multiple devices. SugarSync’s differentiation with its competitors is based on its use of an automated refreshing process which means users don’t need to update their own synced files.

Microsoft’s recent overhaul of its SkyDrive online storage has doubled the storage file limit size, and made sharing as well as file management simpler for users. Just last month, Microsoft released a desktop app for SkyDrive, and allows direct file sharing to Twitter.

On the downside, there may only be a finite amount of users willing to store their data in the cloud, and a lot of competitors vying for a slice of the same pie. What’s good for consumers in terms of free storage or service perks may be difficult to sustain an entire industry of cloud storage competitors. Consolidation of some companies may be necessary.

A recent cautionary note was also founded when the file storage and hosting business Megapuload in Hong Kong was shut down by the U.S. Justice department for assisting in the violation of U.S. copyrights due to the ability of users to upload copyrighted material and distribute a link to it.

Megaupload’s violations bring up a key point in cloud storage, leading to the question as to whether or not Microsoft, Google, Dropbox, and all their competitors must scan files for copyright violations. Should this occur, the market will likely improve for Google with its Content ID already in place. Privacy and trust issues are also key in cloud storage growth. The only online storage company that claims to be unable to view users’ stored data is SpiderOak.

Online storage may be still in a speculative stage, but with data volumes predicted to multiply over forty times by the year 2020, data storage in one form or another is not only here to stay, it’s here to grow. Publicly traded companies such as EMC Corporation, NetApp, Isilon, Amazon, and CSC are providing expanded cloud storage options, and growing in financial leaps and bounds. IBM is working on a cloud system that can create virtual machines, able to integrate data without the costs of standard IT development, and simplifying cloud resources.

Complete data management through the cloud is clearly coming in the near future. Personal computer users and businesses multi-national to mom and pop size, must address data storage. Cloud storage is the go-to storage of the future, protected from human error, disasters, theft, and hardware malfunctions.