Category Archives: Storage and Information Management

Questions Around Uptime Guarantees

Some manufacturers recently have made an impact with a “five nines” uptime guarantee, so I thought I’d provide some perspective. Most recently, I’ve come in contact with Hitachi’s guarantee. I quickly checked with a few other manufacturers (e.g. Dell EqualLogic) to see if they offer that guarantee for their storage arrays, and many do…but realistically, no one can guarantee uptime because “uptime” really needs to be measured from the host or application perspective. Read below for additional factors that impact storage uptime.

Five Nines is 5.26 minutes of downtime per year, or 25.9 seconds a month.

Four Nines is 52.6 minutes/year, which is one hour of maintenance, roughly.

Array controller failover in EQL and other dual controller, modular arrays (EMC, HDS, etc.) is automated to eliminate downtime. That is really just the beginning of the story. The discussion with my clients often comes down to a clarification of what uptime means – and besides uninterrupted connectivity to storage, data loss (due to corruption, user error, drive failure, etc.) is often closely linked in people’s minds, but is really a completely separate issue.

What are the teeth in the uptime guarantee? If the array does go down, does the manufacturer pay the customer money to make up for downtime and lost data?

{Register for our upcoming webinar on June 12th ”What’s Missing in Hybrid Cloud Management- Leveraging Cloud Brokerage“ featuring guest speakers from Forrester and Gravitant}

There are other array considerations that impact “uptime” besides upgrade or failover.

  • Multiple drive failures, since most are purchased in batches, are a real possibility. How does the guarantee cover this?
  • Very large drives must be in a suitable RAID configuration to improve the chances that a RAID rebuild will be completed before another URE (unrecoverable read error) occurs. How does the guarantee cover this?
  • Dual controller failures do happen to all the array makers, although I don’t recall this happening with EQL. Even a VMAX went down in Virginia once, in the last couple of years. How does the guarantee cover this?


The uptime “promise” doesn’t include all the connected components. Nearly every environment has something with a single path or SPOF or other configuration issue that must be addressed to insure uninterrupted storage connectivity.

  • Are applications, hosts, network and storage all capable of automated failover at sub-10 ms speeds? For a heavily loaded Oracle database server to continue working in a dual array controller “failure” (which is what an upgrade resembles), it must be connected via multiple paths to an array, using all available paths.
  • Some operating systems don’t support an automatic retry of paths (Windows), nor do all applications resume processing automatically without IO errors, outright failures or reboots.
  • You often need to make temporary changes in OS & iSCSI initiator configurations to support an upgrade – e.g. change timeout value.
  • Also, the MPIO software makes a difference. Dell EQL MEM helps a great deal in a VMware cluster to insure proper path failover, as do EMC PowerPath and Hitachi Dynamic Link Manager. Dell offers a MS MPIO extension and DSM plugin to help Windows recover from a path loss in a more resilient fashion
  • Network considerations are paramount, too.
    • Network switches often take 30 seconds to a few minutes to reboot after a power cycle or reboot.
    • Also in the network, if non-stacked switches are used, RSTP must be enabled. If not, and anything else isn’t configured correctly, connectivity to storage will be lost.
    • Flow Control must be enabled, among other considerations (disable unicast storm control, for example), to insure that the network is resilient enough.
    • Link aggregation, if not using stacked switches, must be dynamic or the iSCSI network might not support failover redundancy


Nearly every array manufacturer will say that upgrades are non-disruptive, but that is at the most simplistic level. Upgrades to a unified storage array, for example, will involve disruption to file system presentation, almost always. Clustered or multi-engine frame arrays (HP 3PAR, EMC VMAX, NetApp, Hitachi VSP) can offer the best hope of achieving 5 nines, or even greater. We have customers with VMAX and Symmetrix that have had 100% uptime for a few years, but the arrays are multi-million dollar investments. Dual controller modular arrays, like EMC and HDS, can’t really offer that level of redundancy, and that includes EQL.

If the environment is very carefully and correctly set up for automated failover, as noted above, then those 5 nines can be achieved, but not really guaranteed.


The Buzz Around Software Defined Networking

By Nick Phelps, Consulting Architect, LogicsOne


One of the emerging trends in our industry that is stirring up some buzz right now is software defined networking. In this short video I answer the following questions about SDN:


  1. What is Software Defined Networking or SDN?
  2. Who has this technology deployed and how are they using it?
  3. What does SDN mean to the small to mid-market?
  4. When will the mid-market realize the benefits from SDN based offerings?
  5. When will we hear more? When should we expect the next update?


What are your thoughts on SDN? I’d love to hear you’re comments on the video and my take on the topic!



EMC World 2013 Recap

By Randy Weis, Consulting Architect, LogicsOne


The EMC World Conference held last week in Las Vegas demonstrated how EMC has a strong leadership position in the Virtualization, Storage and Software Defined Datacenter markets.

Seriously, this is not the Kool-Aid talking. Before anyone jumps in to point out how all the competitors are better at this or that, or how being a partner or customer of EMC has its challenges, I’d like to refer you to a previous blog I wrote about EMC: “EMC Leads the Storage Market for a Reason.” I won’t recap everything, but that blog talks about business success, not technical wizardry. Do the other major storage and virtualization vendors have solutions and products in these areas? Absolutely, and I promise to bring my opinions and facts around those topics to this blog soon.

What I found exciting about this conference was how EMC is presenting a more cohesive and integrated approach to the items listed below. The ExtremIO product has been greatly improved, some might say so that it is really usable now. I’d say the same about the EMC DR and BC solutions built on RecoverPoint and VPLEX – VPLEX is affordable and ready to be integrated into the VNX line. The VNX product line is mature now, and you can expect announcements around a major refresh this year. I’d say the same about the BRS line – no great product announcements, but better integration and pricing that helps customers and solution providers alike.

There are a few items I’d like to bullet for you:

  1. Storage Virtualization – EMC has finally figured out that DataCore is onto something, and spent considerable time promoting ViPR at EMC World. This technology (while 12 years to market behind DataCore) will open the eyes of the entire datacenter virtualization market to the possibilities of a Storage Hypervisor. What VMware did for computing, this technology will do for storage – storage resources deployed automatically, independent of the array manufacturer, with high value software features running on anything/anywhere. There are pluses and minuses to this new EMC product and approach, but this technology area will soon become a hot strategy for IT spending. Everyone needs to start understanding why EMC finally thinks this is a worthwhile investment and is making it a priority. To echo what I said in that prior blog, “Thank goodness for choices and competition!” Take a fresh look at DataCore and compare it to the new EMC offering. What’s better? What’s worse?
  2. Business Continuity and Highly Available Datacenters: Linking Datacenters to turn DR sites into an active computing resource is within reach of non-enterprise organizations now – midmarket, commercial, healthcare, SMB – however you want to define it.
    1. VPLEX links datacenters together (with some networking help) so that applications can run on any available compute or storage resource in any location – a significant advance in building private cloud computing. This is now licensed to work with VNX systems, is much cheaper and can be built into any quote. We will start looking for ways to build this into various solutions strategies – DR, BC, array migration, storage refreshes, stretch clusters, you name it.  VPLEX is also a very good solution for any datacenter in need of a major storage migration due to storage refresh or datacenter migration, as well as a tool to manage heterogeneous storage.
    2. RecoverPoint is going virtual – this is the leading replication tool for SRM, is integrated with VPLEX and now will be available as a virtual appliance. RP also has developed multi-site capabilities, with up to five sites, 8 RP “appliances” per site, in fan-in or fan-out configurations.
    3. Usability of both has improved, by standardizing management of both in Unisphere editions for both products.
    4. High Performance Storage and Computing – Server-side Flash, Flash Cache Virtualization and workload-crushing all-Flash arrays in the ExtremSF, ExtremSW and ExtremIO product line (formerly known as VFCache). As usual, the second release nails it for EMC. GreenPages was recently recognized as Global leaders in mission critical application virtualization, and this fits right in. Put simply, put an SSD card in a vSphere host and boost SQL/Oracle/EXCH performance over 100% in some cases. The big gap was in HA/DRS/vMotion. The host cache was a local resource, and thus vMotion was broken, along with HA and DRS. The new release virtualizes the cache so that VMs assigned local cache will see that cache even if it moves. This isn’t an all or nothing solution – you can designate the mission critical apps to use the cache and tie them to a subset of the cluster. This make this strategy affordable and granular.
    5. Isilon – This best in class NAS system keeps getting better. Clearly defined use cases, much better VMware integration and more successful implementations makes this product the one to beat in the scale-out NAS market.


Another whole article can be written about ViPR, EMC’s brand new storage virtualization tool, and that will be coming up soon. As promised, I’ll also take a look at the competitive offerings of HP and Dell, at least, in the Storage Virtualization, DR/BC, Server-side flash and scale-NAS solutions areas, as well as cloud storage integration strategies. Till then, thanks for reading this and please share your thoughts.

Catching up with Chuck Hollis: A Storage Discussion

Things are moving fast in the IT world. Recently, we caught up with Chuck Hollis (EMC’s Global Marketing CTO and popular industry blogger) to discuss a variety of topics including datacenter federation, Solid State Drives, and misperceptions surrounding cloud storage.

JTC: Let’s start off with Datacenter federation…what is coming down the road for running active/active datacenters with both HA and DR?

Chuck: I suppose the first thing that’s worth pointing out is that we’re starting to see using multiple data centers as an opportunity, as opposed to some sort of problem to overcome. Five years ago, it seems that everyone wanted to collapse into one or two data centers. Now, it’s pretty clear that the pendulum is starting to move in the other direction – using a number of smaller locations that are geographically dispersed.

The motivations are pretty clear as well: separation gives you additional protection, for certain applications users get better experiences when they’re close to their data, and so on. And, of course, there are so many options these days for hosting, managed private cloud services and the like. No need to own all your data centers anymore!

As a result, we want to think of our “pool of resources” as not just the stuff sitting in a single data center, but the stuff in all of our locations. We want to load balance, we want to failover, we want to recover from a disaster and so on – and not require separate technology stacks.

We’re now at a point where the technologies are coming together nicely to do just that. In the EMC world, that would be products like VPLEX and RecoverPoint, tightly integrated with VMware from an operations perspective. I’m impressed that we have a non-trivial number of customers that are routinely doing live migrations at metro distances using VPLEX or testing their failover capabilities (not-disruptively and at a distance) using RecoverPoint.

The costs are coming down, the simplicity and integration is moving up – meaning that these environments are far easier to justify, deploy and manage than just a few years ago. Before long, I think we’ll see active-active data centers as sort of an expected norm vs. an exception.

JTC: How is SSD being leveraged in total data solutions now, with the rollout of the various ExtremeIO products?

Chuck: Well, I think most people realize we’re in the midst of a rather substantial storage technology shift. Flash (in all its forms) is now preferred for performance, disks for capacity.

The first wave of flash adoption was combining flash and disk inside the array (using intelligent software), usually dubbed a “hybrid array”. These have proven to be very, very popular: with the right software, a little bit of flash in your array can result in an eye-popping performance boost and be far more cost effective than trying to use only physical disks to do so. In the EMC portfolio, this would be FAST on either a VNX or VMAX. The approach has proven so popular that most modern storage arrays have at least some sort of ability to mix flash and disk.

The second wave is upon us now: putting flash cards directly into the server to deliver even more cost-effective performance. With this approach, storage is accessed at bus speed, not network speed – so once again you get an incredible boost in performance, even as compared to the hybrid arrays. Keep in mind, though: today this server-based flash storage is primarily used as a cache, and not as persistent and resilient storage – there’s still a need for external arrays in most situations. In the EMC portfolio, that would be the XtremSF hardware and XxtremSW software – again, very popular with the performance-focused crowd.

The third wave will get underway later this year: all-flash array designs that leave behind the need to support spinning disks. Without dragging you through the details, if you design an array to support flash and only flash, you can do some pretty impactful things in terms of performance, functionality, cost-effectiveness and the like. I think the most exciting example right now is the XtremIO array which we’ve started to deliver to customers. Performance-wise, it spans the gap between hybrid arrays and server flash, delivering predictable performance largely regardless of how you’re accessing the data. You can turn on all the bells and whistles (snaps, etc.) and run them at full-bore. And data deduplication is assumed to be on all the time, making the economics a lot more approachable.

The good news: it’s pretty clear that the industry is moving to flash. The challenging part? Working with customers hand-in-hand to figure out how to get there in a logical and justifiable fashion. And that’s where I think strong partners like GreenPages can really help.

JTC: How do those new products tie into FAST on the array side, with software on the hosts, SSD cards for the servers and SSD arrays?

Chuck: Well, at one level, it’s important that the arrays know about the server-side flash, and vice-versa.

Let’s start with something simple like management: you want to get a single picture of how everything is connected – something we’ve put in our management products like Unisphere. Going farther, the server flash should know when to write persistent data to the array and not keep it locally – that’s what XtremSW does among other things. The array, in turn, shouldn’t be trying to cache data that’s already being cached by the server-side flash – that would be wasteful.

Another way of looking at it is that the new “storage stack” extends beyond the array, across the network and into the server itself. The software algorithms have to know this. The configuration and management tools have to know this. As a result, the storage team and the server team have to work together in new ways. Again, working with a partner that understands these issues is very, very helpful.

JTC: What’ the biggest misperception about cloud storage right now?

Chuck: Anytime you use the word “cloud,” you’re opening yourself up for all sorts of misconceptions, and cloud storage is no exception. The only reasonable way to talk about the subject is by looking at different use cases vs. attempting to establish what I believe is a non-existent category.

Here’s an example: we’ve got many customers who’ve decided to use an external service for longer-term data archiving: you know, the stuff you can’t throw away, but nobody is expected to use. They get this data out of their environment by handing it off to a service provider, and then take the bill and pass it on directly to the users who are demanding the service. From my perspective, that’s a win-win for everyone involved.

Can you call that “cloud storage”? Perhaps.

Or, more recently, let’s take Syncplicity, EMC’s product for enterprise sync-and-share. There are two options for where the user data sits: either an external cloud storage service, or an internal one based on Atmos or Isilon. Both are very specific examples of “cloud storage,” but the decision as to whether you do it internally or externally is driven by security policy, costs and a bunch of other factors.

Other examples include global enterprises that need to move content around the globe, or perhaps someone who wants to stash a safety copy of their backups at a remote location. Are these “cloud storage?”

So, to answer your question more directly, I think the biggest misconception is that – without talking about very specific use cases – we sort of devolve into a hand-waving and philosophy exercise. Is cloud a technology and operational model, or is it simply a convenient consumption model?

The technologies and operational models are identical for everyone, whether you do it yourself or purchase it as a service from an external provider.

JTC: Talk about Big Data and how EMC solutions are addressing that market (Isilon, GreenPlum, what else?).

Chuck: If you thought that “cloud” caused misperceptions, it’s even worse for “big data.” I try to break it down into the macro and the micro.

At the macro level, information is becoming the new wealth. Instead of it being just an adjunct to the business process, it *is* the business process. The more information that can be harnessed, the better your process can be. That leads us to a discussion around big data analytics, which is shaping up to be the “killer app” for the next decade. Business people are starting to realize that building better predictive models can fundamentally change how they do business, and now the race is on. Talk to anyone in healthcare, financial services, retail, etc. – the IT investment pattern has clearly started to shift as a result.

From an IT perspective, the existing challenges can get much, much more challenging. Any big data app is the new 800 pound gorilla, and you’re going to have a zoo-full of them. It’s not unusual to see a 10x or 100x spike in the demand for storage resources when this happens. All of the sudden, you start looking for new scale-out storage technologies (like Isilon, for example) and better ways to manage things. Whatever you were doing for the last few years won’t work at all going forward.

There’s a new software stack in play: think Hadoop, HDFS, a slew of analytical tools, collaborative environments – and an entirely new class of production-grade predictive analytics applications that get created. That’s why EMC and VMware formed Pivotal from existing assets like Greenplum, GemFire et. al. – there was nothing in the market that addressed this new need, and did it in a cloud-agnostic manner.

Finally, we have to keep in mind that the business wants “big answers”, and not “big data.” There’s a serious organizational journey involved in building these environments, extracting new insights, and operationalizing the results. Most customers need outside help to get there faster, and we see our partner community starting to respond in kind.

If you’d like a historical perspective, think back to where the internet was in 1995. It was new, it was exotic, and we all wondered how things would change as a result. It’s now 2013, and we’re looking at big data as a potentially more impactful example. We all can see the amazing power; how do we put it to work in our respective organizations?

Exciting time indeed ….

Chuck is the Global Marketing CTO at EMC. You can read more from Chuck on his blog and follow him on Twitter at @chuckhollis.

EMC Leads the Storage Market for a Reason

By Randy Weis, Consulting Architect, LogicsOne

There are reasons that EMC is a leader in the market. Is it because they come out first with the latest and greatest technological innovation? No, or at least not commonly. Is it because they rapidly turn over their old technology and do sweeping replacements of their product lines with the new stuff? No. It’s because there is significant investment in working through what will work commercially and what won’t and how to best integrate the stuff that passes that test into traditional storage technology and evolving product lines.

Storage Admins and Enterprise Datacenter Architects are notoriously conservative and resistant to change. It is purely economics that drives most of the change in datacenters, not the open source geeks (I mean that with respect), mad scientists and marketing wizards that are churning out & hyping revolutionary technology. The battle for market leadership and ever greater profits will always dominate the storage technology market. Why is anyone in business but to make money?

Our job as consulting technologists and architects is to match the technology with the business needs, not to deploy the cool stuff because we think it blows the doors off of the “old” stuff. I’d venture to say that most of the world’s data sits on regular spinning disk, and a very large chunk of that behind EMC disk. The shift to new technology will always be led by trailblazers and startups, people who can’t afford the traditional enterprise datacenter technology, people that accept the risk involved with new technology because the potential reward is great enough. Once the technology blender is done chewing up the weaker offerings, smart business oriented CIOs and IT directors will integrate the surviving innovations, leveraging proven manufacturers that have consistent support and financial history.

Those manufacturers that cling to the old ways of doing business (think enterprise software licensing models) are doomed to see ever-diminishing returns until they are blown apart into more nimble and creative fragments that can then begin to re-invent themselves into more relevant, yet reliable, technology vendors. EMC has avoided the problems that have plagued other vendors and continued to evolve and grow, although they will never make everyone happy (I don’t think they are trying to!). HP has had many ups and downs, and perhaps more downs, due to a lack of consistent leadership and vision. Are they on the right track with 3PAR? It is a heck of a lot more likely than it was before the acquisition, but they need to get a few miles behind them to prove that they will continue to innovate and support the technology while delivering business value, continued development and excellent post-sales support. Dell’s investments in Compellent, particularly, bode very well for the re-invention of the commodity manufacturer into a true enterprise solution provider and manufacturer. The Compellent technology, revolutionary and “risky” a few years ago, is proving to be a very solid technology that innovates while providing proven business value. Thank goodness for choices and competition! EMC is better because they take the success of their competitors at HP and Dell seriously.

If I were starting up a company now, using Kickstarter or other venture investment capital, I would choose the new products, the brand new storage or software that promises the same performance and reliability as the enterprise products at a much lower cost, knowing that I am exposed to these risks:

  • the company may not last long (poor management, acts of god, fickle investors) or
  • the support might frankly sucks, or
  • engineering development will diminish as the vendor investors wait for the acquisition to get the quick payoff.

Meanwhile, large commercial organizations are starting to adopt cloud, flash and virtualization technologies precisely for all the above reasons. Their leadership needs to drive profitability into the datacenter technologies to increase speed to market and improve profitability. As the bleeding edge becomes the smart bet as brought to market by the market leading vendors, we will continue to see success where Business Value and Innovation intersect.

The Newest Data-Storage Device is DNA?

By Randy Weis

Molecular and DNA Storage Devices- “Ripped from the headlines!”

-Researchers used synthetic DNA encoded to create the zeros and ones of digital technology.

-MIT Scientists Achieve Molecular Data Storage Breakthrough

-DNA may soon be used for storage: All of the world’s information, about 1.8 zettabytes, could be stored in about four grams of DNA

Harvard stores 70 billion books using DNA: Research team stores 5.5 petabits, or 1 million gigabits, per cubic millimeter in DNA  storage medium

IBM using DNA, nanotech to build next-generation chips: DNA works with nanotubes to build more powerful, energy-efficient easy-to-manufacture chips

Don’t rush out to your reseller yet! This stuff is more in the realm of science fiction at the moment, although the reference links at the end of this post are to serious scientific journals. It is tough out here at the bleeding edge of storage technology to find commercial or even academic applications for the very latest, but this kind of storage technology, along with quantum storage and holographic storage, will literally change the world. Wearable, embedded storage technology for consumers may be a decade or more down the road, but you know that there will be military and research applications long before Apple gets this embedded in the latest 100 TB iPod. Ok, deep breath—more realistically, where will this technology be put into action first? Let’s see how this works first.

DNA is a three dimensional media, with density capabilities of up to a zettabyte in a millimeter volume. Some of this work is being done with artificial DNA, injected into genetically modified bacteria (from a Japanese research project from last year). A commercially available genetic sequencer was used for this.

More recently, researchers in Britain encoded the “I Have a Dream” speech and some Shakespeare Sonnets in synthetic DNA strands. Since DNA can be recovered from 20,000 year old wooly mammoth bones, this has far greater potential for long term retrievable storage than, say, optical disks (notorious back in the 90s for delaminating after 5 years).

Reading the DNA is more complicated and expensive, and the “recording” process is very slow. It should be noted that no one is suggesting storing data in a living creature at this point.

Molecular storage is also showing promise, in binding different molecules in a “supramolecule” to store up to 1 petabyte per square inch. But this is a storage media in two dimensions, not three. This still requires temperatures of -9 degrees, considered “room temperature” by physicists. This work was done in India and Germany. IBM is working with DNA and carbon nanotube “scaffolding” to build nano devices in their labs today.

Where would this be put to work first? Google and other search engines, for one. Any storage manufacturer would be interested—EMC DNA, anyone? Suggested use cases: globally and nationally important information of “historical value” and the medium-term future archiving of information of high personal value that you want to preserve for a couple of generations, such as wedding video for grandchildren to see.  The process to lay the data down and then to decode it makes the first use case of data archiving the most likely. The entire Library of Congress could be stored in something the size of a couple of sugar cubes, for instance.

What was once unthinkable (or at least only in the realm of science fiction) has become reality in many cases: drones, hand held computers with more processing power than that which sent man to the moon, and terabyte storage in home computers. The future of data storage is very bright and impossible to predict. Stay tuned.

Here is a graphic from Nature Journal (the Shakespeare Sonnets), “Towards practical, high-capacity, low-maintenance information storage in synthesized DNA”

Click here to learn more about how GreenPages can help you with your organization’s storage strategy

Other References:

Researchers used synthetic DNA encoded to create the zeros and ones of digital technology.

MIT Scientists Achieve Molecular Data Storage Breakthrough

DNA may soon be used for storage

Harvard stores 70 billion books using DNA

IBM using DNA, nanotech to build next-generation chips


Disaster Recovery in the Cloud, or DRaaS: Revisited

By Randy Weis

The idea of offering Disaster Recovery services has been around as long as SunGard or IBM BCRS (Business Continuity & Resiliency Services). Disclaimer: I worked for the company that became IBM Information Protection Services in 2008, a part of BCRS.

It seems inevitable that Cloud Computing and Cloud Storage should have an impact on the kinds of solutions that small, medium and large companies would find attractive and would fit their requirements. Those cloud-based DR services are not taking the world by storm, however. Why is that?

Cloud infrastructure seems perfectly suited for economical DR solutions, yet I would bet that none of the people reading this blog has found a reasonable selection of cloud-based DR services in the market. That is not to say that there aren’t DR “As a Service” companies, but the offerings are limited. Again, why is that?

Much like Cloud Computing in general, the recent emergence of enabling technologies was preceded by a relatively long period of commercial product development. In other words, virtualization of computing resources promised “cloud” long before we actually could make it work commercially. I use the term “we” loosely…Seriously, GreenPages announced a cloud-centric solutions approach more than a year before vCloud Director was even released. Why? We saw the potential, but we had to watch for, evaluate, and observe real-world performance in the emerging commercial implementations of self-service computing tools in a virtualized datacenter marketplace. We are now doing the same thing in the evolving solutions marketplace around derivative applications such as DR and archiving.

I looked into helping put together a DR solution leveraging cloud computing and cloud storage offered by one of our technology partners that provides IaaS (Infrastructure as a Service). I had operational and engineering support from all parties in this project and we ran into a couple of significant obstacles that do not seem to be resolved in the industry.

Bottom line:

  1. A DR solution in the cloud, involving recovering virtual servers in a cloud computing infrastructure, requires administrative access to the storage as well as the virtual computing environment (like being in vCenter).
  2. Equally important, if the solution involves recovering data from backups, is the requirement that there be a high speed, low latency (I call this “back-end”) connection between the cloud storage where the backups are kept and the cloud computing environment. This is only present in Amazon at last check (a couple of months ago), and you pay extra for that connection. I also call this “locality.”
  3. The Service Provider needs the operational workflow to do this. Everything I worked out with our IaaS partners was a manual process that went way outside normal workflow and ticketing. The interfaces for the customer to access computing and storage were separate and radically different. You couldn’t even see the capacity you consumed in cloud storage without opening a ticket. From the SP side, notification of DR tasks they would need to do, required by the customer, didn’t exist. When you get to billing, forget it. Everyone admitted that this was not planned for at all in the cloud computing and operational support design.

Let me break this down:

  • Cloud Computing typically has high speed storage to host the guest servers.
  • Cloud Storage typically has “slow” storage, on separate systems and sometimes separate locations from a cloud computing infrastructure. This is true with most IaaS providers, although some Amazon sites have S3 and EC2 in the same building and they built a network to connect them (LOCALITY).

Scenario 1: Recovering virtual machines and data from backup images

Scenario 2: Replication based on virtual server-based tools (e.g. Veeam Backup & Replication) or host-based replication

Scenario 3: SRM, array or host replication

Scenario 1: Backup Recovery. I worked hard on this with a partner. This is how it would go:

  1. Back up VMs at customer site; send backup or copy of it to cloud storage.
  2. Set up a cloud computing account with an AD server and a backup server.
  3. Connect the backup server to the cloud storage backup repository (first problem)
    • Unless the cloud computing system has a back end connection at LAN speed to the cloud storage, this is a showstopper. It would take days to do this without a high degree of locality.
    • Provider solution when asked about this.
      • Open a trouble ticket to have the backups dumped to USB drives, shipped or carried to the cloud computing area and connected into the customer workspace. Yikes.
      • We will build a back end connection where we have both cloud storage and cloud computing in the same building—not possible in every location, so the “access anywhere” part of a cloud wouldn’t apply.

4. Restore the data to the cloud computing environment (second problem)

    • What is the “restore target”? If the DR site were a typical hosted or colo site, the customer backup server would have the connection and authorization to recover the guest server images to the datastores, and the ability to create additional datastores. In vCenter, the Veeam server would have the vCenter credentials and access to the vCenter storage plugins to provision the datastores as needed and to start up the VMs after restoring/importing the files. In a Cloud Computing service, your backup server does NOT have that connection or authorization.
    • How can the customer backup server get the rights to import VMs directly into the virtual VMware cluster? The process to provision VMs in most cloud computing environments is to use your templates, their templates, or “upload” an OVF or other type of file format. This won’t work with a backup product such as Veeam or CommVault.

5. Recover the restored images as running VMs in the cloud computing environment (third problem), tied to item #4.

    • Administrative access to provision datastores on the fly and to turn on and configure the machines is not there. The customer (or GreenPages) doesn’t own the multitenant architecture.
    • The use of vCloud Director ought to be an enabler, but the storage plugins, and rights to import into storage, don’t really exist for vCloud. Networking changes need to be accounted for and scripted if possible.

Scenario 2: Replication by VM. This has cost issues more than anything else.

    • If you want to replicate directly into a cloud, you will need to provision the VMs and pay for their resources as if they were “hot.” It would be nice if there was a lower “DR Tier” for pricing—if the VMs are for DR, you don’t get charged full rates until you turn them on and use for production.
      • How do you negotiate that?
      •  How does the SP know when they get turned on?
      • How does this fit into their billing cycle?
    • If it is treated as a hot site (or warm), then the cost of the DR site equals that of production until you solve these issues.
    • Networking is an issue, too, since you don’t want to turn that on until you declare a disaster.
      • Does the SP allow you to turn up networking without a ticket?
      • How do you handle DNS updates if your external access depends on root server DNS records being updated—really short TTL? Yikes, again.
    • Host-based replication (e.g. WANsync, VMware)—you need a host you can replicate to. Your own host. The issues are cost and scalability.

Scenario 3: SRM. This should be baked into any serious DR solution, from a carrier or service provider, but many of the same issues apply.

    • SRM based on host array replication has complications. Technically, this can be solved by the provider by putting (for example) EMC VPLEX and RecoverPoint appliances at every customer production site so that you can replicate from dissimilar storage to the SP IDC. But, they need to set up this many-to-one relationship on arrays that are part of the cloud computing solution, or at least a DR cloud computing cluster. Most SPs don’t have this. There are other brands/technologies to do this, but the basic configuration challenge remains—many-to-one replication into a multi-tenant storage array.
    • SRM based on VMware host replication has administrative access issues as well. SRM at the DR site has to either accommodate multi-tenancy, or each customer gets their own SRM target. Also, you need a host target. Do you rent it all the time? You have to, since you can’t do that in a multi-tenant environment. Cost, scalability, again!
    • Either way, now the big red button gets pushed. Now what?
      • All the protection groups exist on storage and in cloud computing. You are now paying for a duplicate environment in the cloud, not an economically sustainable approach unless you have a “DR Tier” of pricing (see Scenario 2).
      • All the SRM scripts kick in—VMs are coming up in order in protection groups, IP addresses and DNS are being updated, CPU loads and network traffic climb…what impact is this?
      • How does that button get pushed? Does the SP need to push it? Can the customer do it?

These are the main issues as I see it, and there is still more to it. Using vCloud Director is not the same as using vCenter. Everything I’ve described was designed to be used in a vCenter-managed system, not a multi-tenant system with fenced-in rights and networks, with shared storage infrastructure. The APIs are not there, and if they were, imagine the chaos and impact on random DR tests on production cloud computing systems, not managed and controlled by the service provider. What if a real disaster hit in New England, and a hundred customers needed to spin up all their VMs in a few hours? They aren’t all in one datacenter, but if one provider that set this up had dozens, that is a huge hit. They need to have all the capacity in reserve, or syndicate it like IBM or SunGard do. That is the equivalent of thin-provisioning your datacenter.

This conversation, as many I’ve had in the last two years, ends somewhat unsatisfactorily with the conclusion that there is no clear solution—today. The journey to discovering or designing a DRaaS is important, and it needs to be documented, as we have done here with this blog and in other presentations and meetings. The industry will overcome these obstacles, but the customer must remain informed and persistent. The goal of an economically sustainable DRaaS solution can only be achieved by market pressure and creative vendors. We will do our part by being your vigilant and dedicated cloud services broker and solution services provider.











Secure Remote Access for Businesses with Limited IT Staff and Budgets

With some of the recent breaches of restaurant chains, I’ve got to think that many of them were related to poor remote access practices. I say this because in all of my years of consulting, I have found that very weak controls around the remote access is a lot more common than one would think. Even today you will commonly find things like POS Servers directly accessible on the Internet via VNC, RDP, or pcAnywhere. I have even seen SQL databases that contain credit card data made directly accessible over the Internet.

Sometimes the organization itself is to blame. Usually because they just don’t know any better. For many, this has been the standard way to connect with their restaurants or stores remotely. They may lack the skills needed to setup secure remote access.  Other times, and this is also very common, a vendor or service provider is responsible. I can’t tell you how many times I have found completely unsecure remote access setup and enabled by the POS vendor or service provider that the merchant didn’t even know about—or at least wasn’t told about as far as the risks and compliance issues this creates. In one case I even found that the service provider had opened up a port on the firewall so they could connect directly to the POS SQL database across the Internet. No matter who is to blame, this needs to be fixed right away.

First, these organizations need to stop allowing systems in their restaurants/stores to be directly accessible across the Internet. It’s actually quite easy fix if you have fairly recent firewall hardware. Set yourself up an IPSEC site-to-site VPN tunnel between each of your stores and the central office using some form of two-factor authentication. Certificate-based along with a pre-shared key for authentication isn’t that hard to set up and meets PCI DSS requirements. Now you can provide vendors and service providers with remote access into your central office where you can centrally log their activities and implement restrictions on what they will have access to at each of the stores. And remember that they also need to be using some form of two-factor authentication to access your environment.

If you are the type of business that doesn’t have full time connectivity from your stores back to your central office then remote access is a bit more complex to manage. Each of your locations needs to be configured to support client-to-site VPN connections from your own IT department as well as from your service providers and vendors. IPSEC or SSL VPNs can be set up on most of today’s small firewalls and UTM devices without much fuss. But remember that two-factor authentication is a requirement and some of these devices don’t support such strong authentication methods. For this type of connectivity, some form of hardware or software token or even SMS-based token code authentication is a good choice. Sometimes this involves the implementation of a separate two-factor authentication solution, but some firewall/UTM devices have two-factor authentication features built in. This is a big plus and makes setting up secure remote access less complex and less expensive. If you go with these types of remote access connections—direct  connections to the stores—it’s very important to get the logs from remote access activity (as well as all other logs of course) from the firewalls pulled back into a central logging server for analysis and audit purposes.

To get started, your first step should be to review your external PCI ASV scans to see if any remote console services are accessible from the Internet. Look for RDP (tcp port 3389), VNC (tcp port 5900), or PCAnywhere (tcp port 5631 and udp port 5632).  Also look for databases such as MS SQL (tcp port 1433), MySQL (tcp port 3306), or PostgreSQL (tcp port 5432). If any of these show up then you should get working on a plan to implement secure and compliant remote access.

If you’re looking for more information, I’ll be hosting a security webinar on July 18th to cover common security mistakes and how your organization can avoid many of them!




Top Takeaways From EMC World 2012

A little over a week has gone by since the end of EMC World, and all the product announcements have gotten out of the bag. So, why another article about EMC World, if there are no “big reveals” left? Because I want to make sense of all of the hype, product announcements, and strategic discussions. What do the over 40 new products mean to GreenPages’ customers—both present and future? How many of those products were just cosmetic makeovers and how many are actual game changers? Why should you, our friends and extended business family, care, and what should you care about?

I will start by saying that this EMC World really did reveal some technology-leading thoughts and products, and proved that EMC has taken the lead in major storage technology strategy. EMC has always been the 800-pound gorilla of the storage industry, but for many years was far from the front of the pack. This has changed, and in a big way. Innovation still takes place mostly in the small companies on the bleeding edge of storage (SSD, virtualization across platforms, innovative file systems), but EMC has become the leading investor in storage R&D, and it shows. While they may not be inventing the coolest and most striking new storage and hardware, their pace of development and integration of that cool stuff has exponentially increased. Time to market and product refresh cycles are picking up pace. Relationships with the people who get the products in front of you (resellers, integrators and distributors) are vastly improved and much friendlier to the commercial world we all live in (as opposed to the rarified heights of the largest enterprises). The relevance of EMC products to the virtualized datacenter is clear, and the storage engineers who ran the technical sessions and laid out all the new storage, DR, and virtualization roadmaps proved that EMC is the leading storage technology firm in the world.

What are the highlights for GreenPages’ world?

Product Announcements:

Probably the biggest technology in terms of impact, IMHO, is Isilon. This is the fastest, most scalable, easy-to-manage NAS systems ever. It can grow to the petabyte range, and there is no downtime or forklift upgrades. It is “scale-out” storage, meaning you add nodes that contain processing (CPU), RAM for Cache and additional bandwidth, along with capacity in three flavors (SSD, 15K and 7.2K).  This is the system of choice for any healthcare PACs application or Life Sciences data storage. It is a fantastic general-purpose NAS system as well. Isilon is the system of choice for anyone managing Big Data (large amounts of unstructured data). The entry point for this system is around 10 TB, so you don’t have to be a large company to find the value here. Isilon also has the advantage of being a true scale-out system. Some technical nuggets around Isilon OneFS Upgrade: 90% greater throughput, or 740 GB/sec; roles-based admin – SEC 17a-4 compliance; better caching (50% reduction in latency of IO intensive apps; VMware Integration: VAAI (vStorage APIs for Array Integration) and VASA (vStorage APIs for Storage Awareness).

If you are going to jump up into the big time storage array arena, the new VMAX line is arguably the one to get, for power, performance and integration with the virtualized datacenter. It has expanded to the VMAX 10, 20 (current), and 40. The top of the line sports 8 controllers, scales up to 4 PB, has up to 32 2.8 GHz Xeon 6-core processors, 1 TB usable RAM, 2.5” drives,  and uses MLC SSD drives (bringing that cost of the flash drive down into the lower atmosphere). The latest development of the auto-tiering software FAST allows IBM and HDS storage to be a “tier” of storage for the VMAX. Other arrays will be added soon.

VNXe 3150 storage system offers up to 50% more performance and capacity in an entry level system. This system includes 10 GbE connectivity, Solid State Storage and the first production storage system (that I have heard of) that uses the latest Intel CPU, Sandy Bridge. Who says EMC product lifecycles are slow and behind the times??

VPLEX Metro/VPLEX Geo solutions have some significant upgrades, including integration with RecoverPoint and SRM, more performance and scalability; and Oracle RAC up to 100 KM apart. If you want to federate your datacenters, introduce “stretch clusters” and have both an HA and DR strategy, this is the industry leader now.

The VNX Series  has  more than a few improvements: lower price SSDs, RAID types that can be mixed in FAST; 256 snaps per LUN; connector for vCOPs; EMC Storage Analytics Suite based on vCOPs; AppSync to replace/improve Replication Manager.

The new VSPEX Proven Infrastructure includes EMC’s VNX and VNXe hybrid storage arrays, along with Avamar software and Data Domain backup appliances. The cloud platform also includes processors from Intel, switches from Brocade, servers from Cisco, and software from Citrix, Microsoft HyperV and VMware.  Avamar and Data Domain products will offer data deduplication to users, while EMC’s Fully Automated Storage Tiering (FAST), will offer data migration between varying disk storage arrays based on data use patterns. There are initially 14 VSPEX configurations, which EMC said represent the most popular use cases for companies moving to cloud computing.

Data Domain & Avamar upgrades include the DD990 with an Intel Sandy Bridge CPU, doubling the performance of the DD890 – 28 PB, 16 TB/hr throughput; tight integration of Avamar with VMware, including Hyper-V, SAP, Sybase, SQL2012 – recovery is 30 times faster than NBU/V-Ray.

Vfcache PCIe NAND Flash Card is a server side I/O enhancement that pushes Flash Cache to the server, but integrates Cache management with the VNX array FAST Cache. This will prove to be a huge deal for mission critical applications running on VMware, since I/O will no longer be a bottleneck even for the most demanding applications. Combine this with Sandy Bridge CPUs and the UCS system with the latest M3 servers and you will have the world’s most powerful server virtualization platform!

DataBridge is a “mash-up” of nearly any storage or system management tool into a common pane of glass, not intended to be a discovery or management tool but, rather, to be a place where all of the discovery tools can deliver their data to one place. This combines EMC infrastructure data sources along with non-EMC data sources with business logic from customer organizations. Stay tuned for more on this.

There are lots of other deep technical messages that were talked about in the sessions that ran for three solid days, not counting the unbelievable Lab sessions. Those Lab sessions are now available for demo purposes. You can see any EMC technology from implementation to configuration just by contacting GreenPages and asking for your Friendly Neighborhood Storage Guy!!

One final thought I would like to stress: efficiency. EMC is sending a smart business message of efficiency, using VNX as example. Storage is far outstripping storage advances and IT budgets. All is not hopeless, however. You can improve efficiency with dedupe/compression, auto tiering; Flash allows storage to keep up with Moore’s Law; you can consolidate file servers with virtual file servers (we have done this with many GreenPages customers when consolidating servers in VMware). Files are the main culprit. How will you manage it, quotas or content management? What will you chose? How will you manage your data without the money or work force you think you might need?

Contact GreenPages if you need help answering these questions! Meanwhile, watch for more storage technology breakthroughs to come from EMC in the coming months.