…On Friday at 5:30 PM PT, we had a planned maintenance scheduled to upgrade the OS on some of our machines.
…In this case, a bug in the script caused the upgrade to run on a handful of machines serving production traffic.
…some master-slave pairs were impacted which resulted in the site going down.
…We were able to restore most functionality within 3 hours, but the large size of some of our databases slowed recovery, and it took until 4:40 PM PT today for core service to fully return.
Wired has a cautionary tale for you to read as you consider the perils as well as the promise of could computing.
New York startup DigitalOcean says that its cloud server platform may be leaking data between its customers.
Kenneth White stumbled across several gigabytes of someone else’s data when he was noodling around on DigitalOcean’s service last week. White, who is chief of biomedical informatics with Social and Scientific Systems, found e-mail addresses, web links, website code and even strings that look like usernames and passwords — things like 1234qwe and 1234567passwd.
The problem started in mid-January, when DigitalOcean introduced a new solid state drive storage service. “The code that wipes the data — that securely deletes the data — was not being activated under the new SSD storage plans,” according to DigitalOcean CEO Ben Uretsky .
Read the details (and weep).