A week of downtime and all the servers were recovered only because the customer had a proper disaster recovery protocol and held backups somewhere else, otherwise Google deleted the backups too
Google cloud ceo says “it won’t happen anymore”, it’s insane that there’s the possibility of “instant delete everything”
Just an FYI in case you don’t follow Cloud news but Google has deleted customers accounts on multiple occasions and has been for literal years. This time they just did it to someone large enough to make the news. I work in SRE and no longer recommend GCP to anyone.
They had backups at multiple locations, and lost data at multiple (Google Cloud) locations because of the account deletion.
They restored from backups stored at another provider. It may have been more devastating if they relied exclusively on google for backups. So having an “offsite backup” isn’t enough in some cases, that offsite location need to be at a different provider.
@Hirom With "offsite" I mean either a different cloud provider or own hardware (if you hold your regular data at some cloud provider, like in this case).
Having a backup at a cloud provider is fine, as long as there is at least one other backup that isn’t with this provider.
Cloud provider seems to do a good job protecting against hardware failure, but can do poorly with arbitrary account bans, and sometimes have mishaps due to configuration problems.
Whereas a DIY backup solution is often more subject to hardware problems (disk failure, fire, flooding, theft, …), but there’s no risk of account problem.
A mix is fine to protect against different kind of issues.
as long as there is at least one other backup that isn’t with this provider.
Which is exactly what I was saying.
Any services used with a cloud provider should be treated as 1 entity, no matter how many geo-locations they claim your data is backed up to because they are a single point from which all those can be deleted.
When I was last involved in a companies backups, we had a fire safe in the basement, we had an off-site location with another fire safe & third copies would go off to another company that provided a backup storage solution so for all backups to be deleted, someone had to go right out of their way to do so. Not just a simple deletion of our account & all backups are wiped.
That company had the foresight to do something similar & it’s saved them. [edited - was on the tube when I wrote this and didnt see the autocorrect had put ‘comment’, not ‘company’]
“This is an isolated, ‘one-of-a-kind occurrence’ that has never before occurred with any of Google Cloud’s clients globally. This should not have happened.
I don’t believe this is what that rare, what I believe is that this was the fist time it happened to a company with enough exposure to actually have in impact and reach the media.
Either way Google’s image won’t ever recover from this and they just lost what small credibility they had on the cloud space and won’t be even considered again by any institution in the financial market - you know the people with the big money - and there’s no coming back from this.
It has 100% happened before and just never been admitted to. I have both 1st hand dealt with the aftermath and heard from other smaller companies about it. I work at medium sized MSP and disaster recovery is in my wheelhouse.
While UniSuper normally has duplication in place in two geographies, to ensure that if one service goes down or is lost then it can be easily restored, because the fund’s cloud subscription was deleted, it caused the deletion across both geographies.
TFW your BCDR gets disastered.
Also “massive misconfiguration” is the “spontaneous disassembly” of cloud computing. i’m sure it’s mutiple systems are misconfigured causing chaos but it sounds hilarious.
They said the outage was caused by a misconfiguration that resulted in UniSuper’s cloud account being deleted, something that had never happened to Google Cloud before.
Bullshit. I’ve heard of people having their Google accounts randomly banned or even deleted before. Remember when the Terraria devs cancelled the Stadia port of Terraria because Google randomly banned their account and then took weeks to acknowledge it? The only reason why Google responded so quickly to this is because the super fund manages over $100b and could sue the absolute fuck out of Google.
This happened to me years ago. Suddenly got a random community guidelines violation on YouTube for a 3 second VFX shot that was not pornographic or violent and that I owned all the rights to. After that my whole Google account was locked down. I never found out what triggered this response and I could never resolve the issue with them since I only ever got automated responses. Fuck Google.
This sort of story is what made me switch away from Google Fi and ultimately mostly degoogling. Privacy was a big part later on, but initially it was realizing that a YouTube comment or a file in my drive could get my cell service turned off.
Hardly. It’s several colocated computers/drives designed to survive major events. It’s insane to me sys admins still think their 7 year old desktop sitting in a closet offers the same level of protections.
thats why i am trying to explain to my family since forever. their answer always amounts to something like “it would be illegal for them to look at my data!” like those companies would care. .
Yeah there’s that, and the fact that you have no control over how much the bill will be each renewal period. Those two things kept me off the cloud for anything important.
And some functions don’t support hard limits, you’d have to set up a script monitoring load and literally take down your service if you get near the max
Are you sure you’ve not just read bad stuff without verification on the internet and feel the need to chime in on something you don’t fully understand?
Then please, point out the specific policies you are referring to. And don’t bother with consumer services. I’m looking for the enterprise services that a business would use. So which policies do you take issue with?
Me too as a programmer that uses Google cloud to store government information. Which bit of the policy says they are going to access your data, shouldn’t take you long to link it to me if you read them as much as you say. Unless what you’re actually doing is spreading misinformation and bullshit.
I’m not the one who you were responding to, but considering google’s history, I don’t believe anything they claim, because they have lied so many times in the past, and because every “privacy guarantee” they provide is practically unprovable. It’s nothing more than wishful thinking to think that google does nothing with government data stored with them, with google classroom data of millions of children, and others. They have shown that they can’t be trusted.
Same way companies know they’ve been hacked. I’m making the assumption you’re non technical, given the question. But there are many ways such as access logs, server monitoring etc
But there are many ways such as access logs, server monitoring etc
Which are all in the control of the company running the servers. If we trust the company, we can trust them giving honest information on these, but if we don’t trust the company… they could just redact logs or even straight out fake them
b2b and audited security standards are a whole different thing - you deal with finance and health you’ve gotta prove to a 3rd party over and over that you have controls and technology in place to make sure you aren’t lying
This. Even if by some miracle Google isn’t accessing everything on corporate cloud, it is an evil company and the policy can change. It’s a very untrustworthy and unreliable base for a business. And I’m not even talking about the fact that businesses that pay for the cloud are financially supporting Google
and you know the security standards that are achievable on google cloud entirely negate your point right? their cloud offering is a totally different beast
It’s absolutely not. If you are at any kind of scale whatsoever, your yearly spend will be a minimum of 2x at a cloud provider rather then creating and operating the same system locally including all the employees, contracts, etc.
Yes, and they’re the company’s resources so they theoretically do what’s best for the company as opposed to hoping Google or (godforbid Microsoft) does it.
The money gets paid either way, and if you have good people it’s often the right call to keep it in house but inevitably somebody read a business book last year and wants to layoff all the IT people and let Google handle it “for savings”. Later directors are amazed at how much money they’re spending just to host and use the data they used to have in-house because they don’t own anything anymore.
There are still benefits - cloud DevOps tools are usually pretty slick, and unless your company has built a bunch of those already or is good about doing it, it might still be worth it in terms of being able to change quickly. But it’s still a version of the age old IT maxim to never own or build it yourself when you can pay someone a huge subscription and then sue them if you have to. I don’t like it, but it’s pretty much iron in the executive suite.
As a result, IT departments or companies spend much more than half of their time - totalling years or decades - moving from whatever they were using to whatever is supposed to be better. Almost all of that effort is barely break-even if not wasted. That’s just the nature of the beast.
If you are a small company then yes. But i would argue that for larger companies this doesn’t hold true. If you have 200 employees you’ll need an IT department either way. You need IT expertise either way. So having some people who know how to plan, implement and maintain physical hardware makes sense too.
There is a breaking point between economics of scale and the added efforts to coordinate between your company and the service provider plus paying that service providers overhead and profits.
If coordinating with service providers is hard for a firm, I would argue the cost effective answer isn’t “let’s do all this in house”. Many big finance firms fall in this trap of thinking it’s cheaper to build v buy, and that’s how you get everyone building their own worse versions of everything. Whether your firm is good at the markets or kitchens or travel bookings, thinking you can efficiently in-source tech is a huge fallacy.
it is not about it being hard. It simply creates effort to coordinate. And this effort needs to be considered. If you do things externally that means there is two PMs to pay, you need QMs on both sides, you need two legal/contract teams, you need to pay someone in procurement and someone in sales…
I agree with you that doing software inhouse when there is good options on the market is usually not a good idea. But for infrastructure i don’t see there to be as much of an efficiency loss. Especially as you very much need experts on how to set things up in a cloud environment and you better look carefully at how many resources you need to not overpay huge amounts.
G Suite is a legitimate option for small-medium businesses. It’s seen as the cheaper, simpler option versus Azure. I usually recommend it for nonprofits as they have a decent free option for 501c3 orgs.
For large businesses, you essentially have two ways to spend money:
OPEX: “operational expenditure” - this is money that you send on an ongoing basis, things like rent, wages, the 3rd party cleaning company, cloud services etc. The expectation is that when you use OPEX, the money disappears off the books and you don’t get a tangible thing back in return. Most departments will have an OPEX budget to spend for the year.
CAPEX: “capital expenditure” - buying physical stuff, things like buildings, stock, machinery and servers. When you buy a physical thing, it gets listed as an asset on the company accounts, usually being “worth” whatever you paid for it. The problem is that things tend to lose value over time (with the exception of property), so when you buy a thing the accountants will want to know a depreciation rate - how much value it will lose per year. For computer equipment, this is typically ~20%, being “worthless” in 5 years. Departments typically don’t have a big CAPEX budget, and big purchases typically need to be approved by the company board.
This leaves companies in a slightly odd spot where from an accounting standpoint, it might look better on the books to spend $3 million/year on cloud stuff than $10 million every 5 years on servers
Excellent explanation, however, technically it does not constitute an “odd spot.” Rather, it represents a “100% acceptable and evident position” as it brings benefits to all stakeholders, from accounting to the CEO. Moreover, it is noteworthy that investing in services or leasing arrangements increases expenditure, resulting in reduced tax liabilities due to lower reported profits. Compounding this, the prevailing high turnover rate among CEOs diminishes incentives for making significant long-term investments.
In certain instances, there is also plain corruption. This occurs when a supplier offering services such as computer and server leasing or software, as well as company car rentals, is owned by a friend or family member of a C-level executive.
Add comment