All: please don't post low-effort comments that merely react to the first association you have. We're trying for curious conversation here, which is something else.
A few years ago I implemented a top to bottom ISO27k1 ISMS for a client handling extremely sensitive and mission-critical data for industry.
One risk I recommended controls for was that of a fire and/or flood at their primary datacentre for their client-facing offerings - this datacentre. I’ve experienced the misery of a datacentre oops myself, firsthand, twice, and it’s a genuine risk that has to be mitigated.
At my insistence, I had them burn hundreds of man-hours ensuring that they could failover to a new environment in a different datacentre with a bare minimum of fuss, as what I arrived to was an all the eggs in one basket situation. It took a fair bit of re-engineering of how deployments worked, how data was replicated, how the environment was configured - but they got there, and the ISMS was put into operation, and was audited cleanly by a reputable auditor, and everyone lived happily ever after.
Except… they were acquired by private equity. Who had no truck with all of this costly prancing about with consultants and systems. Risk register? Why do we need this? What value does it add today? ISO27k1? Don’t be silly. We have that certificate. You don’t need it. Dev team, ops team, leadership — almost everyone — ejected and replaced with a few support staff.
There's that beautiful German word again... schadenfreude. I have had similar discussions multiple times in the last year and the magic thinking around the cloud is so strong that it is sometimes impossible to get through. The fact that cloud stuff can go down and that in the end it is your data and no amount of cloud credits are going to help you if your data is lost seems to be utterly beyond some people's comprehension.
In this case it wasn’t so much “the cloud is invincible!” as “this appears to be a cost centre” - their whole shtick is to boost profitability in the short term by gutting businesses, and then selling them onwards to some other finance sucker.
I am sure that my name is currently being damned in a boardroom in Chicago, as as the person who warned of this, I am likely seen as responsible.
This really has nothing to do with cloud and is more of an "all eggs in one basket" problem. I wish people would stop painting cloud itself as less capable.
The fact is, most cloud providers offer multiple regions, which have the capability of giving you more geographic redundancy than most companies that operate in their own datacenters have.
Whether you choose to adopt a multi-region or multi-datacenter architecture is really orthogonal to whether you choose cloud or on-prem.
You're not far off: the batteries are (probably) made of lithium.
Also, why batteries in a datacenter? When you implement a flush() command at the lowest level you're faced with two choices: 1) actually write to disk, then return from the call, 2) write to some cache/RAM and have just enough battery locally to ensure that you can write it to disk even if all power goes out.
Then there's the other problem of surviving long enough between a power interruption and diesel generators starting up. But this is a smaller problem, rebooting all instances in a datacenter is less bad than losing some data that was correctly flush()ed by software. Bad flush() behaviour can result in errors that cannot be recovered from without a complicated manual intervention (for example if it causes corrupted and unreadable database files).
The batteries in the datacenter are simply there to hold the power until the generators are all up and running, and the phases are in sync.
They create 3 separate arrays of batteries in each back. Each array represents a power phase, A-B-C.. if I remember correctly, each array has a number of low voltage/2000 amp batteries connected in series to make up for a 2000amp 480 volt leg on the other end.
In a tier 4 plus+1 datacenter, they have 4 battery rooms and 4 generators for each data pod. You have a primary generator and UPS battery set, and a backup generator set for each pod. And then that generator set has its own primary and secondary backup set. The end result is that they can work on any piece of equipment without interrupting power. In the event they lost the primary set or needed to take it offline for maintenance, they have the whole secondary redundant set to fallback on.
The servers on the received on the power cord after it passes the switchgear never know that there has been power source changes on the other end.
Everything serious in the telecom/ISP infrastructure sector has a big -48VDC battery plant, or preferably separate A and B side -48VDC battery plants, to provide a significant buffer between power going Grid --> AC-to-DC Rectifiers --> Equipment, and when a generator can start up, warm up, and transfer switch does its job.
Even if a bunch of servers don't have any UPS or battery backup because they're designed to tolerate individual node (or whole rack, or whole row failures) the core network equipment in a datacenter will still have a huge battery plant.
Ideally if you have a chilled water loop for cooling you do not want it anywhere near your big-ass racks of batteries. Or near the racks that contain the rectifiers and DC breakers, distribution bus bars.
If you look at the battery racks in a traditional telco CO in the US for instance you will see that all of the cabling and batteries are a minimum of 1 foot off the floor, so that the whole place could theoretically flood and the DC distribution would remain unaffected. Same principle that applies to very traditional setups with wet-cell 2V lead acid batteries also applies to more modern things if building from scratch.
Very different trade offs in play for google who run with a relatively high tolerance for failure at the individual machine or even rack level. At one point I believe there were batteries in every rack, though I don’t know what they're building these days. A telco DC is gonna have more network interconnect with lower tolerance for failure due to capacity impact that isn’t easy to double.
Think like a fiber termination demarc vs an in-cluster mesh.
What I was saying above is that the 'core' of a google DC has a massive amount of network interconnect and needs for battery backup not very different from a big IX point or traditional "primary CO" for a city in a telco environment.
By square footage maybe 95% of a google DC might have no UPS or battery backup but the core network for things like routers and DWDM transport equipment absolutely will have such.
If they were unlucky enough that the burst cooling loop met with the battery plant for the core gear in a building or small campus of buildings....
The linked article says that there was a leak in a water cooling system, which in turn ended up in the battery system which caused a fire. But yeah it’s not coming from Google but second hand reports.
I can't ignore the feeling that Google Cloud is sub par compared to AWS. How did this again cause a multi zone failure. Why haven't they fixed those dependencies the last few times they had a full region failure.
zones and regions have different definition in google cloud than AWS.
Multiple zones are physically co-located and are not truly availability zones because the physical proximity causes shared fates even when they have independent systems (network, power) that should allow one to fail while another doesn't. Even two datacenters in the same city are prey to the same meteor.
I guess you just discovered the difference between a GCP Zone and an AWS Availability Zone.
By definition, AWS availability zones do not share fault domains, other than a geographic region up to hundreds of miles wide. Even for services used by multiple AZs, such as transit to the Internet and other regions, there are two transit centers operating in separate fault domains.
In contrast, many GCP Zones share the same physical datacenter. What they are actually providing you are simply different racks, rows of racks, or rooms in a single physical facility. Caveat emptor.
Wow, this is a big issue! Is there any way to guarantee AWS-level physical redundancy in GCP without paying the latency/inter-region data transfer (higher than inter-zone outside US: https://cloud.google.com/vpc/network-pricing#egress-within-g...) pricing?
A nice thing about EC2 is that you're getting a pretty dumb, predictable service. There have been multi-zone or global control plane issues but the physical metal has bona fide redundancy between zones/regions.
I don't know what it is like these days but us-east AZs used to be in different datacenters that were on different flood plains and power companies. They were just on a very high capacity (for the day) fiber ring. You still had a couple miles of light-delay latency in between them. A sufficiently big enough meteor, or a powerful enough massive hurricane, could probably take out multiple ones at the same time.
I believe the Google design is one big pool of machines, perhaps spread across a few buildings, but they hope that any failure only affects a few racks.
They will arrange/move workloads such that any one customer will only see an outage in one 'zone'.
It was not only a region outage, but a global outage!
"GCE Global Control Plane: Experienced a global outage, which has been mitigated. Primary impact was observed from 2023-04-25 23:15:20 PDT to 2023-04-26 03:45:30 PDT and impacted customers utilizing Global DNS (gDNS). A secondary global impact for aggregated list operation failures for customers with resources in europe-west9 has also been mitigated. Please see migration guide for gDNS to Zonal DNS for more information: https://cloud.google.com/compute/docs/internal-dns#migrating... "
Depending on the metric. I have not seen any incidents where the isolation between customers was breach. Azure had several.
Their compute offerings are better. We could go on.
On the other hand, Azure was(and still is) upfront about not having AZs - now that they have rolled out, hopefully those are not in the same building.
Not sure what kind of fire there was there, but once those automatic sprinkler systems get going, they are very difficult to stop.
Someone in my freshman college dorm decided to use one as a clothes hanger hook and broke the thermometer in there. The sprinkler damaged the entire floor with water and the floor below had spotty rain as well.
The fire department came and was mainly concerned about evacuating everyone rather than shutting the water off.
The water is typically chemically treated and has been sitting there for years as well -- very nasty stuff.
Always worth tracking down the sprinkler shut off valve in your residence/place of work. If you’re in a high rise it‘ll be the big red wheel on the sprinkler main in the fire stairs. If it’s a spurious activation you can just shut it off yourself, you don’t need to ask anybody’s permission.
The fire department is always going to prioritize safety of life, and after all it’s not their stuff getting soaked.
Fires develop crazy fast, too. Horrifying real-time footage of The Station nightclub fire is on YouTube. Within 2 minutes of ignition, fire is leaping out the windows. By 6 minutes in, the entire building is burning like a torch.
Anyone talking up firefighters like this clearly hasn't been around them much.
They're boys with toys that they don't frequently get to use and they work for the government. Follow the incentives. They'll do their jobs but they don't give a lot of fucks about things like "unnecessary property damage" and "other people's financial well being" and anything else not written in their KPIs.
I used to drive tow truck. I can't count the number of cars they totaled peeling the roof off (granted some were totaled anyway) because that was easy and cutting a door off was hard. And don't get me started on them and their stupid stands they use to prop shit up in the most questionable of ways...
I make many accounts because I don't like people snooping through my history. The Redditism of "I've looked back 10 years in your history, and will dismiss your comment because of something you said in 2015" is not a discussion I'm interested in.
Most of their calls are mundane stuff. And they leave a pretty decently wide path of destruction in doing that. We're talking like mundane situations where there is no urgency and no need to tear shit up in the interest of time.
I once arrived to a minor rollover after the cops but before fire. Nobody injured. Occupant trapped because she was a large lady and couldn't release her seatbelt upside down and was having difficulty unlocking the car because side curtain airbags.
I offered to flip the car and treat it like a lockout. "Customer" was fine with it. Cop was iffy. FD showed up, didn't want to hear it, broke the window, unlocked the car, opened the door, cut her belt rather than release it and dropped her on her face and then had difficulty getting her out. Now I get that they have "procedures" but this seems like a forest for the trees situation.
Or they'll show up, shut down two lanes for a minor fire on the shoulder and not move the trucks until the car is loaded on a tow truck and gone. Supposedly it's to keep them safe from being hit by traffic. Meanwhile here I am not blocking traffic to recover shit that broke down.
Sure, they'll save a life if the situation presents itself but they sure don't care about being tidy about it.
This happened in my freshman dorm as well. The broken sprinkler was on the 3rd or 4th floor and my room which was on the 1st got at least an inch of water.
I have no idea what's used these days but I remember someone telling me the dry powder that is sometimes used is very bad for computer hardware - not immediately, but within a couple of years the metal will show obvious signs of reaction with whatever is in it.
Halon has been banned for years because 1) it's bad for the ozone layer and 2) it'll kill you. Newer systems (FM-200, Inergen, etc.) fight the fire by removing heat instead of removing oxygen.
Halon is still used. Unfortunately the same properties that makes it effective also makes it harm the ozone layer. It does not just remove heat or oxygen, it directly interferes with the reaction involved in combustion, making things stop burning.
Wasn't there also this technique of lowering oxygen levels so much that humans can still survive but fire won't spread as fast? Or did this turn out to be too expensive?
[disclaimer: SRE @ Google, I was involved with the incident, obvious conflicts of interest]
Hey Dang, thanks for cleaning up the thread. One thing to note is that the title is not correct. The entire region is not currently down, as the regional impact was mitigated as of 06:39 PDT, per the support dashboard (though I think it was earlier). The impact is currently zonal (europe-west9-a), so having zone in the title as opposed to region would reflect reality closer.
There's not much emotion as the core team working on the huge outages is more like an "SRE for SRE". They are all people who've been with the company for a long time and they've been in the secondary seat for at least one previous big rodeo. Not to mention that we're all running a checklist that has been exercised multiple times and there's always somebody on the call who could help if a step fails.
Personally, I wasn't part this time for the actual mitigation of the overall Paris DC recovery, as I was busy with an unfortunate[0] side effect of the outage. These generate more anxiety, as being woken up at 6am and being told that nobody understands exactly why the system is acting this way is not great. But then again, we're trained for this situation and there are always at least several ways of fixing the issue.
Finally, it's worth repeating that incident management is just a part of the SRE job and after several years I've understood that it is not the most important one. The best SREs I know are not great when it comes to a huge incident. But, they're work has avoided the other 99 outages that could have appeared on the front page of Hacker News.
Life and experience, if you're looking for a short answer. For example, last year we had an outage in London[0] and the folks who worked on it learnt a lot. Now, they applied the learnings in this incident.
Based on that thread it sounds like only AWS guarantees that their AZs are in physically separate DCs, while for Google and Microsoft AZs could be in separate buildings of the same DC facility.
Yes. Azure and GCPs numbers on the size of their AZs and such are more marketing spin than hard engineering. AWS keeps these in separate physical locations to provide true separation. While there have been tech related regional incidents at AWS a physical event disabling multiple AZs would be extremely unlikely given their much more robust and geographically distributed design. If such a physical event had happened in AWS it would have been a non-event with things just failing over to other AZs.
Other cloud providers mostly just vaguely put things in another part of the building and say it’s “a separate AZ” but as GCPs woes highlighted that’s corner cutting that bites badly when the whole building has a problem.
> If such a physical event had happened in AWS it would have been a non-event with things just failing over to other AZs.
In many cases in AWS an availability zone is actually composed of multiple datacenters, each with their own redundancies. This may not be true for smaller regions, but in large ones it definitely is. In those cases, losing an entire datacenter would maybe take out a percentage of instances in that AZ. This has happened before and our production systems barely noticed other than provisioning new nodes to replace the failed health checks.
I think you misunderstand Google's infrastructure. I'm guessing that each GCP zone is actually a Borg Cell (see: https://storage.googleapis.com/pub-tools-public-publication-... ). Borg cells tend to be isolated from eachother in many ways in the physical layer (networking and management being a big one, not sure about power). So networking or machine management for an entire zone could go down and not affect other cells. Changes also tend to get pushed on a per-cell basis when they are Google wide rollouts.
I don’t know what you’re trying to say with Borg cells, the point of discussion is not that the network etc are separated, but that they’re physically separated in such a way that these kind of flooding wouldn’t affect different AZs, and that GCP is cutting corners here.
Obviously every cloud vendor recommends replicating data between multiple regions, but fact of the matter is that a lot of cloud services work much easier with redundancy within a single region than multi-region redundancy.
I guess it's different types of concerns. My feeling is that Google tries to optimize the resources of a datacenter, and the larger it is, the better things can scale. GCP Zones provide logical separation of machines for management (and network). There may be physical separation, but within a given region, GCP does not advertise this.
I think Google designs their datacenters for their own needs and expect you (a product running in their DCs) to distribute by region. Almost products at Google will be operating in multiple regions given the reach of most of our services, so DC design followed that need.
Based on GCP's docs, they still think region separate is better. Not sure why you wouldn't just do that?
If there is a catastrophic event (a large tornado hit AWS us-east-2), those buildings are pretty close to one another and both likely would be taken out, right? So you could lose multiple AZs since they are physically located so close to one another?
Yeah, you’re not getting what people are saying. AWS’s AZs are much more separated than GCPs. Your recommendation that one could build across regions isn’t what folks are talking about here since there is a big benefit to having geographically separate AZs in the same region. That’s where GCP is falling short here.
Funnily enough floods (GCP) and fires (OVH) are two of the 3 things AWS explicitly mentions in the Well Architected docs. For a lot of companies an AZ going down is an annoyance or bad day but a whole region going down could be a real continuity risk.
> Each Availability Zone is separated by a meaningful physical distance from other zones to avoid correlated failure scenarios due to environmental hazards like fires, floods, and tornadoes.
> but a whole region going down could be a real continuity risk
Very much so - Australia only got a second region this year, so if your work required data to remain in Australia, you just had to hope that ap-southeast-2 didn't have a major issues.
I'm sure there are plenty of other countries with only a single region.
It makes it very easy for me (as someone who comes from a world of physical datacentres) to reason about what an AZ is getting me, and also to understand the benefits of using AWS (not having to think about the details of power routing, blade switch vs top-of-rack vs core switch, storage cabling, blah blah blah).
If I have to think too hard and do too much work about how I lay applications out, I might as well just rent in a colo.
For physical zone separation you need to check the `supportsPzs` attribute when listing the zones (e.g. https://cloud.google.com/compute/docs/reference/rest/v1/zone..., but you should be able to find many other places where this attribute is surfaced).
Random datacenters should start advertising availability zones since they should have different fault domains anyway. Google can get away with this, why can't smaller companies?
Googles advice is not to rely on uptime in every region.
Instead aim for uptime in a few regions, and load balance your users to regions that are healthy.
That design is far cheaper for both google and for you - and, in the typical case, users still get nice low latency to a local datacenter, and only in the rare failure case might they have to wait for latency to some other region.
The recommendations are to run in multiple regions if you need this kind of redundancy. Run everything in a single region and you can be affected by an event like this.
It amazes me that in every market they serve, Amazon has no actual competitors from a feature perspective.
Like, Target does not compete with Amazon. They have a totally different home delivery model that is not in the same category of reliability or service.
I think it's because lots of amazons services are in 'winner takes all' markets.
No random online eshop can offer next day delivery across half the world unless they already have a logistics chain of 100,000 truck drivers spread across the world. But Amazon can.
Likewise, no cloud provider has enough data centers to offer multiple separate data centers in the same city, for hundreds of cities around the world. But Amazon does.
Any competitor can't offer amazons level of service until they get to amazon scale... Which they never will.
When I worked at AWS there was a similar scenario in eu-west-2. There was a fire in one of the availability zones (AZs). The fire suppression system kicked in and flooded the data center up to ankle or knee height. All the racks were powered off and the building was evacuated for hours (I don't remember the duration of the evacuation) until the water was pumped out.
But for the service team I worked for, our AZ-evacuation story wasn't great at the time and it took us tens of minutes to manually move out of the AZ, but at least there wasn't a customer-visible availability impact. Once we did it was just monitoring and baby-sitting until we got the word to move back in, I think it was 1-2 days later.
If you operate on AWS you work with the assumption that an AZ is a failure domain, and can die at any time. Surprisingly many service teams at AWS still operate services that don't handle AZ failure that well (at the time). But if you operate services in the cloud you have to know what the failure domain is.
> urprisingly many service teams at AWS still operate services that don't handle AZ failure that well (at the time)
Ouch, hopefully none of the major services? I recently had to look into this for work (for disaster recovery preparation) and it seemed like ECS, Lambda, S3, DynamoDB and Aurora Serverless (and probably CloudWatch and IAM) all said they handled availability zone failures transparently enough.
I’m familiar with Lambda and DynamoDB. When I left in 2022 they both had strong automated or semi-automated AZ evacuation stories.
I’m not that familiar with S3, but I never noticed any concerns with S3 during an AZ outage. I’m not at all familiar with Aurora Serverless or ECS.
For all AWS services you can always ask AWS Support pointed, specific questions about availability. They usually defer to the service team and they’ll give you their perspective.
Also keep in mind that AWS teams differentiate between the availability of their control and data planes. During an AZ outage you may struggle to create/delete resources until an AZ evacuation is completed internally, but already created resources should always meet the public SLA. That’s why especially for DR I recommend active-active or active-“pilot light”, have everything created in all AZs/regions and don’t need to create resources in your DR plan.
Okay good to know - Lambda seemed to suggest it could handle an availability zone going down without any trouble.
ECS Fargate's default is to distribute task instances across availability zones too but I'm assume if you use EC2 it might not be as straight-forward.
And that makes sense - I remember during the last outage that affected me it was a compute rather than data failure and the running stuff continued fine, just nothing new was getting created.
I can imagine clients who used one DC being impacted. But Google’s services would be designed for a single DC going down, right? Data would be eventually consistent (once they find and plug the hard drives in) but isn’t this the promise of the cloud and they’re (approximately) the best at using it.
I have to assume it’s a fault that not even distributed services can paper over. Eg lots of crucial data in flight and they’re reluctant to drop it. Can an expert weigh in?
I love Google’s post-mortems. This one will be epic.
> But Google’s services would be designed for a single DC going down, right?
Right. But nobody forces GCP's customers to design their services to be tolerant of a single DC failure. In fact as a business, actively not designing for such tolerance is an attractive cost-cutting measure.
Cloud customers have no control on which or how many 'datacenters' are used. That's not something that's even advertised or easily available to customers.
The logical units are regions and availability zones or the equivalent nomenclature in each cloud. One availability zone is expected to be one or more datacenters.
We have thousands of instances in AWS. I do not know - or care - where they are physically located(other than the region name, say, Oregon). I expect at most one availability zone to get impacted if a datacenter goes up in flames (and sometimes, just a portion of one). I mention in another comment that AWS has had issues before and production systems barely got impacted. And recovered with zero intervention - instances with failed health checks get replaced by brand new ones in whatever AZs are still operational.
> Data would be eventually consistent (once they find and plug the hard drives in)
At the level of abstractions cloud operates, no-one is plugging drives in – someone is, but you can never see it.
Most cloud workloads use network attached storage - when you can even see the logical drives (SaaS offerings may not even have that abstraction). We don't know (or care) how many physical hard drives exist, or where they are. Latency requirements probably dictate that they are close to the actual instances, but there's usually data replication going on even across DCs.
In addition to that, at least in AWS, if you have saved any volume snapshots at all, they will be in S3. This data will be replicated and underlying systems can even use it to restore lost or corrupted data without you even noticing and sometimes even without a recent snapshot, as storage keeps track of what blocks have been rewritten since the last snapshot. In a particularly bad case you might have to do a restore.
In almost a decade and number of volumes in the 6 digits (no clue how many drives that is!) we never had a single volume fail on AWS. Some got into a 'degraded' state and then recovered.
We haven't had any failures on GCP either. In the case of GCP, even faulty hypervisors are transparently worked around - we never notice other than some audit logs saying the VM was moved. They even preserve the network connections. AWS requires a stop/start to do the same, but your VM will be up and running in a different hypervisor (sometimes a different datacenter) in a couple of minutes, with all the storage.
Mind you, AWS promises eleven nines(!) of durability for S3.
When you do have locally attached storage, it's treated as ephemeral and it's gone if the instance restarts.
> I have to assume it’s a fault that not even distributed services can paper over.
If a single datacenter fails, since it _should_ be at most one AZ(this case seems to be different) that will depend on how the application is architected. Requests in flight will obviously fail, how big of a deal depends on the problem domain. For most web apps, this will cause a retry and that's the end of the story, others will be specifically engineered to deal with receiving multiple messages or dropping messages. For example, if you need at most once delivery guarantees, you need to take extra measures
Not all applications can survive an entire region going down. Some can, but that usually raises costs if you are continuously replicating data across regions. If you do that, then you should be able to steer traffic to the surviving regions. You can do that old-school by changing DNS records, or you could have fanciers solutions such as global anycast loadbalancers and have a single IP worldwide that still goes to the closest healthy region.
Nobody here with any thoughts for the operations/datacenter engineers trying to deal with stopping and cleaning up the disaster, just customers complaining...
"Thoughts and Prayers" type comments don't make for particularly interesting reading.
I think it's safe to assume that most people feel empathy for others struggling, whether or not they type it out regularly. Then again, some AI evangelists have had me questioning that assumption lately.
I think people who are complaining are stressed out about their own services being down.
If you’ve only ever used the cloud, you’re not necessarily aware of everything that’s involved at data centers. If you’re not familiar with them, I don’t think you’d know how many things can (literally) blow up in your face. If someone sees flooding, they generally aren’t thinking that it’ll lead to fires.
Anyway, just want to think that everyone generally has good intentions and just don’t know what’s ACTUALLY happening in the DC, or how much work it will be for the folks working in the DC to restore services.
Hopefully all the failsafes kicked in and worked and nobody was injured.
It's funny because AWS, at least for the services I knew of when I was there, did rolling deploys to each region over several days. us-east-1 was always the final day because it was the biggest region, so you'd think it'd the safest region since everything getting deployed was well-tested. But while I was there I remember at least 2 COEs where the root cause was basically, "us-east-1 had some hacky legacy configuration that no other region has and that wasn't known/accounted for."
I hope whoever is hosting data in that zone has thoroughly tested and verified backups offline or with another cloud provider. Of course, you can complete DC failover, depending upon service needs, but it costs more resources. Either way, timely tested backups are the only way to survive natural or manufactured disasters. Good luck to Google OPs team and everyone else involved with the GCP region in the EU.
Funny enough, Lucas Film suffered a same outage because their data center was backed up to the lake, and was basically under water, and then the wall started to leak.....
Yeah, me. I was one of the lead designers on the Lucas Presidio campus, including the DC there...
When we were still designing the site/systems/infra we would go to BigRock Ranch a lot for meetings and vendor interviews.
Its the datacenter at bigrock which has one wall which backups to the lake on site, and the DC is in the sub-level parking garage, and its rear wall was leaking due to the pressure from the lake on the wall.
Is your data center suffering an outage due to flooding from a faulty wall you built against the man-made lake you built next to your $ Billion data center??
Call California and Meyers to see if you qualify for PG&E benefits.
Sure is leaky. And a cloud is a bunch of water vapour that eventually comes crashing down to earth. I'll never understand how we decided it was a good metaphor for a place we run our services.
It's more that everyone advertises so many 9's of uptime, but in reality they don't count anything less than a total outage of an entire datacenter as actual downtime against that statistic.
I don’t see how this is a good argument? Yes, it’s someone else’s computers. That they rent me by the month or the hour or the second through a programmatic interface. That’s exactly the product I want. And it’s not surprising that it has outages sometimes because at the end of the day, yeah, it’s still just a bunch of computers someone has put in a room somewhere.
Yeah this is the usual backlash of experts vs marketing vs management.
Someone else managing your shit so you don't have to is a market in just about every industry, and it makes a ton of sense in tech where things don't even have to be on the same continent to work (or very specifically NEED to be on another one if you're international).
There's a ton of companies that have jumped to the cloud that probably shouldn't have, and even more who should've jumped, but not nearly as much as they did. Still it's a useful service.
Now of course it being a useful service, that also happens to be so obscenely expensive to start up barely anyone does it, means it comes with all the miserable obfuscation, bullshit fine print, total lack of support, and every other horrible thing we've come to expect from the modern world, but "It's just someone else's computer!" isn't changing any minds.
Either they already knew that or they never cared.
No, "serverless" is a toxic term. I get that it exists and I can't do anything about it except refuse to use it. It isn't something I misunderstand because I'm not a marketing person though. It is a term that I understand and disagree with.
Cloud Computing I agree is a usual concept that marketing and experts often disagree about.
"Multiple Google Cloud services in the europe-west9 region are impacted.
Description: Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones. We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage"
Emergency shutdown of multiple zones.
As of a few hours ago they changed the status to report just on us-west9-a.
Even the article linked mentions that Google initially reported that only zone A was affected, then got changed to report that the whole region was affected, now it changed to a single zone again.
Do you expect realtime updates whenever Google changes the story?
https://news.ycombinator.com/newsguidelines.html