Announcing AWS Wavelength for delivering ultra-low latency applications for 5G
https://aws.amazon.com/about-aws/whats-new/2019/12/announcing-aws-wavelength-delivering-ultra-low-latency-applications-5g/

AWS Wavelength embeds AWS compute and storage services at the edge of telecommunications providers’ 5G networks, and provides seamless access to the breadth of AWS services in the region. AWS Wavelength enables you to build applications that serve mobile end-users and devices with single-digit millisecond latencies over 5G networks, like game and live video streaming, machine learning inference at the edge, and augmented and virtual reality.

AWS Wavelength brings AWS services to the edge of the 5G network, minimizing the network hops and latency to connect to an application from a 5G device. Wavelength delivers a consistent developer experience across multiple 5G networks around the world

As Sunday’s outage demonstrates, the Amazon Web Services cloud is critical to many of its more than 1 million customers. Data Center Frontier looks at Amazon’s cloud infrastructure, and how it builds its data centers. The company’s global network includes at least 30 data centers, each typically housing 50,000 to 80,000 servers. “We really like to keep the size to less than 100,000 servers per data center,”

Like Google and Facebook, Amazon also builds its own custom server, storage and networking hardware, working with Intel to produce processors

Inside Amazon’s Cloud Computing Infrastructure
http://datacenterfrontier.com/inside-amazon-cloud-computing-infrastructure/

This week we’ll look at Amazon’s mighty cloud infrastructure, including how it builds its data centers and where they live (and why).

Lifting the Veil of Secrecy … A Bit

Amazon has historically been secretive about its data center operations, disclosing far less about its infrastructure than other hyperscale computing leaders such as Google, Facebook and Microsoft. That has begun to change in the last several years, as Amazon executives Werner Vogels and James Hamilton have opened up about the company’s data center operations at events for the developer community.

“There’s been quite a few requests from customers asking us to talk a bit about the physical layout of our data centers,” said Werner Vogels, VP and Chief Technology Office for Amazon, in a presentation at the AWS Summit Tel Aviv in July. “We never talk that much about it. So we wanted to lift up the secrecy around our networking and data centers.”

A key goal of these sessions is to help developers understand Amazon’s philosophy on redundancy and uptime. The company organizes its infrastructure into 11 regions, each containing a cluster of data centers. Each region contains multiple Availability Zones, providing customers with the option to mirror or back up key IT assets to avoid downtime. The “ripple effect” of outages whenever AWS experiences problems indicates that this feature remains underutilized.

Scale Drives Platform Investment

In its most recent quarter, the revenue for Amazon Web Services was growing at an 81 percent annual rate. That may not translate directly into a similar rate of infrastructure growth, but one thing is certain: Amazon is adding servers, storage and new data centers at an insane pace.

“Every day, Amazon enough new server capacity to support all of Amazon’s global infrastructure when it was a $7 billion annual revenue enterprise,”

Amazon’s data center strategy is relentlessly focused on reducing cost, according to Vogels, who noted that the company has reduced prices 49 times since launching Amazon Web Services in 2006.

“We do a lot of infrastructure innovation in our data centers to drive cost down,” Vogels said. “We see this as a high-volume, low-margin business, and we’re more than happy to keep the margins where they are. And then if we have a lower cost base, we’ll hand money back to you.”

A key decision in planning and deploying cloud capacity is how large a data center to build. Amazon’s huge scale offers advantages in both cost and operations. Hamilton said most Amazon data centers house between 50,000 and 80,000 servers, with a power capacity of between 25 and 30 megawatts.

“It’s undesirable to have data centers that are larger than that due to what we call the ‘blast radius’,” said Vogels, noting the industry term for assessing risk based on a single destructive regional event. “A data center is still a unit of failure. The larger you built your data centers, the larger the impact such a failure could have. We really like to keep the size of data centers to less than 100,000 servers per data center.”

So how many servers does Amazon Web Services run? The descriptions by Hamilton and Vogels suggest the number is at least 1.5 million. Figuring out the upper end of the range is more difficult, but could range as high as 5.6 million, according to calculations by Timothy Prickett Morgan at the Platform.

Amazon leases buildings from a number of wholesale data center providers

An interesting element of Amazon’s approach to data center development is that it has the ability to design and build its own power substations. Tha specialization is driven by the need for speed, rather than cost management.

“You save a tiny amount,” said Hamilton. “What’s useful is that we can build them much more quickly. Our growth rate is not a normal rate for utility companies. We did this because we had to. But it’s cool that we can do it.”

But as its operations grew, Amazon followed the lead of Google and began creating custom hardware for its data centers. This allows Amazon to fine-tune its servers, storage and networking gear to get the best bang for its buck, offering greater control over both performance and cost.

“Yes, we build our own servers,” said Vogels. “We could buy off the shelf, but they’re very expensive and very general purpose. So we’re building custom storage and servers to address these workloads. We’ve worked together with Intel to make household processors available that run at much higher clockrates. It allows us to build custom server types to support very specific workloads.”

Amazon offers several EC2 instance types featuring these custom chips, a souped-up version of the Xeon E5 processor based on Intel’s Haswell architecture and 22-nanometer process technology.

AWS uses designs its own software and hardware for its networking, which is perhaps the most challenging component of its infrastructure. Vogels said servers still account for the bulk of data center spending, but while servers and storage are getting cheaper, the cost of networking has gone up.

“The way most customers work is that an application runs in a single data center, and you work as hard as you can to make the data center as reliable as you can, and in the end you realize that about three nines (99.9 percent uptime) is all you’re going to get,”

“Building distributed development across multiple data centers, especially if they’re geographically further away, becomes really hard,”

Netflix, Tinder, Airbnb and other big names were crippled or thrown offline for millions of people when Amazon suffered what’s now revealed to be a cascade of cock-ups.

On Sunday, Amazon Web Services (AWS), which powers a good chunk of the internet, broke down and cut off websites from people eager to stream TV, or hookup with strangers; thousands complained they couldn’t watch Netflix, chat up potential partners, find a place to crash via Airbnb, memorize trivia on IMDb, and so on.

Today, it’s emerged the mega-outage was caused by vital systems in one part of AWS taking too long to send information to another part that was needed by customers.’

In technical terms, the internal metadata servers in AWS’s DynamoDB database service were not answering queries from the storage systems within a particular time limit.

DynamoDB tables can be split into partitions scattered over many servers.

At about 0220 PT on Sunday, the metadata service was taking too long sending back answers to the storage servers.

At that moment on Sunday, the levee broke: too many taxing requests hit the metadata servers simultaneously, causing them to slow down and not respond to the storage systems in time. This forced the storage systems to stop handling requests for data from customers, and instead retry their membership queries to the metadata service – putting further strain on the cloud.

It got so bad AWS engineers were unable to send administrative commands to the metadata systems.

Other services were hit by the outage: EC2 Auto Scaling, the Simple Queue Service, CloudWatch, and the AWS Console feature, suffered problems.

Ask Amazon about its AWS data centres and you’ll get get this response: Amazon doesn’t talk about its data centres. Until its chief technology officer pitches in, that is.

AWS is becoming to many what Windows once was: a platform for doing business. It has started as something that let enterprises free themselves from the yoke of owning their own servers. Now it’s letting them deliver new services.

Another important group of customers are internet pure plays who, again, don’t need to set up and run their own servers and infrastructure. These include everything from fundraising efforts such as Just Giving, a service for individuals and groups to raise funds online, to Omnifore – music streaming infrastructure employed by SiriusXM and Sony Music Unlimited.

Just Giving and Omnifore sit between their customers and the raw AWS infrastructure that for the non-techie is still difficult to knit together. What they rely on are hundreds of thousands of servers and network switches that Amazon has custom designed and built, working with Intel and others. Servers are grouped into, yes, data centres, which comprise Amazon’s Availability Zones, which themslves in turn make up regions – there’s 10 regions and 28 zones.

Each region is comprised of two or more Availability Zones and each zone has at least one data centre. No one data centre serves two Availability Zones, while some Zones are served by up to six data centres. Data centres must also be on different power grids, so no one power outage can take down a Zone.

Availability Zones are AWS’s way to circumvent the problems of back-up and latency that traditionally dog wide-area computing. Traditionally, a company in, say, New York might have disaster back-up in New Jersey, with data also replicated across the US in Los Angeles.

However, according to Vogels: “This old replication was deemed not fit for scale. One transaction is 1-2 milliseconds and replicating that will cost you 100 milliseconds. Then if you have to do fail over from New York to LA it’s a nightmare – failing back is even worse. Integrating a failed system into a live system is a nightmare.”

To solve latency, Amazon built Availability Zones on groups of tightly coupled data centres. Each data centre in a Zone is less than 25 microseconds away from its sibling and packs 102Tbps of networking.

As for those data centres, each is capped at 80,000 servers – determined to be the upper optimum limit – but contains at least 50,000. Servers are built by Amazon, working with Intel and other manufacturers. These aren’t cheap-o boxen, according to Vogels.

“Don’t think these are white-box servers,”

Amazon has also stripped out unwanted features that come with standard, off-the-shelf servers. Gone are audio chips and power transformers

Gartner analyst Kyle Hilgendorf has spotted something very interesting: Amazon Web Services seems to have stopped rating cloud servers based on EC2 compute units (ECUs), its proprietary metric of computing power.

ECUs were an odd metric, as they were based on “… the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor … equivalent to an early-2006 1.7 GHz Xeon”.

Elastic Compute Units have been replaced with processor information and clock speed.

Amazon Web Services (AWS) chief technology officer Werner Vogels believes the cloudy colossus could, in the future consider using ARM CPUs, or even offering RISC-as-a-service to help those on legacy platforms enjoy cloud elasticity.

AWS, he added, is “always looking for efficiency” and as “power management for ARM is considered state of the art” it makes sense to consider it.

Amazon is one of the most technically influential companies operating today – but you wouldn’t know it, thanks to a dearth of published research papers and negligible code contributions to the open-source projects it relies on.

This, according to multiple insiders, is becoming a problem. The corporation is described as a “black hole” because improvements and fixes for the open-source software it uses rarely see the light of day. And, we’re told, that policy of secrecy comes right from the top – and it’s driving talent into the arms of its rivals.

This secretiveness, “comes from Jeff,” claimed another source. “It’s passed down in HR training and policy. It’s all very clear.”

Though a select few are permitted to give public talks, when they do, they disclose far less information about their company’s technology than their peers.

“Amazon behaves a lot like a classified military agency,” explained another ex-Amazonian

Multiple sources have speculated to us that Amazon’s secrecy comes from Jeff Bezos’ professional grounding in the financial industry, where he worked in trading systems. This field is notoriously competitive and very, very hush-hush. That may have influenced his thoughts about how open Amazon should operate, as does his role in a market where he competes with retail giants such as Walmart.

But one contact argued that a taciturn approach may not be appropriate for the advanced technology Amazon has developed for its large-scale cloud computing business division, Amazon Web Services.

“In the Amazon case, there is a particular schizophrenia between retail and technology, and the retail culture dominates,” explained the source. “Retail frugality is all about secrecy because margins are so small so you can’t betray anything – secrecy is a dominant factor in the Amazon culture.

“It’s a huge cost to the company.”

Amazon Web Services (AWS) has drawn up borders within its cloud with a new ‘Geostriction’ feature for its CloudFront service.

CloudFront is AWS’ content distribution offering and speeds downloads for all manner of media, often by locating it closer to users.
Click here

AWS can now ensure that only the users you want – or at least those within borders you desire – can access content served from CloudFront thanks to the Geo Restriction feature. Amazon says the new feature means “you can choose the countries where you want Amazon CloudFront to deliver your content.”

You might wish to do that, AWS says, because “licensing requirements restrict some media customers from delivering movies outside a single country.”

If you have invested in the creation of “golden” Linux images suitable for your on-premises environment, I have some good news for you.