PT, AWS's Service Health Dashboard warned CloudWatch users: "Customers may experience delayed or missing data points for CloudWatch metrics and CloudWatch alarms may transition into 'INSUFFICIENT_DATA' state if set on delayed metrics." It was simply a matter of increased latencies and outright delays in its operations.Īt 4:23 a.m. At no time did EC2 services cease functioning. Likewise, by 3:51 am there were increased error rates for invoking both Spot Instance APIs (the low-priced, demand-sensitive virtual servers offered by Amazon) and new EC2 instance launches. PT alarms went off that CloudWatch, AWS's monitoring service, was slowing down inside US-East-1. The loss of DynamoDB had a swift impact on the more general infrastructure. PT, Amazon's core EC2 service in Ashburn began to experience greater latencies and error rates.
PT notice, Amazon informed its customers it would need to stifle or "throttle" the activity of service APIs in order to work on DynamoDB's recovery. The services that carried a red warning in Northern Virginia, in addition to DynamoDB, included: Amazon Email Service, Amazon Workspaces, Simple Queue Service, Lambda, Amazon CloudFormation, Simple Workflow Service, Simple Notification Service, Amazon CloudWatch, and Auto Scaling. Twelve of them were only slowed or temporarily delayed ten warranted the red, "some customers may experience an outage" symbol on the Service Health Watch dashboard and thirteen received the yellow symbol warning of a persistent slowdown.
Such metadata is critical to how a NoSQL system functions, and how services that depend on DynamoDB function if something goes awry, as it turns out.Ī total of 35 services, many of them in US-East-1, were affected by the DynamoDB outage. The metadata service controls the names of tables and partitions, the attributes of the table's primary key, and the table's read-write requirements, among other things. This is an internal sub-service which manages table and partition information." PT, about two hours after the incident began, the AWS Service Health Dashboard reported: "The root cause began with a portion of our metadata service within DynamoDB. AWS identified the DynamoDB metadata service as the incident's cause within an hour of its start. Just as much of Facebook runs on the NoSQL system Cassandra, much of Amazon depends on the unstructured data system it invented for its own operations, DynamoDB.
0 kommentar(er)
