off-prem<\/p>\n
Maintenance mistake caused immediate issues and took out 27 services<\/p>\n
Azure users who use resources in Microsoft\u2019s West US region endured an uncomfortable day after the tech giant cut off access to its Californian cloud outpost.<\/p>\n
As Microsoft explains in its preliminary post incident review<\/a>, at 14:44 UTC on July 23rd (07:44 AM Pacific Time), the company started \u201croutine device maintenance.\u201d<\/p>\n That effort immediately produced problems.<\/p>\n \u201cMultiple Azure services began to detect and correlate service degradation,\u201d Microsoft\u2019s review reads.<\/p>\n A minute later, Microsoft says folks from its networking and services teams, plus people it describes as \u201cincident responders\u201d began \u201creviewing traffic anomalies, routing behavior, packet loss signals, and recent changes.\u201d<\/p>\n Microsoft says this sort of maintenance job requires isolating specific network paths, and that it uses a process that \u201cconverts these requests into system-readable requests and verifies that at least one of the two redundant paths remains healthy.\u201d We\u2019re told the company makes \u201csafety checks to confirm the work will be impact-less.\u201d<\/p>\n The mere fact you are reading this story shows that process instead delivered an unintended impact.<\/p>\n \u201cIn this case, a bug in the request conversion system incorrectly marked additional devices as a part of the maintenance event and caused a set of IP routes to be removed from more devices than intended,\u201d Microsoft confessed. \u201cThe routes were removed between our datacenter and wide-area network, impacting traffic entering or exiting the region.\u201d<\/p>\n Microsoft\u2019s incident report says the issue \u201cinitially presented as large-scale route churn in our Wide-Area Network\u201d and that its investigations \u201clater found that route removal was from a datacenter in the West US region.\u201d<\/p>\n Some time between 16:00 UTC and 17:45 UTC Microsoft identified \u201crecent fiber maintenance activity, which we correlated to the identified routing behavior.\u201d<\/p>\n At 17:45 the company started rolling back changes, and by 18:26 Microsoft\u2019s WAN was back to its best.<\/p>\n By 19:41 UTC, all impacted services had fully recovered \u2013 and Microsoft joined AWS<\/a> and Google<\/a> in offering recent examples of how clouds can be rather more fragile than advertised. \u00ae<\/p>\n","protected":false},"excerpt":{"rendered":"off-prem Maintenance mistake caused immediate issues and took out 27 services Azure users who use resources in Microsoft\u2019s…\n","protected":false},"author":2,"featured_media":410448,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[7,9,8],"class_list":["post-410447","post","type-post","status-publish","format-standard","has-post-thumbnail","category-california","tag-california","tag-california-headlines","tag-california-news"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/us-ca\/wp-json\/wp\/v2\/posts\/410447","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/us-ca\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/us-ca\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us-ca\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us-ca\/wp-json\/wp\/v2\/comments?post=410447"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/us-ca\/wp-json\/wp\/v2\/posts\/410447\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us-ca\/wp-json\/wp\/v2\/media\/410448"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/us-ca\/wp-json\/wp\/v2\/media?parent=410447"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/us-ca\/wp-json\/wp\/v2\/categories?post=410447"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/us-ca\/wp-json\/wp\/v2\/tags?post=410447"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}