BCN Hosted - Unable to access Microsoft 365 Services – Incident details

This incident has been resolved by Microsoft. Final update from them below via Azure status history | Microsoft Azure

What happened?

Between approximately at 11:45 UTC and 19:43 UTC on 30 July 2024, a subset of customers may have experienced issues connecting to a subset of Microsoft services globally. Impacted services included Azure App Services, Application Insights, Azure IoT Central, Azure Log Search Alerts, Azure Policy, as well as the Azure portal itself and a subset of Microsoft 365 and Microsoft Purview services.

What do we know so far?

An unexpected usage spike resulted in Azure Front Door (AFD) and Azure Content Delivery Network (CDN) components performing below acceptable thresholds, leading to intermittent errors, timeout, and latency spikes. While the initial trigger event was a Distributed Denial-of-Service (DDoS) attack, which activated our DDoS protection mechanisms, initial investigations suggest that an error in the implementation of our defenses amplified the impact of the attack rather than mitigating it.

How did we respond?

Customer impact began at 11:45 UTC and we started investigating. Once the nature of the usage spike was understood, we implemented networking configuration changes to support our DDoS protection efforts, and performed failovers to alternate networking paths to provide relief. Our initial network configuration changes successfully mitigated majority of the impact by 14:10 UTC. Some customers reported less than 100% availability, which we began mitigating at around 18:00 UTC. We proceeded with an updated mitigation approach, first rolling this out across regions in Asia Pacific and Europe. After validating that this revised approach successfully eliminated the side effect impacts of the initial mitigation, we rolled it out to regions in the Americas. Failure rates returned to pre-incident levels by 19:43 UTC - after monitoring traffic and services to ensure that the issue was fully mitigated, we declared the incident mitigated at 20:48 UTC. Some downstream services took longer to recover, depending on how they were configured to use AFD and/or CDN.

What happens next?

Our team will be completing an internal retrospective to understand the incident in more detail. We will publish a Preliminary Post Incident Review (PIR) within approximately 72 hours, to share more details on what happened and how we responded. After our internal retrospective is completed, generally within 14 days, we will publish a Final Post Incident Review with any additional details and learnings. To get notified when that happens, and/or to stay informed about future Azure service issues, make sure that you configure and maintain Azure Service Health alerts – these can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/ash-alerts. For more information on Post Incident Reviews, refer to https://aka.ms/AzurePIRs. Finally, for broader guidance on preparing for cloud incidents, refer to https://aka.ms/incidentreadiness.

Latest update from Microsoft

30 Jul 2024, 16:51 BST

Telemetry shows that the service has remained stable. We're continuing to monitor the service for an extended period of time to confirm resolution.

Latest update from Microsoft

30 Jul 2024, 15:31 BST

We've implemented a networking configuration change and some Microsoft 365 services have performed failovers to alternate networking paths to provide relief. Monitoring telemetry shows improvement in service availability, and we're continuing to monitor to ensure full recovery.

Latest update from Microsoft

30 Jul 2024, 14:14 BST

We've identified a potential networking issue and we're investigating this further.

Next update by:

Tuesday, 30 July 2024 at 16:00 BST

Latest update from Microsoft

30 Jul 2024, 13:54 BST

Our investigation is currently focused on networking infrastructure. This quick update is designed to give the latest information on this issue.

Latest update from Microsoft

Issue ID: MO842351
Affected services: Microsoft 365 suite
Status: Service degradation
Issue type: Incident
Start time: Jul 30, 2024, 1:21 PM GMT+1

User impact
Users may be unable to access multiple Microsoft 365 services.

More info
Users who are able to access Microsoft 365 service may experience latency or degraded feature performance.

Scope of impact
Impact is specific to some users who are served through the affected infrastructure, attempting to utilize multiple Microsoft 365 services or features.

Current status
Jul 30, 2024, 1:28 PM GMT+1
We're reviewing service monitoring telemetry to determine our next troubleshooting steps.
Next update by:
Tuesday, July 30, 2024 at 3:30 PM GMT+1

History of updates
Jul 30, 2024, 1:24 PM GMT+1
We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.