Affected
- ResolvedResolved
Latest update from Microsoft:
Status
Service RestoredImpacted services
Exchange OnlineDetails
Title: Some users may have been unable to send Exchange Online email messagesUser impact: Some users may have been unable to send or receive Exchange Online email messages.
More info: Affected users may have received NDRs or the following error "503 5.5.1 bad sequence of commands"
Final status: We’ve determined that a recent change caused a large influx of free/busy Control Flow Messages (CFM) to be sent to the Exchange online infrastructure, resulting in stale free-busy information. An interceptor rule was applied to discard these CFM messages to mitigate impact for the free/busy issue, which inadvertently caused impact to mail flow.
We've removed the interceptor rule and verified that the impact to the mail flow has now been mitigated. We'll continue to investigate the free/busy issue, which is being communicated under the SI EX648815.
Due to the nature of the issue, the users who had received the NDRs during this event will need to resend those emails.
Scope of impact: Impact was specific to some users who are served through the affected infrastructure.
Start time: Tuesday, July 18, 2023, at 5:00 AM UTC
End time: Tuesday, July 18, 2023, at 8:25 AM UTC
Root cause: We’ve determined that a recent change caused a large influx of free/busy Control Flow Messages (CFM) to be sent to the Exchange online infrastructure, resulting in stale free-busy information. An interceptor rule was applied to discard these CFM messages to mitigate impact for the free/busy issue, which inadvertently caused impact to mail flow.
Next steps:
- We're reviewing our interceptor rule implementation procedures to understand why mail flow was inadvertently affected, in order to help prevent similar impact from happening again in the future.
- MonitoringMonitoring
18 July 2023 at 10:08 BST · Update from Microsoft
Current status: We’ve disabled an interceptor rule that we believe was causing impact to the mail flow. We’re monitoring the service to ensure the mail flow is restored.
- InvestigatingInvestigating
Please be advised that Microsoft have reported the following:
Title: Some users may be unable to send Exchange Online email messages
User impact: Users may be unable to send Exchange Online email messages.
More info: Affected users may receive the following error "503 5.5.1 bad sequence of commands"
Current status: We're reviewing service monitoring telemetry to isolate the root cause and develop a remediation plan.
Scope of impact: Impact is specific to some users who are served through the affected infrastructure.
Further updates to follow.