BCN Hosted Status - Incident history https://bcngroup.instatus.com BCN Hosted Thu, 26 Feb 2026 14:00:00 +0000 Issue with primary data center transit provider within BCN Hosted Services Type: Incident Duration: 6 minutes Affected Components: Hosted Platform, External Connectivity Feb 26, 14:28:07 GMT+0 - Resolved - Services are stable on our secondary provider. We will work with the primary provider on the cause of the issue. Feb 26, 14:00:00 GMT+0 - Monitoring - There is currently an issue with BGP routing on our primary transit provider which will have caused brief drops in connectivity to internet facing hosted services. We have migrated to our secondary provider and will continue to monitor. Type: Incident

Duration: 6 minutes

Affected Components: ,

<p><small>Feb <var data-var='date'> 26</var>, <var data-var='time'>14:28:07</var> GMT+0</small><br><strong>Resolved</strong> - Services are stable on our secondary provider. We will work with the primary provider on the cause of the issue. .</p> <p><small>Feb <var data-var='date'> 26</var>, <var data-var='time'>14:00:00</var> GMT+0</small><br><strong>Monitoring</strong> - There is currently an issue with BGP routing on our primary transit provider which will have caused brief drops in connectivity to internet facing hosted services. We have migrated to our secondary provider and will continue to monitor. .</p> ]]> Thu, 26 Feb 2026 14:00:00 +0000 https://bcngroup.instatus.com/incident/cmm3jyrs400rt10rw2sswum05 https://bcngroup.instatus.com/incident/cmm3jyrs400rt10rw2sswum05 Azure UK South Region Issues Type: Incident Duration: 4 hours and 37 minutes Affected Components: Azure Services Nov 20, 10:05:00 GMT+0 - Investigating - Microsoft have made it known that they are experiencing issues within the UK South region. Further details to follow. Nov 20, 11:12:22 GMT+0 - Investigating - The scope and scale of the impact appears to be limited. Official update pending from Microsoft. Nov 20, 14:42:20 GMT+0 - Resolved - Microsoft appear to have contained the issues, with limited impacted noted throughout the duration of the outage. Type: Incident

Duration: 4 hours and 37 minutes

Affected Components:

<p><small>Nov <var data-var='date'> 20</var>, <var data-var='time'>10:05:00</var> GMT+0</small><br><strong>Investigating</strong> - Microsoft have made it known that they are experiencing issues within the UK South region. Further details to follow..</p> <p><small>Nov <var data-var='date'> 20</var>, <var data-var='time'>11:12:22</var> GMT+0</small><br><strong>Investigating</strong> - The scope and scale of the impact appears to be limited. Official update pending from Microsoft..</p> <p><small>Nov <var data-var='date'> 20</var>, <var data-var='time'>14:42:20</var> GMT+0</small><br><strong>Resolved</strong> - Microsoft appear to have contained the issues, with limited impacted noted throughout the duration of the outage..</p> ]]> Thu, 20 Nov 2025 10:05:00 +0000 https://bcngroup.instatus.com/incident/cmi79m1zd012w11cylem2qsf7 https://bcngroup.instatus.com/incident/cmi79m1zd012w11cylem2qsf7 Azure Outage Type: Incident Duration: 15 hours and 57 minutes Affected Components: Azure Services Oct 29, 16:26:57 GMT+0 - Investigating - Microsoft have advised an ongoing incident impacting Azure. Further information can be found here: [Azure status](https://azure.status.microsoft/en-gb/status) **_Azure Portal Access Issues_** _We are investigating an issue with the Azure Portal where customers may be experiencing issues accessing the portal. More information will be provided shortly. This message was last updated at 16:18 UTC on 29 October 2025_ Oct 29, 16:36:30 GMT+0 - Identified - Latest update from Microsoft **_Azure Portal Access Issues_** _Starting at approximately 16:00 UTC, we began experiencing DNS issues resulting in availability degradation of some services. Customers may experience issues accessing the Azure Portal. We have taken action that is expected to address the portal access issues here shortly. We are actively investigating the underlying issue and additional mitigation actions. More information will be provided within 60 minutes or sooner._ _This message was last updated at 16:35 UTC on 29 October 2025_ Oct 29, 17:03:10 GMT+0 - Identified - Latest Update **_Azure Portal Access Issues_** _Starting at approximately 16:00 UTC, we began experiencing Azure Front Door issues resulting in a loss of availability of some services. In addition. customers may experience issues accessing the Azure Portal. Customers can attempt to use programmatic methods (PowerShell, CLI, etc.) to access/utilize resources if they are unable to access the portal directly. We have failed the portal away from Azure Front Door (AFD) to attempt to mitigate the portal access issues and are continuing to assess the situation._ _We are actively assessing failover options of internal services from our AFD infrastructure. Our investigation into the contributing factors and additional recovery workstreams continues. More information will be provided within 60 minutes or sooner._ _This message was last updated at 16:57 UTC on 29 October 2025_ Oct 30, 08:24:06 GMT+0 - Resolved - Microsoft have confirmed that the incident has been fully mitigated and resolved. Oct 29, 20:54:23 GMT+0 - Monitoring - Latest update from Microsoft as services begin to recover. **_Azure Front Door - Connectivity issues - Observing recovery_** _Starting at approximately 16:00 UTC on 29 October, 2025, customers and Microsoft services leveraging Azure Front Door (AFD) may have experienced latencies, timeouts, and errors. We have confirmed that an inadvertent configuration change was the trigger event for this issue._ **_Current status:_** _We initiated the deployment of our ‘last known good’ configuration, which has now successfully been completed. Customers may have begun to see initial signs of recovery. We are currently recovering nodes and routing traffic through healthy nodes, and as we make progress in this workstream, customers will continue to see improvement._ _Customer configuration changes will remain temporarily blocked while we continue mitigation efforts. We will notify customers once this block has been lifted._ _Some customers may also have experienced issues accessing the Azure management portal. We have failed the portal away from AFD to mitigate these access issues. Customers should now be able to access the Azure portal directly, and while most portal extensions are functioning as expected, a small number of endpoints (e.g., Marketplace) may still experience intermittent loading problems._ _At this stage, we anticipate full mitigation within the next four hours as we continue to recover nodes. This means we expect recovery to happen by 23:20 UTC on 29 October 2025\. We will provide another update on our progress within two hours, or sooner if warranted._ _Although we are seeing signs of recovery and have an estimated timeline, customers may also consider implementing existing failover strategies using Azure Traffic Manager to redirect traffic from Azure Front Door to their origin servers as an interim measure._ _Learn more about Azure Front Door failover strategies for AFD:_ <https://learn.microsoft.com/en-us/azure/architecture/guide/networking/global-web-applications/overview> _This message was last updated at 19:57 UTC on 29 October 2025_ Type: Incident

Duration: 15 hours and 57 minutes

Affected Components:

<p><small>Oct <var data-var='date'> 29</var>, <var data-var='time'>16:26:57</var> GMT+0</small><br><strong>Investigating</strong> - Microsoft have advised an ongoing incident impacting Azure. Further information can be found here: [Azure status](https://azure.status.microsoft/en-gb/status) **_Azure Portal Access Issues_** _We are investigating an issue with the Azure Portal where customers may be experiencing issues accessing the portal. More information will be provided shortly. This message was last updated at 16:18 UTC on 29 October 2025_.</p> <p><small>Oct <var data-var='date'> 29</var>, <var data-var='time'>16:36:30</var> GMT+0</small><br><strong>Identified</strong> - Latest update from Microsoft **_Azure Portal Access Issues_** _Starting at approximately 16:00 UTC, we began experiencing DNS issues resulting in availability degradation of some services. Customers may experience issues accessing the Azure Portal. We have taken action that is expected to address the portal access issues here shortly. We are actively investigating the underlying issue and additional mitigation actions. More information will be provided within 60 minutes or sooner._ _This message was last updated at 16:35 UTC on 29 October 2025_.</p> <p><small>Oct <var data-var='date'> 29</var>, <var data-var='time'>17:03:10</var> GMT+0</small><br><strong>Identified</strong> - Latest Update **_Azure Portal Access Issues_** _Starting at approximately 16:00 UTC, we began experiencing Azure Front Door issues resulting in a loss of availability of some services. In addition. customers may experience issues accessing the Azure Portal. Customers can attempt to use programmatic methods (PowerShell, CLI, etc.) to access/utilize resources if they are unable to access the portal directly. We have failed the portal away from Azure Front Door (AFD) to attempt to mitigate the portal access issues and are continuing to assess the situation._ _We are actively assessing failover options of internal services from our AFD infrastructure. Our investigation into the contributing factors and additional recovery workstreams continues. More information will be provided within 60 minutes or sooner._ _This message was last updated at 16:57 UTC on 29 October 2025_.</p> <p><small>Oct <var data-var='date'> 30</var>, <var data-var='time'>08:24:06</var> GMT+0</small><br><strong>Resolved</strong> - Microsoft have confirmed that the incident has been fully mitigated and resolved..</p> <p><small>Oct <var data-var='date'> 29</var>, <var data-var='time'>20:54:23</var> GMT+0</small><br><strong>Monitoring</strong> - Latest update from Microsoft as services begin to recover. **_Azure Front Door - Connectivity issues - Observing recovery_** _Starting at approximately 16:00 UTC on 29 October, 2025, customers and Microsoft services leveraging Azure Front Door (AFD) may have experienced latencies, timeouts, and errors. We have confirmed that an inadvertent configuration change was the trigger event for this issue._ **_Current status:_** _We initiated the deployment of our ‘last known good’ configuration, which has now successfully been completed. Customers may have begun to see initial signs of recovery. We are currently recovering nodes and routing traffic through healthy nodes, and as we make progress in this workstream, customers will continue to see improvement._ _Customer configuration changes will remain temporarily blocked while we continue mitigation efforts. We will notify customers once this block has been lifted._ _Some customers may also have experienced issues accessing the Azure management portal. We have failed the portal away from AFD to mitigate these access issues. Customers should now be able to access the Azure portal directly, and while most portal extensions are functioning as expected, a small number of endpoints (e.g., Marketplace) may still experience intermittent loading problems._ _At this stage, we anticipate full mitigation within the next four hours as we continue to recover nodes. This means we expect recovery to happen by 23:20 UTC on 29 October 2025\. We will provide another update on our progress within two hours, or sooner if warranted._ _Although we are seeing signs of recovery and have an estimated timeline, customers may also consider implementing existing failover strategies using Azure Traffic Manager to redirect traffic from Azure Front Door to their origin servers as an interim measure._ _Learn more about Azure Front Door failover strategies for AFD:_ <https://learn.microsoft.com/en-us/azure/architecture/guide/networking/global-web-applications/overview> _This message was last updated at 19:57 UTC on 29 October 2025_.</p> ]]> Wed, 29 Oct 2025 16:26:57 +0000 https://bcngroup.instatus.com/incident/cmhc7ki98007xwrhud1dnqt2p https://bcngroup.instatus.com/incident/cmhc7ki98007xwrhud1dnqt2p SonicWall Cloud Backup Incident Type: Incident Duration: 19 days and 51 minutes Affected Components: Vendor/3rd Party Oct 9, 10:10:23 GMT+0 - Investigating - SonicWall have advised the following: "Our investigation into the recent cloud backup security incident has confirmed that configuration files for customers who have used the firewall cloud backup service were accessed by an unauthorized party. While the specific exposure varies by environment, we are treating this matter with the highest priority." <https://www.sonicwall.com/support/knowledge-base/mysonicwall-cloud-backup-file-incident/250915160910330> BCN are working with SonicWall to establish a plan to address this incident. More to follow. Oct 9, 15:22:38 GMT+0 - Identified - Impacted clients have been contacted as BCN mobilise engineers to commence remediation as per the guidelines from SonicWall. Oct 10, 10:20:36 GMT+0 - Identified - ### **Q&A SECTION** **Q: What steps have been taken so far to investigate and remediate the issue?** A: BCN have worked directly with SonicWall and have mobilised a taskforce to identify, and rectify any affected devices. This will take time as there are a larger number of devices affected however we are working around the clock and prioritising devices by their risk factor. **Q: Has our device been affected?** A: Yes - if you received an email regarding this from BCN, your device has been affected. You will receive confirmation from BCN once your device has been remediated. **Q: What further actions are planned to ensure full resolution and protection?** A: SonicWall have provided instruction on the steps taking to remediate the issue, these can be found here - [Remediation Playbook](https://www.sonicwall.com/support/knowledge-base/remediation-playbook/250916130050523). BCN will perform these steps on your behalf. **Q: What can my company do in the meantime to reduce the risks?** A: BCN recommend disabling SonicWall SSLVPN until you have received confirmation that your device has been remediated. This will impact any remote users who rely on SSLVPN to access work resources and as such BCN have not made this change by default however if you wish to disable SSLVPNplease respond to this email to let us know. **Q: Will you let us know when our device has been remediated?** A: Yes - BCN will contact you to confirm upon completion. Oct 28, 11:01:21 GMT+0 - Resolved - The vast majority of devices have been remediated. Any remaining activities will be tracked through individual tickets. Type: Incident

Duration: 19 days and 51 minutes

Affected Components:

<p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>10:10:23</var> GMT+0</small><br><strong>Investigating</strong> - SonicWall have advised the following: "Our investigation into the recent cloud backup security incident has confirmed that configuration files for customers who have used the firewall cloud backup service were accessed by an unauthorized party. While the specific exposure varies by environment, we are treating this matter with the highest priority." <https://www.sonicwall.com/support/knowledge-base/mysonicwall-cloud-backup-file-incident/250915160910330> BCN are working with SonicWall to establish a plan to address this incident. More to follow..</p> <p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>15:22:38</var> GMT+0</small><br><strong>Identified</strong> - Impacted clients have been contacted as BCN mobilise engineers to commence remediation as per the guidelines from SonicWall..</p> <p><small>Oct <var data-var='date'> 10</var>, <var data-var='time'>10:20:36</var> GMT+0</small><br><strong>Identified</strong> - ### **Q&A SECTION** **Q: What steps have been taken so far to investigate and remediate the issue?** A: BCN have worked directly with SonicWall and have mobilised a taskforce to identify, and rectify any affected devices. This will take time as there are a larger number of devices affected however we are working around the clock and prioritising devices by their risk factor. **Q: Has our device been affected?** A: Yes - if you received an email regarding this from BCN, your device has been affected. You will receive confirmation from BCN once your device has been remediated. **Q: What further actions are planned to ensure full resolution and protection?** A: SonicWall have provided instruction on the steps taking to remediate the issue, these can be found here - [Remediation Playbook](https://www.sonicwall.com/support/knowledge-base/remediation-playbook/250916130050523). BCN will perform these steps on your behalf. **Q: What can my company do in the meantime to reduce the risks?** A: BCN recommend disabling SonicWall SSLVPN until you have received confirmation that your device has been remediated. This will impact any remote users who rely on SSLVPN to access work resources and as such BCN have not made this change by default however if you wish to disable SSLVPNplease respond to this email to let us know. **Q: Will you let us know when our device has been remediated?** A: Yes - BCN will contact you to confirm upon completion..</p> <p><small>Oct <var data-var='date'> 28</var>, <var data-var='time'>11:01:21</var> GMT+0</small><br><strong>Resolved</strong> - The vast majority of devices have been remediated. Any remaining activities will be tracked through individual tickets..</p> ]]> Thu, 9 Oct 2025 10:10:23 +0000 https://bcngroup.instatus.com/incident/cmgj9b7b50284kr34ep0zc3bc https://bcngroup.instatus.com/incident/cmgj9b7b50284kr34ep0zc3bc Citrix Licencing Issue Type: Incident Duration: 2 hours and 5 minutes Affected Components: Vendor/3rd Party Nov 8, 09:19:09 GMT+0 - Investigating - Please be advised that Citrix are experiencing issues with licencing which is impacting the ability to use the platform. BCN are working with Citrix to understand timescales and next steps. Nov 8, 10:03:16 GMT+0 - Investigating - Citrix have confirmed the issue is impacting Citrix DaaS in the EU - [Citrix Cloud Status](https://status.cloud.com/incidents/651542) Nov 8, 10:30:03 GMT+0 - Identified - Citrix have confirmed they have identified the root cause of the issue and are working towards resolution. They have advised that some of the impacted customers should see successful DaaS launches as they are recovering. Nov 8, 10:53:52 GMT+0 - Monitoring - We are starting to see impacted instances accept logins again. Awaiting full confirmation from Citrix that the issue has been fully resolved. Nov 8, 11:23:55 GMT+0 - Resolved - All BCN customers experiencing issues with Citrix DaaS have confirmed they are able to access the environment again. Type: Incident

Duration: 2 hours and 5 minutes

Affected Components:

<p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>09:19:09</var> GMT+0</small><br><strong>Investigating</strong> - Please be advised that Citrix are experiencing issues with licencing which is impacting the ability to use the platform. BCN are working with Citrix to understand timescales and next steps..</p> <p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>10:03:16</var> GMT+0</small><br><strong>Investigating</strong> - Citrix have confirmed the issue is impacting Citrix DaaS in the EU - [Citrix Cloud Status](https://status.cloud.com/incidents/651542).</p> <p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>10:30:03</var> GMT+0</small><br><strong>Identified</strong> - Citrix have confirmed they have identified the root cause of the issue and are working towards resolution. They have advised that some of the impacted customers should see successful DaaS launches as they are recovering..</p> <p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>10:53:52</var> GMT+0</small><br><strong>Monitoring</strong> - We are starting to see impacted instances accept logins again. Awaiting full confirmation from Citrix that the issue has been fully resolved..</p> <p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>11:23:55</var> GMT+0</small><br><strong>Resolved</strong> - All BCN customers experiencing issues with Citrix DaaS have confirmed they are able to access the environment again..</p> ]]> Fri, 8 Nov 2024 09:19:09 +0000 https://bcngroup.instatus.com/incident/cm38iyxyx000e7lsvdwtqr4at https://bcngroup.instatus.com/incident/cm38iyxyx000e7lsvdwtqr4at Unable to access Microsoft 365 Services Type: Incident Duration: 15 hours and 47 minutes Affected Components: Vendor/3rd Party Jul 31, 04:29:39 GMT+0 - Resolved - This incident has been resolved by Microsoft. Final update from them below via [Azure status history | Microsoft Azure](https://azure.status.microsoft/en-us/status/history/) **What happened?** Between approximately at 11:45 UTC and 19:43 UTC on 30 July 2024, a subset of customers may have experienced issues connecting to a subset of Microsoft services globally. Impacted services included Azure App Services, Application Insights, Azure IoT Central, Azure Log Search Alerts, Azure Policy, as well as the Azure portal itself and a subset of Microsoft 365 and Microsoft Purview services. **What do we know so far?** An unexpected usage spike resulted in Azure Front Door (AFD) and Azure Content Delivery Network (CDN) components performing below acceptable thresholds, leading to intermittent errors, timeout, and latency spikes. While the initial trigger event was a Distributed Denial-of-Service (DDoS) attack, which activated our DDoS protection mechanisms, initial investigations suggest that an error in the implementation of our defenses amplified the impact of the attack rather than mitigating it. **How did we respond?** Customer impact began at 11:45 UTC and we started investigating. Once the nature of the usage spike was understood, we implemented networking configuration changes to support our DDoS protection efforts, and performed failovers to alternate networking paths to provide relief. Our initial network configuration changes successfully mitigated majority of the impact by 14:10 UTC. Some customers reported less than 100% availability, which we began mitigating at around 18:00 UTC. We proceeded with an updated mitigation approach, first rolling this out across regions in Asia Pacific and Europe. After validating that this revised approach successfully eliminated the side effect impacts of the initial mitigation, we rolled it out to regions in the Americas. Failure rates returned to pre-incident levels by 19:43 UTC - after monitoring traffic and services to ensure that the issue was fully mitigated, we declared the incident mitigated at 20:48 UTC. Some downstream services took longer to recover, depending on how they were configured to use AFD and/or CDN. **What happens next?** Our team will be completing an internal retrospective to understand the incident in more detail. We will publish a Preliminary Post Incident Review (PIR) within approximately 72 hours, to share more details on what happened and how we responded. After our internal retrospective is completed, generally within 14 days, we will publish a Final Post Incident Review with any additional details and learnings. To get notified when that happens, and/or to stay informed about future Azure service issues, make sure that you configure and maintain Azure Service Health alerts – these can trigger emails, SMS, push notifications, webhooks, and more: <https://aka.ms/ash-alerts>. For more information on Post Incident Reviews, refer to <https://aka.ms/AzurePIRs>. Finally, for broader guidance on preparing for cloud incidents, refer to <https://aka.ms/incidentreadiness>. Jul 30, 12:42:37 GMT+0 - Investigating - Microsoft have published that they're seeing issues with access across multiple Microsoft 365 services. Microsoft incident ID MO842351. Jul 30, 12:50:32 GMT+0 - Investigating - Latest update from Microsoft Issue ID: MO842351 Affected services: Microsoft 365 suite Status: Service degradation Issue type: Incident Start time: Jul 30, 2024, 1:21 PM GMT+1 User impact Users may be unable to access multiple Microsoft 365 services. More info Users who are able to access Microsoft 365 service may experience latency or degraded feature performance. Scope of impact Impact is specific to some users who are served through the affected infrastructure, attempting to utilize multiple Microsoft 365 services or features. Current status Jul 30, 2024, 1:28 PM GMT+1 We're reviewing service monitoring telemetry to determine our next troubleshooting steps. Next update by: Tuesday, July 30, 2024 at 3:30 PM GMT+1 History of updates Jul 30, 2024, 1:24 PM GMT+1 We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes. Jul 30, 13:00:31 GMT+0 - Investigating - Latest update from Microsoft 30 Jul 2024, 13:54 BST Our investigation is currently focused on networking infrastructure. This quick update is designed to give the latest information on this issue. Jul 30, 13:21:27 GMT+0 - Identified - Latest update from Microsoft 30 Jul 2024, 14:14 BST We've identified a potential networking issue and we're investigating this further. Next update by: Tuesday, 30 July 2024 at 16:00 BST Jul 30, 14:38:03 GMT+0 - Monitoring - Latest update from Microsoft 30 Jul 2024, 15:31 BST We've implemented a networking configuration change and some Microsoft 365 services have performed failovers to alternate networking paths to provide relief. Monitoring telemetry shows improvement in service availability, and we're continuing to monitor to ensure full recovery. Jul 30, 16:16:10 GMT+0 - Monitoring - Latest update from Microsoft 30 Jul 2024, 16:51 BST Telemetry shows that the service has remained stable. We're continuing to monitor the service for an extended period of time to confirm resolution. Type: Incident

Duration: 15 hours and 47 minutes

Affected Components:

<p><small>Jul <var data-var='date'> 31</var>, <var data-var='time'>04:29:39</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved by Microsoft. Final update from them below via [Azure status history | Microsoft Azure](https://azure.status.microsoft/en-us/status/history/) **What happened?** Between approximately at 11:45 UTC and 19:43 UTC on 30 July 2024, a subset of customers may have experienced issues connecting to a subset of Microsoft services globally. Impacted services included Azure App Services, Application Insights, Azure IoT Central, Azure Log Search Alerts, Azure Policy, as well as the Azure portal itself and a subset of Microsoft 365 and Microsoft Purview services. **What do we know so far?** An unexpected usage spike resulted in Azure Front Door (AFD) and Azure Content Delivery Network (CDN) components performing below acceptable thresholds, leading to intermittent errors, timeout, and latency spikes. While the initial trigger event was a Distributed Denial-of-Service (DDoS) attack, which activated our DDoS protection mechanisms, initial investigations suggest that an error in the implementation of our defenses amplified the impact of the attack rather than mitigating it. **How did we respond?** Customer impact began at 11:45 UTC and we started investigating. Once the nature of the usage spike was understood, we implemented networking configuration changes to support our DDoS protection efforts, and performed failovers to alternate networking paths to provide relief. Our initial network configuration changes successfully mitigated majority of the impact by 14:10 UTC. Some customers reported less than 100% availability, which we began mitigating at around 18:00 UTC. We proceeded with an updated mitigation approach, first rolling this out across regions in Asia Pacific and Europe. After validating that this revised approach successfully eliminated the side effect impacts of the initial mitigation, we rolled it out to regions in the Americas. Failure rates returned to pre-incident levels by 19:43 UTC - after monitoring traffic and services to ensure that the issue was fully mitigated, we declared the incident mitigated at 20:48 UTC. Some downstream services took longer to recover, depending on how they were configured to use AFD and/or CDN. **What happens next?** Our team will be completing an internal retrospective to understand the incident in more detail. We will publish a Preliminary Post Incident Review (PIR) within approximately 72 hours, to share more details on what happened and how we responded. After our internal retrospective is completed, generally within 14 days, we will publish a Final Post Incident Review with any additional details and learnings. To get notified when that happens, and/or to stay informed about future Azure service issues, make sure that you configure and maintain Azure Service Health alerts – these can trigger emails, SMS, push notifications, webhooks, and more: <https://aka.ms/ash-alerts>. For more information on Post Incident Reviews, refer to <https://aka.ms/AzurePIRs>. Finally, for broader guidance on preparing for cloud incidents, refer to <https://aka.ms/incidentreadiness>..</p> <p><small>Jul <var data-var='date'> 30</var>, <var data-var='time'>12:42:37</var> GMT+0</small><br><strong>Investigating</strong> - Microsoft have published that they're seeing issues with access across multiple Microsoft 365 services. Microsoft incident ID MO842351..</p> <p><small>Jul <var data-var='date'> 30</var>, <var data-var='time'>12:50:32</var> GMT+0</small><br><strong>Investigating</strong> - Latest update from Microsoft Issue ID: MO842351 Affected services: Microsoft 365 suite Status: Service degradation Issue type: Incident Start time: Jul 30, 2024, 1:21 PM GMT+1 User impact Users may be unable to access multiple Microsoft 365 services. More info Users who are able to access Microsoft 365 service may experience latency or degraded feature performance. Scope of impact Impact is specific to some users who are served through the affected infrastructure, attempting to utilize multiple Microsoft 365 services or features. Current status Jul 30, 2024, 1:28 PM GMT+1 We're reviewing service monitoring telemetry to determine our next troubleshooting steps. Next update by: Tuesday, July 30, 2024 at 3:30 PM GMT+1 History of updates Jul 30, 2024, 1:24 PM GMT+1 We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes..</p> <p><small>Jul <var data-var='date'> 30</var>, <var data-var='time'>13:00:31</var> GMT+0</small><br><strong>Investigating</strong> - Latest update from Microsoft 30 Jul 2024, 13:54 BST Our investigation is currently focused on networking infrastructure. This quick update is designed to give the latest information on this issue..</p> <p><small>Jul <var data-var='date'> 30</var>, <var data-var='time'>13:21:27</var> GMT+0</small><br><strong>Identified</strong> - Latest update from Microsoft 30 Jul 2024, 14:14 BST We've identified a potential networking issue and we're investigating this further. Next update by: Tuesday, 30 July 2024 at 16:00 BST.</p> <p><small>Jul <var data-var='date'> 30</var>, <var data-var='time'>14:38:03</var> GMT+0</small><br><strong>Monitoring</strong> - Latest update from Microsoft 30 Jul 2024, 15:31 BST We've implemented a networking configuration change and some Microsoft 365 services have performed failovers to alternate networking paths to provide relief. Monitoring telemetry shows improvement in service availability, and we're continuing to monitor to ensure full recovery..</p> <p><small>Jul <var data-var='date'> 30</var>, <var data-var='time'>16:16:10</var> GMT+0</small><br><strong>Monitoring</strong> - Latest update from Microsoft 30 Jul 2024, 16:51 BST Telemetry shows that the service has remained stable. We're continuing to monitor the service for an extended period of time to confirm resolution..</p> ]]> Tue, 30 Jul 2024 12:42:37 +0000 https://bcngroup.instatus.com/incident/clz8er95w3536hgo5g2hm3wg7 https://bcngroup.instatus.com/incident/clz8er95w3536hgo5g2hm3wg7 Global Crowdstrike Outage Type: Incident Duration: 9 hours and 42 minutes Affected Components: Vendor/3rd Party Jul 19, 09:12:42 GMT+0 - Monitoring - BCN have identified customers impacted, and have engineering resource from across all teams reaching out to effected customers to assist with mitigation/remediation. Jul 19, 08:20:48 GMT+0 - Investigating - BCN are aware of global issues Crowdstrike. We are in the process of contacting customers impacting to discuss a remediation plan. Further updates will be posted here. Jul 19, 08:35:08 GMT+0 - Identified - Crowdstrike have advised the following: **Tech Alert | Windows crashes related to Falcon Sensor | 2024-07-19** Cloud: US-1EU-1US-2 Published Date: Jul 19, 2024 --- ## Summary * CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor. ## Details * Symptoms include hosts experiencing a bugcheck\\blue screen error related to the Falcon Sensor. * This issue is not impacting Mac- or Linux-based hosts * Channel file "C-00000291\*.sys" with timestamp of 0527 UTC or later is the reverted (good) version. ## Current Action * CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes. * If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue: ### Workaround Steps: * Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then: * Boot Windows into Safe Mode or the Windows Recovery Environment * Navigate to the C:\\Windows\\System32\\drivers\\CrowdStrike directory * Locate the file matching “C-00000291\*.sys”, and delete it. * Boot the host normally. Note: **Bitlocker-encrypted hosts may require a recovery key**. ## Latest Updates * 2024-07-19 05:30 AM UTC | Tech Alert Published. * 2024-07-19 06:30 AM UTC | Updated and added workaround details. * 2024-07-19 08:08 AM UTC | Updated Jul 19, 10:13:31 GMT+0 - Identified - Crowdstrike are continuing to provide further clarity on ways to remediate the impact including specific versioning of files that are problematic (that caused this issue) and those which aren't. Our technical teams are leveraging this insight and are continuing to work with impacted customers. Jul 19, 11:54:43 GMT+0 - Identified - BCN engineers continue to work with impacted customers, with 40% of those now fully resolved. \*\*\*\*\*\*IMPORTANT\*\*\*\*\*\* Please do not engage with any other 3rd parties claiming they are able to help remediate the issue. If you have any concerns about the outage, please contact BCN directly on 0345 095 7001 if you are not already in dialogue with us regarding this matter. Jul 19, 14:44:18 GMT+0 - Monitoring - There have been no further issues noted from Crowdstrike, with remediation work continuing. 72% of BCN customers impacted by the outage are now fully operational. Jul 19, 16:12:28 GMT+0 - Monitoring - The vast majority of customers are now back up and running following the Crowdstrike outage. BCN will continue to remediate remaining customers and monitor the situation over the weekend. Jul 19, 16:13:13 GMT+0 - Monitoring - BCN to monitor over the weekend to ensure no further issues arise. Jul 19, 18:02:48 GMT+0 - Resolved - All BCN customers have been brought back online following the Crowdstrike outage. BCN will continue to monitor. Type: Incident

Duration: 9 hours and 42 minutes

Affected Components:

<p><small>Jul <var data-var='date'> 19</var>, <var data-var='time'>09:12:42</var> GMT+0</small><br><strong>Monitoring</strong> - BCN have identified customers impacted, and have engineering resource from across all teams reaching out to effected customers to assist with mitigation/remediation..</p> <p><small>Jul <var data-var='date'> 19</var>, <var data-var='time'>08:20:48</var> GMT+0</small><br><strong>Investigating</strong> - BCN are aware of global issues Crowdstrike. We are in the process of contacting customers impacting to discuss a remediation plan. Further updates will be posted here..</p> <p><small>Jul <var data-var='date'> 19</var>, <var data-var='time'>08:35:08</var> GMT+0</small><br><strong>Identified</strong> - Crowdstrike have advised the following: **Tech Alert | Windows crashes related to Falcon Sensor | 2024-07-19** Cloud: US-1EU-1US-2 Published Date: Jul 19, 2024 --- ## Summary * CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor. ## Details * Symptoms include hosts experiencing a bugcheck\\blue screen error related to the Falcon Sensor. * This issue is not impacting Mac- or Linux-based hosts * Channel file "C-00000291\*.sys" with timestamp of 0527 UTC or later is the reverted (good) version. ## Current Action * CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes. * If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue: ### Workaround Steps: * Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then: * Boot Windows into Safe Mode or the Windows Recovery Environment * Navigate to the C:\\Windows\\System32\\drivers\\CrowdStrike directory * Locate the file matching “C-00000291\*.sys”, and delete it. * Boot the host normally. Note: **Bitlocker-encrypted hosts may require a recovery key**. ## Latest Updates * 2024-07-19 05:30 AM UTC | Tech Alert Published. * 2024-07-19 06:30 AM UTC | Updated and added workaround details. * 2024-07-19 08:08 AM UTC | Updated.</p> <p><small>Jul <var data-var='date'> 19</var>, <var data-var='time'>10:13:31</var> GMT+0</small><br><strong>Identified</strong> - Crowdstrike are continuing to provide further clarity on ways to remediate the impact including specific versioning of files that are problematic (that caused this issue) and those which aren't. Our technical teams are leveraging this insight and are continuing to work with impacted customers..</p> <p><small>Jul <var data-var='date'> 19</var>, <var data-var='time'>11:54:43</var> GMT+0</small><br><strong>Identified</strong> - BCN engineers continue to work with impacted customers, with 40% of those now fully resolved. \*\*\*\*\*\*IMPORTANT\*\*\*\*\*\* Please do not engage with any other 3rd parties claiming they are able to help remediate the issue. If you have any concerns about the outage, please contact BCN directly on 0345 095 7001 if you are not already in dialogue with us regarding this matter..</p> <p><small>Jul <var data-var='date'> 19</var>, <var data-var='time'>14:44:18</var> GMT+0</small><br><strong>Monitoring</strong> - There have been no further issues noted from Crowdstrike, with remediation work continuing. 72% of BCN customers impacted by the outage are now fully operational..</p> <p><small>Jul <var data-var='date'> 19</var>, <var data-var='time'>16:12:28</var> GMT+0</small><br><strong>Monitoring</strong> - The vast majority of customers are now back up and running following the Crowdstrike outage. BCN will continue to remediate remaining customers and monitor the situation over the weekend..</p> <p><small>Jul <var data-var='date'> 19</var>, <var data-var='time'>16:13:13</var> GMT+0</small><br><strong>Monitoring</strong> - BCN to monitor over the weekend to ensure no further issues arise..</p> <p><small>Jul <var data-var='date'> 19</var>, <var data-var='time'>18:02:48</var> GMT+0</small><br><strong>Resolved</strong> - All BCN customers have been brought back online following the Crowdstrike outage. BCN will continue to monitor..</p> ]]> Fri, 19 Jul 2024 08:20:48 +0000 https://bcngroup.instatus.com/incident/clysfkcmr67936hloeiwk5ir4a https://bcngroup.instatus.com/incident/clysfkcmr67936hloeiwk5ir4a Third party connectivity issue affecting multiple services Type: Incident Duration: 7 hours and 18 minutes Affected Components: Hosted Platform, External Connectivity Nov 24, 09:44:50 GMT+0 - Identified - We are currently experiencing a third party connectivity issue at one of our Data Centres. This has been escalated and the third party are currently investigating. Nov 24, 12:39:40 GMT+0 - Identified - The third party engineers are performing troubleshooting within the Data Centre and are working on a resolution. We can see that the service has become more stable but work is on going to resolve the issue. Nov 24, 15:51:48 GMT+0 - Monitoring - The third party is continuing to work on a permanent resolution to the problem. The service is currently stable but will remain marked 'at risk' until we are notified that a permanent final solution is in place. Nov 24, 17:03:11 GMT+0 - Resolved - The connectivity services have been stable since 3PM. Services have returned to full operation and we will continue to liaise with the third party and monitor for any further re-occurrence. Type: Incident

Duration: 7 hours and 18 minutes

Affected Components: ,

<p><small>Nov <var data-var='date'> 24</var>, <var data-var='time'>09:44:50</var> GMT+0</small><br><strong>Identified</strong> - We are currently experiencing a third party connectivity issue at one of our Data Centres. This has been escalated and the third party are currently investigating. .</p> <p><small>Nov <var data-var='date'> 24</var>, <var data-var='time'>12:39:40</var> GMT+0</small><br><strong>Identified</strong> - The third party engineers are performing troubleshooting within the Data Centre and are working on a resolution. We can see that the service has become more stable but work is on going to resolve the issue. .</p> <p><small>Nov <var data-var='date'> 24</var>, <var data-var='time'>15:51:48</var> GMT+0</small><br><strong>Monitoring</strong> - The third party is continuing to work on a permanent resolution to the problem. The service is currently stable but will remain marked 'at risk' until we are notified that a permanent final solution is in place. .</p> <p><small>Nov <var data-var='date'> 24</var>, <var data-var='time'>17:03:11</var> GMT+0</small><br><strong>Resolved</strong> - The connectivity services have been stable since 3PM. Services have returned to full operation and we will continue to liaise with the third party and monitor for any further re-occurrence. .</p> ]]> Fri, 24 Nov 2023 09:44:50 +0000 https://bcngroup.instatus.com/incident/clpcfrtp252001bfoi59sqbbm2 https://bcngroup.instatus.com/incident/clpcfrtp252001bfoi59sqbbm2 Microsoft 365 - Unable to access one or more Microsoft 365 services Type: Incident Duration: 1 hour and 37 minutes Affected Components: External Connectivity Nov 10, 10:17:31 GMT+0 - Investigating - **From Microsoft:** 10 November 2023 at 09:54 GMT Title: Unable to access one or more Microsoft 365 services User impact: Users may be unable to access one or more Microsoft 365 services. Current status: We're looking at diagnostic data to identify the underlying cause of the cause. Scope of impact: Impact is specific to a subset of users who are accessing the service from Germany and the United Kingdom. Nov 10, 11:16:07 GMT+0 - Identified - **From Microsoft:** 10 November 2023 at 10:49 GMT Title: Unable to access one or more Microsoft 365 services User impact: Users may be unable to access one or more Microsoft 365 services. Current status: We're continuing to evaluate diagnostic data. Preliminary analysis has identified errors within the network infrastructure and our investigation is currently focused on identifying the cause of these alerts. More info: The majority of reports are for impact to Microsoft Teams. Impact may include, but is not limited to: Microsoft Teams - Users may be unable to send and receive messages. - Users may be unable to create new Chats or Channels. Exchange Online - Users may be unable to send or receive email. Power Apps - Users may be unable to access the service. Power Automate - Users may be unable to access the service. Scope of impact: This issue could potentially impact any user attempting to access the affected services. Nov 10, 11:54:45 GMT+0 - Resolved - **From Microsoft** Status Service Restored Impacted services Microsoft 365 suite Details Title: Unable to access one or more Microsoft 365 services User impact: Users may have been unable to access one or more Microsoft 365 services. Final status: We determined that an artificial increase in synthetic network traffic resulted in DNS issues. We've made configuration changes within the networking infrastructure to remediate impact. We've monitored the service and have confirmed that the issue is resolved. More info: The majority of reports were for impact to Microsoft Teams. Impact may have included, but was not limited to: Microsoft Teams - Users may have been unable to make or receive calls. - Users may have been unable to join meetings. - Users may have been unable to send and receive messages. - Users may have been unable to create new Chats or Channels. - Users may have seen incorrect presence information. Exchange Online - Users may have been unable to send or receive email. Power Apps - Users may have been unable to access the service. Power Automate - Users may have been unable to access the service. Scope of impact: This issue primarily impacted users attempting to access the service from within the European region. However, due to the nature of the issues, users from other regions may have been affected. Start time: Friday, November 10, 2023, at 8:50 AM UTC End time: Friday, November 10, 2023, at 10:42 AM UTC Preliminary Root cause: An artificial increase in synthetic network traffic resulted in DNS issues, causing impact. Next Steps: - We're further reviewing the event and the circumstances that lead to impact, so that we can identify next steps that'll help us improve the reliability and resiliency of our service. Type: Incident

Duration: 1 hour and 37 minutes

Affected Components:

<p><small>Nov <var data-var='date'> 10</var>, <var data-var='time'>10:17:31</var> GMT+0</small><br><strong>Investigating</strong> - **From Microsoft:** 10 November 2023 at 09:54 GMT Title: Unable to access one or more Microsoft 365 services User impact: Users may be unable to access one or more Microsoft 365 services. Current status: We're looking at diagnostic data to identify the underlying cause of the cause. Scope of impact: Impact is specific to a subset of users who are accessing the service from Germany and the United Kingdom. .</p> <p><small>Nov <var data-var='date'> 10</var>, <var data-var='time'>11:16:07</var> GMT+0</small><br><strong>Identified</strong> - **From Microsoft:** 10 November 2023 at 10:49 GMT Title: Unable to access one or more Microsoft 365 services User impact: Users may be unable to access one or more Microsoft 365 services. Current status: We're continuing to evaluate diagnostic data. Preliminary analysis has identified errors within the network infrastructure and our investigation is currently focused on identifying the cause of these alerts. More info: The majority of reports are for impact to Microsoft Teams. Impact may include, but is not limited to: Microsoft Teams - Users may be unable to send and receive messages. - Users may be unable to create new Chats or Channels. Exchange Online - Users may be unable to send or receive email. Power Apps - Users may be unable to access the service. Power Automate - Users may be unable to access the service. Scope of impact: This issue could potentially impact any user attempting to access the affected services..</p> <p><small>Nov <var data-var='date'> 10</var>, <var data-var='time'>11:54:45</var> GMT+0</small><br><strong>Resolved</strong> - **From Microsoft** Status Service Restored Impacted services Microsoft 365 suite Details Title: Unable to access one or more Microsoft 365 services User impact: Users may have been unable to access one or more Microsoft 365 services. Final status: We determined that an artificial increase in synthetic network traffic resulted in DNS issues. We've made configuration changes within the networking infrastructure to remediate impact. We've monitored the service and have confirmed that the issue is resolved. More info: The majority of reports were for impact to Microsoft Teams. Impact may have included, but was not limited to: Microsoft Teams - Users may have been unable to make or receive calls. - Users may have been unable to join meetings. - Users may have been unable to send and receive messages. - Users may have been unable to create new Chats or Channels. - Users may have seen incorrect presence information. Exchange Online - Users may have been unable to send or receive email. Power Apps - Users may have been unable to access the service. Power Automate - Users may have been unable to access the service. Scope of impact: This issue primarily impacted users attempting to access the service from within the European region. However, due to the nature of the issues, users from other regions may have been affected. Start time: Friday, November 10, 2023, at 8:50 AM UTC End time: Friday, November 10, 2023, at 10:42 AM UTC Preliminary Root cause: An artificial increase in synthetic network traffic resulted in DNS issues, causing impact. Next Steps: - We're further reviewing the event and the circumstances that lead to impact, so that we can identify next steps that'll help us improve the reliability and resiliency of our service. .</p> ]]> Fri, 10 Nov 2023 10:17:31 +0000 https://bcngroup.instatus.com/incident/closgrwi515764c2nck5xfufhk https://bcngroup.instatus.com/incident/closgrwi515764c2nck5xfufhk Microsoft 365 - Exchange Online - Some users may be unable to send Exchange Online email messages Type: Incident Duration: 2 hours and 59 minutes Affected Components: External Connectivity Jul 18, 07:36:50 GMT+0 - Investigating - Please be advised that Microsoft have reported the following: Title: Some users may be unable to send Exchange Online email messages User impact: Users may be unable to send Exchange Online email messages. More info: Affected users may receive the following error "503 5.5.1 bad sequence of commands" Current status: We're reviewing service monitoring telemetry to isolate the root cause and develop a remediation plan. Scope of impact: Impact is specific to some users who are served through the affected infrastructure. Further updates to follow. Jul 18, 09:25:26 GMT+0 - Monitoring - 18 July 2023 at 10:08 BST · Update from Microsoft Current status: We’ve disabled an interceptor rule that we believe was causing impact to the mail flow. We’re monitoring the service to ensure the mail flow is restored. Jul 18, 10:35:47 GMT+0 - Resolved - Latest update from Microsoft: Status Service Restored Impacted services Exchange Online Details Title: Some users may have been unable to send Exchange Online email messages User impact: Some users may have been unable to send or receive Exchange Online email messages. More info: Affected users may have received NDRs or the following error "503 5.5.1 bad sequence of commands" Final status: We’ve determined that a recent change caused a large influx of free/busy Control Flow Messages (CFM) to be sent to the Exchange online infrastructure, resulting in stale free-busy information. An interceptor rule was applied to discard these CFM messages to mitigate impact for the free/busy issue, which inadvertently caused impact to mail flow. We've removed the interceptor rule and verified that the impact to the mail flow has now been mitigated. We'll continue to investigate the free/busy issue, which is being communicated under the SI EX648815. Due to the nature of the issue, the users who had received the NDRs during this event will need to resend those emails. Scope of impact: Impact was specific to some users who are served through the affected infrastructure. Start time: Tuesday, July 18, 2023, at 5:00 AM UTC End time: Tuesday, July 18, 2023, at 8:25 AM UTC Root cause: We’ve determined that a recent change caused a large influx of free/busy Control Flow Messages (CFM) to be sent to the Exchange online infrastructure, resulting in stale free-busy information. An interceptor rule was applied to discard these CFM messages to mitigate impact for the free/busy issue, which inadvertently caused impact to mail flow. Next steps: - We're reviewing our interceptor rule implementation procedures to understand why mail flow was inadvertently affected, in order to help prevent similar impact from happening again in the future. Type: Incident

Duration: 2 hours and 59 minutes

Affected Components:

<p><small>Jul <var data-var='date'> 18</var>, <var data-var='time'>07:36:50</var> GMT+0</small><br><strong>Investigating</strong> - Please be advised that Microsoft have reported the following: Title: Some users may be unable to send Exchange Online email messages User impact: Users may be unable to send Exchange Online email messages. More info: Affected users may receive the following error "503 5.5.1 bad sequence of commands" Current status: We're reviewing service monitoring telemetry to isolate the root cause and develop a remediation plan. Scope of impact: Impact is specific to some users who are served through the affected infrastructure. Further updates to follow..</p> <p><small>Jul <var data-var='date'> 18</var>, <var data-var='time'>09:25:26</var> GMT+0</small><br><strong>Monitoring</strong> - 18 July 2023 at 10:08 BST · Update from Microsoft Current status: We’ve disabled an interceptor rule that we believe was causing impact to the mail flow. We’re monitoring the service to ensure the mail flow is restored..</p> <p><small>Jul <var data-var='date'> 18</var>, <var data-var='time'>10:35:47</var> GMT+0</small><br><strong>Resolved</strong> - Latest update from Microsoft: Status Service Restored Impacted services Exchange Online Details Title: Some users may have been unable to send Exchange Online email messages User impact: Some users may have been unable to send or receive Exchange Online email messages. More info: Affected users may have received NDRs or the following error "503 5.5.1 bad sequence of commands" Final status: We’ve determined that a recent change caused a large influx of free/busy Control Flow Messages (CFM) to be sent to the Exchange online infrastructure, resulting in stale free-busy information. An interceptor rule was applied to discard these CFM messages to mitigate impact for the free/busy issue, which inadvertently caused impact to mail flow. We've removed the interceptor rule and verified that the impact to the mail flow has now been mitigated. We'll continue to investigate the free/busy issue, which is being communicated under the SI EX648815. Due to the nature of the issue, the users who had received the NDRs during this event will need to resend those emails. Scope of impact: Impact was specific to some users who are served through the affected infrastructure. Start time: Tuesday, July 18, 2023, at 5:00 AM UTC End time: Tuesday, July 18, 2023, at 8:25 AM UTC Root cause: We’ve determined that a recent change caused a large influx of free/busy Control Flow Messages (CFM) to be sent to the Exchange online infrastructure, resulting in stale free-busy information. An interceptor rule was applied to discard these CFM messages to mitigate impact for the free/busy issue, which inadvertently caused impact to mail flow. Next steps: - We're reviewing our interceptor rule implementation procedures to understand why mail flow was inadvertently affected, in order to help prevent similar impact from happening again in the future..</p> ]]> Tue, 18 Jul 2023 07:36:50 +0000 https://bcngroup.instatus.com/incident/clk7zeb68490797wkofccxq9avj https://bcngroup.instatus.com/incident/clk7zeb68490797wkofccxq9avj Third Party External Connectivity Issues - Affecting BCN Hosted Services Type: Incident Duration: 14 days, 5 hours and 21 minutes Affected Components: Veeam Cloud, Hosted Platform, External Connectivity, Remote Desktop Platform, Hosted Email Services Oct 10, 10:52:39 GMT+0 - Identified - The third party that provide external connectivity to our Ball Green Primary Data Centre are having nation wide connectivity issues. This will affect connectivity to Customers within the BCN Hosted Environment and customers with M247 connectivity services. The third party are expected to provide a further update at 12:00PM Oct 10, 11:07:45 GMT+0 - Identified - Third party NOC engineers are continuing to investigate the outage Oct 10, 11:17:52 GMT+0 - Identified - The issue has been identified on a core router and they third party are currently working on a solution. Updates to follow as soon as possible. Oct 10, 12:11:35 GMT+0 - Identified - A software upgrade on the third party core routing device is ongoing at the moment. Updates to follow as soon as available. Oct 10, 13:17:25 GMT+0 - Monitoring - Connectivity is returning to service and we are monitoring Oct 10, 13:22:44 GMT+0 - Monitoring - The core router upgrade has finished and the third party are currently performing the after checks. Updates to follow as soon as available. Oct 10, 14:47:56 GMT+0 - Monitoring - Failover to the backup router has finished as the firmware upgrade did not solve the issue, after checks show all ldp sessions up. Traffic has returned to normal parameters, but there is still risk of congestion due to a faulty card within the routing device. Tomorrow, a new card will arrive in London DC to replace the faulty one. Until then services might experience connectivity disruption during peak times. Oct 11, 12:45:39 GMT+0 - Monitoring - The ETA for the replacement routing card is 16:00 UTC. Third party engineers will perform installation and switch over following the arrival of the replacement equipment. Oct 11, 16:10:00 GMT+0 - Resolved - Swap activity finished and traffic values returned to normal trends. Completed 17:10 on 11/10/22 Type: Incident

Duration: 14 days, 5 hours and 21 minutes

Affected Components: , , , ,

<p><small>Oct <var data-var='date'> 10</var>, <var data-var='time'>10:52:39</var> GMT+0</small><br><strong>Identified</strong> - The third party that provide external connectivity to our Ball Green Primary Data Centre are having nation wide connectivity issues. This will affect connectivity to Customers within the BCN Hosted Environment and customers with M247 connectivity services. The third party are expected to provide a further update at 12:00PM.</p> <p><small>Oct <var data-var='date'> 10</var>, <var data-var='time'>11:07:45</var> GMT+0</small><br><strong>Identified</strong> - Third party NOC engineers are continuing to investigate the outage.</p> <p><small>Oct <var data-var='date'> 10</var>, <var data-var='time'>11:17:52</var> GMT+0</small><br><strong>Identified</strong> - The issue has been identified on a core router and they third party are currently working on a solution. Updates to follow as soon as possible. .</p> <p><small>Oct <var data-var='date'> 10</var>, <var data-var='time'>12:11:35</var> GMT+0</small><br><strong>Identified</strong> - A software upgrade on the third party core routing device is ongoing at the moment. Updates to follow as soon as available. .</p> <p><small>Oct <var data-var='date'> 10</var>, <var data-var='time'>13:17:25</var> GMT+0</small><br><strong>Monitoring</strong> - Connectivity is returning to service and we are monitoring.</p> <p><small>Oct <var data-var='date'> 10</var>, <var data-var='time'>13:22:44</var> GMT+0</small><br><strong>Monitoring</strong> - The core router upgrade has finished and the third party are currently performing the after checks. Updates to follow as soon as available. .</p> <p><small>Oct <var data-var='date'> 10</var>, <var data-var='time'>14:47:56</var> GMT+0</small><br><strong>Monitoring</strong> - Failover to the backup router has finished as the firmware upgrade did not solve the issue, after checks show all ldp sessions up. Traffic has returned to normal parameters, but there is still risk of congestion due to a faulty card within the routing device. Tomorrow, a new card will arrive in London DC to replace the faulty one. Until then services might experience connectivity disruption during peak times. .</p> <p><small>Oct <var data-var='date'> 11</var>, <var data-var='time'>12:45:39</var> GMT+0</small><br><strong>Monitoring</strong> - The ETA for the replacement routing card is 16:00 UTC. Third party engineers will perform installation and switch over following the arrival of the replacement equipment..</p> <p><small>Oct <var data-var='date'> 11</var>, <var data-var='time'>16:10:00</var> GMT+0</small><br><strong>Resolved</strong> - Swap activity finished and traffic values returned to normal trends. Completed 17:10 on 11/10/22 .</p> ]]> Mon, 10 Oct 2022 10:52:39 +0000 https://bcngroup.instatus.com/incident/cl92nnsue24042hsn3ozc4m6rl https://bcngroup.instatus.com/incident/cl92nnsue24042hsn3ozc4m6rl Veeam Cloud Service Issues Type: Incident Duration: 22 hours and 49 minutes Affected Components: Veeam Cloud Oct 7, 17:30:00 GMT+0 - Identified - Veeam Cloud Services are currently affected by an urgent maintenance issue at our Data Centre. Our engineers are working to resolve this issue. Oct 7, 22:29:06 GMT+0 - Identified - Our engineers are continuing to work on recovering the VeeamCC service. Oct 7, 22:40:05 GMT+0 - Identified - Our engineers are continuing to restore VeeamCC to full service. Estimated fix time of 3AM. Oct 8, 08:03:42 GMT+0 - Identified - Following further complications to the issue our engineers continue to work on recovering the service Oct 8, 10:11:39 GMT+0 - Identified - We are continuing to work on a fix for this incident. Oct 8, 12:29:45 GMT+0 - Monitoring - Services have been restored and our engineers are monitoring and pro-actively resolving any individual customer issues. Oct 8, 16:18:37 GMT+0 - Resolved - The issue with the BCN VeeamCC Services has been resolved. All services are now operating normally. Please contact BCN support for any issues. Oct 7, 22:40:05 GMT+0 - Identified - Our engineers are continuing to work on recovering the VeeamCC service. Type: Incident

Duration: 22 hours and 49 minutes

Affected Components:

<p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>17:30:00</var> GMT+0</small><br><strong>Identified</strong> - Veeam Cloud Services are currently affected by an urgent maintenance issue at our Data Centre. Our engineers are working to resolve this issue. .</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>22:29:06</var> GMT+0</small><br><strong>Identified</strong> - Our engineers are continuing to work on recovering the VeeamCC service. .</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>22:40:05</var> GMT+0</small><br><strong>Identified</strong> - Our engineers are continuing to restore VeeamCC to full service. Estimated fix time of 3AM. .</p> <p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>08:03:42</var> GMT+0</small><br><strong>Identified</strong> - Following further complications to the issue our engineers continue to work on recovering the service.</p> <p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>10:11:39</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident..</p> <p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>12:29:45</var> GMT+0</small><br><strong>Monitoring</strong> - Services have been restored and our engineers are monitoring and pro-actively resolving any individual customer issues. .</p> <p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>16:18:37</var> GMT+0</small><br><strong>Resolved</strong> - The issue with the BCN VeeamCC Services has been resolved. All services are now operating normally. Please contact BCN support for any issues. .</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>22:40:05</var> GMT+0</small><br><strong>Identified</strong> - Our engineers are continuing to work on recovering the VeeamCC service. .</p> ]]> Fri, 7 Oct 2022 17:30:00 +0000 https://bcngroup.instatus.com/incident/cl8yx6ae091187hkn5b6ed8zyo https://bcngroup.instatus.com/incident/cl8yx6ae091187hkn5b6ed8zyo Service Connectivity Issue Type: Incident Duration: 12 hours and 4 minutes Affected Components: Remote Desktop Platform, Hosted Email Services , Hosted Platform, External Connectivity Jul 18, 17:56:28 GMT+0 - Identified - Engineers are now on-site at our datacentre investigating. Jul 18, 23:50:49 GMT+0 - Monitoring - We implemented a fix and currently testing to ensure that all services are available. Jul 19, 05:56:11 GMT+0 - Resolved - All services should now have returned to normal. Jul 18, 17:52:06 GMT+0 - Investigating - We are currently aware of issues at our data centre which may result in intermittent loss of service. Type: Incident

Duration: 12 hours and 4 minutes

Affected Components: , , ,

<p><small>Jul <var data-var='date'> 18</var>, <var data-var='time'>17:56:28</var> GMT+0</small><br><strong>Identified</strong> - Engineers are now on-site at our datacentre investigating..</p> <p><small>Jul <var data-var='date'> 18</var>, <var data-var='time'>23:50:49</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and currently testing to ensure that all services are available..</p> <p><small>Jul <var data-var='date'> 19</var>, <var data-var='time'>05:56:11</var> GMT+0</small><br><strong>Resolved</strong> - All services should now have returned to normal..</p> <p><small>Jul <var data-var='date'> 18</var>, <var data-var='time'>17:52:06</var> GMT+0</small><br><strong>Investigating</strong> - We are currently aware of issues at our data centre which may result in intermittent loss of service..</p> ]]> Mon, 18 Jul 2022 17:52:06 +0000 https://bcngroup.instatus.com/incident/cl5r1nn654471q5ndmzdikrf6 https://bcngroup.instatus.com/incident/cl5r1nn654471q5ndmzdikrf6 Internet Connectivity Issue Type: Incident Duration: 30 minutes Affected Components: Remote Desktop Platform, Hosted Email Services , Veeam Cloud, Storage Platform, Azure Services, Hosted Platform, External Connectivity May 24, 09:40:42 GMT+0 - Investigating - There have been reports of internet outages impacting customers utilising M247 for connectivity. The incident can be tracked here: https://status.m247.com/issue/1236 It is currently understood that most services have been restored/recovered, however intermittent outages may be noted. May 24, 10:10:40 GMT+0 - Resolved - The vendor has confirmed the issue has been resolved. Further information can be viewed here: https://status.m247.com/issue/1236 Type: Incident

Duration: 30 minutes

Affected Components: , , , , , ,

<p><small>May <var data-var='date'> 24</var>, <var data-var='time'>09:40:42</var> GMT+0</small><br><strong>Investigating</strong> - There have been reports of internet outages impacting customers utilising M247 for connectivity. The incident can be tracked here: https://status.m247.com/issue/1236 It is currently understood that most services have been restored/recovered, however intermittent outages may be noted..</p> <p><small>May <var data-var='date'> 24</var>, <var data-var='time'>10:10:40</var> GMT+0</small><br><strong>Resolved</strong> - The vendor has confirmed the issue has been resolved. Further information can be viewed here: https://status.m247.com/issue/1236.</p> ]]> Tue, 24 May 2022 09:40:42 +0000 https://bcngroup.instatus.com/incident/cl3jyvlic2037salnqoc6w92x https://bcngroup.instatus.com/incident/cl3jyvlic2037salnqoc6w92x External Transit Issue - Third party issue effecting some customers Type: Incident Duration: 55 minutes Affected Components: Remote Desktop Platform, Hosted Email Services , External Connectivity, Hosted Platform Apr 25, 10:20:00 GMT+0 - Investigating - An issue has been identified causing connectivity issues between the data centre and customers on certain ISP services. The issue is due to a third party network service between London and Manchester. Apr 25, 11:15:00 GMT+0 - Resolved - The issue was identified as a circuit issue within our primary transit providers network - they have resolved the issue by removing the circuit from service. Type: Incident

Duration: 55 minutes

Affected Components: , , ,

<p><small>Apr <var data-var='date'> 25</var>, <var data-var='time'>10:20:00</var> GMT+0</small><br><strong>Investigating</strong> - An issue has been identified causing connectivity issues between the data centre and customers on certain ISP services. The issue is due to a third party network service between London and Manchester. .</p> <p><small>Apr <var data-var='date'> 25</var>, <var data-var='time'>11:15:00</var> GMT+0</small><br><strong>Resolved</strong> - The issue was identified as a circuit issue within our primary transit providers network - they have resolved the issue by removing the circuit from service. .</p> ]]> Mon, 25 Apr 2022 10:20:00 +0000 https://bcngroup.instatus.com/incident/cl2emw18e173445fqn3yqm5txpd https://bcngroup.instatus.com/incident/cl2emw18e173445fqn3yqm5txpd External Connectivity Issue - Primary Data Centre Type: Incident Duration: 3 minutes Affected Components: External Connectivity Jan 7, 11:59:00 GMT+0 - Resolved - Issue with External Connectivity has been resolved. Jan 7, 11:56:00 GMT+0 - Investigating - We are currently investigating this incident. Type: Incident

Duration: 3 minutes

Affected Components:

<p><small>Jan <var data-var='date'> 7</var>, <var data-var='time'>11:59:00</var> GMT+0</small><br><strong>Resolved</strong> - Issue with External Connectivity has been resolved. .</p> <p><small>Jan <var data-var='date'> 7</var>, <var data-var='time'>11:56:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> ]]> Fri, 7 Jan 2022 11:56:00 +0000 https://bcngroup.instatus.com/incident/cky4in2r53866efobedw6e70p https://bcngroup.instatus.com/incident/cky4in2r53866efobedw6e70p Outage to Services at Primary Data Centre - BCN Hosted Customers Type: Incident Duration: 19 minutes Affected Components: Remote Desktop Platform, Hosted Email Services , Veeam Cloud, Hosted Platform, External Connectivity Oct 29, 11:45:38 GMT+0 - Identified - Our third party transit provider is currently experiencing issues within their network. This will impact access to services for some customers. Oct 29, 12:04:15 GMT+0 - Resolved - The third party transit provider has resolved the issue within their network and all services should now be restored. Type: Incident

Duration: 19 minutes

Affected Components: , , , ,

<p><small>Oct <var data-var='date'> 29</var>, <var data-var='time'>11:45:38</var> GMT+0</small><br><strong>Identified</strong> - Our third party transit provider is currently experiencing issues within their network. This will impact access to services for some customers. .</p> <p><small>Oct <var data-var='date'> 29</var>, <var data-var='time'>12:04:15</var> GMT+0</small><br><strong>Resolved</strong> - The third party transit provider has resolved the issue within their network and all services should now be restored. .</p> ]]> Fri, 29 Oct 2021 11:45:38 +0000 https://bcngroup.instatus.com/incident/ckvcb9zkq325025txn43mcmemzf https://bcngroup.instatus.com/incident/ckvcb9zkq325025txn43mcmemzf