BCN Hosted - Global Crowdstrike Outage – Incident details

All BCN customers have been brought back online following the Crowdstrike outage. BCN will continue to monitor.

BCN to monitor over the weekend to ensure no further issues arise.

The vast majority of customers are now back up and running following the Crowdstrike outage. BCN will continue to remediate remaining customers and monitor the situation over the weekend.

There have been no further issues noted from Crowdstrike, with remediation work continuing. 72% of BCN customers impacted by the outage are now fully operational.

BCN engineers continue to work with impacted customers, with 40% of those now fully resolved.

******IMPORTANT******

Please do not engage with any other 3rd parties claiming they are able to help remediate the issue. If you have any concerns about the outage, please contact BCN directly on 0345 095 7001 if you are not already in dialogue with us regarding this matter.

Crowdstrike are continuing to provide further clarity on ways to remediate the impact including specific versioning of files that are problematic (that caused this issue) and those which aren't. Our technical teams are leveraging this insight and are continuing to work with impacted customers.

BCN have identified customers impacted, and have engineering resource from across all teams reaching out to effected customers to assist with mitigation/remediation.

Crowdstrike have advised the following:

Tech Alert | Windows crashes related to Falcon Sensor | 2024-07-19

Cloud:  US-1EU-1US-2

Published Date: Jul 19, 2024

Summary

• CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor.

Details

• Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.

• This issue is not impacting Mac- or Linux-based hosts

• Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.

Current Action

• CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.

• If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue:

Workaround Steps:

• Reboot the host to give it an opportunity to download the reverted channel file.  If the host crashes again, then:

  • Boot Windows into Safe Mode or the Windows Recovery Environment

  • Navigate to the C:\Windows\System32\drivers\CrowdStrike directory

  • Locate the file matching “C-00000291*.sys”, and delete it. 

  • Boot the host normally.

Note:  Bitlocker-encrypted hosts may require a recovery key. 

Latest Updates

• 2024-07-19 05:30 AM UTC | Tech Alert Published.

• 2024-07-19 06:30 AM UTC | Updated and added workaround details.

• 2024-07-19 08:08 AM UTC | Updated