BCN Hosted - Third Party External Connectivity Issues - Affecting BCN Hosted Services – Incident details

All systems operational

Third Party External Connectivity Issues - Affecting BCN Hosted Services

Resolved
Degraded performance
Started almost 4 years agoLasted 1 day

Affected

Remote Desktop Platform

Degraded performance from 10:52 AM to 4:10 PM

Hosted Platform

Degraded performance from 10:52 AM to 4:10 PM

Hosted Email Services

Degraded performance from 10:52 AM to 4:10 PM

External Connectivity

Degraded performance from 10:52 AM to 4:10 PM

Veeam Cloud

Degraded performance from 10:52 AM to 4:10 PM

Updates
  • Resolved
    UTC
    Resolved

    Swap activity finished and traffic values returned to normal trends. Completed 17:10 on 11/10/22

  • Update
    UTC
    Update

    The ETA for the replacement routing card is 16:00 UTC. Third party engineers will perform installation and switch over following the arrival of the replacement equipment.

  • Update
    UTC
    Update

    Failover to the backup router has finished as the firmware upgrade did not solve the issue, after checks show all ldp sessions up. Traffic has returned to normal parameters, but there is still risk of congestion due to a faulty card within the routing device. Tomorrow, a new card will arrive in London DC to replace the faulty one. Until then services might experience connectivity disruption during peak times.

  • Update
    UTC
    Update

    The core router upgrade has finished and the third party are currently performing the after checks. Updates to follow as soon as available.

  • Monitoring
    UTC
    Monitoring

    Connectivity is returning to service and we are monitoring

  • Update
    UTC
    Update

    A software upgrade on the third party core routing device is ongoing at the moment. Updates to follow as soon as available.

  • Update
    UTC
    Update

    The issue has been identified on a core router and they third party are currently working on a solution. Updates to follow as soon as possible.

  • Update
    UTC
    Update

    Third party NOC engineers are continuing to investigate the outage

  • Identified
    UTC
    Identified

    The third party that provide external connectivity to our Ball Green Primary Data Centre are having nation wide connectivity issues. This will affect connectivity to Customers within the BCN Hosted Environment and customers with M247 connectivity services.

    The third party are expected to provide a further update at 12:00PM