All Simple - [LON1] Maintenance on B-side power systems – Maintenance details

ns2 experiencing major outage

[LON1] Maintenance on B-side power systems

Completed
Scheduled for November 30, 2024 at 9:00 AM – 4:11 PM

Affects

Authoritative DNS

Under maintenance from 9:00 AM to 4:11 PM

ns1

Under maintenance from 9:00 AM to 4:11 PM

DNS Resolvers

Under maintenance from 9:00 AM to 4:11 PM

resolver1

Under maintenance from 9:00 AM to 4:11 PM

resolver2

Under maintenance from 9:00 AM to 4:11 PM

Virtual Server Hosts

Under maintenance from 9:00 AM to 4:11 PM

Updates
  • Completed
    November 30, 2024 at 4:11 PM
    Completed
    November 30, 2024 at 4:11 PM

    We have been advised that the maintenance is now complete and has been successful.

  • Update
    November 30, 2024 at 3:04 PM
    In progress
    November 30, 2024 at 3:04 PM

    The work is still ongoing, we have had the following update from our DC Operator, we will provide further updates when available:

    All works are now complete, however during a firmware upgrade to the various components within the B-side UPS, we have run into an issue which means we still have this UPS in bypass mode at present.

    We are working to a resolution on this as we speak, however at present, B-side power - while restored in terms of delivery to client racks - is running without UPS protection at this time.

    So while full power has been restored to all racks (following the works to the B-side today), full restoration of UPS protection on the B-side is still pending at present.

  • Update
    November 30, 2024 at 12:42 PM
    In progress
    November 30, 2024 at 12:42 PM

    We can see the loss of power to the B-Side PDU, everything is working as expected and we are continuing to monitor the situation.

  • In progress
    November 30, 2024 at 9:00 AM
    In progress
    November 30, 2024 at 9:00 AM
    Maintenance is now in progress
  • Update
    November 30, 2024 at 9:00 AM
    Planned
    November 30, 2024 at 9:00 AM

    Our data centre provider will be running maintenance on their B-Side power systems during this window.

    During the works they will shutdown power from the B-Side.

    Their A-Side power infrastructure will be unaffected.

    We are using redundant power suppliers in core network equipment and servers, the window is at-risk only.

  • Planned
    November 29, 2024 at 4:33 PM
    Planned
    November 29, 2024 at 4:33 PM

    In preparation for this work, supporting hardware has now had the primary power shifted to our A-Side PDU.