USCA_4 Multacom datacenter AC and power failure

Incident Report for Bandwagon Host Status

Resolved

This incident has been resolved.
Posted Jun 07, 2024 - 04:19 PDT

Update

We are happy to report no hardware failures caused by these issues; all services in USCA_4 have been fully restored.

We observe some VMs resuming slower than anticipated, it will take up to another ~30 minutes for all VMs to fully boot.

We are marking this incident as Resolved, but we will continue observing the situation and will make sure all customers' VMs are fully booted as quickly as possible.
Posted Jun 07, 2024 - 04:19 PDT

Update

Customers' VMs are booting normally. We expect all VMs to be up and running within 15 minutes.
Posted Jun 07, 2024 - 04:00 PDT

Update

We are powering on all our equipment.
Posted Jun 07, 2024 - 03:54 PDT

Update

Power has been restored. Temperatures are back to normal. We are performing final checks before powering on all equipment.
Posted Jun 07, 2024 - 03:47 PDT

Update

In addition to the ongoing AC issues we've registered a loss of power in the datacenter.
Posted Jun 07, 2024 - 02:43 PDT

Update

We are observing temperatures slowly going down after peaking at around 62 C.
Posted Jun 07, 2024 - 02:34 PDT

Update

We are observing temperatures close to 60 C / 140 F in the datacenter at the moment.

We managed to gracefully shut down all our equipment before temperatures crossed 45 C. All customer's VMs have been gracefully shut down via ACPI power off. Once AC issue is resolved and we get an OK from the datacenter to power on our equipment, all customer's VMs will start automatically.
Posted Jun 07, 2024 - 02:12 PDT

Update

Multacom advised us that an emergency crew has been called and is on the way to the datacenter to investigate the AC issue.
Posted Jun 07, 2024 - 01:14 PDT

Identified

We have identified rapidly rising temperatures in Multacom datacenter (second time in the past 24 hours). This time it looks worse than the last time. We are shutting down some of our equipment to prevent damage caused by heat, and we're reaching out to Multacom for more information.
Posted Jun 07, 2024 - 00:54 PDT