We are happy to report no hardware failures caused by these issues; all services in USCA_4 have been fully restored.
We observe some VMs resuming slower than anticipated, it will take up to another ~30 minutes for all VMs to fully boot.
We are marking this incident as Resolved, but we will continue observing the situation and will make sure all customers' VMs are fully booted as quickly as possible.
Posted Jun 07, 2024 - 04:19 PDT
Update
Customers' VMs are booting normally. We expect all VMs to be up and running within 15 minutes.
Posted Jun 07, 2024 - 04:00 PDT
Update
We are powering on all our equipment.
Posted Jun 07, 2024 - 03:54 PDT
Update
Power has been restored. Temperatures are back to normal. We are performing final checks before powering on all equipment.
Posted Jun 07, 2024 - 03:47 PDT
Update
In addition to the ongoing AC issues we've registered a loss of power in the datacenter.
Posted Jun 07, 2024 - 02:43 PDT
Update
We are observing temperatures slowly going down after peaking at around 62 C.
Posted Jun 07, 2024 - 02:34 PDT
Update
We are observing temperatures close to 60 C / 140 F in the datacenter at the moment.
We managed to gracefully shut down all our equipment before temperatures crossed 45 C. All customer's VMs have been gracefully shut down via ACPI power off. Once AC issue is resolved and we get an OK from the datacenter to power on our equipment, all customer's VMs will start automatically.
Posted Jun 07, 2024 - 02:12 PDT
Update
Multacom advised us that an emergency crew has been called and is on the way to the datacenter to investigate the AC issue.
Posted Jun 07, 2024 - 01:14 PDT
Identified
We have identified rapidly rising temperatures in Multacom datacenter (second time in the past 24 hours). This time it looks worse than the last time. We are shutting down some of our equipment to prevent damage caused by heat, and we're reaching out to Multacom for more information.