All Systems Operational
Past Incidents
Jun 13, 2024

No incidents reported today.

Jun 12, 2024

No incidents reported.

Jun 11, 2024

No incidents reported.

Jun 10, 2024

No incidents reported.

Jun 9, 2024

No incidents reported.

Jun 8, 2024

No incidents reported.

Jun 7, 2024
Resolved - Confirmed no outstanding issues. We are closing this incident.
Jun 7, 10:59 PDT
Update - Node 1121 is online now. All remaining VMs will start within 20 minutes.

We are performing a final verification of all systems to make sure there are no outstanding items before closing this incident.

Jun 7, 10:21 PDT
Update - Node 1121: RAID controller has been replaced however it requires a firmware update. We're starting the update procedure.
Jun 7, 09:35 PDT
Update - Node 1123 has been restored. All VMs will start within 15 minutes.

The last node 1121 is still being worked on.

Jun 7, 09:19 PDT
Update - Most nodes have been restored.
Node 1120 is starting, all VMs should be up within 15 minutes.

Node 1121 is having RAID issues, we are replacing the controller.
Node 1123 is having CPU issues, we are preparing a replacement chassis to swap all SSDs.

Jun 7, 08:20 PDT
Update - We have restored approx. 50% of affected VMs in USCA_4. Unfortunately, we are facing hardware failures due to earlier datacenter overheating issue.
Jun 7, 07:04 PDT
Update - We are continuing to work on a fix for this issue.
Jun 7, 06:00 PDT
Identified - We have determined that the temperature sensors in all servers show erratic values after the last incident. We are going to gracefully shut down all equipment and attempt to fully reset iDrac controllers.
Jun 7, 05:23 PDT
Investigating - We have noticed degraded performance of the USCA_4 datacenter after the previous issue was resolved. We are investigating this further.
Jun 7, 05:14 PDT
Resolved - This incident has been resolved.
Jun 7, 04:19 PDT
Update - We are happy to report no hardware failures caused by these issues; all services in USCA_4 have been fully restored.

We observe some VMs resuming slower than anticipated, it will take up to another ~30 minutes for all VMs to fully boot.

We are marking this incident as Resolved, but we will continue observing the situation and will make sure all customers' VMs are fully booted as quickly as possible.

Jun 7, 04:19 PDT
Update - Customers' VMs are booting normally. We expect all VMs to be up and running within 15 minutes.
Jun 7, 04:00 PDT
Update - We are powering on all our equipment.
Jun 7, 03:54 PDT
Update - Power has been restored. Temperatures are back to normal. We are performing final checks before powering on all equipment.
Jun 7, 03:47 PDT
Update - In addition to the ongoing AC issues we've registered a loss of power in the datacenter.
Jun 7, 02:43 PDT
Update - We are observing temperatures slowly going down after peaking at around 62 C.
Jun 7, 02:34 PDT
Update - We are observing temperatures close to 60 C / 140 F in the datacenter at the moment.

We managed to gracefully shut down all our equipment before temperatures crossed 45 C. All customer's VMs have been gracefully shut down via ACPI power off. Once AC issue is resolved and we get an OK from the datacenter to power on our equipment, all customer's VMs will start automatically.

Jun 7, 02:12 PDT
Update - Multacom advised us that an emergency crew has been called and is on the way to the datacenter to investigate the AC issue.
Jun 7, 01:14 PDT
Identified - We have identified rapidly rising temperatures in Multacom datacenter (second time in the past 24 hours). This time it looks worse than the last time. We are shutting down some of our equipment to prevent damage caused by heat, and we're reaching out to Multacom for more information.
Jun 7, 00:54 PDT
Jun 6, 2024
Resolved - The incident has been resolved. No customers have been affected by this incident.
Jun 6, 07:05 PDT
Identified - We have identified rapidly rising temperatures in Multacom datacenter. We are shutting down some of our equipment to prevent damage caused by heat, and we're reaching out to Multacom for more information.
Jun 6, 07:00 PDT
Jun 5, 2024

No incidents reported.

Jun 4, 2024

No incidents reported.

Jun 3, 2024

No incidents reported.

Jun 2, 2024

No incidents reported.

Jun 1, 2024

No incidents reported.

May 31, 2024

No incidents reported.

May 30, 2024

No incidents reported.