Server Outage (October 4th, 2025)

[Widower]

Okay, just to recap for those whose days haven't been consumed by this: VM3 went down around 3:15 Eastern. We spent a couple hours trying to see if we could prompt OVH to do something about it, which eventually succeeded. They replaced a CPU in the machine, but when Cipher went to reconnect it, IPMI wasn't working, so we ended up spending another three hours waiting for OVH to do something. It took them about two hours to replace the motherboard, RAM and CPU cooling system, but when Cipher went to reconnect it (yes, something of a pattern here) he found that some virtual disks were missing. Nothing mission critical, but still not good. Until we can figure out what's going on there, VM3 remains out of commission. Cipher set up VMs for Everlasting and Victory on one of the other hosts during one of the waits, so he's spinning those back up and is hoping to have them back up that way. If you were playing at the time of the initial crash, you shouldn't experience more than 17 minutes of rollback, and hopefully much less. 

[Force Redux]

Thank you to the HC team for working to get this resolved ❤️

[Widower]

Everlasting and Victory are up on one of the other hosts. We'll keep working to bring the affected host back up, though the ball is largely in OVH's hands. Hopefully it's back up for Tuesday maintenance and we'll rearrange everything then. Shout if you notice anything awry. Otherwise, go, hunt, kill skuls.

[TauntingMonk]

Thank you! 

[AspieAnarchy]

[ChickenPaddy]

Thanks for your hard work!  My mains are on one of the afflicted servers, so I'm glad I get to see them again

[Battle7]

i haven't seen anything about Vanguard Server, is there some update I'm not aware of? it shows offline at the log in page.

thanks.

[Staceypants]

18 minutes ago, Battle7 said:

i haven't seen anything about Vanguard Server, is there some update I'm not aware of? it shows offline at the log in page.

thanks.

Vanguard was lowest on the priority order. It will be up when it's up.

[lemming]

Thanks to the team for all the work on a Saturday night.

[kieraoona]

Thank you team for all that you do!  Time to go Trick or Treating!

 

[boundgirlie]

All that you all do is very much appreciated.  I use vanguard myself.  You all do your thing :)

[LocaMocha]

Vanguard main here as well, but I'm very glad to see the progress being made getting all of the servers up and running. I take it was a multiple hardware failure on the main authentication server then?

[Cipher]

2 hours ago, LocaMocha said:

Vanguard main here as well, but I'm very glad to see the progress being made getting all of the servers up and running. I take it was a multiple hardware failure on the main authentication server then?

We're unsure of the root cause, but something took out the CPU(s), certain functions of the mainboard such as IPMI, memory, and damaged the disks.

 

We're currently working on Vanguard and expect to have it up within the hour.

[Cipher]

Vanguard has been restored on new VMs, but we are unable to restore Architect Entertainment on Vanguard until we can safely recover data from the downed VM host.

[Krimson]

@Cipher I've been logged in for hours, and I did some casual monster hunting. I didn't encounter any lag spikes or anything like that. It's been rather smooth.

[Cipher]

1 minute ago, Krimson said:

@Cipher I've been logged in for hours, and I did some casual monster hunting. I didn't encounter any lag spikes or anything like that. It's been rather smooth.

Great to hear, thank you!

[LocaMocha]

Thank you very much for the information and update Cipher, going to pop onto Vanguard now!