Server Outage (October 4th, 2025)

[Widower]

Okay, just to recap for those whose days haven't been consumed by this: VM3 went down around 3:15 Eastern. We spent a couple hours trying to see if we could prompt OVH to do something about it, which eventually succeeded. They replaced a CPU in the machine, but when Cipher went to reconnect it, IPMI wasn't working, so we ended up spending another three hours waiting for OVH to do something. It took them about two hours to replace the motherboard, RAM and CPU cooling system, but when Cipher went to reconnect it (yes, something of a pattern here) he found that some virtual disks were missing. Nothing mission critical, but still not good. Until we can figure out what's going on there, VM3 remains out of commission. Cipher set up VMs for Everlasting and Victory on one of the other hosts during one of the waits, so he's spinning those back up and is hoping to have them back up that way. If you were playing at the time of the initial crash, you shouldn't experience more than 17 minutes of rollback, and hopefully much less. 

[Force Redux]

Thank you to the HC team for working to get this resolved ❤️

[Widower]

Everlasting and Victory are up on one of the other hosts. We'll keep working to bring the affected host back up, though the ball is largely in OVH's hands. Hopefully it's back up for Tuesday maintenance and we'll rearrange everything then. Shout if you notice anything awry. Otherwise, go, hunt, kill skuls.

[TauntingMonk]

Thank you! 

[AspieAnarchy]

[ChickenPaddy]

Thanks for your hard work!  My mains are on one of the afflicted servers, so I'm glad I get to see them again

[Battle7]

i haven't seen anything about Vanguard Server, is there some update I'm not aware of? it shows offline at the log in page.

thanks.

[Staceypants]

18 minutes ago, Battle7 said:

i haven't seen anything about Vanguard Server, is there some update I'm not aware of? it shows offline at the log in page.

thanks.

Vanguard was lowest on the priority order. It will be up when it's up.

[lemming]

Thanks to the team for all the work on a Saturday night.

[kieraoona]

Thank you team for all that you do!  Time to go Trick or Treating!

 

[boundgirlie]

All that you all do is very much appreciated.  I use vanguard myself.  You all do your thing :)

[LocaMocha]

Vanguard main here as well, but I'm very glad to see the progress being made getting all of the servers up and running. I take it was a multiple hardware failure on the main authentication server then?

[Cipher]

2 hours ago, LocaMocha said:

Vanguard main here as well, but I'm very glad to see the progress being made getting all of the servers up and running. I take it was a multiple hardware failure on the main authentication server then?

We're unsure of the root cause, but something took out the CPU(s), certain functions of the mainboard such as IPMI, memory, and damaged the disks.

 

We're currently working on Vanguard and expect to have it up within the hour.

[Cipher]

Vanguard has been restored on new VMs, but we are unable to restore Architect Entertainment on Vanguard until we can safely recover data from the downed VM host.

[Krimson]

@Cipher I've been logged in for hours, and I did some casual monster hunting. I didn't encounter any lag spikes or anything like that. It's been rather smooth.

[Cipher]

1 minute ago, Krimson said:

@Cipher I've been logged in for hours, and I did some casual monster hunting. I didn't encounter any lag spikes or anything like that. It's been rather smooth.

Great to hear, thank you!

[LocaMocha]

Thank you very much for the information and update Cipher, going to pop onto Vanguard now!

[Navy Spider]

I was dusting up cons in the Zig... and the server went down. 😵

[AboveTheChemist]

48 minutes ago, Navy Spider said:

I was dusting up cons in the Zig... and the server went down. 😵

 

That's the scheduled weekly maintenance. According to discord the down time should last another hour or so.

[LocaMocha]

Was able to pop onto Vanguard and smack things around for several hours yesterday with no Issues. Thank you very much @Cipher for keeping us all in the loop, even if at some moments I'm sure it felt more like a Mobius Strip!

[starro]

Since the return of COH my favorite features outside the 64 bit client and move to official license has been the responsiveness and stability of HC.  Thank you volunteers for the return to service so quickly and the communication along the way!  Other volunteer projects might have been crippled longer or debilitated permanently from such an unfortunate turn of events.

[Widower]

So, over the past 24 hours we've had a few crashes take down Excelsior and Everlasting (you know, the servers most of you play on). Just in case anyone was wondering, this is unrelated to the issue from the weekend (or clicking too many doors in Trick or Treat :P). The issue is that vRack (the LAN which lets the servers in the cluster talk to each other) keeps failing, for reasons unknown. Hopefully OVH will clear up the issues sooner rather than later, but until then we're kind of at their mercy.

[jkwak]

an SG i m in started a betting pool on when the next crash happens, i have put 100m inf in to the pool, please tell OVH to make me win the pool and not disappoint me

[Krimson]

2 minutes ago, jkwak said:

an SG i m in started a betting pool on when the next crash happens, i have put 100m inf in to the pool, please tell OVH to make me win the pool and not disappoint me

When the clubs open, of course.

[Widower]

Just a PSA: tonight we did some emergency maintenance that hopefully should keep the issue from crashing the servers again. In commemoration, double XP is available on all shards until at least Tuesday. We are considering what to do for those who lost anything on account of the crashes.