[Resolved] Network Issues - July 3rd, 2020

[Jimmy]

Hey all,

 

Just wanted to provide a brief update on our situation.

 

First, some background: Once you're hosting for more than a few hundred concurrent players, the resources required to host a server need to be spread over multiple physical machines (instead of all being virtualized on one machine). In order for this to work it's required that these machines are all in their own private network - however when renting hardware this needs to be virtualized.

 

OVH is one of the few hosts that provide a compatible technology (vRack) at a reasonable price - and unfortunately, it's this part of our infrastructure that we have the least control over, and by extension something we aren't able to build redundancy for.

 

Most of the issues over the past few days have been a result of the vRack connections between the individual machines being disrupted (either directly or because of wider network issues). As you may have noticed, Reunion and Everlasting have been mostly unaffected. After investigation we've discovered that machine hosting Everlasting's database server is located on a completely different server rack to the machines that host Torchbearer, Excelsior and Indomitable (and, of course, Reunion is across the pond!).

 

So - we're re-configuring a few things and moving TB, Excel and Indom's database servers to the 'good' rack and taking the machines on the 'bad' rack offline. This means that we will be running with reduced redundancy, so here's hoping the good rack remains good 🙂 No ETA just yet - we'll let you know when there is one.

 

In the meantime we're in contact with OVH to try and get the issues with the bad rack resolved. Long term we are also going to be investigating alternative hosting providers to try and avoid these issues going forward.

[zmonkey]

Thank you so much for sharing the details of what is going on. I, for one, am happy to donate what I can if you find a better, but more expensive, alternative. (Nothing crazy, but I'll weigh the total cost against the number of likely contributors).

 

Oh, and just to be clear, this won't be a one time donation, but a recurring one. 😉

 

Edited July 3, 2020 by zmonkey

[Justified_Son]

I second zmonkey.

[qaz1qaz1qa]

Cant say how much we appreciate all your hard work.  Game has been fun and this is the first time I have played it.  Your generous support of the game and us gamers is once again appreciated.

 

Because I am very new to game I don't know where donations can be sent.  Please inform.

Edited July 3, 2020 by qaz1qaz1qa

[KC4800]

So we'll be one rack short of a full load? Gotcha!

 

[Yunamoi]

Yes, switch to a better hoster, I would be willing to donate as well for that!

[GadgetDon]

I've been involved in a few server changes in a system FAR simpler than yours. You have my sympathy, empathy, and good luck.

[PSLAnimal]

I'm in favor of nice racks, so this is good news.

[Cruiser_7]

Sad this is happening, but kudos for keeping us informed.

 

And if the current terms and agreements will allow it, I am sure there would be more donations available for collection if it means we're moved to more robust servers.

[dtj714]

Sorry, I'm a bit confused.  If Everlasting is most affected, it seems like that would be on the "bad rack", but you're migrating the other servers?  I have no tech background, so maybe I'm just missing something obvious to others...

[marcussmythe]

Thank you for keeping us up to date and offering a very easy to understand explanation.  Given how long I could not visit Paragon City, I will here merely note that I appreciate your efforts greatly, and I'm sure it will be up as soon as you can make it so.  Thank you so much for your hard work.

[marcussmythe]

1 minute ago, dtj714 said:

Sorry, I'm a bit confused.  If Everlasting is most affected, it seems like that would be on the "bad rack", but you're migrating the other servers?  I have no tech background, so maybe I'm just missing something obvious to others...

Everlasting (and Reunion, being in a different location altogether) are on 'good' racks.  Torchbearer, Indomitable, and Excelsior are on the currently 'bad' Rack, and are being moved.

[Mezmera]

2 minutes ago, dtj714 said:

Sorry, I'm a bit confused.  If Everlasting is most affected, it seems like that would be on the "bad rack", but you're migrating the other servers?  I have no tech background, so maybe I'm just missing something obvious to others...

They said Everlasting and Reunion were on the good ones, the other servers on the bad ones so it seems they're going to try to reconfigure things.  

[Barbaruj]

Thanks for the update and all your hard work 🙂

 

[dahaystack]

After many years of playing this game back in the day, i've finally learned of its new existence and i'm able to bask in all of its glory once again, and to learn its all run by donations its so amazing of you, plus the updates on problems etc are awesome, i would definitely be willing to donate what i could in support!  

Edited July 3, 2020 by dahaystack

[Nightcoil]

I fully trust the capabilities of the Homecoming team and am convinced that it makes the right decisions. For that reason, I will continue to donate to keep City of Heroes alive including the fantastic community.

Regards,
Drallion

[Satanicus]

Thank you!

[tennisguy1221]

Thank you for keeping us up to date.  I would be happy to contribue every month to help with all of this.  

[Coyote]

Willing to donate for better racks.

[arcane]

I too appreciate a nice big rack

[dtj714]

 "As you may have noticed, Reunion and Everlasting have been mostly unaffected. After investigation we've discovered that machine hosting Everlasting's database server is located on a completely different server rack to the machines that host Torchbearer, Excelsior and Indomitable (and, of course, Reunion is across the pond!)."

 

This statement just made it seem like those two servers were in a worse situation, so migrating the others seemed counter-intuitive.  I have no doubt they know what they're doing.

 

Edit - Derp, I read it as "mostly affected", nm 🤣

Edited July 3, 2020 by dtj714
derp

[Jacobi Law]

I've only been back for about a month, but it's been so nice. Especially during these strange and trying times. It's honestly been therapeutic.  I echo the others that will gladly make a donation, or even a reoccurring donation, to fix the issue and keep things going. I'd hate to lose the CoH world again so soon.  

[Apparition]

Thank you for the update, Jimmy.  FWIW, I've heard great things about IBM Cloud (formerly SoftLayer), but I know that they're quite a bit more expensive than OVH unfortunately.  I'm sure that you and the others will find a more suitable host.

[Mender Silos]

19 minutes ago, Jimmy said:

Most of the issues over the past few days have been a result of the vRack connections between the individual machines being disrupted (either directly or because of wider network issues).

Hey, @Mender Derek it looks like it wasn't your fault, after all! What a surprise.

[Banazirgalasbor]

HA, you cant fool me, its a Nemesis plot!