Jump to content
Jimmy

[Resolved] Network Issues - July 3rd, 2020

Recommended Posts

Hey all,

 

Just wanted to provide a brief update on our situation.

 

First, some background: Once you're hosting for more than a few hundred concurrent players, the resources required to host a server need to be spread over multiple physical machines (instead of all being virtualized on one machine). In order for this to work it's required that these machines are all in their own private network - however when renting hardware this needs to be virtualized.

 

OVH is one of the few hosts that provide a compatible technology (vRack) at a reasonable price - and unfortunately, it's this part of our infrastructure that we have the least control over, and by extension something we aren't able to build redundancy for.

 

Most of the issues over the past few days have been a result of the vRack connections between the individual machines being disrupted (either directly or because of wider network issues). As you may have noticed, Reunion and Everlasting have been mostly unaffected. After investigation we've discovered that machine hosting Everlasting's database server is located on a completely different server rack to the machines that host Torchbearer, Excelsior and Indomitable (and, of course, Reunion is across the pond!).

 

So - we're re-configuring a few things and moving TB, Excel and Indom's database servers to the 'good' rack and taking the machines on the 'bad' rack offline. This means that we will be running with reduced redundancy, so here's hoping the good rack remains good 🙂 No ETA just yet - we'll let you know when there is one.

 

In the meantime we're in contact with OVH to try and get the issues with the bad rack resolved. Long term we are also going to be investigating alternative hosting providers to try and avoid these issues going forward.

  • Like 16
  • Thanks 41

Share this post


Link to post
Share on other sites
Posted (edited)

Thank you so much for sharing the details of what is going on. I, for one, am happy to donate what I can if you find a better, but more expensive, alternative. (Nothing crazy, but I'll weigh the total cost against the number of likely contributors).

 

Oh, and just to be clear, this won't be a one time donation, but a recurring one. 😉

 

Edited by zmonkey
  • Like 14

Share this post


Link to post
Share on other sites
Posted (edited)

Cant say how much we appreciate all your hard work.  Game has been fun and this is the first time I have played it.  Your generous support of the game and us gamers is once again appreciated.

 

Because I am very new to game I don't know where donations can be sent.  Please inform.

Edited by qaz1qaz1qa
  • Like 2

Share this post


Link to post
Share on other sites

So we'll be one rack short of a full load? Gotcha!

 

  • Like 2
  • Haha 8

Share this post


Link to post
Share on other sites

Yes, switch to a better hoster, I would be willing to donate as well for that!

  • Like 4

Share this post


Link to post
Share on other sites

I've been involved in a few server changes in a system FAR simpler than yours. You have my sympathy, empathy, and good luck.

  • Like 2

Share this post


Link to post
Share on other sites

I'm in favor of nice racks, so this is good news.

  • Like 3
  • Haha 7

Pleased to meet you!  I am Animal, and I harbor Perverse Sexual Lust.

Share this post


Link to post
Share on other sites

Sad this is happening, but kudos for keeping us informed.

 

And if the current terms and agreements will allow it, I am sure there would be more donations available for collection if it means we're moved to more robust servers.

  • Like 5

Share this post


Link to post
Share on other sites

Sorry, I'm a bit confused.  If Everlasting is most affected, it seems like that would be on the "bad rack", but you're migrating the other servers?  I have no tech background, so maybe I'm just missing something obvious to others...

Share this post


Link to post
Share on other sites

Thank you for keeping us up to date and offering a very easy to understand explanation.  Given how long I could not visit Paragon City, I will here merely note that I appreciate your efforts greatly, and I'm sure it will be up as soon as you can make it so.  Thank you so much for your hard work.

  • Like 2

Great Justice - Invuln/Energy Melee Tank

Ann Atomic - Radiation/Super Strength Tank

Elecutrix - Electric Blast/Super Reflexes Sentinel

Ramayael - Titan Weapons/Bio Scrapper

C'len - Spines/Bio Brute

Share this post


Link to post
Share on other sites
1 minute ago, dtj714 said:

Sorry, I'm a bit confused.  If Everlasting is most affected, it seems like that would be on the "bad rack", but you're migrating the other servers?  I have no tech background, so maybe I'm just missing something obvious to others...

Everlasting (and Reunion, being in a different location altogether) are on 'good' racks.  Torchbearer, Indomitable, and Excelsior are on the currently 'bad' Rack, and are being moved.


Great Justice - Invuln/Energy Melee Tank

Ann Atomic - Radiation/Super Strength Tank

Elecutrix - Electric Blast/Super Reflexes Sentinel

Ramayael - Titan Weapons/Bio Scrapper

C'len - Spines/Bio Brute

Share this post


Link to post
Share on other sites
2 minutes ago, dtj714 said:

Sorry, I'm a bit confused.  If Everlasting is most affected, it seems like that would be on the "bad rack", but you're migrating the other servers?  I have no tech background, so maybe I'm just missing something obvious to others...

They said Everlasting and Reunion were on the good ones, the other servers on the bad ones so it seems they're going to try to reconfigure things.  

  • Like 1

Share this post


Link to post
Share on other sites
Posted (edited)

After many years of playing this game back in the day, i've finally learned of its new existence and i'm able to bask in all of its glory once again, and to learn its all run by donations its so amazing of you, plus the updates on problems etc are awesome, i would definitely be willing to donate what i could in support!  

Edited by dahaystack
  • Like 3

Share this post


Link to post
Share on other sites

I fully trust the capabilities of the Homecoming team and am convinced that it makes the right decisions. For that reason, I will continue to donate to keep City of Heroes alive including the fantastic community.

Regards,
Drallion

  • Like 3

Share this post


Link to post
Share on other sites
Posted (edited)

 "As you may have noticed, Reunion and Everlasting have been mostly unaffected. After investigation we've discovered that machine hosting Everlasting's database server is located on a completely different server rack to the machines that host Torchbearer, Excelsior and Indomitable (and, of course, Reunion is across the pond!)."

 

This statement just made it seem like those two servers were in a worse situation, so migrating the others seemed counter-intuitive.  I have no doubt they know what they're doing.

 

Edit - Derp, I read it as "mostly affected", nm 🤣

Edited by dtj714
derp
  • Haha 1

Share this post


Link to post
Share on other sites

I've only been back for about a month, but it's been so nice. Especially during these strange and trying times. It's honestly been therapeutic.  I echo the others that will gladly make a donation, or even a reoccurring donation, to fix the issue and keep things going. I'd hate to lose the CoH world again so soon.  

  • Like 2

Share this post


Link to post
Share on other sites

Thank you for the update, Jimmy.  FWIW, I've heard great things about IBM Cloud (formerly SoftLayer), but I know that they're quite a bit more expensive than OVH unfortunately.  I'm sure that you and the others will find a more suitable host.

Share this post


Link to post
Share on other sites
19 minutes ago, Jimmy said:

Most of the issues over the past few days have been a result of the vRack connections between the individual machines being disrupted (either directly or because of wider network issues).


Hey, @Mender Derek it looks like it wasn't your fault, after all! What a surprise.

  • Haha 4

Mender Silos

Leader of Ouroboros

 

 

July Event Info: Derek's Disaster!

 

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...