Jump to content

[Resolved] Network Issues - July 3rd, 2020


Jimmy

Recommended Posts

  • City Council

Hey all,

 

Just wanted to provide a brief update on our situation.

 

First, some background: Once you're hosting for more than a few hundred concurrent players, the resources required to host a server need to be spread over multiple physical machines (instead of all being virtualized on one machine). In order for this to work it's required that these machines are all in their own private network - however when renting hardware this needs to be virtualized.

 

OVH is one of the few hosts that provide a compatible technology (vRack) at a reasonable price - and unfortunately, it's this part of our infrastructure that we have the least control over, and by extension something we aren't able to build redundancy for.

 

Most of the issues over the past few days have been a result of the vRack connections between the individual machines being disrupted (either directly or because of wider network issues). As you may have noticed, Reunion and Everlasting have been mostly unaffected. After investigation we've discovered that machine hosting Everlasting's database server is located on a completely different server rack to the machines that host Torchbearer, Excelsior and Indomitable (and, of course, Reunion is across the pond!).

 

So - we're re-configuring a few things and moving TB, Excel and Indom's database servers to the 'good' rack and taking the machines on the 'bad' rack offline. This means that we will be running with reduced redundancy, so here's hoping the good rack remains good 🙂 No ETA just yet - we'll let you know when there is one.

 

In the meantime we're in contact with OVH to try and get the issues with the bad rack resolved. Long term we are also going to be investigating alternative hosting providers to try and avoid these issues going forward.

  • Like 16
  • Thanks 41

Got time to spare? Want to see Homecoming thrive? Consider volunteering as a Game Master!

Link to comment
Share on other sites

Thank you so much for sharing the details of what is going on. I, for one, am happy to donate what I can if you find a better, but more expensive, alternative. (Nothing crazy, but I'll weigh the total cost against the number of likely contributors).

 

Oh, and just to be clear, this won't be a one time donation, but a recurring one. 😉

 

Edited by zmonkey
  • Like 14
Link to comment
Share on other sites

Cant say how much we appreciate all your hard work.  Game has been fun and this is the first time I have played it.  Your generous support of the game and us gamers is once again appreciated.

 

Because I am very new to game I don't know where donations can be sent.  Please inform.

Edited by qaz1qaz1qa
  • Like 2
Link to comment
Share on other sites

So we'll be one rack short of a full load? Gotcha!

 

  • Like 2
  • Haha 8
  • Victory: reserved for future use
  • Indom: Schtick, Pummel Pete, Plymouth, Pilkington
  • Reunion: Ghost Legacy, 7s7e7v7e7n7, Mind Funk, Bluto
  • Excelsior: Phrendon Largo, Fred Bumbler, John van der Waals,Allamedia Jones, Tzapt, Sn1pe
  • Torchbearer: Phrendon Largo, Kenny Letter,  Bewm, La Merle, Enflambe', Rock Largo, Bulk of the Weather, Retired Phrendon
  • Everlasting: Phrendon Largo, Krown, Buzz Words, Bicycle Repairman, Dee Fender, Carmela Soprano, Radmental Boy, Beet Salad, Sporanghi,Sue Ahn Cuddy, Fukushima Technician, Snow Globe Girl, Thug Therapist, Apple Brown Betty
Link to comment
Share on other sites

Sad this is happening, but kudos for keeping us informed.

 

And if the current terms and agreements will allow it, I am sure there would be more donations available for collection if it means we're moved to more robust servers.

  • Like 5
Link to comment
Share on other sites

Sorry, I'm a bit confused.  If Everlasting is most affected, it seems like that would be on the "bad rack", but you're migrating the other servers?  I have no tech background, so maybe I'm just missing something obvious to others...

Link to comment
Share on other sites

Thank you for keeping us up to date and offering a very easy to understand explanation.  Given how long I could not visit Paragon City, I will here merely note that I appreciate your efforts greatly, and I'm sure it will be up as soon as you can make it so.  Thank you so much for your hard work.

  • Like 2

Great Justice - Invuln/Energy Melee Tank

Ann Atomic - Radiation/Super Strength Tank

Elecutrix - Electric Blast/Super Reflexes Sentinel

Ramayael - Titan Weapons/Bio Scrapper

C'len - Spines/Bio Brute

Link to comment
Share on other sites

1 minute ago, dtj714 said:

Sorry, I'm a bit confused.  If Everlasting is most affected, it seems like that would be on the "bad rack", but you're migrating the other servers?  I have no tech background, so maybe I'm just missing something obvious to others...

Everlasting (and Reunion, being in a different location altogether) are on 'good' racks.  Torchbearer, Indomitable, and Excelsior are on the currently 'bad' Rack, and are being moved.

Great Justice - Invuln/Energy Melee Tank

Ann Atomic - Radiation/Super Strength Tank

Elecutrix - Electric Blast/Super Reflexes Sentinel

Ramayael - Titan Weapons/Bio Scrapper

C'len - Spines/Bio Brute

Link to comment
Share on other sites

2 minutes ago, dtj714 said:

Sorry, I'm a bit confused.  If Everlasting is most affected, it seems like that would be on the "bad rack", but you're migrating the other servers?  I have no tech background, so maybe I'm just missing something obvious to others...

They said Everlasting and Reunion were on the good ones, the other servers on the bad ones so it seems they're going to try to reconfigure things.  

  • Like 1
Link to comment
Share on other sites

After many years of playing this game back in the day, i've finally learned of its new existence and i'm able to bask in all of its glory once again, and to learn its all run by donations its so amazing of you, plus the updates on problems etc are awesome, i would definitely be willing to donate what i could in support!  

Edited by dahaystack
  • Like 3
Link to comment
Share on other sites

I fully trust the capabilities of the Homecoming team and am convinced that it makes the right decisions. For that reason, I will continue to donate to keep City of Heroes alive including the fantastic community.

Regards,
Drallion

  • Like 3
Link to comment
Share on other sites

 "As you may have noticed, Reunion and Everlasting have been mostly unaffected. After investigation we've discovered that machine hosting Everlasting's database server is located on a completely different server rack to the machines that host Torchbearer, Excelsior and Indomitable (and, of course, Reunion is across the pond!)."

 

This statement just made it seem like those two servers were in a worse situation, so migrating the others seemed counter-intuitive.  I have no doubt they know what they're doing.

 

Edit - Derp, I read it as "mostly affected", nm 🤣

Edited by dtj714
derp
  • Haha 1
Link to comment
Share on other sites

I've only been back for about a month, but it's been so nice. Especially during these strange and trying times. It's honestly been therapeutic.  I echo the others that will gladly make a donation, or even a reoccurring donation, to fix the issue and keep things going. I'd hate to lose the CoH world again so soon.  

  • Like 2
Link to comment
Share on other sites

Thank you for the update, Jimmy.  FWIW, I've heard great things about IBM Cloud (formerly SoftLayer), but I know that they're quite a bit more expensive than OVH unfortunately.  I'm sure that you and the others will find a more suitable host.

Link to comment
Share on other sites

  • Ouroboros
19 minutes ago, Jimmy said:

Most of the issues over the past few days have been a result of the vRack connections between the individual machines being disrupted (either directly or because of wider network issues).


Hey, @Mender Derek it looks like it wasn't your fault, after all! What a surprise.

  • Haha 4

Mender Silos

Leader of Ouroboros

 

Got time to spare? Want to see Homecoming thrive? Consider volunteering as a Game Master!

 

 

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...