Map Servered.. How many until calling it a day?

[Apparition]

11 minutes ago, TrelNargil said:

   Ok, as most of you on Excelsior know, the server went down yet again, with only 681 people on it, and a total of 1363 people in the game at the time period. And while we are all blessed that this game we love is here and (semi) running by these selfless GM's who are generally pretty amazing and all, I find it the pinnacle of irony that on a day of server maintenance, this major mapserver issue is not only still not resolved, but might be in fact getting worse (at least for Excelsior). And just to let you know, I upgraded my internet and PC speed, giving me higher ping and internet speeds/connectivity, so it is most assuredly NOT client side that is the issue, but server side.

   As a friend of mine suggested recently, maybe it's time for the GM"s to have another "emergency fund raiser drive" so they can get enough money together so they can upgrade their hardware to be able to handle running things, because as it stands now, it's not looking all that good. Or maybe, just maybe, have a GM or two contact some of the great people here with experience and technical knowhow to be able to help fix things, because again, it's not getting better as today's issues clearly show (especially on a maintenance day). It only takes one person with a good idea to maybe see what the problem is, and maybe, just maybe, its someone here. 🙂

   But just sitting around and waiting for things to get fixed, when it's clearly not, and with even less communication by the GM's on the matter, isn't good either. However, there is definitely one positive thing that has come out of this can of worms, and it is this... it has sorta quieted down all the "talk" (arguments) about the server's being joined, because after all, if the hardware can't even handle the load that we have now, how is it gonna handle the load when there is only one or two servers? 🙂 So at least that's is one way to make lemonade out of lemons lol...

 

Peace...

 

 

It's been explained by Homecoming staff on the Homecoming Discord that hardware issues have been ruled out.  The hardware is fine, as it's shared among all of the shards and only Excelsior is having issues.  What appears to be happening is that too many people consolidated on Excelsior and the shard has occasionally become more active than the Freedom server was on live.  As a result, Excelsior is occasionally hitting hard City of Heroes database limits as Cryptic Studios never planned for a shard to have as many people nor as much activity as Excelsior.

 

It isn't a coincidence that Excelsior seems to be going *boom* more often when a Hamidon or mothership raid is being run or recently ended.

[TrelNargil]

   Oh ok, cools. As I don't use Discord, I did not know about the Hardware not being the case... supposedly. The only reason though that I still have my "semi" doubts, is because the issue is NOT confined to just Excelsior as others I have seen talking in chat have experienced the same issues. The problem may not be to the extent that it is on Excelsior (most likely) due to the population differences, but it is still happening. And as they say, where there is smoke, there is fire, so something is going wrong; it's just not clear what exactly it may be (or even several things at once even as has been said already... like the Hami raids for example... thanks for that connection @Apparition).

   I just wish we could help in some way "technically speaking" ya know? Seeing as the GM's have done so much for us and all in this great game of our's (let alone provide it to us again). I am sure someone here would have a few ideas' in their back pockets "if" they knew what was going on behind the curtain so to speak. Thanks again everyone for your input... 🙂 

 

 

Peace...

Edited June 22 by TrelNargil
Dang spelling errors... :(

[Arc-Mage]

Yes, It would be nice if the "Homecoming staff on the Homecoming Discord" would pop by the forums and say that themselves. 

 

 

[Krimson]

54 minutes ago, Arc-Mage said:

Yes, It would be nice if the "Homecoming staff on the Homecoming Discord" would pop by the forums and say that themselves. 

I'd rather they actually identify the problem, then addresses the problem, and THEN worry about explaining it. They could be busy at the moment. 

[TheZag]

They have been on the forum giving updates.  It just wasnt a dedicated thread started by a team member.

 

 

On 6/20/2022 at 10:45 AM, Telephone said:

For those of you who hunger for technical details, the current belief is that the issue is rooted in what are called 'queue barriers'. The SQL queue on each shard is actually processed by many parallel threads in the dbserver; operations relating to a particular entity are usually routed to the same thread to ensure ordering, but sometimes certain operations require additional synchronization (for example, multiple entities might be involved) and barriers are used to provide this synchronization.

Edited June 22 by TheZag

[srmalloy]

20 hours ago, TrelNargil said:

And just to let you know, I upgraded my internet and PC speed, giving me higher ping and internet speeds/connectivity, so it is most assuredly NOT client side that is the issue, but server side.

If you upgraded your internet and PC speed and got higher ping, take it back and have them fix it; it's broken. A better connection will give you a lower ping. The second part is correct, though.

[Arc-Mage]

3 hours ago, Krimson said:

I'd rather they actually identify the problem, then addresses the problem, and THEN worry about explaining it. They could be busy at the moment. 

I have to disagree. An acknowledgement of the problem goes a long way. How long does it take to post something to the effect, "We are aware of the problem and are looking into it."?

A minute? If you don't you get exactly what we got, people guessing, complaining, arguing and getting upset and leaving frustrated.  

 

3 hours ago, TheZag said:

They have been on the forum giving updates.  It just wasnt a dedicated thread started by a team member.

 

 

I never would have looked under "Help and Support" to find out if anyone knew there where problems. I never said it needed to be a dedicated thread started by a team member. It would have been nice to see a GM post someplace in this thread though acknowledging they where aware of the problem, that's all I'm saying.

[Krimson]

11 minutes ago, Arc-Mage said:

I have to disagree. An acknowledgement of the problem goes a long way. How long does it take to post something to the effect, "We are aware of the problem and are looking into it."?

A minute? If you don't you get exactly what we got, people guessing, complaining, arguing and getting upset and leaving frustrated.  

Submit a ticket and DEMAND answers? Give those work slackers the what for, perhaps?

[TrelNargil]

15 hours ago, srmalloy said:

If you upgraded your internet and PC speed and got higher ping, take it back and have them fix it; it's broken. A better connection will give you a lower ping. The second part is correct, though.

 

   Thank you for the correction @srmalloy; yeah I have lower ping, not higher. Just my addled brain making a tech term error is all lol, so thanks again for the correction... 👍

 

12 hours ago, Krimson said:

Submit a ticket and DEMAND answers? Give those work slackers the what for, perhaps?

 

   That was not the impression on what @Arc-Mage was saying there @Krimson as "Demand" is a very extreme term here I would think. However, for the sake of those who do not know where to find such info, nor that it even exists, a "Request" is not out of the order I would think, even if it's just a link to said web page (and thank you for posting it too @Arc-Mage). Understanding & empathy, on all sides of the isle here, is and would be good here I think... as well as COFFEE at this hour of the morning <em raises cup to you all in greetings>... Cheers! 🙂 

 

Peace...

[Troo]

It does seem interesting that the Excelsior server issue is so much more prevalent.

The other servers appreciate any work being done by the HC team to ensure what happens on Excelsior stays on Excelsior.

 

 

 

Edited Thursday at 11:35 PM by Troo

[Arc-Mage]

6 hours ago, TrelNargil said:

 

   Thank you for the correction @srmalloy; yeah I have lower ping, not higher. Just my addled brain making a tech term error is all lol, so thanks again for the correction... 👍

 

 

   That was not the impression on what @Arc-Mage was saying there @Krimson as "Demand" is a very extreme term here I would think. However, for the sake of those who do not know where to find such info, nor that it even exists, a "Request" is not out of the order I would think, even if it's just a link to said web page (and thank you for posting it too @Arc-Mage). Understanding & empathy, on all sides of the isle here, is and would be good here I think... as well as COFFEE at this hour of the morning <em raises cup to you all in greetings>... Cheers! 🙂 

 

Peace...

Absolutely this. I was just suggesting communication.

[Apparition]

1 hour ago, Arc-Mage said:

Absolutely this. I was just suggesting communication.

 

For better or for worse, most communication is done on the Homecoming Discord server.

[GM Kal]

On 6/22/2022 at 6:23 PM, Arc-Mage said:

I have to disagree. An acknowledgement of the problem goes a long way. How long does it take to post something to the effect, "We are aware of the problem and are looking into it."?

A minute? If you don't you get exactly what we got, people guessing, complaining, arguing and getting upset and leaving frustrated.  

 

I never would have looked under "Help and Support" to find out if anyone knew there where problems. I never said it needed to be a dedicated thread started by a team member. It would have been nice to see a GM post someplace in this thread though acknowledging they where aware of the problem, that's all I'm saying.

We are aware of the problem and looking into it!


Admin staff (the folks that run and maintain the server) have created a new type of monitoring and logging program to see exactly what is wrong with Excel, and it's been getting good data the last few days. They have several ideas of what could be wrong, but do not want to change server code till they are sure that it is in fact the problem.

I have every confidence in them, they have already solved some long standing server inefficiencies by fixing server code that have been around since live days.
The 2020 Peregrine Island crash during trick or treat was one.

 

[TrelNargil]

8 hours ago, GM Kal said:

We are aware of the problem and looking into it!


Admin staff (the folks that run and maintain the server) have created a new type of monitoring and logging program to see exactly what is wrong with Excel, and it's been getting good data the last few days. They have several ideas of what could be wrong, but do not want to change server code till they are sure that it is in fact the problem.

I have every confidence in them, they have already solved some long standing server inefficiencies by fixing server code that have been around since live days.
The 2020 Peregrine Island crash during trick or treat was one.

 

 

   Good morning, and thank you very much for posting this update @GM Kal. As a person who grew up in a home with a Dad who was a communications' officer & naval war hero during WW2 (he was very much involved in the battle of midway btw, and the development of radar too btw), he always stressed the importance of communication in everything we do (his favorite quote was, "A lack of communication can get people killed."). So any form of it about this ongoing issue would/will (especially detailed one's) go a long way in easing troubled minds so to speak, and I for one appreciate the effort; maybe have such a update once a week or so (either here or in the other post), that could potentially head off such posts as this. Just a suggestion... 😄 In either case, thanks again for the reply @GM Kal; you all rock, so keep up the great work! 👍

 

Peace (and coffee... Hmmmmm... yum...)...

[GM Kal]

Not a problem, but as Apparition said, most timely and smaller updates and news are on discord.

 

We really didn't have clear news to share on the forum yet.

 

 

Edit:

Per Telephone on the discord just now:

 

Excelsior is now available for play.

 

Today we optimized a temporary table used by the shard to store effects on players ('attribmods'). Previously, this table was stored on-disk even though the data is cleared at shard restart. The table was moved to be entirely in-memory, which should reduce a significant amount of contention in the database.

Attribmod save operations (potentially resulting in what the database server calls high latch contention) could be especially heavy when many high-level players leave a map or quit at the same time (e.g. after an instanced Rikti raid or after a Hami raid), due to the number of buffs that might still be present.

This change is experimental; we believe it to be stable but if issues or instability are encountered we may perform an emergency restart of Excelsior.

In addition, we've discovered a few other potential sources of issues for which a small patch may be issued on our normal patch day (this coming Tuesday). Unfortunately, I don't believe we've made a lot of progress yet on the superpack/email issues, but we are continuing to work on them.

Thanks again for your patience as we continue resolving this issue.

Please report any unusual issues on Excelsior in #report-server-issues-here or file a petition.

Edited Friday at 01:40 PM by GM Kal

[Troo]

☏

[Arc-Mage]

11 hours ago, GM Kal said:

We are aware of the problem and looking into it!


Admin staff (the folks that run and maintain the server) have created a new type of monitoring and logging program to see exactly what is wrong with Excel, and it's been getting good data the last few days. They have several ideas of what could be wrong, but do not want to change server code till they are sure that it is in fact the problem.

I have every confidence in them, they have already solved some long standing server inefficiencies by fixing server code that have been around since live days.
The 2020 Peregrine Island crash during trick or treat was one.

 

lmao, Was that so hard?

[Arc-Mage]

20 hours ago, Apparition said:

 

For better or for worse, most communication is done on the Homecoming Discord server.

Yah, Discord Servers.....

[Troo]

3 hours ago, Arc-Mage said:

Discord

[TrelNargil]

   I am with you guys on this one...

   Discord...

 

 

[Troo]

On 6/22/2022 at 11:28 AM, TheZag said:

They have been on the forum giving updates.  It just wasnt a dedicated thread started by a team member.

 

Different topic.

Not everyone getting booted from the game is doing hundreds or thousands of super packs and having trouble claiming the associated rewards.

Folks getting routinely knocked off a game, even a small community or volunteer run game, could wonder or look for an official update on a wide spread issue.

 

 

Edited Saturday at 05:26 AM by Troo

[TheZag]

Thats why i linked it.  People in this topic were asking,  and what they wanted to see was in a different thread that they probably wouldnt have looked at.  The quick solution remains the same though - transfer off excelsior.  Its traffic is heavier than the database was designed for and players get booted among other problems.  The slower solution is wait for the team to find a fix.  But a fix can be anywhere from right now to several weeks from now.  In the meantime ill be enjoying a stable gameplay experience on not excelsior and then still on not excel after a fix is found.

[Krimson]

3 hours ago, TheZag said:

Thats why i linked it.  People in this topic were asking,  and what they wanted to see was in a different thread that they probably wouldnt have looked at.  The quick solution remains the same though - transfer off excelsior.  Its traffic is heavier than the database was designed for and players get booted among other problems.  The slower solution is wait for the team to find a fix.  But a fix can be anywhere from right now to several weeks from now.  In the meantime ill be enjoying a stable gameplay experience on not excelsior and then still on not excel after a fix is found.

But the problems are not exclusive to Excelsior. If I transfer any of my toons from my two accounts, my email is still full of items I haven't collected, and if that has anything to do with the problem, then all location change means is I get to solo somewhere less populated with even less chance of teaming with someone I know.  However, if the solution is to nuke Excelsior, and then wait and see what the next problematic shard will be, Everlasting has a LOT of BIG bases, then I will wait and see what HC Staff has to say.

 

Right now, I don't think they have communicated the problem to us because I don't think they have identified the problem. Hence, no reason to come in the thread and say, "Umm... well it's a textbook definition of a SNAFU. In some ways, the Shards are Normal, and in others they are Effed Up."

Edited Saturday at 04:26 PM by Krimson

[GM Kal]

Per Admin Number Six:

Quote

 

Number Six — 

We did some (more) database tuning and have not seen any SQL issues for the last 72 hours.

 

I can't say for sure it's totally fixed, but it's starting to look that way to me.

[Krimson]

Been pretty smooth so far.