Super Packs & Email Issues

[Telephone]

I just wanted to follow up and note that we're aware of the recent database/email/temporary server hang issues (largely affecting Excelsior but also affecting the other shards).

 

The shards each have what is called an SQL Queue which contains pending database operations. Under normal operations, the queue is minimal - our servers are more than adequately provisioned to handle load. Some things can cause a large amount of database load (Hami raids, bulk email claims, and bulk superpack openings are among these). Thanks to some work Six did early on in Homecoming, we can normally handle these as well without significant issue.

 

Recently we've been noticing SQL Queue spikes, which cause issues like the recent multi-minute hangs of Excelsior, and which are likely related to email claiming and superpack opening issues. We're adding additional instrumentation to find out what is causing these, and once we find the cause we'll fix it as quickly as possible.

 

With luck, the full shard restart today will help ameliorate the issue for some of the people who have been suffering it. We know from past experience that some people may still be affected; please be assured we're working on the issue and will resolve it.

 

Edit: This issue also can affect transfers between shards; we're working to resolve that as well.

[Bionic_Flea]

I blame Yomo and Ukase!

 

*Runs and hides in a bank vault*

[Apparition]

Oh, so it's Excelsior screwing up e-mail, superpack opening, and server transfers for everyone. 

[Tenebrose]

Thanks for the reply, @Telephone! After decades of database work, my SQL-sense was tingling on this. 🙂

[Destruktor]

Indeed, thanks @Telephone for the update, very much appreciated

[TalonBlue]

Also happening to me as well. I've been opening buying stacks of Super Packs and having the same issue with my inbox. Thank you for the update.

[Tenebrose]

Still happening, and it doesn't seem to be directly related to Super Packs. Today I claimed a bunch of unslotters (210), put them up on the auction house, then went back to claim more unslotters. It didn't allow anything else to be pulled from the email. I didn't open or buy any Super Packs.

[Apparition]

Last night on Torchbearer, I tried converting a bunch of enhancements, and after a little bit it stopped working entirely.  I clicked the button several times but nothing happened.  I thought that maybe it was lag or my connection, but I could do other things.  Converting enhancements just stopped working.  I tried again a little later and it worked again.

[Clashcityrocker]

Also still happening to me. Cannot claim any items from character item tab (Excelsior and Reunion).

[Tenebrose]

I was in the middle of a task force and couldn't claim any items from email. At this point, it's gone from a nuisance to being extremely frustrating.

[Dark Current]

It just happened to me on Excelsior with all my character items in my email including new things and older items.

 

Can't open new packs or claim old items

[Andreah]

On 5/27/2022 at 10:58 AM, Tenebrose said:

Still happening, and it doesn't seem to be directly related to Super Packs. Today I claimed a bunch of unslotters (210), put them up on the auction house, then went back to claim more unslotters. It didn't allow anything else to be pulled from the email. I didn't open or buy any Super Packs.

I fairly consistently get this problem.

 

I can cause email claim to stall most of the time doing these steps.

 

1. Open up my email window and the auction house at the same time. Assume I have, say, unslotters to claim from email, and an item of any kind in my auction house storage panel.

2. Claim a few unslotters from email.

3. Retrieve an item from auction storage.

4. Claim another unslotter from email.

 

On step 4., there will at least always be a noticeable delay in the unslotter being delivered. And fairly often it will not deliver for many seconds or even minutes. Any subsequent attempts to claim items from email will produce an error messages that the claim failed, another item claim is still pending.

[Yomo Kimyata]

On 5/24/2022 at 10:11 AM, Telephone said:

I just wanted to follow up and note that we're aware of the recent database/email/temporary server hang issues (largely affecting Excelsior but also affecting the other shards).

 

The shards each have what is called an SQL Queue which contains pending database operations. Under normal operations, the queue is minimal - our servers are more than adequately provisioned to handle load. Some things can cause a large amount of database load (Hami raids, bulk email claims, and bulk superpack openings are among these). Thanks to some work Six did early on in Homecoming, we can normally handle these as well without significant issue.

 

Recently we've been noticing SQL Queue spikes, which cause issues like the recent multi-minute hangs of Excelsior, and which are likely related to email claiming and superpack opening issues. We're adding additional instrumentation to find out what is causing these, and once we find the cause we'll fix it as quickly as possible.

 

With luck, the full shard restart today will help ameliorate the issue for some of the people who have been suffering it. We know from past experience that some people may still be affected; please be assured we're working on the issue and will resolve it.

 

Edit: This issue also can affect transfers between shards; we're working to resolve that as well.

 

I just saw this and wanted to say (belatedly) thank you so much for the post.  I'm very grateful for constructive communication!

 

There are clearly still issues, but I feel validated!!

[Yomo Kimyata]

On 5/24/2022 at 10:24 AM, Bionic_Flea said:

I blame Yomo and Ukase!

 

*Runs and hides in a bank vault*

 

Why is there a flea in my vault?

[Tenebrose]

I was gone for the weekend, so I haven't been on since Friday morning. Logged in a few moments ago, claimed influence from sold items in the AH, then attempted to claim one unslotter. Nothing.

[Telephone]

I just wanted to follow up and say that we've discovered a few potential causes for the Excelsior issues. We'll be extending maintenance tomorrow and implementing some of the fixes to see if they help.

 

Note that these are expected to help with the server slowdowns, but I do not believe they will help with superpack or auction claiming. If you've been having persistent issues with claiming and they persist beyond tomorrow's restart, please file a ticket and ask the GM team to direct it to me so that I can look into your account's database records in more detail. If you've already filed a ticket, please follow up on it with a note to bring it to my attention, and reference this post.

[Bionic_Flea]

On 6/4/2022 at 1:51 PM, Yomo Kimyata said:

Why is there a flea in my vault?

 

Fleas are ubiquitous!

[Tenebrose]

2 hours ago, Telephone said:

I just wanted to follow up and say that we've discovered a few potential causes for the Excelsior issues. We'll be extending maintenance tomorrow and implementing some of the fixes to see if they help.

 

Note that these are expected to help with the server slowdowns, but I do not believe they will help with superpack or auction claiming. If you've been having persistent issues with claiming and they persist beyond tomorrow's restart, please file a ticket and ask the GM team to direct it to me so that I can look into your account's database records in more detail. If you've already filed a ticket, please follow up on it with a note to bring it to my attention, and reference this post.

 

Thanks, @Telephone! I know all too well how elusive these SQL issues can be.

[Telephone]

Just to provide a quick update:

 

Last maintenance (not today's) we rebuilt the SQL indices for all shards and global services. While this did have some benefits, it didn't give us the improvement we were seeking nor resolve the issues seen on Excelsior.

 

Over the last week we managed to catch some of the Excelsior issues live and did some debugging on the SQL server with the instrumentation we deployed a few weeks ago. Unfortunately, we weren't able to track down the exact cause of the issue, even though we could see the symptoms clearly.

 

Today, we deployed additional instrumentation to Excelsior to help diagnose the issue. We're hopeful this change will give us the remaining information we need to solve the problem.

 

We also deployed some minor adjustments to various operations which can lag the SQL queue in the hope that this will help the issue.

 

Lastly, several of you have followed up on your tickets; I've seen the forwards from the GM team and we'll continue working on them along with this issue.

[Troo]

1 hour ago, Telephone said:

Just to provide a quick update:

 

Last maintenance (not today's) we rebuilt the SQL indices for all shards and global services. While this did have some benefits, it didn't give us the improvement we were seeking nor resolve the issues seen on Excelsior.

 

Over the last week we managed to catch some of the Excelsior issues live and did some debugging on the SQL server with the instrumentation we deployed a few weeks ago. Unfortunately, we weren't able to track down the exact cause of the issue, even though we could see the symptoms clearly.

 

Today, we deployed additional instrumentation to Excelsior to help diagnose the issue. We're hopeful this change will give us the remaining information we need to solve the problem.

 

We also deployed some minor adjustments to various operations which can lag the SQL queue in the hope that this will help the issue.

 

Lastly, several of you have followed up on your tickets; I've seen the forwards from the GM team and we'll continue working on them along with this issue.

 

appreciated.

 

any chance a few folks causing or exasperating the issue will be asked to knock it off?

[EmperorSteele]

2 hours ago, Troo said:

 

appreciated.

 

any chance a few folks causing or exasperating the issue will be asked to knock it off?

 

"Please avoid opening more than 13 super packs or e-mail items in between server ticks while Mercury is in Retrograde".

 

I dunno if Mercury is even IN  retrograde right now, or what that even MEANS i just heard it once and was given the impression that is causes STUFF to happen, IDK.

[Tenebrose]

2 hours ago, Troo said:

 

appreciated.

 

any chance a few folks causing or exasperating the issue will be asked to knock it off?

 

Unfortunately, it's difficult to get people to "knock it off" when we don't really know what "it" is. All we see are symptoms. The root cause could be something completely unrelated to email, AH, or super packs (although I'm putting my bet on the Auction House causing deadlocks with the SQS queue). I've ran into these problems while just trying to claim a single item from email or opening a single pack immediately after logging in during times of very light server load. I also had email lock up for me, while dozens of others stated they were not affected. I haven't seen these issues prior to about May 9th. It's unlikely players suddenly started doing something new that week and have continued to do for the last month that they haven't in the past few years.

[Andreah]

3 hours ago, Tenebrose said:

It's unlikely players suddenly started doing something new that week and have continued to do for the last month that they haven't in the past few years.

I've occasionally had difficulties claiming items from email and also opening superpacks for at least two years. It might be a little worse lately so far as I can tell, but it's not new in the last few months. And this has been on Everlasting, not Excelsior.

[Tenebrose]

3 hours ago, Andreah said:

I've occasionally had difficulties claiming items from email and also opening superpacks for at least two years. It might be a little worse lately so far as I can tell, but it's not new in the last few months. And this has been on Everlasting, not Excelsior.

That's odd. A buddy and I have been opening packs by the dozens daily for at least 6 months. I think I started around October last year. I don't recall having any issues before May. Definitely nothing like what is happening now.

 

We're on Torchbearer. I think the belief is that the issues on Excelsior may be causing the queue to get "stuck" for all the servers. As I understand, there's one email and auction house system shared between the servers. If one server is causing havoc with it, that could be felt on all the servers.

[Yomo Kimyata]

On 6/14/2022 at 8:59 PM, Tenebrose said:

That's odd. A buddy and I have been opening packs by the dozens daily for at least 6 months. I think I started around October last year. I don't recall having any issues before May. Definitely nothing like what is happening now.

 

We're on Torchbearer. I think the belief is that the issues on Excelsior may be causing the queue to get "stuck" for all the servers. As I understand, there's one email and auction house system shared between the servers. If one server is causing havoc with it, that could be felt on all the servers.

 

I echo this on just about every level.  I suspect the problem is a little bit server side and a little bit client side.  This may be observer bias, but my issues started with the inability to access my "global" Character Items, but only on Excelsior.  It was fine on other shards.  Then the problem migrated to other shards.  I even started using Reunion to try to access the Character Items but my record is spotty at best.

 

My "whistling to scare away tigers" method (which assumes the problem is client side) is to run ccleaner, shut down every program I have running on my computer except CoH, sacrifice a goat, delete as many emails as I can (I currently have about 30) and try to access the Character Items menu *WHILE THE /AH IS NOT OPEN (this anecdotally helps a lot)*.  It sometimes even works.  But not always.

 

I wonder too if my problem is simply that I have too much stuff in my CI menu.  The irony is that I'm unable to get rid of that stuff since I cannot consistently access it.