Super Packs & Email Issues

[Andreah]

37 minutes ago, Yomo Kimyata said:

Listen, I have to keep score somehow!

Influence ... is ... for ... Hoarding!!

 

[UltraAlt]

On 6/25/2022 at 1:54 PM, Tenebrose said:

Multiple times, I've logged in for the first time in 24 hours, bought one pack (no other packs), opened the pack, claimed exactly one converter, bought another pack, opened it, then attempted to claim a second convertor. It fails on the attempt with claiming the second converter.

 

Okay. I'll try to claim some converters.

I never use the things.

 

On 6/25/2022 at 1:54 PM, Tenebrose said:

Today, I bought 60+ packs. Was able to open 50 of them over the course of about 5 minutes. Claimed a few items as well. Then it locked up on opening the last few packs.

 

Do you remember what kind of items you claimed?

 

On 6/25/2022 at 1:54 PM, Tenebrose said:

do agree that it's possibly an in-game behavior. It's also quite possibly account specific, considering it's not affecting you. But I don't think it's due to the number of packs being opened, nor the number of overall items in email.

 

I have a whole lot of card pack items in email.

Generally, I'm only pulling and using or pulling and storing enhances. I don't pull out more than 100 so merit at a time, so that's pretty much nothing. I pull out about 100 brainstorms at a time when I do that. Everything else is piecemeal.

I have a huge surplus of unslotters, converters, etc.

 

On 6/25/2022 at 1:54 PM, Tenebrose said:

Perhaps a certain ATO is the cause, or one of the very rare drops (Character respect, windfall, etc.).

 

I would have to pull and store at another base to check this. My ATO storage base is pretty much full at this point.

 

On 6/25/2022 at 1:54 PM, Tenebrose said:

It might be worth checking what exactly is dropping in the packs before claiming items to see if there's a pattern.

 

Have you been opening any Winter packs?

I have had no need to buy any of those since the end of the even where I had hundreds of the things and stockpiled a base full for my alts.

[UltraAlt]

20 hours ago, Yomo Kimyata said:

If I have 100 alts with 100 market slots each (and that's conservative, and there could be hundreds more people like that), that's 10,000 market slots that I may or may not be doing stuff with.  On Live, I don't remember how many total market slots an account could end up with, but it was a small fraction of 10k.  We have a markedly smaller population, but a vastly larger ability to put stress on a database (the /AH) that is made out of baling wire and cork.

 

I had 150+ characters. It was 10 market slots per character, I had the max character slots per server, characters in very slot, and all were working the market.

 

I have 128 characters some with at or heading on 200 slots.  would say probably half of them maybe full of purchases, sales, and solds.

I have reigned it in recently because I want to play more. I probably rotated through 2-5 characters goods in a day at this point and those are my main trading characters versus the ones that I rotate goods on when I get around to playing them.

I play at odd times, and I can even tell when I'm putting stress on the market at those low population times, so I can only imagine what you guys are doing to the market during prime times on weekends.

 

I don't think the Homecoming servers can handle the number of players that the live servers could.

[UltraAlt]

2 hours ago, UltraAlt said:

On 6/25/2022 at 1:54 PM, Tenebrose said:

Multiple times, I've logged in for the first time in 24 hours, bought one pack (no other packs), opened the pack, claimed exactly one converter, bought another pack, opened it, then attempted to claim a second convertor. It fails on the attempt with claiming the second converter.

 

Okay. I'll try to claim some converters.

I never use the things.

@Tenebrose

 

Okay, I bought 2 card packs, opened them, claimed a converter, bought 2 more packs, opened them, claimed a converter, bought 2 more packs, opened them, claimed a converter ... no slowdown on my end.

But I did make the mistake of doing one round of "get all inf" before buying the first round of packs.

[Tenebrose]

Are there any updates on this issue? I haven't been on in a while but had about 15 minutes to log in at lunchtime. I was able to grab some items out of email, put things on the AH, but became frustrated and logged out when I couldn't claim more than a handful few convertors over the course a minute. So I can't tell if the original locking issues persists, but the timer to claim items is nearly as bad when you're trying to claim a few hundred convertors or brainstorm ideas.

[Apparition]

4 hours ago, Tenebrose said:

Are there any updates on this issue? I haven't been on in a while but had about 15 minutes to log in at lunchtime. I was able to grab some items out of email, put things on the AH, but became frustrated and logged out when I couldn't claim more than a handful few convertors over the course a minute. So I can't tell if the original locking issues persists, but the timer to claim items is nearly as bad when you're trying to claim a few hundred convertors or brainstorm ideas.

 

[Andreah]

I did a bunch of email claims right after the patch yesterday, and it still jammed up occasionally. Was it better than before? Maybe. Can I be sure? No. The pauses I experienced yesterday were short, lasting just a few seconds. Those aren't serious problems, and in the past, there's been much longer pauses. I'm holding off judgement until I've had more opportunities to stress it and see if it will jam for longer. 

[Telephone]

Just to follow up again (this will also be posted on Discord).

 

Over the past few weeks during maintenances, we deployed multiple rounds of instrumentation and mitigations. We believe that the most recent round of mitigations has finally significantly ameliorated if not cured the issues seen on Excelsior.

Among the large number of SQL Queue optimizations were significant fixes to performance issued caused by large team, league, and SG operations as well as corner cases encountered during certain character loading and saving operations. We also optimized the saving of certain temporary data to prevent it from requiring disk operations, and we found a few powers that were behaving very poorly and will be fixed in future patches.

In addition, during the process we implemented a number of rate-limiting measures to reduce database load. One of these was a rate-limiter for Character Items. Note that even though Character Items shows up under the Email tab, it's actually a separate system with its own limitations. If you see the text 'Please wait a moment', your Character Items actions have been rate-limited to ensure server performance, and you can try again in a few moments.

We're going to continue reviewing tickets, but if you've filed a ticket as I requested earlier in this thread, please follow up and see if your problem is related to Character Items (and if you're getting 'Please wait a moment' or some other error), or if it's related to the email system proper. If you're only receiving 'Please wait a moment' and then the problem goes away, or if the problem no longer occurs, please close your ticket.

The AH is an entirely separate issue; it has its own rate-limiter (which is not new, and which has a different message) and is a known problematic system that we will continue to work to improve.

Thanks again for your patience!

[Bionic_Flea]

Thanks for the work and the update.

[Tenebrose]

8 hours ago, Telephone said:

Just to follow up again (this will also be posted on Discord).

 

Over the past few weeks during maintenances, we deployed multiple rounds of instrumentation and mitigations. We believe that the most recent round of mitigations has finally significantly ameliorated if not cured the issues seen on Excelsior.

Among the large number of SQL Queue optimizations were significant fixes to performance issued caused by large team, league, and SG operations as well as corner cases encountered during certain character loading and saving operations. We also optimized the saving of certain temporary data to prevent it from requiring disk operations, and we found a few powers that were behaving very poorly and will be fixed in future patches.

In addition, during the process we implemented a number of rate-limiting measures to reduce database load. One of these was a rate-limiter for Character Items. Note that even though Character Items shows up under the Email tab, it's actually a separate system with its own limitations. If you see the text 'Please wait a moment', your Character Items actions have been rate-limited to ensure server performance, and you can try again in a few moments.

We're going to continue reviewing tickets, but if you've filed a ticket as I requested earlier in this thread, please follow up and see if your problem is related to Character Items (and if you're getting 'Please wait a moment' or some other error), or if it's related to the email system proper. If you're only receiving 'Please wait a moment' and then the problem goes away, or if the problem no longer occurs, please close your ticket.

The AH is an entirely separate issue; it has its own rate-limiter (which is not new, and which has a different message) and is a known problematic system that we will continue to work to improve.

Thanks again for your patience!

 

Thanks for the update! I tried hammering the system to see how things responded:

- Bought and opened 80 packs

- Pulled 70 ATO's, Brainstorm ideas, some convertors*

- Put most stuff up on the AH

- Pulled more ATO's and some more convertors

- Bought and opened 10 packs

- etc

 

Usually it would have easily locked up by the time I bought more packs. It didn't lock up once! So, that seems to have fixed the issue... with one QoL elephant in the room...

 

*So this gets more into "Suggestions" I guess, as I am assuming the timer on claiming stuff from email is going to be needed going forward. It's generally not an issue. Even claiming 70 ATO's was not a problem at all. However, as I was claiming Convertors, Brainstorm Ideas, and Unslotters, it became suspect that the "Please wait a moment" message was less keyed off a system resource restriction and more of a built in timer on the button itself: As long as I clicked the button almost exactly once every second, the message never appeared. If I click it faster, it almost always showed up. Again, I get it and understand why it was likely necessary. However, there could be some QOL improvements here:

 

1. Don't let the "Claim" button be clickable until 1 second (or whatever the timer is) passes. It feels almost like a "punishment" for clicking the button before it was ready. If it is clicked too soon, you get stuck with the "Please wait a moment" message for around 3 seconds. No sense allowing it to happen just to punish the player.

 

2. Change the "5x" for Convertors and Unslotters to work like other salvage that is awarded in groupings: You can claim all 5 with one click. I get that would mean we would have a place in our email for "1 Convertor/Unslotter" salvage and "5x Convertor Unslotter" salvage. But it's not hard to open a few packs and end up with 30-60 items to claim. It shouldn't take a full minute of painfully clicking one button every second just to claim them. Alternatively, perhaps only have them in groups of 5x and rebalance the rewards table to handle that? Just throwing ideas at the wall.

 

3. (Maybe a pipe dream): Allow claiming 5/10/20 of an item with one click. I'm figuring this would still wreck havoc with the email system, as if the user clicked the button 5/10/20 times, but perhaps preparing the system to handle them in groups would make it possible?

 

Please don't take these suggestions as criticism of how the original issue was resolved. As a fellow software (and specifically SQL) developer, I get that sometimes "Can't you just make the computer do stuff faster?" isn't realistic. So, on that end, a heartfelt THANKS is in order for all involved! 

[Destruktor]

On 7/12/2022 at 2:39 PM, Bionic_Flea said:

Thanks for the work and the update.

Indeed, thank you to all who have assisted and continue to assist - your efforts are greatly appreciated.