AH Lag

[Ukase]

I place this here not because I'm not familiar with it - but because some folks may not be familiar with how the AH works (or doesn't work) from time to time. 

Most of us more experienced marketers have seen it. We just adjust, shrug and move on. 

Some of you may have noticed that in the past couple of days, rare salvage wasn't getting bought at the usual 500k price. I mean, not even my overnight bids at 500K were filled. 

This pic was taken just now. It's not just an instant in time. It took me about 30 seconds to pull up the snip & sketch and to stretch out the AH UI so you could all see how many pieces were listed. 

Look at the bids - 34938 bidding for rare salvage. 
Do you honestly think all of those bids are for only 5 inf or less? I do not. The server that's running this is moving fairly slowly. This is why some items that you'd think would sell for 4M easy aren't selling (at least, not that you'd notice). My guess is these sales were "instant" as far as the system goes - but it's taking my system time to get the information from the server. 

Mind you - in the minutes it's taken for me to type this out - the items have sold, so don't try to bargain shop! 
Well, go ahead and try, but you won't get mine. 

But this is why, I believe, that your reasonable bids don't go through when you make them. If influence is important for you, you may be well served to wait about 90 seconds after each bid to see if it actually worked. Items on weekend nights (anecdotally for me) do not seem to sell or get bought right away. 

[Yomo Kimyata]

Well, two things. 

 

First, to your point, there *is* certainly some processing lag going on.  I just conducted two experiments (and I've done this before, but not all that recently):

 

1.  I had 1,000 yellow uncommon salvage on an alt in 100 stacks of 10.  I posted all of them at 1 (after making sure that there were a few thousand bids outstanding).  It took me 1:09 to post them all, and the last one transacted at 3:01.  Most of those trades went to a really low bid, and I'm guessing that it was between 100-150 total transactions (since the system considers a block of 10 one transaction, and I'm guessing that that cut down on the processing time.  Regardless, 1,000 items took just over 3 minutes to transact and allocate when there were guaranteed counterparties.

 

2.  I then bought 1,000 white common salvage, in 100 stacks of 10.  To guarantee that I would buy them, I bid 10,000 inf (which is the seeded price, so I know it is a guaranteed transaction).  It took me 0:23 to make my bids, and I bought the last one at 4:12.  I am guessing that this was a slower transaction, because there were probably a lot of singles for sale rather than blocks of 10.

 

We all know that the programming of the /AH is considered to be kludgy at best.  Or so we are told!

 

Second, keep in mind that I can really very easily imagine a scenario where in the 30 seconds it took for you to post then screenshot that you were not the lowest outstanding offer.  Based on anecdotal evidence from these forums, most people looking to dump either post at 1 inf (guaranteed transaction if there is any bid at all) or 5 inf (which guarantees you will not transact at a price below your minimum posting fee.  If someone had dumped a few hundred salvage at a lower price just before you posted at 6, you wouldn't even be first in the queue.  There are plenty of other things that could screw things up as well.  Well maybe not plenty, but a few.

 

I've bought and sold a lot of items on the AH over the last few years.  In my experience, any delay in transaction time in the /AH has been measured in seconds for me unless I am transacting large numbers of items, and then I've been chalking it up to processing time.  

 

my two inf

[Ura Hero]

Going to have to side with @Ukase on this one.  I've had it take up to 30 seconds to process a bid when I know there are available pieces at the target price.  I haven't looked at the market code, but based on the way that some of the data is stored, I am betting it is a hot mess.  Most things like the auction house are done on some sort of tick cycle so part of it probably is due to how close you are to the next tick along with how many are in the queue and how many get processed before the tick cycle terminates and moves on (assuming that it does, which isn't outside the realm of possibility).  

 

I don't know what the devs are doing towards the future, but hopefully they are moving away from blob and flat file storage and redoing the underlying code to use SQL or some other database that is more efficient.

[Ukase]

My anecdotal data supports the theory that the AH just takes time to process during certain times. 
Over 30K bids strongly suggests that unless they're all for between 1 and 5 inf, that mine should have sold instantly, or at worst, within a second or two of the figurative hamster turning the wheel within the server. There really is no other reasonable explanation. 

I cannot accept that there would be so many bids for such a tiny amount of influence, when everyone knows that rare salvage generally sells for 500k, on average. 
It is possible, of course. Just not very likely. Sure, I can see 100 folks selling 20 pieces or so of rare salvage for the tiny price, but given as my alt account had standing bids for them at 500k at the same time....nah, not buying that my pieces weren't first in queue in this example. Possible? sure. Likely? Nope. 

 

[Yomo Kimyata]

6 hours ago, Ukase said:

My anecdotal data supports the theory that the AH just takes time to process during certain times. 
Over 30K bids strongly suggests that unless they're all for between 1 and 5 inf, that mine should have sold instantly, or at worst, within a second or two of the figurative hamster turning the wheel within the server. There really is no other reasonable explanation. 
 

 

Well, a very reasonable explanation could be that someone else happened to dump a bunch of rares at a lower price, and that was transacting because it wasn't your turn in the queue.  It doesn't matter if your offer is lower than any bids until your offer is the *lowest* offer.  Until then, you just need to wait your turn while the hamster wheel processes and it gets to you.

 

I've been thinking of the /AH as a combination of theory and execution.  The theory is how the queue system works, how it matches up buyers and sellers.  It's a model.  It's not very complex (but probably more complex than people think), but in a Platonic model, everything would transact instantly and exactly.  Then there is the execution, which is going to deal with how it was programmed, and how often the batches are processed, and how many people are in the queue, or how hot it is where the server is located, or any of a number of other factors.  

 

So yes, it does take time to process trades, even if you are the lowest outstanding offer or the highest outstanding bid.  But that's not always the sole reason for it.  I'd like to do a writeup on how I see the queue system working, but there is what I think of as a fundamental problem that I don't want to highlight.  It can be exploited, but it's fundamentally not a universal or profitable exploit.  I've sent in bug reports before, and been told "working as intended" but I don't think that printing a guide on how to gum up the system is good for anyone.

[Yomo Kimyata]

Here's some more anecdotal evidence:  with two of the servers down, I'm noticing a lot more lag on the AH than usual.  In particular, when I switch from one character to another and open /AH, I am getting the previous characters /AH.  This has happened many times before in regular play, but today it has been three for three within a few minutes.  (My workaround is to put a new item in the /AH, and that usually sets it to the current character).

 

Another thing is when I click on an IO in inventory in order to send it to the /AH, there's a more noticeable delay.  This also happens in regular play, but not often.

 

It makes intuitive sense to me that more strain on the server might slow things down, but I'm a rudimentary programmer at best, so I'll listen to the experts out there.

[Aeroprism]

On 11/12/2021 at 7:51 PM, Ukase said:

It took me about 30 seconds to pull up the snip & sketch and to stretch out the AH UI so you could all see how many pieces were listed. 
 

 

Windows+Shift+S. 

Select (frame) what you want.

It is now in your clipboard.

CTRL-V it in a post.

 

You're welcome.

 

(Since the Market forum often mean using screenshots, I figured it would be useful to at least a few of you guys)

 

Happy marketing.

[Yomo Kimyata]

good to know:

 

[iBot]

The AH is a big data issue.  First remember that it is a large scale insertion sort most likely pulled from a type of merge sort or quick sort.  If they implemented it as a bubble sort then good grief but highly unlikely.  It is also a highly fragmented system of data.  Think poorly maintained HD from the 1990s when everyone one in tech would say "speed up your HD ... run disk defrag once in a while."  Something we would need to know is what kicks off a single process pass.  Is it placing an item for sale or is it when you try to buy something?  Most likely it is a form of transaction lists that are merged into completed transactions for transfer.

 

Things that happen fast are probably when you place an item at the edges of those lists.  Anything in the middle requires more processing time to find.  There is also the issue of mismatched transactions.  For example someone bids "I want 4 of these at 100 inf" but the market has "2 at 75, 1 at 80, and 65 at 101".  Well you get 3 and then have to wait for it to either find another list for under 100 or for someone else to sell under 100 which fills your final piece.  Remember, the items are bucketed not the prices so the AH still has to keep all those individual prices stored.  So it is not as simple as match highest bid with lowest price.  You have to take into account the quantities.  When listing a stack of 10 for say 150 inf it is possible for that stack to sell at uneven prices.  So the AH has to maintain the data listing each of those fragmented transaction.  This requires more processing to filter those lists.

 

It is also probably a low priority thread.  It also runs the AH for all servers so it makes sense that if one of the other shards is down the AH would waste time trying to communicate with that shard which results in a delay as the timeout ticks.  The server doesn't instantly know that the transaction is going to fail because the shard is down.  That time out is normally in seconds so if there are a lot of transactions waiting to be fulfilled the AH could waste a lot of time just waiting before moving on to the next transaction.

 

And of course as always I could be 100% wrong since I've never laid eyes on the code.  This is just what my opinion is based on my limited knowledge of these sorts of data systems.

[Andreah]

I recall, here one of the HC developers posted some info about how it works, and iirc, (big iirc) it was a tree-implementation of a priority queue.

[wjrasmussen]

On 11/12/2021 at 8:23 PM, Yomo Kimyata said:

Well, two things. 

 

First, to your point, there *is* certainly some processing lag going on.  I just conducted two experiments (and I've done this before, but not all that recently):

 

1.  I had 1,000 yellow uncommon salvage on an alt in 100 stacks of 10.  I posted all of them at 1 (after making sure that there were a few thousand bids outstanding).  It took me 1:09 to post them all, and the last one transacted at 3:01.  Most of those trades went to a really low bid, and I'm guessing that it was between 100-150 total transactions (since the system considers a block of 10 one transaction, and I'm guessing that that cut down on the processing time.  Regardless, 1,000 items took just over 3 minutes to transact and allocate when there were guaranteed counterparties.

 

2.  I then bought 1,000 white common salvage, in 100 stacks of 10.  To guarantee that I would buy them, I bid 10,000 inf (which is the seeded price, so I know it is a guaranteed transaction).  It took me 0:23 to make my bids, and I bought the last one at 4:12.  I am guessing that this was a slower transaction, because there were probably a lot of singles for sale rather than blocks of 10.

 

We all know that the programming of the /AH is considered to be kludgy at best.  Or so we are told!

 

Second, keep in mind that I can really very easily imagine a scenario where in the 30 seconds it took for you to post then screenshot that you were not the lowest outstanding offer.  Based on anecdotal evidence from these forums, most people looking to dump either post at 1 inf (guaranteed transaction if there is any bid at all) or 5 inf (which guarantees you will not transact at a price below your minimum posting fee.  If someone had dumped a few hundred salvage at a lower price just before you posted at 6, you wouldn't even be first in the queue.  There are plenty of other things that could screw things up as well.  Well maybe not plenty, but a few.

 

I've bought and sold a lot of items on the AH over the last few years.  In my experience, any delay in transaction time in the /AH has been measured in seconds for me unless I am transacting large numbers of items, and then I've been chalking it up to processing time.  

 

my two inf

Is there a list of seeded prices?  I don't want to overpay.

[Andreah]

19 hours ago, Andreah said:

I recall, here one of the HC developers posted some info about how it works, and iirc, (big iirc) it was a tree-implementation of a priority queue.

I found it, it was not a developer, but UberGuy who inspected some of the publicly available code.

 

 

Edited November 16, 2021 by Andreah

[Yomo Kimyata]

1 hour ago, wjrasmussen said:

Is there a list of seeded prices?  I don't want to overpay.

 

@Jimmy (I think) posted a while ago that common salvage (10k), uncommon salvage (100k), rare salvage (1mm), and Brainstorm Ideas (I don't remember the price) were seeded, originally at 10mm in quantity apiece.  Also, we know that Hero Packs have been seeded at 10mm (both kinds), and it appears that when/if they change prices on Winter Packs they reseed with a refreshed quantity of 10mm.  That's all that leaps to mind.

[wjrasmussen]

19 minutes ago, Yomo Kimyata said:

 

@Jimmy (I think) posted a while ago that common salvage (10k), uncommon salvage (100k), rare salvage (1mm), and Brainstorm Ideas (I don't remember the price) were seeded, originally at 10mm in quantity apiece.  Also, we know that Hero Packs have been seeded at 10mm (both kinds), and it appears that when/if they change prices on Winter Packs they reseed with a refreshed quantity of 10mm.  That's all that leaps to mind.

I knew about the packs, but didn't know about the other parts.  Thanks for this!!!