Retrieving character list from server taking longer than it should

[Hopestar]

Hi, this has been a problem since I made the account but haven't felt the need to ask about it until I've seen other people with the same issue. I have a desktop that takes an average of 55 seconds to load the character list no matter if it's 1 or 10 characters. My laptop on the other hand loads within 5 seconds every time for the same account. The desktop is a core i3-2120 @ 3.3 GHz cpu and Geforce GTX 1050 Ti 4GB gpu and sits at ~15% processing while CoH is running; it is also directly connected to the modem. The laptop is about the same specs and wirelessly connected.

 

EDIT: I wanted to add, nothing changed by turning off my AV or firewall on the desktop.

[_NOPE_]

FYI, this is a known issue, and from what I've heard, someone might be looking into it in the near future... maybe.

[Impish Kat]

Yup... same thing here.

 

No matter what server, no matter how few characters I have (some servers I only have one), and even when the server load is "medium" with two green dots; the character list takes a long time to load.

 

Other than that, the game runs smoothly for me, even at peak hours and on weekends.

Technically, I suppose this should be in the Bug forums.  ;)

[_NOPE_]

Question: Do you two have Windows 7, Windows 10, or some other operating system?

 

I experienced this issue on Window 7 after a certain patch, and then I upgraded to a new PC with 10, and the problem went away. I'm wondering if it has something to do with that.

[Impish Kat]

Question: Do you two have Windows 7, Windows 10, or some other operating system?

 

I experienced this issue on Window 7 after a certain patch, and then I upgraded to a new PC with 10, and the problem went away. I'm wondering if it has something to do with that.

 

I do run Win7... but it seems odd, since I ran the game on Win7 on this same rig in the past and there was no problem then.

 

[Erydanus]

I think what made it hard to suss out is at original relaunch the server was legit hammered. Sometimes it took MANY MINUTES to get a character select screen.

 

At some point that got under control, and the menu loaded faster.

 

Then something changed and Win 7 machines now take about 50-60 seconds to load the character select menu. In fact, this past weekend I got on the test server to prep for Sunday's event. When I logged in, the character select screen came up IMMEDIATELY. I rolled a single character then logged out and restarted (to install the popmenus for test server that spawn IOs and set your level to 50). Guess what? In the middle of the night with less than 20 people on and only 1 character on my account, my Win 7 machine took about 55 seconds to load my single character.

[Hopestar]

So then windows 7 is the culprit? My desktop does indeed run Windows 7 while the laptop is Windows 10, so I suppose it's a client issue. I can't even begin to figure out what the problem would be between the 2 windows versions unless Windows 8 too takes 5 seconds like 10.

[_NOPE_]

So then windows 7 is the culprit? My desktop does indeed run Windows 7 while the laptop is Windows 10, so I suppose it's a client issue. I can't even begin to figure out what the problem would be between the 2 windows versions unless Windows 8 too takes 5 seconds like 10.

Yup, sounds like something in Windows 7 specifically, related to a patch that the HC team released after live. I wish I'd noted when I first started seeing the bug, then they could trace it down to which patch and look at the code that that patch contained.... que sera, sera.

[Impish Kat]

Could it be related to some vestige of the queue coding?

 

(I'm not a coder, I'm just tossing theory out)

 

[Erydanus]

Well I do think it kicked in around that time but they were doing so much it's hard to pin down exactly what point it started happening.

[WanderingAries]

Yeah, it feels like it's possibly been this way for a long time and perhaps there Could be an after affect from the que adjustments. I'm on Win7 as well, but IDK how it should be faster on Win10 when it Should be a server side issue.

[Erydanus]

I just logged in 2 minutes ago (9:17pm eastern) and the queue popped up 5 seconds after server selection! It's an anomaly!

 

Update: 9:49pm eastern I logged to char select and was able to select a different character in only 22 seconds!

[goblinite]

So then windows 7 is the culprit?

 

Windows 10 was designed to be lower overhead than Windows 7.  Basically given the exact same specs and a fresh install of both you should see a performance boost on Win 10 v 7.

That said, I doubt CoH is taking advantage of those performance tweaks.  I would look at your RAM and paging file sizes if you see Win 10 having a better performance than Win 7.

 

I don't want to sound like a pompous ass, even if I am one, but I think it's important to understand the basics of Windows memory management.

 

RAM is your PC's ability to store information without referring back to long-term storage like a HDD or SDD.  Compared to them RAM is very, very fast.  The more RAM you have the faster a system is (to a limit) because it doesn't have to 'Ask' the hard drive for information.

 

Back when there were hard caps on RAM, Windows 3 had a limit of 16MB (yes Mega not Giga, and you wondered why old video games looked bad?) and 3.1 maxed out a 256MB (1/4 of a GB).  Windows XP 32-bit (what CoH was written for) had a 3GB limit.

 

Because many programs needed to exceed these limits Windows has a paging file, also called virtual memory.  This is a section of the hard drive that is an open file that writes information to that is needs access to quickly, but not as quickly as what it needs in RAM.  As an analogy, RAM is your brain, a HDD is an encyclopedia (are those still a thing?) and the paging file is a note pad.  It can't hold as much as an encyclopedia, it isn't as convenient as your brain, but it does pretty good for a quick reference point.

 

Do you know why sometimes you need to reboot Windows?  It's is usually the paging file.  All day it is written to, erased, then rewritten.  Just like a sheet of paper, you can only do that some many times before the writing is indecipherable.  Restarting Windows essentially gets a fresh sheet of paper if we keep up the analogy.

 

While CoH only 'requires' 256MB of RAM that is because most of the information it needs is written to the paging file.  Let's look at Zone population and Draw Distance.  Draw Distance is how far out you can 'See'.  It's the limit where CoH puts the image files into RAM (more or less).  The rest of the zone is in the paging file.  As you move, stuff moves outside your draw distance (goes back to the paging file) and enters (goes to RAM).  For a great demonstration of this watch Boundary Break on Youtube.  There is a lot of talk about culling and loading to conserve resources.

 

As your RAM gets too full the game slows down.  Anyone remember Rikti Invasions that were like watching a slideshow?  That was because your RAM had to have all the characters loaded, all their costume parts, all the Rikti, all of their parts, all of the powers and power effects, all the buildings and so on.  It got full so it bled over into the paging file (slower) and if that was maxed out it had to pull from the HDD (slowest).  It was constantly moving things in and out, in and out, and I’m getting very warm here.

 

Now we get into the coding of CoH.  I will preface this by saying these are just educated guesses based on 20 years of IT work and 40 years of video game playing with some light programming work here and there.  I have no firsthand knowledge of the code, servers, database, or data.  Nor do I want it.

 

In general you can handle memory usage 2 ways: Do it by hand or let Windows do it.  For something simple you would want to do it by hand.  The character select screen is something simple.  You query the database with something like SELECT * FROM CHARACTERS WHERE USENAME='bubba_gump' AND SERVERNAME='Excelsior'.  This returns a list of characters, attributes, what have you.  The Engine takes this and makes the pretty display.

 

Well, this needs to be responsive, so you load it all into RAM so people can get to playing fast.  A database query and display is the fundamentals of most programming and is as simple as it gets.

 

BUT: The original CoH had a relatively small cap on characters per server (12-20, I forget, basically one page back in the day).  I don't even know what we have now, but it is a lot more than the past.  This then slows down the database query.  Before it had a build in limiter.  If I knew the limit was 20 I would say SELECT TOP 20 * FROM...  The query stops once I get 20 hits and then high fives the command interpreter and gets a beer.  But, we can't do that anymore since we have XXXXX number of characters per server.

 

Since it is likely every players' characters (at least the listing of them) are stored on the same table this means you know have to search the ENTIRE TABLE every time someone goes to character select.

 

Once the list is made you now have to load all the costume pieces and idle animations for each character into RAM (see above) for display.  Whether it be 20 or 2,000.  Now this doesn't happen all at once.  The query runs, returns the first row, the character is displayed, returns the second row, character is displayed it happens sequentially.  It even accounts for deleted characters by having skipped slots on the page.  We have now exceeded the intentions/limits of the original character select program by shoving more data into the RAM.

 

We also know that the servers have capacity issues.  Capacity is finite.  No matter how powerful a computer is or how fast the internet connection is there is a limited number of commands/processes/mathematical functions that can be done at once or so close to once it doesn't matter.  Once this limit is reached you have to wait.

 

The process is: query database, show character, query database, show character, etc.

When the server is taxed: wait for server to respond, query database, wait for server to respond, show character, wait for server to respond, query database, wait for server to respond, show character, etc.

When you have too many characters: unload character from RAM, query database, show character, unload character from RAM, etc.

When both happens: wait for server to respond, query database, wait for server to respond, show character, unload character from RAM, wait for server to respond, query database, wait for server to respond, show character, unload character from RAM, etc.

 

So what is the solution?

Not everyone can add more RAM.  There also maybe a RAM limit written into the game we don't know about, CoH was written when 3GB was the practical limit and there could be a reason they would write to not exceed this limit and make it a hard cap.

 

There is a practical limit to the size of the servers.  More capacity and higher speeds mean more money.  Same goes for the internet bandwidth.

 

They can put a limit on the number of characters.  This would reduce the load on both the servers and the RAM.  But then how many of us would bitch about how all our favorite characters can't be in the same SG on the same server?

 

The practical answer is this is just something we're going to have to learn to live with. 

 

[Erydanus]

I think you're going WAY off on a tangent talking about machine memory Goblinite. The machine is waiting for the info, it's not doing anything else. Fresh power-on of the machine with absolutely nothing else launched isn't having any effect on this. We know these delays started after a certain point in time when the HC team were messing with queuing. I saw it go from not happening with 0 characters on the test server to immediately happening after I rolled 1 character and restarted the client.

 

I think the issue is something in the server architecture or in how it's communicating with the Win 7 machines differently than Win 10 machines. I think the server patch notes specifically said this was being looked at? Yes, from the Justin Stress test announcement:

 

Why?

We want to put Justin through its paces in order to test a new database architecture - so please help us out and join in with the festivities!

 

Assuming all goes well, this should result in significantly increased stability and reduced load time for the character select screen.

(emphasis mine)

 

The delay suddenly seemed much reduced 2 days ago. Last night I was tired and didn't think to time how long it took to get to character select but I don't feel like I was waiting so long that I might as well get up and go to the bathroom or get a beverage. It wasn't horribly painful to swap between my crafting and active character, I'm guessing it was about a half minute again.  Honestly I'd like it to be more like 10-15 seconds wait tops but this is a marked improvement.

 

I just do find it terribly weird that the server would play nicer with Win 10 than Win 7.

[goblinite]

I just do find it terribly weird that the server would play nicer with Win 10 than Win 7.

 

I don't.

 

Windows 10 has more efficient memory usage than Windows 7 by design.

 

Windows 7 is also 10 years old v 5 years for Windows 10.  Both Bus and RAM speeds as well as HDD response times have improved since Windows 7 released.  Independent of what the server is doing this will improve performance on the client side and that's before CPU speeds, 7 v 14 v 28nm architecture.  Comparing performance on an old Win 7 machine v Win 10 is not a valid comparison.

 

Comparing Windows 7 v Windows 10 on the same system would be interesting but I got a dollar in my pocket that says everything else being identical Windows 10 still has better performance for CoH over Windows 7.

[Impish Kat]

If I ran the game on my Win7 6 years ago without issue, and now I run the game on the exact same Win7 rig and there is only one issue (in regards to performance).... where do you think the issue really lies?

 

It takes about 90 seconds to load the character screen. 

I have no other delayed loading or zoning issues.

I have no other play or performance issues.

 

Here's a question.... Has anyone asked if Leo's server had this issue?

[Cipher]

Hey all - if you're still experiencing this problem after the maintenance / server patch that we did on the 20th then please let me know (through Discord - Cipher#0002, through this thread, or by sending me a private message).

 

We're working on ways to diagnose this on the client side and can provide steps to help us gather information.

 

Thanks!

[goblinite]

If I ran the game on my Win7 6 years ago without issue, and now I run the game on the exact same Win7 rig and there is only one issue (in regards to performance).... where do you think the issue really lies?

 

It takes about 90 seconds to load the character screen. 

I have no other delayed loading or zoning issues.

I have no other play or performance issues.

 

Here's a question.... Has anyone asked if Leo's server had this issue?

 

Outside of the network/server related issues?

 

Click start> run> msconfig

 

You will have a Startup tab.  Many programs create a start up entry that eats up a little bit of RAM in order to reduce load time when the program is launched.  Check those.  You can also check your all programs>start up for the same.  If you are comfortable editing your registry ( a process that can FUBAR a system) start>run>regedit look under hkey_local_machine>Software>Microsoft>Windows>Current Version>Run for more programs.  You can also check under control panel>administrative tools>services.  Google Update and a few others will install themselves as services.  Disabling unneeded services can improve performance.

Start with disabling Google and Adobe updates, they are notorious resource hogs and see if you have a performance improvement.  I would also turn off anything Apple related then turn them on manually when I wanted to update my Iphone or use Itunes, Bonjour and Itunes don't play well with with Windows.

 

I would then Defrag my hard drive.  Type in Defrag in the start>search box, it should be the first entry.  If your RAM storage is exceeded Windows will write back to the page file.  If the hard drive is fragmented the page file may be fragmented as well which will slow down response as it has to 'search' for the bits and pieces of the file.  This will also improve costume rendering as the graphics for the parts are stored locally from what I have seen.

 

For your RAM issue CoH is having to compete with all the other programs running at the same time and you have an accumulated six more years of usage and the collected garbage software manufacturers dump on an end user.  You can also improve performance.

 

I think my giant ass dissertation on RAM mislead people.  It isn't entirely a RAM based issue.  Server capacity, bandwidth, how the client interacts with the database, these all factor in.  But they are outside of your control.  RAM and general PC management are under your control.  You can't tweak the servers for better performance, but you can tweak your PC.  An understanding of how data flows and what memory does can help this.

[Impish Kat]

So... I was hopping around servers today and I ran across something that might be pertinent here:

 

When I quit to character screen, as opposed to server or desktop, I got the delayed Char List screen. 

 

When I hit the "back" button to go back to the server screen, I got this...

(click to embiggen)

 

 

 

So... It seems that the game thinks I'm in queue for some reason... when there is no queue.

 

Edit to Add:

Just had hubby try the same thing on his rig, which is running Win10.  He was able to hit the back button before the char screen came up and did NOT get this notice.

 

The other difference between our setups is that I am using Island Rum as a launcher, and he is using Tequila.  We both have the launchers set to close upon the game opening.  The only thing I can think of, where this might be a factor, is if IR somehow missed downloading part of a patch somewhere along the line.

 

Edit again:

Just re-validated all game files using Tequila.  Made no difference.  Still getting queue message.

[Cipher]

So... I was hopping around servers today and I ran across something that might be pertinent here:

 

When I quit to character screen, as opposed to server or desktop, I got the delayed Char List screen. 

 

When I hit the "back" button to go back to the server screen, I got this...

(click to embiggen)

 

 

 

So... It seems that the game thinks I'm in queue for some reason... when there is no queue.

 

Edit to Add:

Just had hubby try the same thing on his rig, which is running Win10.  He was able to hit the back button before the char screen came up and did NOT get this notice.

 

The other difference between our setups is that I am using Island Rum as a launcher, and he is using Tequila.  We both have the launchers set to close upon the game opening.  The only thing I can think of, where this might be a factor, is if IR somehow missed downloading part of a patch somewhere along the line.

 

Edit again:

Just re-validated all game files using Tequila.  Made no difference.  Still getting queue message.

 

Well this is interesting, thanks!

 

I'll reach out on Discord today/tomorrow (I haven't slept yet :P - Monday the 24th) with some instructions to help gather information to figure this out.

[Erydanus]

I have exactly the same thing Kat does with the queue notice if I back out and I have for weeks. My delay time has been more like 30 seconds since the 20th but longer yesterday during peak times. I'd be happy to do any testing/debugging also Cipher.

 

goblinite, what you're not seeming to understand is that this is like when you load a web page and in the corner of your browser it flashes "waiting for aws/lightbeam/adserver du jour/etc". Our machines are *waiting for data* not having memory efficiency problems handling the data they're given. Why they're waiting for data is the question. Network protocols I'd believe, memory handling issues I don't.

 

[WanderingAries]

If I ran the game on my Win7 6 years ago without issue, and now I run the game on the exact same Win7 rig and there is only one issue (in regards to performance).... where do you think the issue really lies?

 

It takes about 90 seconds to load the character screen. 

I have no other delayed loading or zoning issues.

I have no other play or performance issues.

 

Here's a question.... Has anyone asked if Leo's server had this issue?

 

This, I'm using the Exact same hardware with roughly the same filepaths and ever since (I think) the introduction of the Que system it has taken ages to get onto a new alt. I could probably /afk-Bio and it'd be about as long to load. I'll have to check my Win10 Atom machine to test, but I'm pretty sure it has something to do with how the que is setup as the message you get when clicking "go back to shard" arrow triggers "do you want to leave the que?" message.

[WanderingAries]

Just tested on my Win7 x64 Machine: (3770k @ 3.5Ghz, 32Gb RAM, Samsung Evo 860 SSD / OS, Apps only)

 

Average US server load is roughly 1550, EU ~400

Server->Character List

US Servers: ~60s  / Ranging from 0-40+ alts each

EU Server: ~17s    / 0 alts

 

Game->Character List

Torch: ~60s

 

[WanderingAries]

Outside of the network/server related issues?

 

You're general troubleshooting and procedures are exemplary, but I'm pretty sure Kat and my machines are both running optimally.

[WanderingAries]

Just tested on my Win10 x86 Machine: (Atom 1.33Mhz, 1Gb RAM, Flash storage)

 

Average US server load is roughly 1550, EU ~400

Server->Character List

US Servers: ~2-7s  / Ranging from 0-40+ alts each

EU Server: ~2s    / 0 alts

 

Holy difference batman!

 

Besides the basic specs, the only difference is that the character file doesn't include the spacing I've done on my normal machine (some pages have character gaps), but this has been going on longer than I've been organizing the alt list.