Introducing the Diagnostic Client

[Number Six]

Hello heroes, villains, and everything in between!

 

In the ever-ongoing pursuit of perfection, we're rolling out a new opt-in program to help track down some of the more elusive client-side bugs. The crash reporting system has helped to track down and fix a lot of nasty bugs, including quite a few that date back almost to the release of the game. With the low-hanging fruit gone, now it's time to focus more on the ones that aren't so simple to catch.

 

By running the diagnostic client, you can help us find and squash those really slippery bugs that are difficult to pin down.

 

What is the diagnostic client?

 

It's a special build of the game client that has some extra debugging features enabled. A few of the highlights:

• Extra-stringent memory allocator checks enabled, this will immediately crash if certain types of bugs are detected, such as double free or writing past the end of a buffer.
• Some previously dev-mode-only assertions enabled; they will cause an immediate crash if something that's supposed to be impossible happens, instead of a probable crash down the road somewhere.
• Most release mode optimizations are enabled, but a few that can cause headaches when analyzing crash dumps have been disabled.
• Has the ability to flag specific sections of code to retain extra data that would be normally optimized out, but on an as-needed basis based on other crash reports we're looking into.

 

That's great, it makes my game crash more often, why on earth would I want to do that?!

 

To help make the game better! The things that make it crash in diagnostic mode are things that would be very difficult to find the cause for in the regular release build. There are a few of these; every release we get a small number of crash reports that happen in consistent places, but the state in the memory dump makes no sense and should be impossible. The most likely cause is that corruption is being caused somewhere else in the code, but in rare enough cases that it doesn't happen in a way we can reproduce. This build will catch those earlier and help us narrow the root cause to eliminate them.

 

Also, crash reports sent in by the diagnostic client get a special flag in our bug tracker so we know they have richer data. That means they get priority and are looked at first since they're more likely to lead to a solution.

 

Is there any downside to running the diagnostic client?

 

It is slightly more likely to crash early when there's a problem. In most cases, these situations would result in a crash anyway, just later on. But there are some bugs that might never cause a crash in the release client because they corrupt something that's otherwise unused.

 

Because of the extra sanity checks and optimization differences, expect about a 5-8% performance loss on average compared to the normal release client.

 

How do I opt-in?

 

The diagnostic client is available in the Homecoming Launcher. It is not available in manifest-based launchers. Also, unlike the release client, it is 64-bit only and does not support legacy operating systems.

 

To run it, you just need to go into the launcher settings and unhide it so that it shows up in the list:

 

 

 

Then, simply "install" the diagnostic client. It uses the same data files as the standard Homecoming client, so it is a very fast install and only needs to download 2 additional files.

 

 

When you're ready to play, just run the diagnostic client instead of the regular one! It shares all settings with the live client, so your characters, graphics settings, bind files, and so on will all be there.

 

 

If you do encounter a crash when running the diagnostic client, please pick the memory dump option if possible!

[starro]

 My two favorite homecoming COH improvements:  64 bit client and the Homecoming client.

When Homecoming started i was crashing easily 2-3 times an hour.

 

I do have a semi persistent error logging out, selecting another character and logging back in and hope this provides info to squash that bug.

Edited April 30, 2023 by starro
clarification

[TheZag]

Perhaps a link to this topic in the homcoming launcher news to increase visibility would be appropriate.  There are probably several people that would participate but dont regularly visit the forum and they all see the launcher.

 

[Replacement]

Excellent that this is a separate launch option instead of the whole new launcher exe I feared when I started reading.

It means if you're trying to get something done and have crashed a few times (and thus sent data dumps), you could still launch normally to just get that TF win!

 

12 hours ago, Number Six said:

Expect about a 5-8% performance loss on average

 

Is this at all times, like expect slideshow events to get choppy 5-8% sooner?

 

Or more limited to things like initial client launch, map loads?

 

[TheZag]

8 hours ago, Replacement said:

 

Is this at all times, like expect slideshow events to get choppy 5-8% sooner?

 

Or more limited to things like initial client launch, map loads?

 

 

ive been running it all day and i havent noticed any dip in performance.  i had 20-21 frames during the hami raid today when i usually have 16-17.  still close enough to be due to character composition but there was no problems for me while using it.

 

[Greycat]

So, question.

 

At what point do these get sent? Does COH itself have to send an error or... ?

 

Asking because I'm currently trying to log in a character (granted, dual boxing ATM,) I can select the character, it starts loading, gets to - oh, 99% and just hangs.  I have to end it in task manager.

Did it twice with the regular client, then the diagnostic client because - well, hanging. Windows caught that there was a "faulting application," but other than the Windows "this program encountered an error and needs to close," I didn't see anything else - *would* that have sent in a log? Or is there somewhere it saves a "this is happening..." that we can manually attach if needed?

[Number Six]

Bah, bit of an issue. Sharing the data package seemed like a good idea to not have to duplicate all the data, but causes an issue when updating. If you have both Homecoming and Homecoming (Diag) installed, you won't be able to update either one of them. When you try to update one, the launcher (correctly) detects that the data version is different than the one required by the other profile and refuses to update because of the conflict.

 

I'll work on a solution but for now you'll need to uninstall the Diag profile in order to update the game, then reinstall it.

[Greycat]

So that's what's going on. I was hopping back here to check.

[Bionic_Flea]

But Six, why do you break things in order to fix them?

[Number Six]

I pushed out a fix that changes the dependency type in the diag package to 'newer' to break the cycle, but if you already have it installed you'll still need to remove it temporarily.

 

When the next update comes along it should work better. It may still show up as failed if the diag package tries to update first (going to have autoupdate default to false for new installs from now on), but once the main HC client is updated it'll succeed if you retry.

[KaizenSoze]

3 minutes ago, Number Six said:

I pushed out a fix that changes the dependency type in the diag package to 'newer' to break the cycle, but if you already have it installed you'll still need to remove it temporarily.

 

When the next update comes along it should work better. It may still show up as failed if the diag package tries to update first (going to have autoupdate default to false for new installs from now on), but once the main HC client is updated it'll succeed if you retry.

I uninstalled the diag client, then re-updated the regular client and the error went away.

 

Edited May 2, 2023 by KaizenSoze

[Number Six]

1 minute ago, KaizenSoze said:

What is the correct way to remove the diagnostic client? Uninstall?

 

Yes, make sure it's not hidden, click the '...' on the "Homecoming (Diag)" tile in the launcher, and select Uninstall from the menu.

[Number Six]

On 5/1/2023 at 2:49 AM, Greycat said:

At what point do these get sent? Does COH itself have to send an error or... ?

 

It only gets sent if the game crashes or triggers an assert, at which point it'll prompt you about submitting a report, or preferably a memory dump.

 

It doesn't have any kind of extra telemetry or data reporting or anything like that, so it unfortunately won't help with hangs or network issues like losing connection to mapserver. That might be something I add to it later, but if so it'll be opt-in.

[Greycat]

7 minutes ago, Number Six said:

 

It only gets sent if the game crashes or triggers an assert, at which point it'll prompt you about submitting a report, or preferably a memory dump.

 

It doesn't have any kind of extra telemetry or data reporting or anything like that, so it unfortunately won't help with hangs or network issues like losing connection to mapserver. That might be something I add to it later, but if so it'll be opt-in.

 

OK, so the COH-specific crash box (that as I recall has a place to type in what was going on / what you were doing  - been a bit) = yes, it's sent.

Windows "The program has stopped responding" - nothing saved, nothing sent, nothing to send.

Hangs like I described - nothing saved, nothing sent, nothing to send.

 

I was half thinking/wondering if it would be set (or could... pretty much by running the client) to record whatever's happening to a log, at least, to send in if it didn't get to that point. I don't know how much useful information the rather short "There was an application crash" Windows sometimes records would have by comparison, thought it might have something to go along with it. (Ended up logging into a *different* character, logging them out, then could log the first in, just as a followup to what led to that question.)

 

Thanks!

[BlackSpectre]

What about clients running on Macs? Good idea to use diagnotics client, or no?

[Injunuity]

any chance of getting a list of "hey, we need a diagnostic run  of **THIS AREA** of the game, like an MOTD ingame message or forum post? or on the launcher itself even

[Greycat]

On 7/21/2023 at 11:20 AM, Injunuity said:

any chance of getting a list of "hey, we need a diagnostic run  of **THIS AREA** of the game, like an MOTD ingame message or forum post? or on the launcher itself even

 

They mentioned this was client side, so my *guess* is that unless (say) "Loading the second Levantra mission (saving longbow squad) is causing frequent crashes" comes up, it'll mostly be for capturing data when someone's going "My game's doing weird things nobody else is seeing" or "I always crash here."

[Luminara]

I've used the diagnostic client exclusively for the last four months.  I have to say, I really like it.

 

First reason is game focus.  Even back on the original servers, this... irritation was present.  I don't know if it's the game itself, or something I've done on my OS, but when I start the game, it's only given focus about a third of the time.  What I mean by that is, I'll load the game, get to the login screen, press a key and some other window I have open responds.  My browser, or the task manager, or Mids, or Notepad, something not Co*.  The desktop manager just doesn't reliably switch focus to the game.  It doesn't happen in any other game, not even older games like SS2, or, hell, the first System Shock.  It's only ever happened with Co*.

 

When I use the diagnostic client, it always takes me straight to the login screen.  I don't have to sit here with a dumbfounded expression, trying to figure out why I'm hearing odd sounds and my password isn't entering.  I don't have to Alt+Tab through windows until I get to Co*.  The diagnostic client resolved the focus issue, and if I had nothing else to say about it, I'd praise it for that reason alone.

 

Second, I'm one of those people who disassembles machines just to see what's inside and how everything works together.  Programs are, as I see it, just another type of machine, made of alphanumeric characters, or, more simply, on/off states, instead of materials like plastic or metal, and the diagnostic client console window is like a tool that allowed me to look at the various bits and bobs inside the game's machine and see how it works.  I haven't had an opportunity like this since in forty years and I'm enjoying it immensely.  This has become one of my reasons to keep playing, just so I can load up the game, run some missions, and Alt+Tab to the console and scroll through, see the sprockets and chains and linkages all doing their things.

 

I can't seem to trigger a crash, like I did with the contact window bug, so I have no game-breaking problems to report.  I can't even say that the client itself is causing any problems, it's fine.  Sorry.  I have encountered some minor anomalies that I'll drop here, though.

 

Warning: Message from server, 57 (SERVER_CONTACT_STATUS), was sent before we are receiving entity updates!
Warning: Message from server, 60 (SERVER_TASK_STATUS), was sent before we are receiving entity updates!
 

These two are always present.  It doesn't prevent anything from loading, nor cause any crashes, but it does suggest that this could be optimized.

 

Sky builder can't find 'dark_astoria_sun__' in the object_libray.
anim: Bad geometry request Model: 'dark_astoria_sun__'

Sky builder can't find 'dark_astoria_sunglow__' in the object_libray.
anim: Bad geometry request Model: 'dark_astoria_sunglow__'

Sky builder can't find 'sunOffsetforSpaceglow__' in the object_libray.
anim: Bad geometry request Model: 'sunOffsetforSpaceglow__'
 

These three are always popping up, as well.  It might have something to do with the sky in my base (my first guess), but even if it is that, it'd be a good idea to track those down and get them properly sorted.

 

Really, though, I just wanted to say how much I enjoy having the diagnostic client.  I've been in nerd heaven for months.

[Luminara]

Here are a couple of console messages I haven't noticed before.

 

Switching from state SIMPLE to state DEAD.
client said yes
Switching from state DEAD to state RESURRECT.
Switching from state RESURRECT to state SIMPLE.

 

It occurred to me to check the console after I completed the Alignment Mission, An Unstable Trigger (vigilante).  Something in this mission always causes buff icons to disappear at least once (right after Maelstrom is defeated the first time, and it sometimes occurs the second time you defeat him), and I wondered if the console would reflect the unusual activity.  Can't say with absolute certainty that those lines were triggered by the buff icon disappearance, but... well, it might help, it might not.

The next mission I started was the Morality Mission, The End Justifies the Means.  And this popped up in the console.

 

ERRORLOG FILEERROR: MAPS/MISSIONS/UNIQUE/GR_MORALITY_MISSIONS/GR_MORALITYMISSION_ROGUE_6-7_01/GR_MORALITYMISSION_ROGUE_6-7_01.TXT
File: MAPS/MISSIONS/UNIQUE/GR_MORALITY_MISSIONS/GR_MORALITYMISSION_ROGUE_6-7_01/GR_MORALITYMISSION_ROGUE_6-7_01.TXT
No loading screen match found!

 

And this...

 

Sky builder can't find 'moonglow__' in the object_libray.
anim: Bad geometry request Model: 'moonglow__'

 

which, I'm positing, is because the end of the mission map is outdoors.

[lumberjack jill]

I just installed this and ran where I typically have mapserver issues in DFB.  It's always in DFB, DiB, and the cave mission of the ITF.  I'll try to get another crash in DiB for y'all to see. Thank you!

[Koopak]

I've been using this client since it was released and iv noticed a tendency to have rendering stop but the game not crash. By this I mean the screen freezes but sound continues to play and I continue to be able to interact with the game via macros but a new frame is never drawn. Is this something that this debug client is able to catch and report or should I get on compiling details for a proper bug report?

[Mr_Enigma]

On 9/27/2023 at 8:16 PM, Koopak said:

I've been using this client since it was released and iv noticed a tendency to have rendering stop but the game not crash. By this I mean the screen freezes but sound continues to play and I continue to be able to interact with the game via macros but a new frame is never drawn. Is this something that this debug client is able to catch and report or should I get on compiling details for a proper bug report?

So I've been getting the same issue as Koopak, although for me the frozen screen typically only lasts a few seconds, then unfreezes, or it just eventually crashes to desktop.  Other times I will just immediately CTD with no warning. 

 

I've been getting random crashes here and there on the regular client for a while, which is why I've usually opted for the diagnostic client of late.  What's frustrating is that I cannot remember the last time I was able to submit a crash report from the diagnostic client, as the crash reporter window never opens for me.  Which seems to kind of defeat the entire purpose of the diagnostic client (unless there is other data being collected independent of the crash reporter). 

 

I seemed to get a crash reporter pretty reliably on the regular client, so I may revert to that.  At least I can submit SOME crash data that way.  Is there some option to disable the crash reporter that I may have selected at some point and just forgot?

[Jacke]

The Homecoming (Diag) client on Page 7 is crashing when I tried to connect to Shard Everlasting and load my characters.  Sent in 2 memory dump crash reports.  Regular client appears to be working fine.

[lemming]

Been running the diag client without issue today, but there does seem to be some people having issues.

[Jacke]

After the Hotfix earlier tonight, tried the Homecoming (Diag) client again, trying to load Everlasting Shard, second time Indomitable Shard, which has only ever had 1 anchor toon.  Both times client crashed out immediately when told to load a Shard, similar to the 2 crashes before the Hotfix; sent 2 more memory dump crash reports.

 

Regular Homecoming client still working okay.

 

4 total memory dump crash reports on the Diag client, @Number Six, 2 before the Hotfix, 2 after.

Edited February 21, 2024 by Jacke