Cyclops Log Parsing Tool 1.2

[KaizenSoze]

First, I would like to thank @Carnifax for providing me the source code from their tool. This greatly sped up the initial development. Please, send any flowers to them first.

 

What is this?

A new tool for parsing City of Heroes log files for analysis.

 

Why should I use this instead of Carnifax's?

If you like Carnifax's, please keep using it. This tool runs on your machine instead of Carnifax's site and has some different features.

 

Important upgrade notes:

• I recommend installing the application in a new directory
• 1.1 data is not compatible with 1.2 data, so don't copy the output directory over from the previous version.
  • You will need to re-process your logs due to many database and structure changes
    • Use the directory parse option in the UI. Parsing takes no more than 3 seconds per file.

 

New features for 1.2:

• Realized too late that I forget to update the version number on the HTML pages. It still says 1.1, will fix next release.
• I'll update the screen shots in a bit.
• Improved data quality
• New features
  • Merge Rows Best Guess
    • Uses a string algorithm to merge rows with similar but not identical names.
      • Mastermind pets should be easier to group together for instance.
  • Overkill percentage
    • Help show much over/under kill powers are doing.
    • Attack Summary By Power and Damage Dealt By Power or Mob tables only.
    • New column, Overkill percentage.
    • New drop down, select minion level and the overkill column will adjust.
  • Data geeky stuff
    • Simulated hits, helps with damage auras and pseudo pets that don't report hits consistently
      • See next post down for more details.
    • Auto pseudo pet merging
      • Merge a small number of pseudo pets with their activation to provide better data quality.
      • See next post down for more details.

 

Features (screenshots below):

• Index page that lists all the logs you have parsed
• Total rewards, experience, mobs defeated
• General combat summary
• Attack summary by power
• DPS in intervals. See README for more details.
• Damage dealt by type
• Damage taken by type
• Damage taken by mob
• Damage taken by mob power
• Damage dealt by power or powers that damage a type of mob

 

How do I get started?

• README
• Download the zip from here.
  • File: cyclops-1.2-beta.zip
  • It's just a bit too big to attach to the post
• Unzip the file where every you want
• Window users can use cyclops.bat to start the application
• Or open a command line in the new directory and type: cyclops.exe
• Browse: http://127.1.1.1:11227

 

Future work:

• Bugs fixes, lots of bug fixes
• Depends on feedback I get
• Charts
• Timeline of major events, dps spikes, damage taken spikes, etc...
• Exporting
• Statistical improvements
• Log comparisons

 

Geek stuff:

• Github page
• Tech used
  • Rust language - first project so code quality is meh, but improving
  • Sqlilte database
  • Actix web server
  • Tera templating
  • HTMX for the UI

 

Screenshots:

Index page:

Summary:

 

Attack Summary By Power:

 

 

Intervals, Damage dealt, taken by type and mob:

Damage taken by mob and mob power:

 

Damage dealt by power or mob:

Edited September 9 by KaizenSoze

[KaizenSoze]

Detailed discussion of features for those that care and general data geeking.

 

• Pseudo pets
  • There is often a disconnect between the power name that activates the power and the name of pseudo pet that does the damage. Also, activation and pets are logged with different formats.
    • Examples:
      • Chain Induction spawns up to four pseudo pets called Chain Induction Jump 1-4.
    • Causes a lot of manual merging in the powers table to get valid accuracy and damage per activation or hit.
    • Solution
      • New file ./resources/pseudo_pets.csv
        • Format: Activating power, Pseudo Pet name, Merged name.
      • Changes both the name of the activating power and the damage pseudo pet to a common name.
      • You can edit the file yourself, but you have to restart each time.
        • Use %% to represent wildcards.
      • Post to this thread if you need help.
• Damage Auras, which also applies to other some other powers that do an status effect and damage like Frozen Aura
  • To reduce the number of messages sent from the server to the client damage aura have an annoying behavior if you are trying to parse logs.
    • Only the initial hit roll is logged or if there is a miss, then the next hit is logged.
    • All misses are logged.
    • This causes damage auras to massively under report their hits causing accuracy to look horrible.
    • Also, without accurate hits, damage per hit, is wrong.
  • Solution
    • New file ./resources/sim_hit_powers.csv
      • Format: Power name and that primary damage type
        • Damage is there to prevent double counting for powers that do two types of damage.
      • A hit row is added for each damage row record to give a better representation of the accurate and damage per hit.
      • You can edit the file yourself, but you have to restart each time.
        • Use %% to represent wildcards.
      • Post to this thread if you need help.

 

 

 

Edited September 9 by KaizenSoze

[Excraft]

Nice work on this!  I can see it being very helpful to those interested in maxing out a build and analyzing their DPS.

 

With that said, I'd like to ask a couple of questions about this too.  Please know I am not asking these with any sort of malice toward you, accusing you of anything or implying anything about you personally.  My questions are in no way a condemnation of you personally or the work you have put into this. 

 

With that out of the way, my questions are;  does this record and track other players on your team?  If so, how can they opt out being recorded?

 

The reason I ask is, I've seen these kinds of things being used to discriminate against other players and shame them in other games, most recently in STO.  I'd really hate to see a useful tool such as this be used for that purpose here.

 

Thanks.

[KaizenSoze]

2 minutes ago, Excraft said:

Nice work on this!  I can see it being very helpful to those interested in maxing out a build and analyzing their DPS.

 

With that said, I'd like to ask a couple of questions about this too.  Please know I am not asking these with any sort of malice toward you, accusing you of anything or implying anything about you personally.  My questions are in no way a condemnation of you personally or the work you have put into this. 

 

With that out of the way, my questions are;  does this record and track other players on your team?  If so, how can they opt out being recorded?

 

The reason I ask is, I've seen these kinds of things being used to discriminate against other players and shame them in other games, most recently in STO.  I'd really hate to see a useful tool such as this be used for that purpose here.

 

Thanks.

Good question. You *cannot* see other players *damage* stats, damage dealt, taken, or hit rolls. Some buffs might appear, especially autohit AOE buffs.

You will see if they buff or heal you, defeat a mob, or are defeated.

 

In general buff/heal/debuff logging are a hot mess and not tracked in the tool at the moment. I will keep your concern in mind. I have zero interest in creating something that is used to exclude players.

[KaizenSoze]

2 hours ago, Excraft said:

Nice work on this!  I can see it being very helpful to those interested in maxing out a build and analyzing their DPS.

 

With that said, I'd like to ask a couple of questions about this too.  Please know I am not asking these with any sort of malice toward you, accusing you of anything or implying anything about you personally.  My questions are in no way a condemnation of you personally or the work you have put into this. 

 

With that out of the way, my questions are;  does this record and track other players on your team?  If so, how can they opt out being recorded?

 

The reason I ask is, I've seen these kinds of things being used to discriminate against other players and shame them in other games, most recently in STO.  I'd really hate to see a useful tool such as this be used for that purpose here.

 

Thanks.

To specifically answer your question. I ran a Yin with a full team and looked at the logs.

 

You will see X hit you with a buff or heal, but nothing about the amount of heal or buff.

 

Many buffs don't even tell you who cast them like Fulcrum Shift.
This is all that appears in the logs:

2024-06-05 14:46:58 The Fulcrum Shift has increased the damage of your attacks!

 

Everything non-damage related is a hot mess and lacks a lot of detail.

Edited June 5 by KaizenSoze

[Carnifax]

Nice, great work Kaizen and glad my rambling code helped.

 

On hols at the mo but I'll definitely have a look once back. 

 

Really like the index page idea. I considered it but then I'd be getting into usernames and accounts and I was too lazy 😀

 

One thing which is also cool about an "on pc" application is it can monitor/update the info while you play so hoping that's something on your radar (stream-parsing the active log file as you play). 

[KaizenSoze]

4 hours ago, Carnifax said:

Nice, great work Kaizen and glad my rambling code helped.

 

On hols at the mo but I'll definitely have a look once back. 

 

Really like the index page idea. I considered it but then I'd be getting into usernames and accounts and I was too lazy 😀

 

One thing which is also cool about an "on pc" application is it can monitor/update the info while you play so hoping that's something on your radar (stream-parsing the active log file as you play). 

Yes, I am considering it trying to do something Hero Stats, but that is a next year project.

[Dombloo]

Just wanted to stop by and give a huge thanks to you for working on this! Been having a lot fun throwing my combat logs at it; it's been easy to work with and summarizes the data nicely. 🫡

[Hedgefund]

I just found this thread.  I'm a data junkie so I downloaded asap.  This is outstanding.

 

Since I'm a good test case of "even this guy could do it" to evaluate directions, if I had followed them to the letter, I would have been fine to get it working the first time, but I did miss a step initially.  I only bring this up to let someone sitting on the fence about using this that 1) it works 2) follow directions to a T.

 

Right away I found some really eye opening results, like I wasn't expecting to see the interface proc be my highest source of damage for this MM I used to test this..  I can't wait to start using this after gaming sessions now.

Edited June 21 by Hedgefund

[KaizenSoze]

38 minutes ago, Hedgefund said:

I just found this thread.  I'm a data junkie so I downloaded asap.  This is outstanding.

 

Since I'm a good test case of "even this guy could do it" to evaluate directions, if I had followed them to the letter, I would have been fine to get it working the first time, but I did miss a step initially.  I only bring this up to let someone sitting on the fence about using this that 1) it works 2) follow directions to a T.

 

Right away I found some really eye opening results, like I wasn't expecting to see the interface proc be my highest source of damage for this MM I used to test this..  I can't wait to start using this after gaming sessions now.

Glad you are enjoying it. Please, occasionally hand verify really surprising result to make sure there isn't a bug.

 

There are definitely bugs dealing with pseudo pets hit/miss ratios and crit percentage. Regular pets are fine as far as a I know.

I also found that Interface was doing a lot of damage.

[Hedgefund]

Just fyi, it may not even matter but one of my quality checks is just to check number of items, in the README.md file.  By doing that I found a discrepancy, the text listing doesn't include "Pet Combat" which is shown on the image below it. 

[KaizenSoze]

3 minutes ago, Hedgefund said:

Just fyi, it may not even matter but one of my quality checks is just to check number of items, in the README.md file.  By doing that I found a discrepancy, the text listing doesn't include "Pet Combat" which is shown on the image below it. 

Fixed. Thank you that is exactly the double checking I need.

[EmperorSteele]

So, I was having trouble getting this to work, but I'll document here what I did to solve the issue, for any others who are confused on how this is supposed to work.

You can't just double-click the .bat file and have it work. The Options in the README cannot be input into the program itself, like what one may expect. Instead, you have to EDIT the .bat file using Notepad, and input the options that you want (so that it says something like start cmd /k .\cyclops.exe --logdir D:\Homecoming\accounts\YourName\Logs , then save the file, close it, then run it). This isn't listed in the ReadMe anywhere, as it's probably second nature to folks used to running command line programs and such. But it took me a solid half hour to figure this out.

Hopefully, others find this useful, as well.

[EmperorSteele]

Aaand no sooner do I get it working once when I run into an issue...

Command program said this, and refused to spit out a parse:

thread 'main' panicked at src\db_actions.rs:705:29:
Unable to update summary row: DatabaseError(CheckViolation, "CHECK constraint failed: (last_line_number > first_line_number)")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

*Shrugs*

@KaizenSoze, any idea what that's all about?

Okay, so, what happened was I was in a mission, died, signed out and did a parse. It worked. I went back in, tried the mission over again, completed it, tried to parse again... and that's what wouldn't work. I revived the old chat log, copypasta'd the text from the new log into the old one, put it back in my log directory, and THAT now worked.

?_?

Edited June 23 by EmperorSteele

[KaizenSoze]

55 minutes ago, EmperorSteele said:

So, I was having trouble getting this to work, but I'll document here what I did to solve the issue, for any others who are confused on how this is supposed to work.

You can't just double-click the .bat file and have it work. The Options in the README cannot be input into the program itself, like what one may expect. Instead, you have to EDIT the .bat file using Notepad, and input the options that you want (so that it says something like start cmd /k .\cyclops.exe --logdir D:\Homecoming\accounts\YourName\Logs , then save the file, close it, then run it). This isn't listed in the ReadMe anywhere, as it's probably second nature to folks used to running command line programs and such. But it took me a solid half hour to figure this out.

Hopefully, others find this useful, as well.

Thank you for the feedback. You don't actually have to edit the bat file.

 

After you run the cyclops.bat. Browse to http://127.1.1.1:11227

 

Updated the OP to make that more clear.

 

You should get the index page, which has options to select files or directories.

 

If the index page is busted, then send me a screenshot.

 

The command line options are optional. Mostly there for me when developing.

Edited June 23 by KaizenSoze

[KaizenSoze]

19 minutes ago, EmperorSteele said:

Aaand no sooner do I get it working once when I run into an issue...

Command program said this, and refused to spit out a parse:

thread 'main' panicked at src\db_actions.rs:705:29:
Unable to update summary row: DatabaseError(CheckViolation, "CHECK constraint failed: (last_line_number > first_line_number)")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

*Shrugs*

@KaizenSoze, any idea what that's all about?

It's usually an issue with the parsing having trouble find a session start. Happens sometimes when first setting up the logging.

 

If you would like, please send me the log file. I see if I can bullet proof against that in the future.

 

I the mean time if you have all the log configuration done. Stop the client. Deleting today's log file and logging back in. Back should get everything in a good state.

 

 

[EmperorSteele]

21 minutes ago, KaizenSoze said:

Thank you for the feedback. You don't actually have to edit the bat file.

 

After you run the cyclops.bat. Browse to http://127.1.1.1:11227

 

Updated the OP to make that more clear.

 

You should get the index page, which has options to select files or directories.

 

If the index page is busted, then send me a screenshot.

 

The command line options are optional. Mostly there for me when developing.

 

I get an "unable to connect" message when I click that link... shouldn't it be 127.0.0.1? I tried changing it to that, and it still wouldn't work. Issue with Firefox, maybe?

But yeah, the parse buttons didn't work on the html interface when I open them directly. If I try using ctrl + shift + C, it opens up Firefox's debug panel, not intended, I presume?

Also, sent the log that wouldn't open. Happy hunting 😃

Edited June 23 by EmperorSteele

[Dungeoness Eloora]

On 6/23/2024 at 6:56 PM, EmperorSteele said:

Aaand no sooner do I get it working once when I run into an issue...

Command program said this, and refused to spit out a parse:

thread 'main' panicked at src\db_actions.rs:705:29:
Unable to update summary row: DatabaseError(CheckViolation, "CHECK constraint failed: (last_line_number > first_line_number)")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I was getting these errors as well. I had to clean out the output folder from a previous test run in order for it to stop throwing that error. After directing it to any log file, though, it got to the end and threw this:

 

thread 'actix-rt|system:0|arbiter:16' panicked at src\db_actions.rs:647:10:
Error saving new summary: DatabaseError(Unknown, "no such table: summary")

 

Any ideas, @KaizenSoze?

[KaizenSoze]

25 minutes ago, Dungeoness Eloora said:

I was getting these errors as well. I had to clean out the output folder from a previous test run in order for it to stop throwing that error. After directing it to any log file, though, it got to the end and threw this:

 

thread 'actix-rt|system:0|arbiter:16' panicked at src\db_actions.rs:647:10:
Error saving new summary: DatabaseError(Unknown, "no such table: summary")

 

Any ideas, @KaizenSoze?

I have a fix for the line number issue. This issue though is a bit unclear.

 

Surprised it's failure for all logs. Are you using the command line or browser to select the file?

I would, stop the program, delete the output directory entirely. Then try again. Thank you for the bug report.

[Dungeoness Eloora]

I'm using the browser, directing it to an existing log file. Cyclops throws multiple lines of this, and the browser switches to a page that looks like I have no internet connection:

 

Latest Request: "J:\\Homecoming\\accounts\\XX\\Logs\\chatlog 2024-07-09.txt"
File opened for processing: J:\Homecoming\accounts\XX\Logs\chatlog 2024-07-09.txt
Line count: 14505, Data point count: 14505
Matching and conversion done.
thread 'actix-rt|system:0|arbiter:9' panicked at src\db_actions.rs:647:10:
Error saving new summary: DatabaseError(Unknown, "no such table: summary")

 

It does this regardless if there's any files in the output directory.

Edited July 9 by Dungeoness Eloora

[KaizenSoze]

New release beta 1.1. See first post for details.

 

Thank you in advance for feedback.

[Dombloo]

Loving the addition of the "Damage By Power or Mob" section, it has me digging through a bunch of older combat logs to compare engagements. Again, appreciate you working on this! 

[KaizenSoze]

1 hour ago, Dombloo said:

Loving the addition of the "Damage By Power or Mob" section, it has me digging through a bunch of older combat logs to compare engagements. Again, appreciate you working on this! 

You're welcome. Next I'll be adding damage per hit to the main power table, plus the option to select mob level to tell how much overkill damage is occurring. Or not.

[SeraphimKensai]

A couple months late, but this is pretty interesting stuff. Numbers are fun to munch.

[KaizenSoze]

On 8/3/2024 at 2:17 PM, Dombloo said:

Loving the addition of the "Damage By Power or Mob" section, it has me digging through a bunch of older combat logs to compare engagements. Again, appreciate you working on this! 

Has anything interesting popped out since you started look?