Jump to content

Cyclops Log Parsing Tool


Recommended Posts

Posted (edited)

First, I would like to thank @Carnifax for providing me the source code from their tool. This greatly sped up the initial development. Please, send any flowers to them first.

 

What is this?

A new tool for parsing City of Heroes log files for analysis.

 

Why should I use this instead of Carnifax's?

If you like Carnifax's, please keep using it. This tool runs on your machine instead of Carnifax's site and has some different features.

 

Features (screenshots below):

  • Index page that lists all the logs you have parsed
  • Total rewards, experience, mobs defeated
  • General combat summary
  • Power by usage
  • DPS in intervals. See README for more details.
  • Damage dealt by type
  • Damage taken by type
  • Damage taken by mob
  • Damage taken by mob power

 

How do I get started?

  • README
  • Download the zip from here.
  • Unzip the file where every you want
  • Window users can use cyclops.bat to start the application
  • Or open a command line in the new directory and type: cyclops.exe
  • Browse: http://127.1.1.1:11227

 

Future work:

  • Bugs fixes, lots of bug fixes
  • Depends on feedback I get
  • Charts
  • Timeline of major events, dps spikes, damage taken spikes, etc...
  • Exporting
  • Statistical improvements
  • Log comparisons

 

Geek stuff:

  • Github page
  • Tech used
    • Rust language - first project so code quality is meh
    • Sqlilte database
    • Actix web server
    • Tera templating

 

Screenshots:

Index page:
image.thumb.png.6221ac26a45b815765cec90d1de1c174.png

Summary:

image.thumb.png.a203f92343e5626c4e41df77d053b15a.pngPowers table:
image.thumb.png.50d28aa28b5fea644e7c0fa45455806f.png

 

DPS intervals, Damage dealt, taken by type and mob:
image.thumb.png.de3d2c6899039233f659a1a89ae83564.png

image.png

Edited by KaizenSoze
  • Thanks 3
  • Thumbs Up 3
Link to comment
Share on other sites

Nice work on this!  I can see it being very helpful to those interested in maxing out a build and analyzing their DPS.

 

With that said, I'd like to ask a couple of questions about this too.  Please know I am not asking these with any sort of malice toward you, accusing you of anything or implying anything about you personally.  My questions are in no way a condemnation of you personally or the work you have put into this. 

 

With that out of the way, my questions are;  does this record and track other players on your team?  If so, how can they opt out being recorded?

 

The reason I ask is, I've seen these kinds of things being used to discriminate against other players and shame them in other games, most recently in STO.  I'd really hate to see a useful tool such as this be used for that purpose here.

 

Thanks.

Link to comment
Share on other sites

2 minutes ago, Excraft said:

Nice work on this!  I can see it being very helpful to those interested in maxing out a build and analyzing their DPS.

 

With that said, I'd like to ask a couple of questions about this too.  Please know I am not asking these with any sort of malice toward you, accusing you of anything or implying anything about you personally.  My questions are in no way a condemnation of you personally or the work you have put into this. 

 

With that out of the way, my questions are;  does this record and track other players on your team?  If so, how can they opt out being recorded?

 

The reason I ask is, I've seen these kinds of things being used to discriminate against other players and shame them in other games, most recently in STO.  I'd really hate to see a useful tool such as this be used for that purpose here.

 

Thanks.

Good question. You *cannot* see other players *damage* stats, damage dealt, taken, or hit rolls. Some buffs might appear, especially autohit AOE buffs.

You will see if they buff or heal you, defeat a mob, or are defeated.

 

In general buff/heal/debuff logging are a hot mess and not tracked in the tool at the moment. I will keep your concern in mind. I have zero interest in creating something that is used to exclude players.

Link to comment
Share on other sites

Posted (edited)
2 hours ago, Excraft said:

Nice work on this!  I can see it being very helpful to those interested in maxing out a build and analyzing their DPS.

 

With that said, I'd like to ask a couple of questions about this too.  Please know I am not asking these with any sort of malice toward you, accusing you of anything or implying anything about you personally.  My questions are in no way a condemnation of you personally or the work you have put into this. 

 

With that out of the way, my questions are;  does this record and track other players on your team?  If so, how can they opt out being recorded?

 

The reason I ask is, I've seen these kinds of things being used to discriminate against other players and shame them in other games, most recently in STO.  I'd really hate to see a useful tool such as this be used for that purpose here.

 

Thanks.

To specifically answer your question. I ran a Yin with a full team and looked at the logs.

 

You will see X hit you with a buff or heal, but nothing about the amount of heal or buff.

 

Many buffs don't even tell you who cast them like Fulcrum Shift.
This is all that appears in the logs:

2024-06-05 14:46:58 The Fulcrum Shift has increased the damage of your attacks!

 

Everything non-damage related is a hot mess and lacks a lot of detail.

Edited by KaizenSoze
  • Like 1
Link to comment
Share on other sites

Nice, great work Kaizen and glad my rambling code helped.

 

On hols at the mo but I'll definitely have a look once back. 

 

Really like the index page idea. I considered it but then I'd be getting into usernames and accounts and I was too lazy 😀

 

One thing which is also cool about an "on pc" application is it can monitor/update the info while you play so hoping that's something on your radar (stream-parsing the active log file as you play). 

  • Like 1
Link to comment
Share on other sites

4 hours ago, Carnifax said:

Nice, great work Kaizen and glad my rambling code helped.

 

On hols at the mo but I'll definitely have a look once back. 

 

Really like the index page idea. I considered it but then I'd be getting into usernames and accounts and I was too lazy 😀

 

One thing which is also cool about an "on pc" application is it can monitor/update the info while you play so hoping that's something on your radar (stream-parsing the active log file as you play). 

Yes, I am considering it trying to do something Hero Stats, but that is a next year project.

Link to comment
Share on other sites

  • 2 weeks later

Just wanted to stop by and give a huge thanks to you for working on this! Been having a lot fun throwing my combat logs at it; it's been easy to work with and summarizes the data nicely. 🫡

  • Like 1
Link to comment
Share on other sites

I just found this thread.  I'm a data junkie so I downloaded asap.  This is outstanding.

 

Since I'm a good test case of "even this guy could do it" to evaluate directions, if I had followed them to the letter, I would have been fine to get it working the first time, but I did miss a step initially.  I only bring this up to let someone sitting on the fence about using this that 1) it works 2) follow directions to a T.

 

Right away I found some really eye opening results, like I wasn't expecting to see the interface proc be my highest source of damage for this MM I used to test this..  I can't wait to start using this after gaming sessions now.

Edited by Hedgefund
Link to comment
Share on other sites

38 minutes ago, Hedgefund said:

I just found this thread.  I'm a data junkie so I downloaded asap.  This is outstanding.

 

Since I'm a good test case of "even this guy could do it" to evaluate directions, if I had followed them to the letter, I would have been fine to get it working the first time, but I did miss a step initially.  I only bring this up to let someone sitting on the fence about using this that 1) it works 2) follow directions to a T.

 

Right away I found some really eye opening results, like I wasn't expecting to see the interface proc be my highest source of damage for this MM I used to test this..  I can't wait to start using this after gaming sessions now.

Glad you are enjoying it. Please, occasionally hand verify really surprising result to make sure there isn't a bug.

 

There are definitely bugs dealing with pseudo pets hit/miss ratios and crit percentage. Regular pets are fine as far as a I know.

I also found that Interface was doing a lot of damage.

Link to comment
Share on other sites

Just fyi, it may not even matter but one of my quality checks is just to check number of items, in the README.md file.  By doing that I found a discrepancy, the text listing doesn't include "Pet Combat" which is shown on the image below it. 

Link to comment
Share on other sites

3 minutes ago, Hedgefund said:

Just fyi, it may not even matter but one of my quality checks is just to check number of items, in the README.md file.  By doing that I found a discrepancy, the text listing doesn't include "Pet Combat" which is shown on the image below it. 

Fixed. Thank you that is exactly the double checking I need.

Link to comment
Share on other sites

So, I was having trouble getting this to work, but I'll document here what I did to solve the issue, for any others who are confused on how this is supposed to work.

You can't just double-click the .bat file and have it work. The Options in the README cannot be input into the program itself, like what one may expect. Instead, you have to EDIT the .bat file using Notepad, and input the options that you want (so that it says something like start cmd /k .\cyclops.exe --logdir D:\Homecoming\accounts\YourName\Logs , then save the file, close it, then run it). This isn't listed in the ReadMe anywhere, as it's probably second nature to folks used to running command line programs and such. But it took me a solid half hour to figure this out.

Hopefully, others find this useful, as well.

  • Thanks 1
Link to comment
Share on other sites

Aaand no sooner do I get it working once when I run into an issue...

Command program said this, and refused to spit out a parse:

thread 'main' panicked at src\db_actions.rs:705:29:
Unable to update summary row: DatabaseError(CheckViolation, "CHECK constraint failed: (last_line_number > first_line_number)")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

*Shrugs*

@KaizenSoze, any idea what that's all about?

Okay, so, what happened was I was in a mission, died, signed out and did a parse. It worked. I went back in, tried the mission over again, completed it, tried to parse again... and that's what wouldn't work. I revived the old chat log, copypasta'd the text from the new log into the old one, put it back in my log directory, and THAT now worked.

?_?

Edited by EmperorSteele
Link to comment
Share on other sites

Posted (edited)
55 minutes ago, EmperorSteele said:

So, I was having trouble getting this to work, but I'll document here what I did to solve the issue, for any others who are confused on how this is supposed to work.

You can't just double-click the .bat file and have it work. The Options in the README cannot be input into the program itself, like what one may expect. Instead, you have to EDIT the .bat file using Notepad, and input the options that you want (so that it says something like start cmd /k .\cyclops.exe --logdir D:\Homecoming\accounts\YourName\Logs , then save the file, close it, then run it). This isn't listed in the ReadMe anywhere, as it's probably second nature to folks used to running command line programs and such. But it took me a solid half hour to figure this out.

Hopefully, others find this useful, as well.

Thank you for the feedback. You don't actually have to edit the bat file.

 

After you run the cyclops.bat. Browse to http://127.1.1.1:11227

 

Updated the OP to make that more clear.

 

You should get the index page, which has options to select files or directories.

 

If the index page is busted, then send me a screenshot.

 

The command line options are optional. Mostly there for me when developing.

Edited by KaizenSoze
Link to comment
Share on other sites

19 minutes ago, EmperorSteele said:

Aaand no sooner do I get it working once when I run into an issue...

Command program said this, and refused to spit out a parse:

thread 'main' panicked at src\db_actions.rs:705:29:
Unable to update summary row: DatabaseError(CheckViolation, "CHECK constraint failed: (last_line_number > first_line_number)")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

*Shrugs*

@KaizenSoze, any idea what that's all about?

It's usually an issue with the parsing having trouble find a session start. Happens sometimes when first setting up the logging.

 

If you would like, please send me the log file. I see if I can bullet proof against that in the future.

 

I the mean time if you have all the log configuration done. Stop the client. Deleting today's log file and logging back in. Back should get everything in a good state.

 

 

Link to comment
Share on other sites

21 minutes ago, KaizenSoze said:

Thank you for the feedback. You don't actually have to edit the bat file.

 

After you run the cyclops.bat. Browse to http://127.1.1.1:11227

 

Updated the OP to make that more clear.

 

You should get the index page, which has options to select files or directories.

 

If the index page is busted, then send me a screenshot.

 

The command line options are optional. Mostly there for me when developing.

 

I get an "unable to connect" message when I click that link... shouldn't it be 127.0.0.1? I tried changing it to that, and it still wouldn't work. Issue with Firefox, maybe?

But yeah, the parse buttons didn't work on the html interface when I open them directly. If I try using ctrl + shift + C, it opens up Firefox's debug panel, not intended, I presume?

Also, sent the log that wouldn't open. Happy hunting 😃

Edited by EmperorSteele
Link to comment
Share on other sites

  • 3 weeks later
On 6/23/2024 at 6:56 PM, EmperorSteele said:

Aaand no sooner do I get it working once when I run into an issue...

Command program said this, and refused to spit out a parse:

thread 'main' panicked at src\db_actions.rs:705:29:
Unable to update summary row: DatabaseError(CheckViolation, "CHECK constraint failed: (last_line_number > first_line_number)")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I was getting these errors as well. I had to clean out the output folder from a previous test run in order for it to stop throwing that error. After directing it to any log file, though, it got to the end and threw this:

 

thread 'actix-rt|system:0|arbiter:16' panicked at src\db_actions.rs:647:10:
Error saving new summary: DatabaseError(Unknown, "no such table: summary")

 

Any ideas, @KaizenSoze?

@dungeoness and @eloora on Excelsior

<Federation of United Cosmic Knights>

Link to comment
Share on other sites

25 minutes ago, Dungeoness Eloora said:

I was getting these errors as well. I had to clean out the output folder from a previous test run in order for it to stop throwing that error. After directing it to any log file, though, it got to the end and threw this:

 

thread 'actix-rt|system:0|arbiter:16' panicked at src\db_actions.rs:647:10:
Error saving new summary: DatabaseError(Unknown, "no such table: summary")

 

Any ideas, @KaizenSoze?

I have a fix for the line number issue. This issue though is a bit unclear.

 

Surprised it's failure for all logs. Are you using the command line or browser to select the file?

I would, stop the program, delete the output directory entirely. Then try again. Thank you for the bug report.

Link to comment
Share on other sites

Posted (edited)

I'm using the browser, directing it to an existing log file. Cyclops throws multiple lines of this, and the browser switches to a page that looks like I have no internet connection:

 

Latest Request: "J:\\Homecoming\\accounts\\XX\\Logs\\chatlog 2024-07-09.txt"
File opened for processing: J:\Homecoming\accounts\XX\Logs\chatlog 2024-07-09.txt
Line count: 14505, Data point count: 14505
Matching and conversion done.
thread 'actix-rt|system:0|arbiter:9' panicked at src\db_actions.rs:647:10:
Error saving new summary: DatabaseError(Unknown, "no such table: summary")

 

It does this regardless if there's any files in the output directory.

Edited by Dungeoness Eloora
  • Thumbs Up 1

@dungeoness and @eloora on Excelsior

<Federation of United Cosmic Knights>

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...