Carni's Online Combat Parser

[Carnifax]

WARNING : This is Alpha Code. Things will refuse to upload sometimes and sometimes things will be wrong

 

Ahem. Now we have that out of the way. I've spent the last few days working on a Java Parser for Log files and a simple webservice which lets you use it. You can find it here. 

 

https://www.carnifax.org

 

Instructions: 

1: Click "Choose File" and find your City of Heroes log file for a given day. Try and pick smaller files as at the moment the Uploader is a bit crap (working on it)

2: Click "Upload"

3: Wait a while. The tier of AWS I'm currently using seems a bit underpowered and slow and things take a while to upload.

4: Choose a Combat Session to parse

5: Browse the resulting JSON Object in the Viewer to see the various things harvested from your CombatLog.

 

You can also, if you copied it down, use the Retrieve by UUID to find a previous job and rerun Combat Sessions and download previously generated summaries. 

 

Remember you need to have logging switched on ingame (/logchat will stop/start it). Also go to your ingame Combat tab, right-click it and add all the Pet tabs in there. A lot of powers actually put things in Pet Channels rather than damage, healing etc. 

 

Known Issues.

• The uploader seems slow to me. To the point that sometimes it craps out entirely. I'm working on various options. 
• Figures may be all over the shop. Again I'm working on it. 
• Index Page is NOT pretty. I was in a hurry to get a minimum viable up and running. Anyone with suggestions about easy to set up frontend frameworks which don't involve me learning a whole load of new crap, send them on. I went the route of HTML + Javascript + CSS just cos it's quick and dirty. 
• DPS figures don't match Total / combatDuration for some reason. 
• "Hits" in powers / subPowers is off. Since Power damage includes all subpowers then Hits should too (or the other way around)
• Additional of new, initial session means there are very small "Unknown" combat session logs from between game start and the "Welcome to City of Heroes" being shown. 

 

 

How does it work?

The Logger has two phases. File Parsing and Combat Parsing. 

 

File Parsing takes one Log file and splits it into Global Channel and Combat Session logs on my server. I then delete the Global ones (cos I don't care about those). 

 

Combat Parsing takes a Combat Session log file and generates a summary JSON object from it by whizzing through it, capturing various types of lines, like Damage Dealt, pulling out interesting bits from each line (like what Power did it, what Type it was and how much it was for) and storing them inside objects. Once complete the Object generates some summaries and then writes the entire object to a summary file to view / download. 

 

What the hell is a "Combat Session"?

A combat session is just a subsection of the full log file. The system starts one for every new Character log in (based on the "Welcome to City of Heroes" message it shows as you load in). 

 

You can also control sessions by saying things in the Local channel ingame : 

 

/local STARTPARSE $name

will make the Parser split into a new session when it reads this

 

/local ENDPARSE

Will stop parsing until a new Start condition is hit (you alt or you do a /local STARTPARSE).

 

These let you control exactly when to stop / start logging. 

 

I downloaded one and it's bloody huge. What gives?

Downloaded summary_*.json files are more verbose than the Displayed version (because the Browser Viewer was struggling). They include Instance nodes for each time a power triggers, inside the Powers section. They also include a second by second breakdown of DPS at that point in time inside the DPS section. 

 

Your figures are arseways and you should feel bad.

Very likely. This was written over the last 4 days, and for periods of those days I was full of various intoxicants (Bank Holiday Weekend and all). At least some of the parser part was written while 3 Rums in. 

 

But the point of releasing this is so people can tell me what's arseways and (hopefully) why. 

 

How do we know you're not harvesting ChatLogs for evil?

Well it'll only parse what you send. I am storing the chatlogs on the server but I don't care about them and will start deleting them in the next release. Like anything on d'internet don't send anything if you're not sure. 

 

Why is it online? 

I'm mainly a backend and web Java developer. I don't do Client stuff and I don't like Java on clients anyway given their patch record. The Parser code is on GitHub so if you want to see how that part works you can download it, build it and run it locally on your machine. The "FileMain" class parses an entire City of Heroes log into seperate summaries for you. 

 

https://github.com/Jonfon0/cohcombatparser

 

This is also why there is a CombatParser project / jar and a seperate one for my webservices. 

 

There's nothing showing me XP or Inf or Drops

That'll be in the next version. Those aren't too difficult to monitor overall so it shouldn't take long. 

 

What's next

For the moment I'll be working on the backend. Improving the infrastructure, making it quicker to upload files and improving the parser. 

 

The webpage provided is just a proof of concept. In theory a frontend developer could write their own which just calls my services and shows pretty things like graphs etc. My main initial focus is finishing off these parts, adding new Services like "retrieval one of my older uploads based on a UUID or my history". Ideally we'd end up with something where you could just ping someone a link so they could see the parsed data. 

 

I'd also love to have the ability to link with other CoH Tools and Utilities and maybe even a thin desktop client eventually but baby steps. 

 

ChangeLog

• 05/04/2021 : Added Total XP and Inf tracking. Added pseudopet healing. Added CombatStart and CombatEnd. Normalized some fieldnames and removed some additional unneeded output. 
• 06/04/2021 : Added DamageOverTime, totalDamage. Added Pseudopet Hit, Miss and Damage Over Time. Refactored filenames so they sort naturally and allow full names again. Added "Unknown" combat sessions for when we can't see a "start session" string. 

 

Edited April 28, 2021 by Carnifax

[Carnifax]

Updated again. 

• I'd managed to miss DoT effects which I only noticed when Frost and Freezing Touch were strangely missing from the Damage section of a parse of my Ice Scrapper. Those are there now. 
• All Data sections now split Damage into damageTotal, damage and damageOverTime. damageTotal = damage + damageOverTime. I thought this was neat because it gives you a breakdown of "immediate" damage versus damage over time. Thinking about storing "Activations" in the JSON, then you could figure out a powers average damage per activation and tell how much DoT a power "wastes" on dead things. 
• Added CombatStart and CombatEnd. I've defined those as "the first time I did damage" and "the last time I did damage". Calculating combatDuration (in seconds) based off those
• Added totalInf, totalXP and quick and dirty calculates for damagePerCombatSecond, infPerCombatSecond and xpPerCombatSecond. 
• Removed the dps node entirely from the in-browser view cos the figures don't make a lot of sense. However they (and Power Instance nodes) are available if you download a summary. 

 

And again

• Added Pseudopet Hit, Miss and Damage Over Time. Refactored filenames so they sort naturally and allow full names again. Added "Unknown" combat sessions for when we can't see a "start session" string. 

 

This last change involved refactoring filenames so older uploads were deleted on the server. 

Edited April 6, 2021 by Carnifax

[Sovera]

This is lovely to have. My biggest thanks for adding this to the tools available for players, Carni!

[Carnifax]

7 minutes ago, Sovera said:

This is lovely to have. My biggest thanks for adding this to the tools available for players, Carni!

You're very welcome. Thank *you* for the Brunker builds. I love my Fire/Claws Giant Flaming Chicken. 

 

Spin is a silly power on Tanks.

[Sovera]

3 minutes ago, Carnifax said:

You're very welcome. Thank *you* for the Brunker builds. I love my Fire/Claws Giant Flaming Chicken. 

 

Spin is a silly power on Tanks.

 

Spin is. Martial Arts may be the better all rounded combo but nothing beats that *crunch* sound of Spin doing a super hero landing and HP bars depleting all around me.

[Carnifax]

Small update. Thanks to @Handmixer we realised that /local STARTPARSE said by someone else would cause YOUR log to split. 

 

To solve this you now have to put your own name after the STARTPARSE so the parser can confirm you said it. 

 

In other words : 

/l STARTPARSE $name

 

Or, if you just want Macros to do it for you

 

/macro STA "local STARTPARSE $name"

/macro END "local ENDPARSE $name"

[Carnifax]

Haven't forgotten this. I've added a summary of Drops to the parser now. 

 

 

I've also updated the index page very slightly to allow previous UUIDs to be passed in the URL. 

https://www.carnifax.org/?uuid=2b1dcadd-6b40-4de4-a7db-2a87d9798e1c

 

For example

[Carnifax]

Added a "Share" link now too. Just click it and it'll be copied to the clipboard. 

 

[Carnifax]

Crit-addicts update. I've added a Min Damage and Max Damage for powers and overall (they all use the same class internally so any time non-Damage Over Time Dam is added it'll see if it's the new Min or Max).

 

This is just for you Big Orange Number junkies. 

[Carnifax]

And for all the Controllers (except Elec cos you've weird powers I'm still looking at) a very general update for recording Mezzing (including normal knockbacks). Again since they belong to the same internal class used for both the Char and Powers you'll see them at both levels)

 

As you can see in the Mez section there may be some false positives because the pattern I'm using is very broad. I'll look into those (what's matching those 3 "reduce" hits?)

 

Elec Control > Static Field and Chain Confuse thing are logged weirdly, so I'll need to look at those individually. Earth Control and Dark Control might also have weird powers. 

[Carnifax]

Pretty visuals update. Reworked the DPS section and have added some graphs based on the imcoming data. I've put them at the bottom, under the CombatLog viewer section (cos it's collapsable). I've omitted legends because they're interactive anyway. 

 

Also, @GM Impervium / @Jimmy, I can has sticky?

 

[Dungeoness Eloora]

First of all, thanks for creating this! I was just looking for a combat log analyzer, but I was actually hoping to find one that analyzed damage taken. I think that would be helpful for examining the survivability of a given build. Is that functionality on your radar and/or is parsing that accurately even possible?

 

Also, would you be willing to add a pie graph that displayed skills by DPA (damage per activation)?

[Carnifax]

Damage taken I'll need to look at the logs, but I think it should be doable in terms of recording hits / misses and damage taken, along with heals and absorb (regen isn't logged).

 

DPA is trickier because the parser is "dumb". It knows nothing about power stats and I think the log timestamps are per second, not fractional. If I can make calls to City of Data I may be able to do something once I clear it with Uberguy.

[Dungeoness Eloora]

Thanks for the consideration. I look forward to future releases!

[Carnifax]

Defensive Parsing Update #1

 

 Because the CSS Stylesheet has changed you may need to clear your Browser Cache to see the updated look with the graphs on the righthand side 

 

I've split the stats into Offensive and Defensive blocks. Offensive is where what we had before is logged (so your Attacks, Misses, Damage done and so on). Defensive is the same thing but recording Attacks / Misses made against you and damage made against you, which the parser now calculates. Since both use the same (refactored) object that means things like MaxDamage something did to you is also parsed out for you. 

 

For example here you can see that in this parse I was attacked 546 times, 427 missed so the overall enemy accuracy was 21%. Total damage taken was 8044. Heaviest attack against me did 260 damage. 

 

This refactoring means you can now see how much damage you receive and attacks made against you. You can even see which attacks are doing the most damage. 

 

Graphs also refactored. So after the 4 Offensive ones (how much damage did I do) you can now see graphs of defensive stats (how much damage did I take, what types and from what powers). 

 

 

Couple of other changes also included

• Simple Accuracy for both the overall character and per power as a %. (so Hits * 100 / (Hits + Misses). Available Offensively and Defensively (so you can see how often you're getting hit too).
• Recording Criticals as a new stat. Again because it is in the "data" object both the Character and each power has you can see overall Crits and a per-power breakdown. 
• Refined the "Mez" recorded as it was getting some false positives from powers like Build Up being "tap" mez. At the moment it should just record Hold, Immobilize, Stun, Confuse, Terrorize and Taunt as "mez" effects. 
• Did some GUI stuff to make better use of the real estate. Graphs now appear down the right hand side of the page for easier access. You may need to clear Browser Cache to see these. 

 

 

Edited May 18, 2021 by Carnifax

[Psy-]

This is some spectacular work Carni, really starting to show the areas of weakness and strength well and identify gaps .. this is going to be incredibly useful!  🎖️🎖️🎖️🎖️

[Carnifax]

Finally got Annotations going. This allows you to stick names, powers etc into the Markers box. The first and last time the Strings are detected will be shown on your graphs. 

 

It uses a Cookie to remember the last values you used in the Marker Phrases box so should be somewhat persistant. 

 

For example here on a SBB I can show on the damage chart when I started & finished fighting the minotaur bloke.

 

 

[Carnifax]

New update. I've added a Custom Graph at the top of the Graphs section, which will create a Pie chart of any Power Attribute (base stat, mez or damage type).

 

So you can easily see how Holds break down

 

Or drill into a Damage Type and see what is doing that damage

 

Or what powers are missing a lot (oh Flames, I wish I knew what you were doing with all those ToHit rolls and misses)

 

Works for Defensive as well, telling you which enemy powers were the most likely to hit you

 

Known Issues : 

• I'm not picking up the Taunt from Taunt itself correctly. The Log String is a little different for those so it needs its own regex (bit like knockback). 
• Some Pseudopet mezzes don't appear to be recorded correctly. For example a Lockdown +2 proc in Traps Poison Gas isn't being picked up. 
• Electric Control : Still haven't adjusted / added specific regexes for the Chains. 

 

Edited May 20, 2021 by Carnifax

[Hardship]

I just want to say, thank you for sharing Carnifax! I haven't used this yet, but I full intend to soon. It looks great.

[Teirusu]

Thank you Carnifax!  I really like this utility.  Finally a decent way to compare characters and powers that doesn't require running specific maps/missions or targets.  This seems to allow running regular missions in strings and comparing the results with ease.

One of things I've noticed though is that, sometimes, it's missing displaying tags for certain powers.  For example, in the below image, Nova doesn't show up a label in the 'Damage by power' but clearly has a large slice of pie and it's own color.  Also, using the custom graph, it doesn't seem to show hits or activation for the interface procs, at least for the one that I was running.  It be nice to know how many times an interface proc goes off, for instance.

 

https://www.carnifax.org/?uuid=b4700942-95b3-4629-8ace-c6ce33730347

 

 

[Carnifax]

On 6/4/2021 at 3:00 AM, Teirusu said:

Thank you Carnifax!  I really like this utility.  Finally a decent way to compare characters and powers that doesn't require running specific maps/missions or targets.  This seems to allow running regular missions in strings and comparing the results with ease.

One of things I've noticed though is that, sometimes, it's missing displaying tags for certain powers.  For example, in the below image, Nova doesn't show up a label in the 'Damage by power' but clearly has a large slice of pie and it's own color.  Also, using the custom graph, it doesn't seem to show hits or activation for the interface procs, at least for the one that I was running.  It be nice to know how many times an interface proc goes off, for instance.

 

https://www.carnifax.org/?uuid=b4700942-95b3-4629-8ace-c6ce33730347

 

 

Sorry I missed this 1st time around. 

 

>  For example, in the below image, Nova doesn't show up a label in the 'Damage by power' but clearly has a large slice of pie and it's own color.

Yep, the graphs are limited to a 340x340 box (or thereabouts). So what happens is that the Legend gets truncated simply because there's no room for all of it. I'll probably let them become max-able someone with a Fancy Box so you could show an almost-fullscreen version which would allow the whole Legend to get shown. For example if I mess with the width and height via Browser Inspect you can see it more clearly

 

 

>  Also, using the custom graph, it doesn't seem to show hits or activation for the interface procs, at least for the one that I was running.  It be nice to know how many times an interface proc goes off, for instance.

 

Again I have reasons for this. Simply Interface (and other procs) never give you any Hit or Miss lines (or any Activation lines). 

A normal single target power (lets say Brawl) is nice and neat :

 

Brawl Activated : I catch this and record a new Instance of a Power being cast

Brawl Hit / Missed with the chance and the roll : I record this, associating it with the last Instance 

Brawl did X [Type] damage to Y : Again I record this and mark it against the last instance 

 

This works really well for lots of things, even with AoEs I could tell how many times it was activated and how many it hits / misses. 

 

But Procs and the like are trickier

Proc Activated : This is never logged

Proc Hit / Missed : Ditto

Proc did X [Type] Damage to Y : This is actually the only bit I can log. 

 

Which makes it really tricky to record in a way which will work but ALSO won't bugger up our recording of Brawl from above. For example if we said "Every time we do damage we should add a hit" then you'd be adding lots of "false" hits to our Brawl records. Damage over Time would also act weirdly. 

 

Part of this is due to my insistance that the Parser should have 0 foreknowledge. For example we know Interface is a proc, so I could go off and code special rules for Interfaces, procs and the like. But then I'd need to know the full list of these and have a big IF / OR block of code to catch them. 

 

I might be able to come up with a design for handling these better though. For example I could look at giving the system an alogrithim for these. Something like an "Is Autopower" function. That would trigger the 1st time a power does damage. If it is a power that has just done damage despite never having created a Power instance or made a hit/miss roll we know it must be some sort of proc and mark it as such. Then every time it does damage we can also acrue a HIT. That should work. Provided there's no DoT procs (I don't think there are). 

 

*Edit : Turns out I already did this for most procs. For example your Armageddon proc here did 94 instances of Damage. 

But it looks like I need an alternative / additional pattern for Interfaces. 

 

Edited June 21, 2021 by Carnifax

[Carnifax]

Your log file gives me two interesting powers, Interface and Lightning Field. Both of these never log Hits, just misses (in the case of Lightning Field). 

 

Spoiler

2021-05-31 15:48:18 You hit Guardian with your Spectral Interface for 8.69 points of Negative Energy damage over time.
2021-05-31 15:48:18 You hit Conscript with your Spectral Interface for 8.69 points of Negative Energy damage over time.
2021-05-31 15:48:18 Conscript HITS you! Rikti Sword power had a 6.50% chance to hit and rolled a 2.15.
2021-05-31 15:48:18 Conscript MISSES! Rikti Sword power had a 6.50% chance to hit, but rolled a 16.59.
2021-05-31 15:48:18 Conscript hits you with their Rikti Sword for 81.12 points of Lethal damage.
2021-05-31 15:48:18 Conscript hits you with their Rikti Sword for 27.82 points of Energy damage.
2021-05-31 15:48:18 Guardian MISSES! Rikti Pistol power had a 6.50% chance to hit, but rolled a 19.80.
2021-05-31 15:48:18 Chief Mentalist HITS you! Mental Blast power had a 48.68% chance to hit and rolled a 10.87.
2021-05-31 15:48:18 Guardian MISSES! Rikti Pistol power had a 6.50% chance to hit, but rolled a 7.27.
2021-05-31 15:48:18 Chief Mentalist hits you with their Mental Blast for 87.51 points of Psionic damage.
2021-05-31 15:48:18 Conscript MISSES! Rikti Rifle power had a 6.50% chance to hit, but rolled a 55.50.
2021-05-31 15:48:18 You activated the Nova power.
2021-05-31 15:48:18 You hit Conscript with your Spectral Interface for 8.69 points of Negative Energy damage over time.
2021-05-31 15:48:18 You hit Advanced Drone with your Spectral Interface for 13.47 points of Negative Energy damage over time.
2021-05-31 15:48:18 You hit Conscript with your Spectral Interface for 8.69 points of Negative Energy damage over time.
2021-05-31 15:48:18 You hit Conscript with your Spectral Interface for 8.69 points of Negative Energy damage over time.
2021-05-31 15:48:18 You hit Conscript with your Spectral Interface for 8.69 points of Negative Energy damage over time.
2021-05-31 15:48:18 Infantry MISSES! Rikti Pistol power had a 6.50% chance to hit, but rolled a 70.42.
2021-05-31 15:48:19 You hit Guardian with your Spectral Interface for 8.69 points of Negative Energy damage over time.
2021-05-31 15:48:19 You hit Conscript with your Spectral Interface for 8.69 points of Negative Energy damage over time.
2021-05-31 15:48:19 Communications Officer MISSES! Rikti Pistol power had a 6.50% chance to hit, but rolled a 50.51.
2021-05-31 15:48:19 You hit Conscript with your Spectral Interface for 8.69 points of Negative Energy damage over time.
2021-05-31 15:48:19 You hit Advanced Drone with your Spectral Interface for 13.47 points of Negative Energy damage over time.
2021-05-31 15:48:19 You hit Conscript with your Spectral Interface for 8.69 points of Negative Energy damage over time.
2021-05-31 15:48:19 You hit Conscript with your Spectral Interface for 8.69 points of Negative Energy damage over time.
2021-05-31 15:48:19 Chief Mentalist MISSES! Mesmerize power had a 48.68% chance to hit, but rolled a 62.33.
2021-05-31 15:48:19 You hit Guardian with your Lightning Field for 20.98 points of Energy damage.
2021-05-31 15:48:19 You hit Guardian with your Lightning Field for 1.72 points of their endurance.
2021-05-31 15:48:19 You hit Conscript with your Lightning Field for 20.98 points of Energy damage.
2021-05-31 15:48:19 You hit Conscript with your Lightning Field for 1.72 points of their endurance.
2021-05-31 15:48:19 You hit Advanced Drone with your Lightning Field for 27.09 points of Energy damage.
2021-05-31 15:48:19 You hit Advanced Drone with your Lightning Field for 1.72 points of their endurance.
2021-05-31 15:48:19 You hit Conscript with your Lightning Field for 20.98 points of Energy damage.
2021-05-31 15:48:19 You hit Conscript with your Lightning Field for 1.72 points of their endurance.
2021-05-31 15:48:19 You hit Conscript with your Lightning Field for 20.98 points of Energy damage.
2021-05-31 15:48:19 You hit Conscript with your Lightning Field for 1.72 points of their endurance.
2021-05-31 15:48:19 You hit Infantry with your Lightning Field for 20.98 points of Energy damage.
 

 

So I'll add a "NoHits" rule for these along the lines of the above which should give you some info about how often they actually hit things. 

 

Now there is an obvious downside there. Spectral is DoT and each "tick" will be recorded as a new hit. But nothing I can do about that really. 

Edited June 21, 2021 by Carnifax

[Carnifax]

New wee Update. Powers like Lightning Field, other Damage auras and Interface will now record hits as well. The DoT aspect of Spectral might throw it a bit but at least it gives you an idea of how many Hits there are. 

 

Also DoT seemed to be wrong there, Spectral is all DoT but was recording both as Damage and DoT (because some idiot forgot his ELSE in front of one of the IF blocks). 

 

  

Edited June 21, 2021 by Carnifax

[FallenHero]

This tool is fantastic. Thank you for your hard work (and for providing the code).

[Carnifax]

42 minutes ago, FallenHero said:

This tool is fantastic. Thank you for your hard work (and for providing the code).

You're very welcome! If you spot anything weird let me know.

 

(Note to self, check in the updated code for the parser later. I've been lax recently)