Visualize City of Heroes Logfiles

[valcryst81]

Hi everyone,

 

i got bored one day, so i thought i should look into CoX logging, since there where some changes somewhere in the last patches.
It went realy well and i got most of the interesting parts parsed from the logs, but i also ran into multiple problems.

 

1) Duplicates: It will happen when you use AOE-Powers and Hit alot of enemies.
    - You hit targets at the "same time" with the "same name" and the "same amount of damage"

 

2) Confusing: Some power log entries are just wrong in spelling and i dont know yet, if thats end-drain or some sort of damage? 
   - 2020-05-22 10:49:01 You hit D.U.S.T. Leader with your ESD Arrow for 253.82 points of their endurance.

 

3) ToHit: It seems to me that not every time you activate a skill there will be a roll for ToHit? Is that a bug or intented?

 

And sadly there is no possible solution to track how many enemies you hit with a specific skill other that guessing and using some black magic.

Anyway i wanted to share what i got so far, maybe someone is interested in this and wants to help rounding it up 🙂

 

(English is not my native language, so please be kind) 

 

SETUP:

Note: I already had a TIG-Stack running in my home network, so it was just connecting the wires for me.
Note2: I wont give you any support on installing/configure the needed tools, there is enough documentation for that already.

- Requirements

  ! All tools are available for Linux + Windows !

 

InfluxDB 1.8 (dont use 2.x) - https://portal.influxdata.com/downloads/ 
Telegraf 1.14.x                    - https://github.com/influxdata/telegraf/releases
Grafana 7.x                         - https://grafana.com/grafana/download

 

City of Heroes with Chatlog enabled
(dont forget to add Pet Damage stuff for Blizzard/Rain of Fire etc)

 

- Instructions

First you should install InfluxDB, in most cases you dont need to configure anything, just install and start it.
After starting it, InfluxDB should listen on 127.0.0.1:8086 for connections.

 

Install Telegraf on your Machine and register it as a service and dont start it yet.

-> https://github.com/influxdata/telegraf/blob/master/docs/WINDOWS_SERVICE.md

With default settings Telegraf will sent metrics to 127.0.0.1:8086
 

Optional: If you also want to get your NVIDIA Stats while Gaming you can activate this too
-> https://github.com/influxdata/telegraf/tree/master/plugins/inputs/nvidia_smi

 

 

Now we setup log parsing for City of Heroes, for that you have to open the telegraf.conf file
wich you should find in "C:\Programm Files\Telegraf\" - open it with "Notepad++" or an editor that you like.
Go to the end of the file and add the following code snipped, but edit paths and account to fit your needs.
 

[[inputs.tail]]
  files = ["D:\\coh\\ACCOUNT-NAME\\logs\\chatlog*.txt"]
  from_beginning = false
  watch_method = "poll"
  name_override = "city_of_heroes"
  data_format = "grok"
  grok_patterns = ['%{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02 15:04:05"} You hit %{GREEDYDATA} with your %{GREEDYDATA:power_name:tag} for %{NUMBER:damage:float} points of %{GREEDYDATA:damage_type:tag}.',
                   '%{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02 15:04:05"} %{GREEDYDATA}:  You hit %{GREEDYDATA} with your %{GREEDYDATA:power_name:tag} for %{NUMBER:damage:float} points of %{GREEDYDATA:damage_type:tag}.',
                   '%{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02 15:04:05"} You have defeated %{GREEDYDATA:defeated}',
                   '%{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02 15:04:05"} %{WORD:tohit_result:tag} %{GREEDYDATA:tohit_target} Your %{GREEDYDATA:power_name:tag} power had a %{NUMBER:tohit_chance:float}% chance to hit, you rolled a %{GREEDYDATA:tohit_rolled:float}.',
                   '%{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02 15:04:05"} You activated the %{GREEDYDATA:power_activation} power.',
                   '%{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02 15:04:05"} You received %{GREEDYDATA:drop}.']
  grok_timezone = "Local"
  [inputs.tail.tags]
    account_name = "ACCOUNT-NAME"

 

After you have saved the file, you should clean up your City of Heroes logfiles before you start Telegraf,
because in this setup Telegraf will open file listeners on all logfiles he will find in that directory, so a little housekeeping is good 🙂

 

Now install Grafana, the tool we use to visualize the data we gathered.
After installing/starting Grafana it will listen on http://127.0.0.1:3000 and serve the webui.
Login to Grafana with default creds (admin/admin) and setup the Datasource (InfluxDB) and point to 127.0.0.1:8086
When everything is working you can now import my attached Dashboard into your Grafana.

Start Playing 🙂

(Yes i know this is far from perfect or good, but it works. I'm happy for any input/suggestions to make this better)

Note: If you want to parse old logs, check the option "from_beginning" in Telegraf inputs.tail - true will parse logs from the beginning.
 

Pictures:

 

NVIDIA:

City of Heroes-1590237832719.json

Edited May 23, 2020 by valcryst81

[valcryst81]

Updated Dashboard with a little more infos 🙂

(See Attachments)

 

 

City of Heroes-1590408810722.json

[Eclipse.]

I went ahead and added Healing/Endurance sources effecting the player.

'%{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02 15:04:05"} %{GREEDYDATA} hits you with their %{GREEDYDATA:power_name:tag} granting you %{NUMBER:endurance:float} points of endurance.',
'%{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02 15:04:05"} %{GREEDYDATA} heals you with their %{GREEDYDATA:power_name:tag} for %{NUMBER:health:float} health points.',
'%{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02 15:04:05"} You are healed by your %{GREEDYDATA:power_name:tag} for %{NUMBER:health:float} health points.'

I spent a decent amount of time trying to figure out the grok_pattern for XP/INF. Unfortunately since the default INT & FLOAT don't deal with thousands seperators AND influxdb can't convert from string to int I kinda gave up. Oh Well.

I need to practice with Grafana anyways for a project at work. I might try to make the install process for this a bit less painful for people using windows using Docker Compose if there is interest. No Promises when that might actually happen if there is though, work keeps me far too busy lately...and honestly, getting XP/INF working would take higher priority for me.

City of Heroes-1593337221623.json

[valcryst81]

Hi @Eclipse.nice work! 🙂

 

I may be able to help on this, but it was a tough one even for me 😄

> select experience,influence from city_of_heroes limit 10
name: city_of_heroes
time                experience influence
----                ---------- ---------
1591520769000000000 6579       4604
1591520770000000000 13158      15788
1591520771000000000 6579       4604
1591520780000000000 6579       4604
1591520783000000000 13158      15788
1591520789000000000 13158      15788
1591520793000000000 39472      49339
1591520798000000000 13158      15788
1591520801000000000 6579       4604
1591520812000000000 6579       4604
>

You have to add this into your telegraf config file.

[[processors.regex]]
  namepass = ["city_of_heroes"]
  [[processors.regex.fields]]
    key = "experience"
    pattern = "[,]*"
    replacement = "${1}"
  [[processors.regex.fields]]
    key = "influence"
    pattern = "[,]*"
    replacement = "${1}"

[[processors.converter]]
  namepass = ["city_of_heroes"]
  [processors.converter.fields]
    integer = ["experience"]

[[processors.converter]]
  namepass = ["city_of_heroes"]
  [processors.converter.fields]
    integer = ["influence"]

The GROK pattern looks like this

'%{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02 15:04:05"} You gain %{GREEDYDATA:experience} experience and %{GREEDYDATA:influence} influence.'

 

So first we grab the XP/Influence values and pipe them into the regex processor, to remove the comma, then we pipe it trough the converter to convert string to integer 🙂

(For other readers: Its important that you dont save the numbers as strings, cause you cant do math/functions with strings)

 

Edit: Hmm but we have to make another line for the other side, dont we? Wasnt Influence called something different there?

 

Regards

Edited June 28, 2020 by valcryst81

[Eclipse.]

24 minutes ago, valcryst81 said:

 

'%{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02 15:04:05"} You gain %{GREEDYDATA:experience} experience and %{GREEDYDATA:influence} influence.'

 

Awesome! I was trying to figure out how to do pre/post processing on captured fields before sending it to influxdb, but my google foo for Telegraf just wasn't good enough apparently.

As an aside, I'd replace influence in the quoted line with %{GREEDYDATA} otherwise if you're ever a villain/rogue it'll stop capturing.

[valcryst81]

2 minutes ago, Eclipse. said:

Awesome! I was trying to figure out how to do pre/post processing on captured fields before sending it to influxdb, but my google foo for Telegraf just wasn't good enough apparently.

As an aside, I'd replace influence in the quoted line with %{GREEDYDATA} otherwise if you're ever a villain/rogue it'll stop capturing.

Ah yes that was the thing that came into my mind after i posted it 🙂

 

For Telegraf it helps sometimes to look into the plugin dirs on github, inside there is always a readme with examples.

 

https://github.com/influxdata/telegraf/tree/master/plugins/processors/converter

https://github.com/influxdata/telegraf/tree/master/plugins/processors/regex

 

Regards

[Eclipse.]

9 minutes ago, valcryst81 said:

For Telegraf it helps sometimes to look into the plugin dirs on github, inside there is always a readme with examples.

 

Gotcha, good to know.
More accustomed to splunk and being able to sed/regex at query time. Working now though, so all good. Thanks!

[Mikewho]

This is awesome stuff. If you ever find a way to package this into a single app somehow, I would totally use it!

[Eclipse.]

btw @valcryst81, in the "Defeated" count you included your hostname in the json. Probably want to remove that before sharing. Makes that chart not work for anyone else, also for privacy reasons

Side note, here is what I've got so far.  Incorporated the XP, also made a section to check on incoming damage instead of just outgoing.

Edited July 1, 2020 by Eclipse.

[valcryst81]

@Eclipse.Wow pretty cool 🙂

Oh that hostname thing wasnt intendet, now you know that i'm a HG2G fan 😶

I was about to create a github repository that could hold our stuff, what do you think?

 

Regards 

Edited July 2, 2020 by valcryst81

[LaconicLemur]

Hi there. Someone led me to this thread when I was asking about potential combat tracking tools. I wanted to let you know that I managed to install the TIG stack as a non-Grafana user. I did take a few extra steps. These might seem obvious to you, but thought others like me might appreciate knowing them:

 

(1) On install of Influxdb, I needed to create a database called "telegraf"

(2) On Win 10, I needed to install telegraf.exe and telegraf.conf on C:\Program Files (x86)\telegraf. Any other directory resulted in an error when attempting to start the service via Net start or the Services console

(3) When you update the telegraf.conf file with the CoH settings, you can  use the command C:\Program Files (x86)\telegraf>telegraf.exe --config telegraf.conf --test to check the .conf file is still ok and that you didn't accidentally introduce an error on the edit

-> If the telegraf service doesn't start in Windows, check the Events> Application logs for the error when service start was attempted

(4) In Grafana, to make in easy, name your data source to "InfluxDB" to match the CoH json file provided. Change the time zone to your local time zone; it is set to CET

(5) In CoH, make sure your log file is updating in your folder. I think you have to turn it on manually for each session.

 

I found it helpful to test a "bare" install using the steps from this site: https://litefoote.com/2018/03/27/setup-telegrafinfluxdbgrafana-to-monitor-windows/

 

The steps to get logging to happen for CoH in windows:

(1) Start influxdb by running \influxdb-1.8.4-1\influxd.exe from a cmd line terminal (leave it on, don't close it)

(2) Start the telgraf service either using net start or Services console

(3) Open a browser to localhost:3000 and login to Grafana. You may need to refresh the screen periodically (there is a little icon on the top right to do this).

 

Thanks for taking the time to put the original post and the .json files together. It's really neat to have a look at the GUI.

 

[valcryst81]

Thank you @LaconicLemur for reporting it, i will update the start post soon and will add your steps to it 🙂

[Carnifax]

Hi folks. I've written an online parser in Java. Dunno if any of it is any use to you but the project is online at https://www.carnifax.org

 

The codebase is on GIT and the Patterns I've used for various parsings are all in one spot too

https://github.com/Jonfon0/cohcombatparser

https://github.com/Jonfon0/cohcombatparser/blob/main/combatparser/src/main/java/org/coh/carnifax/combat/data/CombatDefs.java

 

Feel free to use anything of use or if I can help in any way let me know. I'm not familiar with Grafina but I'm always willing to learn, even the way I decided to handle things like "Power Instances" etc might be of help to you. 

Edited April 9, 2021 by Carnifax

[valcryst81]

Hey @Carnifax thats awesome, i may have to check if its possible to re-use your patterns to extend the grok parser inside telegraf 🙂

Are you somehow visualizing your data?

[Carnifax]

13 minutes ago, valcryst81 said:

Hey @Carnifax thats awesome, i may have to check if its possible to re-use your patterns to extend the grok parser inside telegraf 🙂

Are you somehow visualizing your data?

No, my focus is all based on generating a JSON file. I am planning a session based thing where you could Start a session, then a webservice would accept a block of new lines to add to it and return updated JSON, to make it continous. 

 

Essentially I want to return a JSON data file which others can use to do whatever they like. 

 

If you use b7da7cbb-1e9e-4a80-b298-56be60489b87 in the "Retrieve by UUID" and you'll see the split up Combat logs for each "session" from my playing yesterday. I'm just displaying the JSON Object (or a simplier version of it) in the browser using a JS library I found. 

 

You can use whatever you like from my project, I'm all about sharing knowledge and resources. The patterns I'm using aren't radically different to yours. From reading your example I understand the format so if you want to pick my brains on one please do. 

[Carnifax]

On 6/28/2020 at 6:02 PM, Eclipse. said:

Awesome! I was trying to figure out how to do pre/post processing on captured fields before sending it to influxdb, but my google foo for Telegraf just wasn't good enough apparently.

As an aside, I'd replace influence in the quoted line with %{GREEDYDATA} otherwise if you're ever a villain/rogue it'll stop capturing.

Oooh, there's some I didn't consider. Looks like I'll have to edit my version of this pattern. 

 

Could you change your String to 

'%{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02 15:04:05"} You gain %{GREEDYDATA:experience} experience and %{GREEDYDATA:influence} inf%{GREEDYDATA:inftype}.'

 

And just ignore inftype?

[LaconicLemur]

On 4/8/2021 at 5:10 AM, valcryst81 said:

Thank you @LaconicLemur for reporting it, i will update the start post soon and will add your steps to it 🙂

 

No, thank you! I do have a question: If I use multiple accounts, how do I get both to show in grafana? I assume it has to do with the telegraf.conf file, but I'm not sure if I copy/paste and simply change the account name, or if I have to also change the name of the input.tails name_override to something else.  Or.. something else?