Sif's Pigg/bin/Parse7 parser

[Progor]

Reposting @Sif's work here so we don't take over @The Philotic Knight's thread:

 

GitHub (source): https://github.com/IndexObscurum/index_datamanip

Crates.io (Rust package): https://crates.io/crates/index_datamanip

Docs.rs (docs): https://docs.rs/index_datamanip/

 

This contains the code for extracting data from Piggs, enough handling around MessageStores to go from the P-strings to localized text (the rest of the data is parsed, but discarded currently), and for deserializing Parse7-encoded bins into the passed in data structures. Additionally, it contains all of the structures (in the objects module) used for deserializing Powers/Powerset/PowerCategories/Classes (Classes has something like 2 fields unimplemented).

 

I have some additional high level documentation on the formats I've written up that I intend to merge into the library's docs. Some work would need to be done to make this a CLI / callable from non-Rust languages (I have some ideas for both, but... I also rolled a new Blaster...)

[Progor]

I think there's value here as well as @RubyRed's API.  This library can give you *all* the raw data, while RubyRed's API focuses on giving you the important bits for building tools without all the extras.  I'll work on a command line application for exporting the results as JSON so it can be just as accessible from downstream apps.

 

@Sif Alright, quick usage check.  I don't know rust well enough to write correct syntax yet, but here's what it looks like I need to do to load powers data and write it out as json:

 

let pigg = Pigg::new("path/to/file.pigg");
let messages = parse_messages::get_pmessages(pigg.get_data("bin/clientmessages-en.bin"));
let powers = defs::decode<objects::Power>(pigg.get_data("bin/powers.bin"));
foreach (power in powers) power.fix_strings(messages);
writeFile("powers.json", serde_json::to_string(powers));

 

Does that look about right (other than slaughtering the syntax, I'm sure), or am I missing something?

[_NOPE_]

Thanks for starting a new thread. 

[Sif]

The main thing to note is that while most of the bins are in bin.pigg, the powers are bin_powers.pigg (they're also the only thing in it).

[Mikewho]

Awesome, thanks for this!

[Progor]

Yep, noticed that.  So, got that figured out and made the first early, hard-coded, cobbled-together, command-line app.

 

https://github.com/dpwhittaker/sif_parser

 

Just "cargo run" it with bin.pigg and bin_powers.pigg in the parent directory to generate a powers.json in the parent directory.

 

and here's the output for the powers.bin:

 

https://1drv.ms/u/s!AuGvcX8-qHq2luNtIXUr_nVnmKheqA?e=nMZT8N

 

So, next steps are:

 

• output the other bins as json
• come up with a "default value" for each field and only serialize the field if it is different from the default to cut down on file size.  242MB uncompressed is a bit much for a web app
• split it up into categories similar @RubyRed's API

This post-processing will probably happen in a language I'm more comfortable with - Node or Python.  In any case, this should get it down to something that fits comfortably in github pages limits, so I'll publish it there as a "static API" similar to RubyRed's.

[Progor]

Alright, I went ahead and did the defaults and split into categories.  This made all the files around 1MB or less (a few big ones up to 5MB), which is much more doable for a web app that's pulling in around 5-10 files to get the primary, secondary, pools, boosts, and pets for  a single build.  If you're going to do much more than this you probably just want to load the 41MB powers.json with everything in it, including the powers possessed by Hellions in the Sewer trial.  Maybe I'll make a powers.Player.json that has everything accessible to a player.

 

There's also the powers.default.json.  This file has the default values for any value that shows up in more than 20% of the powers.  At each level of this object, the "sub" field represents the defaults of the object contained in that field.  For instance: sub.effects.tag tells you the default value that should go in effects: [ { tag: ...} ].  This structure is more deeply nested than I thought (effects can have effects of their own), so I used a recursive function to find the defaults, and you'll probably need one to rebuild the original object from the defaults if you want to.

 

Here's a shot of the files that are output (197 total).  Feel free to look around in the github project, or you can access them API-style like this:

 

https://dpwhittaker.github.io/sif_parser/powers/Tanker_Melee.json

 

I'll work on the archetypes and powersets next to act as an index into these files.  I'll probably end up adding some folder structure here, so don't start building anything on this yet 🙂

 

 

edit: removed the powers. prefix and moved them into a powers folder.

Edited March 8, 2020 by Progor

[_NOPE_]

Great work @Progor

[Olly]

@Progor and @Sif, thanks for this!!  

 

Question: I am interested in the powers.json, and specifically the "Boosts" (aka Enhancements) info.  I see it's in @Progor's .json file in the following format:

 

Spoiler

"full_name": "Boosts.Attuned_Will_of_the_Controller_C.Attuned_Will_of_the_Controller_C",
    "crcfull_name": 2261644448,
    "source_file": "DEFS/POWERS/BOOSTS_ATTUNED_WILL_OF_THE_CONTROLLER_C.POWERS",
    "name": "Attuned_Will_of_the_Controller_C",
    "source_name": "Boosts.Attuned_Will_of_the_Controller_C.Attuned_Will_of_the_Controller_C",
    "system": 0,
    "auto_issue": false,
    "auto_issue_save_level": false,
    "free": false,
    "display_name": "Will of the Controller: Endurance/Recharge",
    "display_help": "UNIQUE -- No more than 1 enhancement of this type may be slotted by a character. Reduces the endurance cost of a power by {Boost.Attrib.Endurance.Scale}% and reduces recharge time by {Boost.Attrib.RechargeTime.Scale}%. This enhancement can only be slotted by Controllers and may be slotted into most control powers.",
    "display_short_help": "Reduces Endurance Cost and Increases Attack Rate",

 

Does anyone know where and how to get the actual data, for example, for max level of each IO?  I assume it's related to the "{Boost.Attrib.[SomeAttribute].Scale}%"  I'd be happy to search for it myself but also happy to download a file in case it's already been parsed.  

 

Thank you again!!

[Sif]

20 minutes ago, Olly said:

Does anyone know where and how to get the actual data, for example, for max level of each IO?

Check out the min_slot_level and max_slot_level values. One thing to note is that the Attuned version of the enhancement is distinct from the "Crafted" version, they're different "powers".

 

How much the enhancement does would be under Effects, with the "value" being effects.attrib_mod.scale, multiplied by something (at least normal powers would use the values in your associated Class, not investigated to see what things like enhancements would use, possibly something in there also).

Edited March 10, 2020 by Sif

[Number Six]

On 3/8/2020 at 4:00 PM, Progor said:

This structure is more deeply nested than I thought (effects can have effects of their own)

[Progor]

4 hours ago, Olly said:

@Progor and @Sif, thanks for this!!  

 

Question: I am interested in the powers.json, and specifically the "Boosts" (aka Enhancements) info.  I see it's in @Progor's .json file in the following format:

 

  Reveal hidden contents

"full_name": "Boosts.Attuned_Will_of_the_Controller_C.Attuned_Will_of_the_Controller_C",
    "crcfull_name": 2261644448,
    "source_file": "DEFS/POWERS/BOOSTS_ATTUNED_WILL_OF_THE_CONTROLLER_C.POWERS",
    "name": "Attuned_Will_of_the_Controller_C",
    "source_name": "Boosts.Attuned_Will_of_the_Controller_C.Attuned_Will_of_the_Controller_C",
    "system": 0,
    "auto_issue": false,
    "auto_issue_save_level": false,
    "free": false,
    "display_name": "Will of the Controller: Endurance/Recharge",
    "display_help": "UNIQUE -- No more than 1 enhancement of this type may be slotted by a character. Reduces the endurance cost of a power by {Boost.Attrib.Endurance.Scale}% and reduces recharge time by {Boost.Attrib.RechargeTime.Scale}%. This enhancement can only be slotted by Controllers and may be slotted into most control powers.",
    "display_short_help": "Reduces Endurance Cost and Increases Attack Rate",

 

Does anyone know where and how to get the actual data, for example, for max level of each IO?  I assume it's related to the "{Boost.Attrib.[SomeAttribute].Scale}%"  I'd be happy to search for it myself but also happy to download a file in case it's already been parsed.  

 

Thank you again!!

There's also some additional files that I don't think any of the projects have parsed yet.  One is boostsets.bin.  I may try to reverse engineer that one myself based on ourodev code, but I don't have access to homecoming code, so I'm at a disadvantage.  I'm not sure what information it will or won't provide.

[Progor]

3 hours ago, Number Six said:

At least the recursion depth is limited by the size of the file, or I may have never emerged from the abyss!

[RubyRed]

1 hour ago, Progor said:

There's also some additional files that I don't think any of the projects have parsed yet.  One is boostsets.bin.  I may try to reverse engineer that one myself based on ourodev code, but I don't have access to homecoming code, so I'm at a disadvantage.  I'm not sure what information it will or won't provide.

boostsets doesn't look all that complicated, comparatively speaking. Looks like it contains the sets linked to the boosts that are in them as well as the set bonuses.

[Olly]

I have to admit I don't quite understand exactly where I'm supposed to look, but I'll hang around the thread and keep an eye out for more info. Not a dev, so I really couldn't use it for anything except my personal learning.  But man, is it fun!

 

[Sif]

3 hours ago, Progor said:

One is boostsets.bin

I've done a bit of work on this, I have it partially parsing, but ran into some peculiarities. I'll poke at it some more another time.

[Progor]

I started trying to parse boostsets as well, and found an awesome tool to help:

 

https://ide.kaitai.io/# part of the https://kaitai.io/ kaitai-struct project.

 

It gives you a DSL made specifically for parsing binary files, and more importantly, an IDE that highlights the section of the binary file that each parse covers.  Once you've got the file defined in their simple YAML format, it generates code in a bunch of languages to do the actual parsing.

 

The only difficulty is the parser output is pretty verbose, mentioning every array size, string length, and offset pointer you had to use to define the file.  I fixed this by naming the important variable at each level "value" and replacing the parent object with the value field whenever I see it.

 

I've got boostsets and attrib_names (wanted to start with something easy to evaluate the tool) in a new project:

 

https://github.com/dpwhittaker/coh-parse7

 

With the following "API" available so far:

 

https://dpwhittaker.github.io/coh-parse7/boostSets.json

https://dpwhittaker.github.io/coh-parse7/attribNames.json

 

I'll probably do client messages in this format next, then try to tackle powers to see if it ends up being even more straightforward than the rust implementation.  The easier it is to find and fix format changes after a release, the faster tools that rely on it can get updated.

[Sif]

I've pushed 0.5.2, which includes support for boostsets.bin using the BoostSet struct.