Making numbers more visible

[Psyonico]

I’ve gone down a rabbit hole with this one…

 

So, uberguy is MIA so there is no foreseeable update to City of Data 2.0

 

i has the idea of potentially trying to re-create CoD a 3rd time with the little bit of open source code that still exists… then it hit me:

 

we have devs that are not tied to a for-profit studio anymore.  Rather than try to reverse engineer and parse hex code data that is in the various .bin files, can we just get a view of the tables that the dev team uses to manipulate powers before they’re compiled?

 

My research tells me that on live, all of this data was in spreadsheets that were then exported to .def files for testing before finally being compiled into .bin files and compressed into .pigg files.  

I suspect that a similar process is being used now.  So, my suggestion is: can we get a site that has an export of these spreadsheets into table format?  Then players will always have the most accurate data available for items like arcanatime and other fields that are not available in game.  I am not suggesting a complex site setup, simply access to a database where some intrepid player could, if they wanted, create a complex site and just pull the data.

 

Anyway, lots of rambling, and I don’t have high hopes for this, but one can dream.

[Cipher]

Quote

My research tells me that on live, all of this data was in spreadsheets that were then exported to .def files for testing before finally being compiled into .bin files and compressed into .pigg files.  

I suspect that a similar process is being used now.

 

For better or worse, we don't use a sheets-driven approach like what may have been done earlier into City's development; we work directly with the domain-specific text representation that those sheets were originally compiled to which are not easily understood if you are not a powers developer, and have a tool which parses and generates these files with a GUI which some on the team use. Unfortunately, this means that it is not as simple as just publishing sheets somewhere for user consumption.

 

That said, it was a past desire of mine to parse and build a version of some of the powers data in a format that is easier for the community to work with for purposes like keeping Mids' up to date, but it's not a priority at the moment and would be unlikely to happen in the near future.

[Psyonico]

Thanks so much for the reply!

 

sounds like you guys had to reverse engineer the system, which is how it looks like it was being done with CoD.  I’m assuming the powers team is looking at the same hex format that I came across (I haven’t actually been able to access the actual data yet because the code I found isn’t working right now and it was written in Rust, which I have no experience in)

[Jacke]

@Psyonico, the best people after the HC Powers devs to talk with about all this stuff are the MRB staff:  @Felis Noctu, @Metalios, and @Zed.  They are very irritated with UberGuy, as his work building on the people I mention below is all hidden away closed-source.  And the CoD2 website isn't really walkable by any automated tool (like wget or curl) to extract its data.  Last update to CoD2 was done in 2024 July because of pressure from an HC dev.  And UberGuy is now walkabout.

 

The only open-source stuff on it (besides .json files here and there) is the parser @RedTomax used for the original CoD.  And what @RubyRed did, which I think is the parser in Rust.

 

 

[Psyonico]

Yes, I found the parser in Rust, and from what I found on the way back machine (the blog post doesn’t exist anymore) the Rust version i came across is designed for i25 forward, whereas the red to max version works for pre i25 data only.  Something changed in i25 that required an update to the parser.

 

unfortunately, when I try to run the parser, one of the .bin files throws an error about the data chunk size not matching what is expected.

 

i have never worked in Rust and the only programming I’ve done in the past 20 years has been in SQL and VBA, neither of which involved the kind of pointer searching or hex code parsing that this requires.  Maybe 20 years ago I could have worked my way through it when Java and C++ were still fresh in my mind, but not today.

[Michiyo]

6 hours ago, Jacke said:

@Psyonico, the best people after the HC Powers devs to talk with about all this stuff are the MRB staff:  @Felis Noctu, @Metalios, and @Zed.  They are very irritated with UberGuy, as his work building on the people I mention below is all hidden away closed-source.  And the CoD2 website isn't really walkable by any automated tool (like wget or curl) to extract its data.  Last update to CoD2 was done in 2024 July because of pressure from an HC dev.  And UberGuy is now walkabout.

 

The only open-source stuff on it (besides .json files here and there) is the parser @RedTomax used for the original CoD.  And what @RubyRed did, which I think is the parser in Rust.

 

 

Uber *IS* working on updating CoD, there were some power struct changes he's having to work around.  The last update before P2 that effected powers *WAS* in 2024, as has been mentioned elsewhere.

[Troo]

10 hours ago, Jacke said:

And UberGuy is now walkabout.

 

3 hours ago, Michiyo said:

Uber *IS* working on updating CoD