City of Data v2.0

[Bill Z Bubba]

14 hours ago, Captain Powerhouse said:

Don't try to figure out why Claws does what Claws does. It uses it's own formula that has been lost to archeology.

I blame Castle, BAB and Arcanaville.

[Galaxy Brain]

 

 

Edit: 

 

Awesome job on this!!! Only thing that would be awesome to add is a dropdown or even page with all the Power Categories listed out. For example, I can get to here:

 

 

But I do not see an easy way to get a list of all Power Categories if I were to look up various enemy groups?

Edited February 1, 2021 by Galaxy Brain

[UberGuy]

25 minutes ago, Galaxy Brain said:

But I do not see an easy way to get a list of all Power Categories if I were to look up various enemy groups?

I don't understand what you mean. Power Categories don't map directly to enemy groups in any consistent / obvious way. I mean, some do, by virtue of obvious names, but not all. The only way to really know is to look at an entity.

 

Edit: but look upthread for a way to get a list of everything.

Edited February 1, 2021 by UberGuy

[Galaxy Brain]

Just now, UberGuy said:

I don't understand what you mean. Power Categories don't map directly to enemy groups in any consistent / obvious way. I mean, some do, by virtue of obvious names, but not all. The only way to really know is to look at an entity.

I'm referring to looking up NPC powers similar to how this is organized:

 

 

 

Right now, I can search for enemy groups, which can bring up a category:

 

  > 

 

 

 

 

Which based on there being a page seems like there is a directory where they can be listed out?

[UberGuy]

PCs and NPCs aren't assigned their powers in the same ways. Each NPC entity gets its own list of powers, powers and powercats it can use. You can't build a directory like that for NPCs in a consistent way like you can for players.

 

Those categories you can see are "bags" of powers that a given group's powers will probably be found in, but they can have powers from other powercats all together.

Edited February 1, 2021 by UberGuy

[UberGuy]

23 minutes ago, Galaxy Brain said:

Which based on there being a page seems like there is a directory where they can be listed out?

But as mentioned up thread, for a raw list of all the categories... 

On 1/31/2021 at 11:30 AM, UberGuy said:

If you can work with just a list (which you could use to input into the search box or, with a wee bit of figuring out, the address bar), use this.

 

[Galaxy Brain]

12 minutes ago, UberGuy said:

PCs and NPCs aren't assigned their powers in the same ways. Each NPC entity gets its own list of powers, powers and powercats it can use. You can't build a directory like that for NPCs in a consistent way like you can for players.

 

Those categories you can see are "bags" of powers that a given group's powers will probably be found in, but they can have powers from other powercats all together.

I gotcha, that makes sense! Thanks for the link too, I must have missed that going through the thread!

[Number Six]

On 1/31/2021 at 2:23 PM, Bopper said:

It could be. Powerhouse can probably clear that up for us. I used to think this was a common thing back when I'd see Null effects, I thought those were old effects that got removed but had to remain for who knows why (makes database happy? I dunno). But I found out recently those are possibly there for animation purposes. I don't know if this would be a case of that. Probably not. It definitelly seems like an oversight, but then I see PvP having the delayed damage effect and wonder why there's a discrepancy.

The Cryptic devs had a rule that attribmods should never be removed from a power, so in cases where an effect was removed and there was nothing to put in its place, things were often changed to kNull or a 0% chance.

 

While I don't have complete information from anyone with firsthand knowledge, I can speculate with some degree of certainty as to the genesis of that rule.

 

There are a number of places in the code that directly use an index to the power's attribmod array as a shortcut. Even in the modern framework with nested effect groups, the engine creates a synthetic flattened list of attribmods for compatibility with that code. It's used for a number of things such as the network protocol for sending buff icons, but the one relevant to this rule is that references to that index are persisted in the database for long-duration buffs/debuffs that last across play sessions.

 

That means that if order of the mods in that list changes between versions, when the server is started up, as players load in they can end up getting attribmods attached based on indexes from stale data. In the most benign case, later mods in the same power are shifted up the array, and the index ends up pointing at the wrong mod. That would manifest as an unexpected effect, with the magnitude and duration from before the character logged off, but the type of the effect switched to whatever is now in that slot.

 

A much worse version happens when the list is shortened and the index now points off the end of the array. That's almost certain to cause a mapserver crash.

 

Strictly speaking, I don't think that rule is needed anymore. We ran into those issues a couple of times since HC went live, because power developers were not aware of the rule and removed effects from the middle of the list. The details are a little fuzzy, but I remember one instance where a fairly common inherent power had saved mods that changed types across releases and caused weird issues. Later there was a case of a small number of unlucky characters (I want to say one VEAT in particular) that would cause a map crash when they tried to log in after a patch.

 

As a result of those issues, we added code to the dbserver to clear saved attribmods on server startup. That means that long-running buffs still last when you log out for a while, but they don't persist across server restarts. That makes the rule obsolete and those can be removed.

[Bionic_Flea]

Uber, thanks for all the time and effort you put into this.  And it seems like it was a lot of both.

[RubyRed]

This is some really spectacular work, @UberGuy. So happy to see something cool and useful for the players come out of the reverse engineering I did.

[UberGuy]

1 hour ago, Bionic_Flea said:

Uber, thanks for all the time and effort you put into this.  And it seems like it was a lot of both.

It was. But the good news was it was something I wanted to do, for a variety of reasons.

 

Foremost among them was the fact that I personally missed having a working CoD a great deal. I used it a lot, even after it fell behind. But in the last year or two it became so far behind that even if the old site still worked, it would be wrong a lot of the time. Combine that with the fact that it was written in an unsupported version of PHP not compatible with any forums currently associated with CoH, and it was well and truly dead in its old form.

 

Next was the fact that I saw doing this an an excuse to learn (a) Rust and (b) modern JavaScript. Rust is what RubyRed's bin file parser tooling is written in, and JavaScript is, of course, largely unavoidable when writing a non-trivial web page.

 

I'm an IT person by profession (specifically, a Site Reliability Engineer - no, not at Google, they just "invented" the discipline as its widely recognized). Most of my programming skills are in languages used by Linux system admins, like Python and Bash, with some languages from my past and current jobs, like Java and C/C++. Rust is an up-and-coming language I was interested in. JavaScript is a consistent top-rank language mostly due to its ubiquity in web-based applications, which have, of course, taken over the world. It's been years since any of my work projects required me to understand JavaScript, and it's evolved a ton since then. Actually designing a web site from the ground up would certainly require me to catch up with it.

 

There were a lot of iterations of learning to do. I had to not just learn what the Rust code was doing, you know, because I didn't know Rust. I had to figure out what the program was doing, because I knew very little about how to read the powers bin files. Progress between those goals would ping pong back and forth - learning how Rust works would help me understand what sections of code were doing, and figuring out what code was doing would help me understand what certain constructs in Rust meant.

 

Once I had a decent grasp of how the Rust code worked, I started tinkering with it. I wanted certain outputs it produced in different formats, for various reasons that depended on the thing. I did make a variety of changes, initially small but more complex over time. But I quickly learned that adding all the changes I was envisioning was going to take me longer than I wanted. Mostly, that comes down to me not being a strong Rust coder. I knew I could do what I wanted easily in Python, even though it would not be as performant as Rust code. So I had a choice - spend the time learning to code faster in Rust, or invest in one burst of Rust learning to create a bridge between Rust and Python (for which there was existing support). I never considered ditching the Rust code - it did too much too well.

 

I opted for the Python bridge, allowing me to treat the Rust library as a Python module that would expose iterators I could use to parse all the things. It took me a good week to build this, but then I could write Python code like this:

# This thing is Rust code but can be loaded into Python as a library
from coh_bin_parser import parse_bins

# The bin files are read here
coh_data = parse_bins.parse()
for powercat in coh_data.all_powercats():
	for powerset in powercat:
		for power in powerset:
			# do useful stuff

I'm glossing over a lot of detail, but hopefully that makes some sense - one of Python's claims to fame is it is relatively human-readable.

 

Make no mistake - the processing time for doing a lot of the power handling in Python instead of Rust is at least an order of magnitude longer (as in slower). But my speed in writing the code in Python was probably weeks faster.

 

There's another benefit of having the data in Python. It's very easy (and pretty fast) to dump Python objects and reload them later. After I parse a release of the game just once, I can load all the data into an interactive Python session and search it. @Boppercan attest to how often this comes in handy for things like "find all / any examples of powers that X".

 

On the front-end side I needed to learn not just JavaScript, but brush up on HTML and CSS, and learn about a few new browser features. I also ended up learning a JS web UI framework (Vue.js) - you don't want to reinvent the wheel when people have built sports cars for you. But "car" is a good analogy - there are a ton of moving parts and learning to use such a framework is a whole adventure of its own. (Props to @Leandrofor pointing me at Vue.js as the strong option with the lower learning curve than the main alternatives.)

 

Basically, I probably increased my job marketability and salary potential significantly by engaging in something I enjoyed doing anyway, will use extensively while CoH-ing, and helps other people too.

Edited February 2, 2021 by UberGuy

[RubyRed]

3 minutes ago, UberGuy said:

an an excuse to learn (a) Rust and (b) modern JavaScript. Rust is what RubyRed's bin file parser tooling is written in

Oh jeez, I'm so sorry... I hope you didn't use my bin parser as a model for Rust programs, I literally just vomited that code out as I went along specifically because I didn't feel like overthinking or planning ahead. I've written far more coherent code in my lifetime, I swear 😆

[UberGuy]

7 minutes ago, RubyRed said:

Oh jeez, I'm so sorry... I hope you didn't use my bin parser as a model for Rust programs, I literally just vomited that code out as I went along specifically because I didn't feel like overthinking or planning ahead. I've written far more coherent code in my lifetime, I swear 😆

You commented it pretty well IMO, so places you were being "non-Rust idiomatic" seemed obvious to my untrained eyes. Plus, you'd probably be moderately horrified at how I did some of the serialization I added. 😂

 

Edit: I actually watched some on-line courses on Rust, so your code wasn't my only learning source. I'm not so masochistic that I try to learn a programming language Thirteenth Warrior style!

Edited February 2, 2021 by UberGuy

[Bionic_Flea]

Can we get a link to new COD on the top line like the Wikis?

[Bopper]

1 hour ago, Bionic_Flea said:

Can we get a link to new COD on the top line like the Wikis?

I think that would be a @Jimmyor @Cipher question. It certainly would be good to have.

[UberGuy]

I made a variety of bugfixes / feature updates today. The main ones that affect live Homecoming powers relate to CoD not previously displaying information specific to powers with "Chain" areas of effect. There are a few attributes unique to these powers which weren't displayed. This included expressions for dynamic chain behavior, like targeting priority and "chain effectiveness" (which modifies the strength of an effect based on the number of targets it has jumped to.) The simpler chain-related values are shown at the bottom of the power's "Activation Details" in a new section for "Chain Attributes".

 

Also missing was display of max target expressions, which dynamically affect how many max targets a power can effect. This does come into play for a number of chain powers, but is not chain specific.

 

You can see an example of these on this power. A number of Electrical Affinity powers are chains, so this update probably really makes powers in that powerset make more sense.

 

All of the new expressions are "whole power" attributes and can be quite long, so I display them above the tables, in the same place power "Requirements" go.

Edited February 8, 2021 by UberGuy

[Grindingsucks]

@UberGuy This is incredibly useful.  Thanks very much!  It's what I used to primarily use Mids for, actually; looking up more detailed and complete explanations of power effects, rather than planning builds.  This has been extremely useful for understanding how various powers function in the game! 

[Kala]

First off, thank you so much for doing this! It's really great to have a web resource for this stuff, especially for those of us who don't have easy access to Mids 100% of the time and didn't otherwise have a resource at all. I'm genuinely excited that this exists now and I can already tell it's going to be incredibly useful for me.

Do you have any sort of raw data or source code that you're willing to or plan to make available? I haven't reliably been able to find comprehensive mob power and resistance data before now, and I think it would be possible to do some across-the-board analysis regarding things like the efficacy of damage types or defenses, or calculating how well any given powerset matches up against various mob groups. It seems like your data would be set up to be able to answer those questions programmatically, but I'm not seeing a great way to do that, short of using your internal endpoints as a makeshift API, which doesn't seem ideal for either side of things.

[Dispari]

Yay City of Data is back! And my wife helped! I'm so excited.

Data has always been important for getting intricate details that even Mids won't show you, pet stats and powers, or badguys powers, being good examples.

We've really come a long way. Does anyone remember the super old character planner that used Brawl Index as a measure of damage? (Pepperidge Farm remembers)

[UberGuy]

On 2/8/2021 at 7:14 AM, Kala said:

Do you have any sort of raw data or source code that you're willing to or plan to make available? I haven't reliably been able to find comprehensive mob power and resistance data before now, and I think it would be possible to do some across-the-board analysis regarding things like the efficacy of damage types or defenses, or calculating how well any given powerset matches up against various mob groups. It seems like your data would be set up to be able to answer those questions programmatically, but I'm not seeing a great way to do that, short of using your internal endpoints as a makeshift API, which doesn't seem ideal for either side of things.

Regarding source code: TBD. Bear in mind that CoD is not something I was able to keep up-to-date with recent changes to CoH in a vacuum. Whether its source code can be made public may be complicated by the same constraints that affect Homecoming's own source code. I am feeling out what constraints may exist, if any.

 

Unrelated to any legal constraints, CoD's current back-end tooling is not set up in a way that makes it consumable by the vast majority of normal humans. Probably the biggest reason for that is that I do my development on Linux, not Windows. Both Python and Rust work fine on Windows, so it can unquestionably run there, but I almost certainly need to rework some of the tools I use  (for things like building new data bundles and pushing code update the site) for them to work on Windows. A number of things are currently bash scripts, for example. Making a tool like this easy for non-dev/non-admin types to use is actually a lot of work and creates its own support burden, so even if I end up free to do so, it will not happen for a while.
 

As far as exporting critter data, that's another area where I'm asking for Dev feedback. It's one thing to let people know how their own characters' powers work, which is the primary focus of the site. Because old CoD also exposed NPC powers in its drop downs, I felt free to do so as well. And I obviously expose information about critters which players can summon, since that is a subset of "how their own powers work". However, I am sensitive to the devs opinions about how easy it should be to know everything about totally arbitrary critters. I very much share the feels when it comes to things like understanding critter resistances or mez abiltiies, and see great value in the community being able to research such things independently. However, I also understand that this is rather spoilery, and our current devs are adding new critters with new capabilities. Because of this, I have asked for the devs' opinions on this before I enable whole-hog everyone-can-see-behind-the-curtain features in CoD.

 

Re: using the site as an API - my default position is "please don't". While hosting it is not expensive, I do pay for its use based on traffic. Well-behaved access to the data is not a problem, but I know from experience how easy it is to accidentally be a very bad actor when "scraping" another site.

 

On 2/8/2021 at 9:45 AM, Dispari said:

 Does anyone remember the super old character planner that used Brawl Index as a measure of damage? (Pepperidge Farm remembers)

I actually still have a copy of it laying around - both the installer and an installed version in an imaged copy of my game machine at the time. I was using it in the first few weeks after the initial game release.

[Bill Z Bubba]

1 hour ago, UberGuy said:

I actually still have a copy of it laying around - both the installer and an installed version in an imaged copy of my game machine at the time.

You're a smart human. I'm still kicking myself for losing WAY too much CoH info during a less than careful system rebuild.

[UberGuy]

I got a big ole green light on critter data, so look forward to UI support for that in a future update. I'm thinking weeks, not days, just because I have to work on this in my spare time, but I don't expect it to be at all technically hard to add.

[UberGuy]

3 hours ago, Bill Z Bubba said:

You're a smart human. I'm still kicking myself for losing WAY too much CoH info during a less than careful system rebuild.

On the other hand, it doesn't seem very useful to have it, as it doesn't seem able to run on a modern Windows environment. I should try firing it up in an XP VM, just for giggles.

Hero Planner 1.51. Last modified date 4/24/2004. 😵

I also have a copy of SherkSilver's Hero Builder, circa 2/24/2006. This actually still works.

 

And yes, the day of the month on those really does match.

 

Edit: Oh snap, I was wrong. It does work. They both do.

 

Edited February 9, 2021 by UberGuy

[Bionic_Flea]

I remember Lil Pips' Hero Builder.  I think that was the original. 

[roleki]

So nice to have insight into what pseudopets are up to again; also interesting to see the Rube Goldberg stuff related to newer sets like Electrical Affinity.  Good show, @UberGuy!