City of Data v2.0

[BillyMailman]

On 8/7/2022 at 12:11 AM, BalrogLord said:

Bug report: all the End Cost data for Toggles is wrong, or at least a lot of it.   😞

It's actually completely correct, but you need to know to also look at the Activate Period. Toggles don't actually have and Endurance-Per-Second cost like the game displays; they have a fixed endurance cost that they remove from you every time they activate, same as click powers. But toggles activate themselves over and over, and the Activate Period tells you how often.
 

So, grabbing a couple random examples, Tough shows in CoD as costing .163 endurance, and having an activate period of .5s. That means it will drain 0.163 endurance twice a second, totalling 0.326 end/s. On the other hand, something like Tanker Dark Armour's Death Shroud will drain the listed 1.04 endurance once every 2s, meaning 0.52 end/s.

Edited August 9, 2022 by BillyMailman

[BalrogLord]

Ok thanks, makes sense. However then I think CoD should also display somewhere the end/sec for toggles, since

that's what the game shows, and how people compare them?

[CaptainLupis]

14 hours ago, BillyMailman said:

It's actually completely correct, but you need to know to also look at the Activate Period. Toggles don't actually have and Endurance-Per-Second cost like the game displays; they have a fixed endurance cost that they remove from you every time they activate, same as click powers. But toggles activate themselves over and over, and the Activate Period tells you how often.
 

So, grabbing a couple random examples, Tough shows in CoD as costing .163 endurance, and having an activate period of .5s. That means it will drain 0.163 endurance twice a second, totalling 0.362 end/s. On the other hand, something like Tanker Dark Armour's Death Shroud will drain the listed 1.04 endurance once every 2s, meaning 0.52 end/s.

I'm guessing it's probably just a typo, but I think that should be 0.326 end/s for your example. Normally I wouldn't bother mentioning for something like that, where I'm fairly certain it's only a typo, but I think there is enough things in the CoD that confuse some people that it was worth flagging it for you here.

[Luminara]

12 hours ago, BalrogLord said:

I think CoD should also display somewhere the end/sec for toggles, since

that's what the game shows, and how people compare them?

 

That's what Mids' is for.  City of Data is a different tool.

[UberGuy]

I don't think displaying the end cost / time of toggles is a bad idea, but I definitely wouldn't replace the existing info. I'd probably add a derived "endurance burn" value somewhere that only appeared for toggles.

[UberGuy]

I've made some back-end updates that, right now, are only visible on the Alpha data. Most of it has to do with "requirements" expressions.

 

As I've posted about in the past, CoH stores the logical and math expressions found in power, critter and other data in what's called postfix notation (AKA "Reverse Polish Notation" or "RPN").  This is an efficient, straightforward way to compute things, which matters a lot in places like attribmods of powers, which can cause many hundreds of evaluations per server "clock tick". However, it's not very friendly for most people to read. Those of us used to math or logical expressions are used to this:

 

(a + b) - (c + d) / 4

 

not this

 

a b + c d + 4 / -

 

Now, CoD has long handled this conversion. It's something I spent a lot of time on early in the site's life. However, a general postfix-to-infix conversion is not exactly easy. You can find code that does it for simple cases (that handle "binary" operators like +, -, * and /) very easily with a good search engine. Code that gracefully handles things like unary operators (like negation or logical "not"), and proper emission of parenthesis are harder to come by.

 

Early on, my approach to handling this was to use a symbolic math library called SymPy. This has functionality for simplifying expressions, which was a handy way to deal with the fact that my postfix-to-infix conversions were simplistic and so often produced visually ugly infix expressions. For example

 

Input: a b + c + d + e +
Output: ((((a + b) + c) + d) + e)

 

Using SymPy to simplify expressions often provides a nice side benefit of taking expressions that make sense as efficient postfix operations, but render as ugly compound infix expressions. My goal in displaying expression info on CoD is for players to (hopefully) be able to comprehend what the expression does, not to preserve its native operation flow. SymPy's simplication logic usually makes expressions easier to mentally digest, combining repeated terms, reducing numeric constants to their simplest forms, etc.

 

However, SymPy isn't able to do everything I want. First and foremost, it refuses to process otherwise valid expressions that combine boolean logic and regular arithmetic. For example, this is a valid CoH expression.

 

(a || b || c) * 1000

 

That says "multiply 1000 times (if a or b or c)". Said out loud like that, it doesn't make much sense. But CoH's expression parsing engine is written in C, where "boolean" values are really integers. "True" is 1 and "False" is 0, so the expression above really means "1000 if a or b or c, else 0". It's actually quite an efficient way to express a certain subset of  "X or 0" values. But SymPy, written in Python (where True and False are not integers) understandably refuses to process such constructs. When this happens, I fall back to the base postfix/infix converter.

 

In such cases I lose not just the nicety of having SymPy rearrange grody expressions, but also it cleaning up after my wildly-nested parenthesis - something that often look particularly bad in long boolean expressions.

 

What's changed is two things.

1. I finally undertook improving the base converter, giving it the ability to emit the least parenthesis for a correct infix form of a postfix expression.
2. Now that the "base" converter now emits nicely formatted infix expressions, I could see that in a lot of cases they're more nicely formatted than the raw SymPy version.

SymPy still does extremely nice things my converter does not, usually making complex expressions easier to read. However, it does two things I don't like.

1. It writes some things without spaces that I would prefer it spaced out. It writes "a + b" exactly like that, but writes "a * b" as "a*b" (no spaces).
2. It follows Python operator precedence rules, which have some small differences from C, meaning it omits some parenthesis that could change the meaning of some expressions. This is an edge case, but once I saw it happening, I couldn't unsee it.

Here's what the converter used to do:

1. Convert postfix into "raw" infix
   1. If the conversion fails, emit the raw postfix as a string that says it couldn't be parsed
2. Mutate the infix expression into valid Python/SymPy expression that still preserves its meaning
3. Simplify the mutated expression with SymPy
   1. If SymPy cannot parse the expression, emit the "raw" infix expression
   2. Else emit the simplified infix

Here's what it does now:

1. Convert postfix into "raw" infix
   1. If the conversion fails, emit the raw postfix as a string that says it couldn't be parsed
2. Mutate the infix expression into valid Python/SymPy expression that still preserves its meaning
3. Simplify the mutated expression with SymPy
   1. If SymPy cannot parse the expression, emit the "raw" infix expression
   2. Else
      1. Convert the simplified infix back into postfix. Note that this still preserves changes SymPy made to the expression
      2. Convert this post-processing postfix back into infix
      3. Emit the final infix

The last 3 steps above may seem like madness, but they let me apply the spacing and parenthesis grouping rules I choose instead of the ones that SymPy chooses, while still retaining the benefit of other, more significant expression reductions that SymPy applies.

 

Important Note: I don't do much processing of expressions in CoD, and what I do process is done exclusively in the back end. (That's to say the website does not parse any expression data on-the-fly.)  That said, I definitely do some parsing of expression data on the back-end, and when I started making these expression changes I immediately broke a few of them. In every case this was a sign that I was doing something wrong in the new code, not just that the layout of an expression changed slightly. After I got everything working without errors I checked over the modified Alpha data pretty carefully to try and ensure that only expected things changed, but I definitely might have missed something. If you see anything weird, let me know!

 

Note for pedants: The different postfix/infix conversion stages are not exactly the same, since I have to account for the fact that SymPy and C have some different operator precedence, and so have different parenthesis wrapping rules for their infix expressions. Mostly all that changes is a map of operators to their priority, with just a few being different between stages. For the terminally curious, what's different there is that SymPy has higher priority for equality comparisons ("==" and "!=") than for boolean algebra operators ("&&" and "||"), which are themselves higher priority than the inequality comparison operators (">", ">=", "<", "<="). This is different than C, where "&&" and "||" outrank all the comparison operators, and is also actually different from what's documented for the Python language (where all equality and inequality operators have equal priority) which is what SymPy's docs say it follows.

 

Edited August 11, 2022 by UberGuy

[BillyMailman]

So, I have a request. Dunno how useful it will be to other people, dunno how hard it is, so feel free to ignore it. And definitely back-burner it if the new update's a pain at any point. But basically, I'd like the Group Name thing to be a clickable link, to some sort of page listing Entities in the group.

 

I think CoD has the info needed to be able to list things like "At level X, this group can spawn the following random enemies:", which would be amazing, but honestly just a big page listing 'em all like the Tag page, would be great.

[CaptainLupis]

5 hours ago, BillyMailman said:

So, I have a request. Dunno how useful it will be to other people, dunno how hard it is, so feel free to ignore it. And definitely back-burner it if the new update's a pain at any point. But basically, I'd like the Group Name thing to be a clickable link, to some sort of page listing Entities in the group.

 

I think CoD has the info needed to be able to list things like "At level X, this group can spawn the following random enemies:", which would be amazing, but honestly just a big page listing 'em all like the Tag page, would be great.

I dare say it could be done, but I think it would be a lot of work for a feature not many people are likely to use when that kind of information is already available. A hyperlink to the Unofficial homecoming wiki page would likely be simpler and accomplish the same thing if it was required. https://homecoming.wiki/wiki/Council#Villain_Types In the subgroups it tells you the levels they spawn at.

[UberGuy]

I'll definitely look into it, because it doesn't seem like it should be hard to do. When I generate the site I have all this stuff in memory at once, and I walk all the lists of things like powers and entities one at a time to export them each as their own files. So it's usually relatively easy to build indexes of stuff like that as I go.

 

And hey, I'm working on new features anyway. 

[Bionic_Flea]

10 minutes ago, UberGuy said:

And hey, I'm working on new features anyway. 

 

[UberGuy]

One small one went live tonight - the "burn rate" stat for toggles.

 

 

For powers that are toggles with a non-zero endurance cost and a non-zero activate period, you get that parenthetical display of cost / interval divided by intervals / sec.

[UberGuy]

I am updating the site for Page 4. This release has relatively minor data changes so the bin parser required some small updates, already made based on the Alpha server.

 

I was also in the middle of significantly rearranging some data structures around boosts (enhancements). This was all boosts, not just set pieces.

 

To try and summarize what I was up to: boosts are just powers. A damage SO has all the core data attached that, say, a blaster blast or tanker armor has. Powers are usually the largest data structures in CoH. but powers that are boosts usually have, much fewer interesting bits of info that we really care about than non-boost powers. I wanted to expose this boost-specific data in multiple places, like both the powers page and the boostset page, without having to load each boost's whole power object to get at it. So I started breaking the boost-specific stuff out into a smaller object I could nest into the larger power dataset but also pass around on its own.

 

One way I want to use this is to improve how boosts look when displayed as powers. A lot of boost specific info currently has no specific visual display, and the only way to learn about it is to look at the JSON data. Clearly this can be better, giving viewers more "enhancement specific" details more readily when the power they're viewing is, in fact, a boost.

 

When I added boostsets to CoD, I added a bunch of code to handle boosts and to map their raw data to the ways we usually talk and think about enhancements. Think stuff like how "Heal" enhancements boost multiple attributes (Healing, Regeneration and MaxHP) and so on. But because I did it to support boostsets (IOs) I tangled the concepts of boosts and boostsets together in the code. For example, only boosts that were in sets actually got any extra inspection to map out what they enhance.

 

Set pieces (IOs, ATOs, etc.) are not the only fancy enhancements around. Think about the GR "prestige enhancements" given out at character creation. And someday maybe more things like that could exist. To make sure things like that are processed for nice display, I needed to break boost and boostset concepts apart in the code. Boosts that are in sets still get extra info that "plain" boosts don't (like the name of the set it belongs to, etc.) but anything else needed to be inspected on every power that is a boost.

 

Of course achieving that required ripping apart a bunch of code I had not really looked at in close to a year. I broke it a bunch, then slowly found (hopefully) all the places I broke it and pieced it back together.

 

I was (I think) at the end of that today when Page 4 landed. I'm going through what I hope is some final testing and will update CoD shortly.

 

Edit: Still squashing bugs. Getting close though. I think... 

 

Edited August 23, 2022 by UberGuy

[UberGuy]

Updates are publishing. This update contains the following changes:

• Data changes for today's patches for both Live and Alpha. Live data is now Page 4, and Alpha is whatever was added there on top of Page 4 today.
• Minor additional info displayed for powers that are boosts (AKA enhancements).
  • The fact that a power is a boost is called out, just above its main data tables (below the description)
  • If the boost has requirements, those are displayed next. Such requirements are used to restrict ATOs to their ATs, restrict "Superior" enhancements to level 50s, and create unique enhancements.
  • The background disc used when the power is viewed as an enhancement in game is now placed behind the power's icon, so it looks more clearly like an enhancement.

Less visibly, there are now moderately significant changes to the power info for boosts, moving the boost-specific fields into their own sub "object" and adding a a fair bit of additional data. This will be used to add more visual elements to both the powers page (when the displayed power is a boost) and to the boostsets page. Like many other things, these will likely be icons with "pop up" info indicating interesting info about the boost. For example: is it attuned? Can it be catalyzed and what does it turn into if so? Can it be boosted? Combined? Traded? And so on.

 

I also plan to add enhancement borders to the icon displayed on the powers page for boosts. I have the beginnings of this code already on the boostsets page, but it only handles borders for boosts we find in boostsets, so I need to generalize it a bit and make it into a widget I can reuse on both pages.

[UberGuy]

I haven't yet put yesterday's Alpha patch data on CoD. A few small changes in it that broke long-standing assumptions about a narrow bit of powers data. Nothing changed in data structures, but more how they're used. I'm working on some changes that make fewer assumptions about the affected stuff, and hopefully will have it done by the weekend.

[UberGuy]

OK, I got it working faster than I thought I would. Nothing new to see on the site (other than updated data). The code changes this time were just back-end improvements.

[UberGuy]

Found some bugs in how enhancement names were being rendered by the code that does so. Pretty sure these have always been wrong - stuff like "Slow" being displayed as "MoveSpeed" and things that are debuffs missing the "debuff" part of the name. The data's been updated with the fixes - this really only affected how the Boost Sets pages display the component enhancements.

 

Next up I'm going to see about adding a bit more description to simple procs. In some ways this will be easier than the enhancement naming, because the sign of the scale of the attribmod determines whether it's a debuff or not. With enhancements, the scale of the strength effect is (almost) always positive - the only thing you can look at to know if it's a debuff is to inspect what the boost power is allowed to be slotted in.

 

Even so, there's some fiddly stuff I'm probably just gonna leave as "Proc", and people can click on it to see the actual power for more info.

[Griffyn]

Great work on CoD!  It's extremely useful. Thanks for the effort! 

 

There is one issue that I'm seeing that I hope can be added. I'm only seeing Resistance on Pets_Reverberant

 

Thanks again!

[UberGuy]

That isn't a bug. That's actually the only power directly attached to the entity definition.  It gains the rest of its powers via GrantPower Activation Effect attached to other powers in the Symphony Control powerset.

 

I do see a bug there though. It looks like Activation Effects can affect "CasterAndPets" and not just the caster. I'll have to fix a bit of display logic there, as it is currently showing "Self Only" for the AEG's GrantPower, which wouldn't make a lot of sense in this case. You can see the correct target spec in the raw data though.

Edited September 4, 2022 by UberGuy

[Erratic1]

On 8/10/2022 at 8:04 PM, UberGuy said:

I've made some back-end updates that, right now, are only visible on the Alpha data. Most of it has to do with "requirements" expressions.

 

As I've posted about in the past, CoH stores the logical and math expressions found in power, critter and other data in what's called postfix notation (AKA "Reverse Polish Notation" or "RPN").  This is an efficient, straightforward way to compute things, which matters a lot in places like attribmods of powers, which can cause many hundreds of evaluations per server "clock tick". However, it's not very friendly for most people to read. Those of us used to math or logical expressions are used to this:

 

(a + b) - (c + d) / 4

 

not this

 

a b + c d + 4 / -

 

Thos of us who came up using HP calculators would. 😊

 

 

[UberGuy]

Like this one? 

 

 

I'm pretty sure not all HP calculators are RPN ones though!

[Erratic1]

44 minutes ago, UberGuy said:

Like this one? 

 

My first:

 

 

 

44 minutes ago, UberGuy said:

 

 

I'm pretty sure not all HP calculators are RPN ones though!

 

Not anymore. I'd blame Carly but at this point that seems like piling on.

[UberGuy]

Some small fixes this morning:

• Dark Mode now more visibly shows the headers for power effect group sections. Previously the text was black on a very dark gray, which was basically illegible. It now matches the color of other text in either mode.
• Activation Effect groups that include pets in the target spec now display "self and own pets". The only other option is "self only". This is a bit of a hack, since the power spec can technically be set to things that would allow other kinds of targets, but Activation Effects are always applied with the caster as the main target (no matter what the base power says), so other target specs ultimately can only effectively result in the two variations "self only" and "self and own pets".
• A couple of universal set categories had their icons tweaked so they look less like generic enhancements of that type.
• Some of the power enhancement/set icon display code was cleaned up a bit, though this should make no functional difference.

I'm still working on back-end code that lets me improve display of powers that are boosts. It has resulted in a major code reorganization and expansion which is still ongoing.

[UberGuy]

OK, I think I'm finally done reworking how the back-end handles boost powers to enable easier display processing in the web UI. Now I just have to update the web side to do something useful with it! Here's hoping I'll be able to wrap that up by next weekend.

[UberGuy]

Sadly not much progress for the last couple of weeks due to real life complications (health issues with family members).

 

FWIW, all is ... as well as it can be - the family member in question was not in great health to start with, and is largely back to their original state, but that's probably about as good as it's gonna get. So the crisis per se is probably past and my time commitments will hopefully be back to mostly normal.

[BillyMailman]

Back again with another bug report, sorry.

 

Current messed-up power is Soul Extraction. I'm seeing nothing but a non-heal heal, and checking the raw JSON shows that the power does indeed do a lot more.