City of Data v2.0

[UberGuy]

That's useful info. It's still maybe manageable, as long as it doesn't jump around too much. I might be able to do something a like I did with Entities (critters) where I show their name by level range. That was an exercise in just this sort of collapse of data into ranges when the data "natively" exists as something defined at each and every level. In other words, the game data doesn't say "it's called a Wild Rikti Monkey for levels 34-37". The game data gives you a distinct entry for every level for which Rikti Monkeys exist, and I look at that and suss out the ranges for the names.

 

If, say, only certain things like salvage and level of doodad created vary, I could pick that out the same way and display one page for a recipe with tables showing the salvage and whatnot by level range.

 

But if too much varies, or some things vary too much to build useful ranges out of ... maybe not. I'm optimistic, but CoH implementation details sometimes have a way of crushing optimism. 😆 But recipes really should be pretty consistent over kinda broad ranges. If they're not, at least we can build a list and maybe get them cleaned up.

[UberGuy]

PS: The whole recipe thing is probably going to be something of a slow burn. It's kind of a rabbit hole, because recipes refer to a whole bunch of other things that the bin parser doesn't know about, like salvage, workbenches and so on, that the recipe either crafts or requires to do the crafting. To get recipe parsing to work, I have to teach it how to read the data on those things too, or it won't know what it's doing as it pulls out the recipe data. I will probably be able to get away with skipping some of what the recipes refer to (like info on what workbenches let you craft it), but I still have to go through and figure out what I need and don't. I'm basically doing that in layers. Get it to read salvage, then recipes, for example. Really weird baby steps.

Edited June 4, 2021 by UberGuy

[UberGuy]

New shinies!

• Per page search box on powers pages. This was pretty easy to do. You can search for powers directly from an existing search page. Unlike the search box found on the main page, once you pick a power from the picker you're sent to that power immediately. (On the main page, picking a power updates the three links to the right so you have a way to get to powercats and powersets, not just powers.)
  
  
   
• I changed how duplicate display names are handled for...
  • Powersets on the power category page
  • Powers on the powerset page
  • Inherent powers on archetype pages

Previously, duplicate display names would be displayed with the internal name next to it in parenthesis. Unfortunately, this approach had problems.

• Some internal names are terribly long, so needed to be truncated. Making the table column a lot wider to allow for this was a huge waste of space on pages where powers had no duplicate display names.
• Long internal names that were truncated were no better at disambiguating the power than the display name.

So I've replaced the parenthetical names with a lightweight popup that only appears for non-unique display names.

 

Before

 

After

 

To try and deal with the occasional really long name, I display the names by replacing underscores with spaces so they can word wrap.

Edited June 5, 2021 by UberGuy

[UberGuy]

A minor fix to an issue I'm not even sure anyone noticed: the power category page offers a breadcrumb popup that links you to the AT page for ATs that can use powersets in that powercat. This works great for player ATs. For NPC classes, not so much - those would send you to a blank page due to errors, since NPC ATs lack the powerset references the AT page uses to build its display. The powercat page popup still lists the NPC classes, but they are no longer links, and it sorts them below the PC classes.

 

Also, if the powercat has no associated ATs, the entire breadcrumb text + icon is hidden. (Technically, it's not rendered at all.)

 

Edited June 5, 2021 by UberGuy

[UberGuy]

There are now help pages for these pages:

• Boostset Groups
• Boostset
• Entity

The help new reflects that pages that show duplicate display names will show a popup instead of a parenthetical internal name entry.

 

Number labels on all page images have been made larger so they're easier to read without zooming in on the image.

 

The main outstanding things I know of offhand for help pages are:

• Update the settings page help for new theme features
• Add an icon "legend" to the power page help
• Update the power page help to reflect that it now includes a search box

Edited June 6, 2021 by UberGuy

[UberGuy]

While working on the help, I either discovered new things I want to do with the site, or were reminded of issues / suggestions I still need to work on. So I actually have a decent little to-do list. In no particular order, this is what's on it now.

• Search result scrolling. This is a nice-to-have I may not be able to provide without a significant rewrite of the search box. I've never been satisfied with how results are limited and you can't scroll through matches. DONE!
• Allow regex-style anchors in search boxes. So things like "^" for "start of string" and "$" for "end of string". This would be particularly useful if I can't get the feature above working with reasonable performance. I need to make sure this has reasonable performance, as what I do now to support "*" in searches is already pretty expensive, but ensuring it only updates the search results every so often (instead of every time you type something) seems to work pretty well. DONE!
• Support internal power names on power searches. Entity search already works this way. DONE!
• Sort display name matches in search results before internal names. Most folks are going to be searching by display name, and right now the internal names sort first if they "win" alphabetically. DONE!
• Create pages listing all critters that share a power tag, like "Electronic" or "Undead" . Link to these pages from both entity and power pages. DONE!
• Create a "conversion simulator" that shows all the other sets you can convert a piece of a given set into, based on either its level or whether it's attuned.
• Break out data for set bonuses by attribmod and scale, to empower search/display for sets based on what bonuses they provide.
• Display more more boostset info in a table when viewing lists of boostsets from the Boostset Groups page.
  • Max level the set exists at
  • Lowest level the set's bonuses work at
  • Lowest level at which you can slot the set (which is not always linked to the lowest level for bonuses)
• Check why there are no AT assignments for some power categories. At least some NPC power categories have no NPC ATs mapped to them, which is surprising to me. See why this is (not) happening in the Rust code.
• Clean up raw data files for compressed bundling and add download links

Edited June 20, 2021 by UberGuy

[UberGuy]

Some decent updates from the above list.

• Power and entity name searches now support explicit searches against "internal names" of powers and entities. The default is to search against display names. For example, the internal name of "Blaster Support.Devices" is "blaster_support.gadgets"
  
   
  • Matches against the internal name are displayed with the internal name showing and the matched part highlighted.
    
     
  • If a search string matches a mix of both display names and internal names, the display name matches are listed first, then the internal names. Matches are otherwise sorted by display name, even for matches against the internal name
  • The internal matching checkbox is not yet persisted - you have to turn it on each time. I will add persistence, but wanted to get this out for wider testing.
  • There is no meaningful benefit to searching internal names for boostsets, so the internal name search is not enabled there.
• The search box code got cleaned up and more "regular expression" features added.
  • You can use "^" to indicate that something must be at the beginning of the string. "^Coun" will only match things like "Council...." and not "AnimusArcana.Counter Spell...."
  • You can use "$" to indicate that something must be at the end of the string. "Blaster$" will only match things like "Epic.Mace Mastery.Summon Blaster" and not "Blaster Ranged..."
  • You can use "|" to indicate "or"
    
     
  • Including "^", "$", "|" or "*" signifies that your search string should be treated as regular expression-like.
    • This changes parenthesis to be a match grouping operator, as in the example above. Without the parenthesis, the "or" would take precedence over the wildcard, and it would search for "(Council) or (5th*Kick)", which would give very different results.
    • If your search doesn't contain the characters above, parenthesis are not special and just match in the name
    • There aren't many display names with parenthesis in them, but there are some.
    • If you need both regular expressions and to search for parenthesis in a name, you can use the backslash to "escape" your parenthesis to say they should be treated as literal strings and not grouping operators.
      
       
  • All searches show highlighted match text. Previously only ones using a wildcard (*) would do so.

 

All of this needs to go into the help. The search boxes are going to get their own help page.

Edited June 7, 2021 by UberGuy

[UberGuy]

@Apparition, if you could give the above a whirl on iOS when you have a chance, I would much appreciate it.

[Apparition]

13 minutes ago, UberGuy said:

@Apparition, if you could give the above a whirl on iOS when you have a chance, I would much appreciate it.

 

Just tried all of the new options on Brave Browser for iPadOS without any issues.

[UberGuy]

Awesome. With that seeming to work well, I've updated the site to remember the new "search internal names" setting across pages and between visits. (As always, this setting is browser/browser-profile specific.)

[UberGuy]

More search tweaks. I'm hoping this one is a big deal for heavy users of the site.

• The search box scrolls. You can use the cursor keys, the page up/down keys, a mouse wheel, or click on the scroll indicators at the top and bottom of the list.
   
• When the current search has no more data either up, down or both, the respective scroll indicators at the top or bottom darken and turn solid, to indicate you can't scroll further in that direction. I want to do some cleanup so that they disappear if the current search matches fit fully on screen, but that's a little complicated currently, so narrow search results will have rather glaring "you can't scroll this" indicators for now.
  • Unfortunately, adding a scroll bar was beyond the scope of what I could easily achieve. I do believe it is possible (and have what I believe are relevant examples), and once/if I get that working, scrolling (via any input method) should be much more visually smooth. Plus, moving around the lists with the scroll bar will be much faster, though the page keys do currently work nicely.
• The search box now remembers two things it used to immediately forget if you touched its inputs
  • It remembers the most recent selected search result. Previously, if you interacted with the search box, it would forget this, at least visually.
    • This has an important meaning with the new scroll behavior - the most recently picked item is now used to anchor the drop down when you go to search.
      
      
      
      
      
      This is meant to facilitate poking around in the list near the most recent thing you looked at.
  • The search box also remembers the most recent search input if you let the search box lose focus without picking anything. This way you don't have to re-type a long/complex search if you accidentally clicked off the search box or hit escape.
  • These two combine - if you have something picked, then type a search but don't pick anything, when it loses focus the box will show the last thing you picked, but if you give it focus again, it will remember what you typed. Only when you actually pick something will what you typed will be cleared out.

• The search boxes at the top of various pages also now default to the object of that page. So if you navigate to Scrapper Siphon Life, as shown above, that power is already picked in the search box and the dropdown will be located at that power.

Some caveats

• This works reasonably well on a desktop computer. Without a scroll bar, you can't do a lot with it on a mobile device. Arrow keys and page up/down keys, if you happen to have them on your soft keyboard*, do work, but otherwise you can't scroll, as far as I can tell. The up/down "bars" at the top of the bottom of the lists don't seem to honor mouse-like presses on a tablet the way they do an actual mouse on a PC.
• Because the search rows are not all the same height (because of the display of internal names on some rows), occasionally you get a row that's half visible. Scrolling one row will usually fix this, depending on what's visible, but because it only advances in "whole item" increments, certain combinations of visible entries just refuse to re-align until you get back to a place where everything is one row tall.
• Also, while I tried hard to squash these bugs, you may find that the highlighted item wanders out of the viewport. If you lose track of it, just click off the search area and back. Or just click / select something with the mouse / your finger.
• All the highlight selection bugs should be addressed fairly naturally once/if I get a scrollbar working.

 

Edit: Just fixed a bug where scrolling with mouse wheel in the match list would also scroll the page itself if it had enough info visible to have a scroll bar. That's what I get for testing without the "raw data" icon checked.

 

* For Android, "Hacker's Keyboard" offers arrow keys and, on the "Fn" page of keys, page up/down keys.

Edited June 11, 2021 by UberGuy

[UberGuy]

OK, folks. It took several days of learning, coding, breaking things wildly and then figuring out how to fix them, but I've implemented a fast scroll box that has a real scroll bar.

 

 

Most of the original scroll interactions from my first go at this have been preserved. The main thing that's changed is the scroll indicators at the top and bottom of the list are gone, replaced by the legit scroll bar.

• The cursor up/down keys move the currently selection up and down, and scroll once you reach the ends of the visible list.
• The page up/down keys scroll roughly a whole screen at a time. Sometimes they go more or less than a whole page because it snaps to the nearest top/bottom row.
  • If the mouse is off the scroll list, the page keys move the current selection to the top (page up) or bottom (page down).
  • If the mouse is over the list, it's position determines the selected item.
• The mouse wheel will scroll up and down, without momentum.
• The list will initially put the last searched item in the search box at the top, unless that's not possible because the remaining part of list is too short. The last searched item is initially selected, whether it's at the top of the list or not.

New behaviors are:

• The dropdown remembers its last position if you leave it without picking something. This is not really a feature, it's just default unless I do something different. Personally, I kind of like it.
• You have scrollbars. They scroll very fast even if you have all 28k powers matched by your current search.
• You can (finally) scroll on mobile using traditional swipe up and down motions on the list.

Sadly, the behavior of the swipe scroll on mobile is not nearly as nice as it looks on a regular PC. This seems to be a limitation of the platforms - an actual difference in how all the mainstream mobile browsers handle scroll events. On desktop, many events fire as you scroll, creating a smooth visualization. On mobile, the scroll events fire near the end of the scrolling, meaning a fast scroll creates a long period of nothing visible in the list. I will continue looking for ways to address this - the one I found was out of date and discontinued, and most articles talk about just sucking it up.

 

But hey, at least you can scroll on mobile now.

[UberGuy]

So, on further research, it seems the update lag when scrolling on mobile is not a (missing/bad) browser feature thing. The articles I was seeing suggesting that are out of date. Browser tech has changed fast during the last 5 years, so there's a lot of old info out there.

 

It seems it is honestly just a performance thing.

Each time you update the search box, a new list of matched entries is built. When you scroll around, the code has to calculate where in that list you're currently at in order to know what list entries to display. This is an optimization to avoid rendering several thousand entries (10s of thousands for powers) on page load.

 

This calculation of what to display is non-trivial because not all the entries are the same display height - duplicate display names and matches against internal names both cause the internal name to be shown underneath the display name, making some entries taller in a way that varies  based on the current search box input. If every row is the same height, you can just figure out where you are in the list by dividing the current scroll position by the height of the rows, but with variable height, you have to get fancy.

 

The brute force approach would be to walk the list of display items until their heights add up to the current scroll offset. The performance on this would be abysmal for large lists. Instead, I used  a tried-and-true approach called a "binary search", where you start in the middle and say "am I above or below where I want to be", and then keep dividing the remaining list in half until you end up in the right place. Worst case performance on the brute force approach is proportional to the number of things in the list  - a hefty 21,800 (not 28k like I said above) in the worst case of the powers list. But for a binary search, it's proportional to the log() function of this number. For the ~22,000 items in the full power list, it cuts down finding your place to 14 tries tries or less, with an average of probably around 7-8.

 

But it still has to do this on every scroll update.

 

But wait, the Entity search list has about 7,700 entries, which still seems to result in 12 tries or less, and the entity scrolls noticeably more smoothly on mobile. That tells me its probably not the time spent finding one's location that's the dominant factor.

 

Power names, with their need to display the category, powerset and power names together, can be really long. So the search box (and the dropdown list) for powers is much wider than the Entity or Boostset search boxes. And a wider list box is more area the browser has to repaint on each scroll refresh.

 

And that seems to make the dynamic scrolling updates to the list box slow enough that they fall behind when you scroll fast. Which... there isn't a lot I can do about, since displaying less than the full names would make the power search considerably less useful.

 

I'm going to keep poking around, but so far it doesn't seem like I can do a lot about this. The best advice seems to be:

• Don't slap the search list with a massive swipe action that will make it rip through the list super fast. Unless you don't care if you can't see what's scrolling by, in which case, go nuts. 
• AND put something sane in the search box so the code has less to search through a smaller list

Edited June 17, 2021 by UberGuy

[UberGuy]

A few relatively small updates.

 

AT modifiers links

On the AT modifiers table page, when you select a modifier from the pick list, the address bar turns into a URL that will take you directly to the page with that modifier pre-selected.

 

Entity Tag Pages

Some powers have additional, conditional effects against critters that have specific "tags" applied to them. Examples are things like "Undead" or "Electronic", applied to zombies and robots, respectively, and these tags let powers like the Undead Slaying Axe or EM Pulse do special things to these targets.

 

There are now pages which show you either all the critters that have a specific tag or all the powers that affect critters with that tag.

 

 

Like several other pages, these pages include a "deduplication" popup when you hover over a row that has a non-unique display name (pictured above).

 

You can get to these pages from both critter pages and power pages. On power pages, these tags are referenced in effect group condition expressions. In the expression, the tag names are now links.

 

 

Also, the entity pages treat all the critter's applicable tags as links.

 

 

AT Links in Condition Expressions

While I was in adding links to power expressions, AT/class references are now also links. Player AT links point to the base AT page listing inherent powers and pri/sec/pool powersets, while NPC classes point to their attributes data pages.

 

Here's an example from a Scrapper critical hit effect.

 

 

And here's one of Boxing's many AT-specific effect lines (also for a Scrapper).

 

 

This probably isn't nearly as useful as the tags thing above, but it was easy to add given what I was already doing for tags.

 

Bug Fixes

I fixed a couple of bugs.

• A big one was that I broke a lot of (maybe all) the links across the site for maybe 10 minutes yesterday. Hopefully no one noticed!
• A small one was that the deduplication popups like the one pictured above always had a bug where, if your mouse was in just the right place, the popup would appear where it would block the mouse hover target that made it appear. The result was it would flicker on and off very annoyingly. This is fixed.

 

What's Next

Other new features on my todo list are slightly larger than the above, so my next task is to get back to working on the help pages, particularly around search.

Edited June 18, 2021 by UberGuy

[UberGuy]

Some updates today. 

 

• Some minor fixes to the search boxes to help them behave more like they did on the previous (scrolling without a scrollbar) version. There's a lot of variables and state management to keep track of, so bugs are kind of easy to cause. So if you see anything wonky, let me know.
• The help pages have been updated to cover the recent search updates. Search has its own dedicated section in the help.
• I cleaned up the styling on the help pages so they only show one scroll bar, and are hopefully more mobile friendly.

 

I still need to add a help page for the settings page. Also, I used collapsible sections in the new search page that I think might help make some of the other long help pages look more manageable, so I might update a few of them with it.

Edited June 20, 2021 by UberGuy

[UberGuy]

The settings page now has a help section.

[UberGuy]

I pushed a few small changes over the last few days. Most were tweaks to search to improve performance and reliability of some of its behavior. I also went through and made sure all the pages that work off of URL query parameters show useful messages if you give them bad (or no) parameters instead of just giving you a mostly-blank page and a spam of errors in the browser console. This also cleaned out a legacy input box on the power page that would appear if the requested data couldn't be found - that's super redundant and inferior to the search interface that's now on the power pages.

 

I've been peering at my copy of the game's code to make sure I really understand rules for things like when a set piece's bonuses stop working, so I can add more of that info to the boostset / boostset groups pages.

 

Fun fact - the game  uses the rules for SO/DO/TO degradation with level difference to determine if a crafted IO will give you bonuses or not. This is where the "IO level -3" cutoff comes from. An SO more than 3 levels lower than your combat level gives you 0 boost. The code says (paraphrasing) "include this boost in the power's set piece count if that logic would give you more than 0 boost." (This made it hard to find the logic in question - it's not at all clear from a casual perusal that this is what that bit of code is doing.)

 

Edited June 22, 2021 by UberGuy

[UberGuy]

Teensy tweak - the far end of the search input boxes now show how many entries are matched by your current input. Usually what you type in the box is way shorter than the input box itself gives room for, since its width is set to display your final selection, so I don't expect this number display to get in the way visually. (Plus, the number should get smaller the more you type.) Let me know if you run into places where it does interfere.

 

 

 

There were some more small fixes. Mainly, using a link to a search selection on the main page (or navigating back / refreshing the page) wasn't populating the powercat/powerset/power links. Also, I shortened the text for each of those links to only display the name of the thing in question, rather than the fully qualified name.

 

 

This was done because the full name of some powers is huge, so the power link in particular would sometimes overflow and mess with the page layout.

 

Edited June 23, 2021 by UberGuy

[UberGuy]

Just a status update, I'm working on building the code to perform dedicated data exports  that will facilitate a set bonus search/lookup interface. However, I'm pretty tied up with work for the next few weeks, and then I go on vacation, so I probably won't get a lot of new feature work done for the next month or maybe two. Obviously, updates will appear here when I have them. 🙂

[Bionic_Flea]

Have fun on vacation!

 

You've done such a good job on COD, that I nominate you to tackle CIT next! 

 

[UberGuy]

Small update!

 

Based on reports of issues with the checkboxes in the header (PvP/E and Raw Data) not always applying on page load (which I have sometimes seen myself), I made some relatively small tweaks to how these are loaded. Hopefully, they will now consistently apply the checkbox setting to the page, rather than occasionally requiring you to re-toggle them to get them working. 

 

For the curious about how the sausage gets made... Visual elements on CoD pages are "components", each with their own code, and so loosely coupled. The main component on the powers page (itself made of other components) knows nothing about the checkboxes in the page header, per-se. That architecture is useful in a lot of ways, but it makes the way things communicate shared state (like the whether checkboxes are checked) relatively complex - more so than you'd probably think for something as conceptually simple as a checkbox. To solve for this, checkboxes "announce" their state when it changes (like when you click on them), and things that care "listen" for these announcements. On initial page load, the checkboxes "announce" their remembered state after they load it from your browser storage. But this means there's a so-called race condition - what if the checkboxes "announce" their initial status before the other components on the page that care about it are ready to listen for it?

 

To deal with this, I have the components that need to know about checkbox state "announce" a request for all checkboxes to broadcast their state only after they're ready to receive such an announcement. Some of my decisions about how to build that were, in retrospect, probably not so well-informed about how the web framework I'm using actually works. In other words, I n00bed. I have moved when these things  happen to an earlier stage in the page render process and also added some retry logic to them. Now if a components that cares about checkbox state don't see a status response for all the checkboxes it cares about, it asks for a new status broadcast again, and will do this multiple times until it knows everything it needs to.

 

At least, that's the plan. 

Edited July 25, 2021 by UberGuy

[macskull]

On 6/30/2021 at 2:25 PM, Bionic_Flea said:

Have fun on vacation!

 

You've done such a good job on COD, that I nominate you to tackle CIT next! 

 

WTB the version of CIT a different server has so I don't have to guess which character has that random PvP IO sitting on it...

[BillyMailman]

Bug Report:

 

The Entity Search knows that 'Super Stunner' exists, but their internal name is apparently something like 'freakshow_supr_st|_|n3r', and the link produced by clicking the result in the search is https://cod.uberguy.net/html/entity.html?entity=freakshow_supr_st%7C_%7Cn3r, which doesn't load.

Edited July 28, 2021 by BillyMailman

[UberGuy]

13 hours ago, BillyMailman said:

Bug Report:

 

The Entity Search knows that 'Super Stunner' exists, but their internal name is apparently something like 'freakshow_supr_st|_|n3r', and the link produced by clicking the result in the search is https://cod.uberguy.net/html/entity.html?entity=freakshow_supr_st%7C_%7Cn3r, which doesn't load.

 

Thanks for that. That name entity's internal name is a bane on existence in multiple ways. I am forced to sanitize it because using it as a filename is invalid in nearly every operating system, and CoD uses a separate file for each entity. Clearly I missed lining up the generated link with the (sanitized) file name.

[UberGuy]

The above issue with Super Stunners is fixed.