City of Data v2.0

[UberGuy]

At this point, the only "concern" I have (if you can call it that) with releasing my code is that it's really not in great shape for anyone else to use. It's not at all "plug and play", in no small part because I develop and run it on Linux rather than Windows. (Sounds like that would be no problem for you though.) Getting the code into shape for others to use and build on is 100% something I would like to get to, but it taken a back seat to actually keeping the site up to date and feature complete. Since all of this is work I do in my spare time, I haven't yet put in the time to make the code more user-friendly, even for a fellow Penguinista.

 

What I could do, and I have given thought to how I'd most usefully go about it, is bundle up the "raw" data as a zip file for others to use, downloadable from the site. This is not the data I actually feed the site's web pages, but the very raw data that comes out of my forked version of RubyRed's Rust code, before I feed it into the Python code that massages it for use on the site. That's something I can do with pretty limited (but non-zero) effort.

 

By the way, Ruby's code actually does emit enhancement data. Just turn off the filter for the power category "boosts" in the config file. By default it's set up to filter out anything that's not an actual player character power/powerset, but that filters out enhancements. I'm pretty sure Boostsets support was added too, but I haven't used Ruby's version since before that update, so I'm not sure how/if it's exported.

[carroto]

 

 

1 hour ago, UberGuy said:

bundle up the "raw" data as a zip file

Why not the finished product?  It would be appreciated regardless.

 

1 hour ago, UberGuy said:

Ruby's code actually does emit enhancement data.

Thanks for the tip.  I hadn't looked at the code to this point and just presumed that what I was seeing was the full product.  Are you doing much translation/interpretation of the output of that program, or is that left to the front-end code?  As I said I'd prefer to avoid reinventing the wheel.

 

Regardless, it looks like even in the worst case I can probably get what I need without a tremendous amount of digging around in .bin files.  So I guess there's nothing stopping me from moving forward.  RIP my free time.  I'll have to do a little soul-searching to be sure I want to continue.  Thanks for the reply.

[UberGuy]

On 5/28/2021 at 7:00 PM, carroto said:

Why not the finished product?  It would be appreciated regardless.

Two main reasons.

• The finished product is massaged in ways that make it consumer friendly, but may obfuscate its relationship to the bin data. If you look at my raw data on the site, it's going to be confusing to map some of it back to the internal names. Why? Because the internal names are sometimes complete butt in the sense that they would confuse a regular player. Hell, they often confused me. But I had the benefit of source code and some help from our devs to translate. Also, some of the structures just were more efficient to flatten or otherwise remap in order to make sane JSON that performed well to be re-parsed by a website.
• I don't dump everything to the site, though I do dump most of it. Since the focus is on powers, some things just aren't relevant to how CoD works and get trimmed.

IMO the raw data is more valuable to someone writing new tooling, because they can pick and choose what data they need how it's displayed or massaged, instead of having to (re)parse what I transformed it into for CoD. Ultimately, this is why I took Ruby's code bolted Python onto it. I don't actually operate on the output of Ruby's original code, which does similar transformations to what I describe above. I modified it to emit its internal, "raw" representation of the bin data, which is much more "pure", and feed that directly into Python. Basically I replaced Ruby's approach to making the data "human readable" with one that made it "CoD" readable for my interpretation of what CoD should do. I could have operated on Ruby's output, but it would have made some of what I did a lot harder. Not because Ruby's choices were bad, but because they were optimized for slightly different goals.'

 

Edit: It's worth noting that Ruby did add a feature to export "raw" data as well, just to files instead of cranking it into Python data. (I could read the "raw" output and read it into Python, but that would be even slower than what I do now.)

 

There's no reason I can't bundle up the finished data too, but I think that would be more useful to regular players than to a tool writer. And regular players probably get a way nicer view of that data on the real site rather than peering at raw JSON. The JSON is provided on the pages so people can find things I don't handle (yet), or spot things I handle badly.

 

Edited May 31, 2021 by UberGuy

[KaizenSoze]

First, I love your website! So, informative, it has helped me so many times already.

 

There does appear to be an issue with Trick Arrow.Disruption Arrow across all ATs that can use it, Defender, Controller, Corruptor.

 

It appears in the search drop down, but the power page itself is not found.

 

https://cod.uberguy.net./html/power.html?power=corruptor_buff.trick_arrow.disruption_arrow

[UberGuy]

1 hour ago, KaizenSoze said:

It appears in the search drop down, but the power page itself is not found.

 

https://cod.uberguy.net./html/power.html?power=corruptor_buff.trick_arrow.disruption_arrow

 

Weird, it works for me off of the above link. Are you getting a browser "file not found" page, or a CoD "power not found" page? I wonder if you're seeing the default info on the page due to a Javascript error. It will say something about the power not being found until the data is loaded, and if the data load blows up, you'll be left with that message. (Something I should probably address.)

 

If you're getting the "power not found" message, can you try a force refresh of the page? It's usually Control-F5 (or Command-F5 on a Mac).

[Bionic_Flea]

2 hours ago, KaizenSoze said:

https://cod.uberguy.net./html/power.html?power=corruptor_buff.trick_arrow.disruption_arrow

This comes up for me too.

 

[KaizenSoze]

48 minutes ago, UberGuy said:

 

Weird, it works for me off of the above link. Are you getting a browser "file not found" page, or a CoD "power not found" page? I wonder if you're seeing the default info on the page due to a Javascript error. It will say something about the power not being found until the data is loaded, and if the data load blows up, you'll be left with that message. (Something I should probably address.)

 

If you're getting the "power not found" message, can you try a force refresh of the page? It's usually Control-F5 (or Command-F5 on a Mac).

 Control-F5 worked, even though I had hit the page multiple times in Firefox.

 

Chrome worked on the first time. I will run those steps before I post an error next time.

[UberGuy]

Very small update - the help pages are still out of date, but at least they obey the theme setting now.

 

Updating the help is slow just because (a) it's prose and (b) I have a lot of screenshots to update. The annotations on the screenshots are particularly slow to redo. I'm using this update to change the annotations to layers on top of the base screenshot image so I don't have to recreate the annotations on each image from scratch each time the screenshot changes.

[Apparition]

Just FYI, CoD is broken today on both Brave Browser on Windows and Brave Browser on iPadOS.  It worked last night just fine.

 

 

 

 

 

It looks like that on both Windows and iPadOS.

[UberGuy]

2 hours ago, Apparition said:

Just FYI, CoD is broken today on both Brave Browser on Windows and Brave Browser on iPadOS.  It worked last night just fine.

 

My bad. Another deploy script bug. Trying to tune the site a bit and some of the small files ended up flagged as compressed when they weren't due to a variable not being cleared in a loop. Unsurprisingly, browsers don't like being told to uncompress something that was never compressed. One of those small files that said it was compressed and wasn't? The new code for handling light/dark switching, which is loaded on every page.

 

The site should mostly be working now. Some edge places may not if they rely on data files that had the above issue. The simplest way to make sure all the data files are correct is to re-upload them all, so I'm running that now. It will take about 20 mins from the time of this post.

 

Edit: Done some time ago.

Edited May 30, 2021 by UberGuy

[Apparition]

It is mostly fixed, but actual power pages come up blank on iPadOS.

 

@MJB confirmed this, and also added that it doesn't work on Safari on macOS either.

Edited June 1, 2021 by Apparition

[UberGuy]

1 hour ago, Apparition said:

It is mostly fixed, but actual power pages come up blank on iPadOS.

 

@MJB confirmed this, and also added that it doesn't work on Safari on macOS either.

 

I wonder if those platforms aren't sending requests expecting/allowing compressed files back. I would not expect that of any modern browser. I may have to turn on some logging. Also doing some Googling about those browsers.

[UberGuy]

From what I can tell, Safari should be handling this. To understand why it's not, I would need to see the errors from its developer console.

 

I may be able to do that myself later today, at least on a mac, but I have no access to an iPad.

 

Chrome is available on iPads. Does it suffer this issue?

 

Edit: Were caches cleared?

Edited June 1, 2021 by UberGuy

[Apparition]

6 minutes ago, UberGuy said:

From what I can tell, Safari should be handling this. To understand why it's not, I would need to see the errors from its developer console.

 

I may be able to do that myself later today, at least on a mac, but I have no access to an iPad.

 

Chrome is available on iPads. Does it suffer this issue?

 

Edit: Were caches cleared?

 

Yes, the cache was cleared.  All browsers on iPadOS/iOS use Safari internally, so all browsers are impacted.  I tried on Brave Browser and Safari.

[UberGuy]

That's effectively true on iOS, but not macOS, at least so far. Chrome on macOS is its own, independent thing, with a different rendering engine and everything. Does Chrome work correctly there? I am trying to narrow this down to Safari/Webkit specifically.

Edited June 1, 2021 by UberGuy

[Apparition]

22 minutes ago, UberGuy said:

That's effectively true on iOS, but not macOS, at least so far. Chrome on macOS is its own, independent thing, with a different rendering engine and everything. Does Chrome work correctly there? I am trying to narrow this down to Safari/Webkit specifically.

 

 

I don't have a Mac, only an iPhone and iPad Air.  I asked @MJB to check.

[UberGuy]

I may see the issue - Safari likes to send OPTIONS preflight requests, and I think those may be being rejected. There was a similar problem when I first stood up the site, but this has a different cause. Will update here if I can get those requests flowing and we'll see if that fixes Safari.

[UberGuy]

Any chance this has improved? I don't thnk what I was looking at was likely to actually be blocking at all, but I did change something, so there's a chance I fixed it.

 

Absent that I'd need some console logs from the Safari browser, probably from the macOS version.

[Apparition]

30 minutes ago, UberGuy said:

Any chance this has improved? I don't thnk what I was looking at was likely to actually be blocking at all, but I did change something, so there's a chance I fixed it.

 

Absent that I'd need some console logs from the Safari browser, probably from the macOS version.

 

Negative, still broken.

[UberGuy]

It should be fixed now. I got my hands on a mac and debugged it from there.

 

What I posted in Discord:

Quote

Turns out it had nothing to do with the site-level settings I was changing yesterday. It will have been broken since last Friday when I enabled Dark mode. There are Javascript regular expressions used to convert the game color code markup used in power info text into HTML, and one of them used a Regex feature that Safari is bizarrely singular in not supporting, despite it being standard (if specialized). I replaced it with something equivalent that achieves the same goal.

Finding this was painful because Safari doesn't emit a line number for the error, as mentioned. I had to work backwards using its description, and ironically the error message is not even correct for the actual issue. WTF, Apple.

 

Edit: Access to the mac I used also let me spot a couple of other errors that didn't show up (or at least weren't obvious) in Chrome, and those are also now fixed. I pushed out incomplete help page updates to get the fixes out, so a couple of the help pages are more correct now too.

Edited June 2, 2021 by UberGuy

[UberGuy]

Some minor tweaks:

• A fairly large number of powers, mostly for critters, reference power icons which simply don't exist. Sometimes they just have the name wrong, but sometimes they reference files that clearly no longer exist, or perhaps never existed. Historically the process that builds the powers data had no concept of whether the power referred to an icon I could actually extract, and would blindly point to it no matter what. Now the process checks if the icon was extracted, and if not, the power is flagged so the site will display a question mark icon instead of a broken image. This will also reduce a known source of "file not found" errors in the site's access logs, reducing noise and helping me find other, potentially more serious missing file errors.
• I discovered some PvP/PvE parsing was busted, and has  been for a while. I rewrote the parser that handles expressions like this a while back, and there were some lingering bugs I happened to stumble across. They are now fixed, I hope. I said that last time. 🤞
• Some more help pages were updated. That process continues, slowly. I'm probably 2/3rds done. The power page is a beast to document.
• A few small visual tweaks mostly meant to make the site look a bit better on narrow devices like tablets or (heaven help you) phones.

 

[UberGuy]

I finally uploaded the updated Power Page help. It's still not done because I want to add keys for the copious number of icons you can find on this page. But I updated the page for other new stuff that's not icon-based.

 

Main help pages todo list:

1. Add help pages for the new Boostset and Entity Search pages (That's two help pages.)
2. Go back and make the numbers on most of these pages' images bigger. Fortunately that won't be too time consuming since they're text layers in Photoshop images.
3. Document the icons. I'll probably do this in phases, with AttribMod icons last, because there are a ridiculous number of them.

The only site features I'm looking at adding right now are:

• An on power-page search box. I really like being able to search directly from the Boostset and Entity pages, and miss it on the Power pages. I need to play around with it and see how much space it needs. Power names can be really long, which is why the search box on the landing page is so wide. Case in point: "Boosts.Attuned Mark of Supremacy F.End/Pet +Resist(All) and +Regen Aura (Mastermind)"
• Maybe something for recipes. I am poking at teaching the Rust code to read them, and if I get it working, I might as well display the info. I'm not sure yet what display approach would make sense though. I want to see the data first. Unlike boosts, for which the enhancement's level is effectively a attribute, there's a unique recipe for crafting boosts at every level you can craft them for. So the recipe for a level 25 Mako's acc/damage enhancement is totally distinct from the recipe for a level 50, even though they basically craft the same thing with a different number set on it.

[Bionic_Flea]

6 hours ago, UberGuy said:

Unlike boosts, for which the enhancement's level is effectively a attribute, there's a unique recipe for crafting boosts at every level you can craft them for. So the recipe for a level 25 Mako's acc/damage enhancement is totally distinct from the recipe for a level 50, even though they basically craft the same thing with a different number set on it.

 

Ugh!  That sounds like more trouble than it's worth unless you are planning to integrate with Mids or have some kind of build planner of your own.  That's a lot of granularity for what I expect will be minimal usage.

[UberGuy]

4 hours ago, Bionic_Flea said:

 

Ugh!  That sounds like more trouble than it's worth unless you are planning to integrate with Mids or have some kind of build planner of your own.  That's a lot of granularity for what I expect will be minimal usage.

 

The main thing I'm thinking of doing with the data is use it to audit recipe descriptions and levels against actual enhancements. But I figure if I get the data, maybe I can show it too. But as you say, it may not be worth the effort.

 

It sort of depends on how uniform they are. If they're well-behaved, I'm thinking maybe I can "collapse" ones that vary only by, say, level to a single thing myself. And if they are inconsistent, maybe we can get that fixed, and then I can collapse them. 

[AboveTheChemist]

1 hour ago, UberGuy said:

It sort of depends on how uniform they are. If they're well-behaved, I'm thinking maybe I can "collapse" ones that vary only by, say, level to a single thing myself. And if they are inconsistent, maybe we can get that fixed, and then I can collapse them.

There are only two oddities that I know of in the recipe data, and only one that might cause an issue with collapsing them by level range.

 

The recipe for Gladiator's Javelin: Chance for Toxic Damage uses one Boresight at level 10, and at levels 11-25 it instead uses one Computer Virus. At level 26 it uses one Improvised Cybernetic, and at levels 27-40 it instead uses one Circuit Board.

 

The other oddity is that the recipe for Mako's Bite: Accuracy/Damage is the only recipe to use multiples of salvage (2 Steel for levels 30-40, and 2 Chaos Theorem for levels 41-50), but I don't think this would cause any issues with collapsing that recipe by level range.