UberGuy

June 15, 2022

10 hours ago, BillyMailman said:

...of course, this is me, so I saw that because I was coming here with a bug report anyway. All versions of Twilight Grasp are failing to show what the Execute Power bit is executing, which they used to do at some point in the past. Poking around, it looks like it's every Execute Power, Fulcrum Shift is also failing to display properly.

Hopefully that won't be too hard to find the source of. Off the top of my head, that sounds like something I may have broken when I last touched the attribmod parsing back in April. Thanks as always for the report!

Edit: Hrm, nope, the logic is good, what's different is the data.

"params": {
  "type": null,
  "count": 1,
  "category_names": [],
  "powerset_names": [],
  "power_names": [
    "Redirects.Dark_Miasma.Twilight"
  ]
},

That "type" field should be "Power". I will have to go digging into the parser to figure out what's going on there. I have an idea where the issue is.

June 14, 2022

On 12/11/2021 at 5:52 AM, LQT said:

As of issue 27 page 3 attuned Air Burst IOs still do not give an option to convert by category.

This should not have been new to Page 3. This is a long-standing issue with how the conversion process works for Attuned IOs. The problem is that Attuned pieces count as their lowest available crafted level for conversion processing. Air Burst is the only Ranged AoE set that exists at level 10, so there's no way to convert it to any other set. That means the only option is to convert it to another Uncommon.

June 14, 2022

Current Alpha (Cryptic) data has been uploaded. If there's anything anyone wanted to check about stuff that's currently in testing, it should be there now.

June 14, 2022

So quick life update - I got married! This has been in the pipeline for a good year and a half, but we planned basically the whole thing ourselves so my free time the last couple of months has been spent getting everything ready for that. The dual good news is that it all went amazingly awesomely, and now I should have more time again for CoH and CoD.

My first order of business will be to get the latest Beta and Live builds out. That should happen tomorrow. Then I plan to get back into working some features I've wanted to work on forever, updating the site help, and publishing the JSON data maps (not necessarily in that order).

May 7, 2022

On 5/2/2022 at 5:48 PM, AlishaShatogi said:

The help pages appear to be broken (on Chrome? haven't tested other browsers).

Bah. I noticed that before and thought I had fixed it... Clearly not.

I believe it's fixed now. It loads for me currently in Chrome. I did have to force refresh my browser, but that was immediately after I made changes to the CDN configuration, so hopefully 10 minutes after I did that, everyone should see it no matter what.

April 23, 2022

Apologies if anyone hitting the site around mid-day today got broken pages. I added some increased site security settings that I had to then loosen since they initially broke a bunch of the site's functionality. Oops.

Tag me if you see any new breakage, as it's possible I overlooked something.

I'll likely clean up some of the things that broke today so that more strict rules can be (re)applied in the future, but that probably won't happen until I take the to revamp the site more thoroughly. I'm not talking about changes to how the site works for end users. This is more implementation-only stuff, like what version of Vue.js is used to render the dynamic content.

Suggested improvements or new features I haven't gotten to yet will still be a thing, of course.

April 9, 2022

lol. Called it! (That I'd break it.) Looking now.

Edit: Fixed 😅

April 6, 2022

Yep. Looks like it might not be trivial to add, as that attribmod has no special handling currently, and the attribmod handling code is a giant cascading if/else tree that I usually break when I touch, but it should definitely be doable and might not be hard.

Edit: Definitely not too hard.

April 5, 2022

Ended up running the extract last night and uploaded the data this morning.

April 4, 2022

I didn't expect these enemies to have new powers, or indeed to be new entities at all. I thought they pre-existed from last year.

I'll see if I can find anything new tomorrow.

March 23, 2022

On 3/21/2022 at 6:05 PM, Bionic_Flea said:

What if you reverse the polarity of the blilateral kelilactirals then route it through the Heisenberg compensators? That just might work!

Only if I can get it to 1.21 gigawats first. Or maybe 88 miles per hour. Or both...

March 21, 2022

Time for Uber's forum post book of the month.

Still not much progress on the data map. I hope to work on it later this week. But I did make some small, backend-only changes as an experiment.

Building all the CoD site data for one "version" of the game (Alpha or Live) takes a little over 8 minutes on the hardware I usually use. That's extracting everything, icons, data files, etc. but not including uploading it to where it's hosted, which can take a couple of minutes but is usually much faster because only changed files need to be uploaded.

When I build both Live and Alpha site data together, I build them both at the same time by running two copies of the program in parallel, one handling each build, so the time is about the same whether I build one or both.

As a reminder for context, CoD's data is pulled out of the game files using Rust code derived from that written by @RubyRed. This does all the low-level work of actually turning the very raw *.bin file data into data structures it can then pass around, manipulate, link together, etc. The Rust code handles this plus stitches powers into powersets and powersets into power categories ("powercats"), as well as things like figuring out which ATs use a power, plus a few more things.

At that point, the Rust code disgorges the extracted data into Python. This transfer about as efficient as it can be, as the Rust code is loaded into Python (and invoked) as though it were native Python code (just much faster). A final transformation of that data is done in actual Python code, turning all the raw data from Rust into dedicated Python objects (so there's an object type for a Powerset, a Power, an Atrribmod, etc.), then, slicing and dicing those into the JSON output used by the CoD website. I originally did this this way because it was easier for me to code the transformation in Python. That's because I'm very experienced with Python, and because Rust's design, which is meant to protect programmers from common coding/design mistakes, makes it hard(er) to do certain things that are easy in other languages. Or at least harder to think about doing them in new ways.

Anyway, the downside is that the bits done in Python are much slower than the stuff done in Rust. But there's a version of Python, called "PyPy", that's often way faster than the standard version. I wondered if PyPy could speed things up for me, at least a bit.

A speed improvement of 3-6 times is considered typical for PyPy. PyPy's is pretty darn compatible with regular Python, but the way it runs the code is very different than the standard Python interpreter. These differences aren't easy to summarize for non-programmers, but they tend to do two things for you: make the code just plain faster, and allow effective multi-threading. Regular Python supports multi-threaded programs, but the interpreter design hamstrings the ability of multiple threads to do really heavy duty work in parallel. PyPy doesn't have that problem.

My CoD Python code isn't multithreaded and really would not be easy to make it so, but I still wondered if PyPy still shave off some execution time if I just dropped my code on it mostly as-is.

So I spent some time today figuring out how to get the Rust/Python library to play nice with PyPy. This mostly involved upgrading a few things in the project and then fixing a few things that this broke. Getting everything working with PyPy actually wasn't very hard, but I did spent a couple of hours stumbling over a couple of issues that were actually quite simple in the end but just weren't document well.

So what happened?

The PyPy version actually took about 30 seconds longer to run. Oops.

Why? Well, based on watching the system while it ran, and knowing a little about how PyPy works, I think the problem is that what my program is doing is kind of a bad deal for PyPy. To explain why, I have to go a little into the territory I said would be hard to summarize. Programming runtimes need to deal with things, like variables or data structures, that a program was using but is now done with. If it doesn't, the program will just fill up memory with unused junk until it runs out of places to put things. There are, broadly, three ways to handle this problem:

Make the programmer responsible for cleaning up after themselves. You ask for it, you have to explicitly clean it up when you're done with it. This is, except in the case of extremely low-level, high performance code, not that common in modern languages. (Note: CoH's client and engine code, written in C, works this way.) The historic problem with this approach is that it is easy to get wrong and create not just memory leaks, but situations where you trying to use memory that's not yours to use any more. (This is the source of whole range of security vulnerabilities in the real world.)
The language or runtime automatically cleans up things that are not used any more as soon as this becomes "obvious". There are a lot of ways compilers or runtimes can tell when a variable (and/or what it pointed to) is no longer used, and whatever it pointed to can then be cleaned up on the spot. For the most part, this is the approach taken by regular Python. It's convenient and relatively easy to design, but does have some performance implications and sometimes needs some help from a garbage collector anyway (see next point) so it's not always preferred.
Create a "garbage collection" mechanism. Basically, these languages figure out when something is unused after the fact, sometimes using very complicated analysis of what's in memory. Stuff that's been marked "old enough" is slated for cleanup by a reaper that runs in its own thread(s), so that, in theory, cleanup of unused stuff doesn't steal any time from the main program. This is the approach taken by PyPy's version of Python, as well as well-known languages like Java, C# and Go.

It turns out that PyPy's "garbage collector" expects most things a program uses to stick around for a while. A program that creates tons of transient objects and then rapidly discards them (which CoD's Python code definitely does) breaks PyPy's tuning assumptions, leading to a buildup of junk that isn't cleared out as fast as it could be. I think PyPy actually would have run a little faster than my code, except near the end it used a ton more memory than the regular Python version. This caused the VM it ran it in to use swap memory, slowing down its performance.

There's another reason, though. I already do most of the heavy lifting in my code in other, faster languages, rather than pure Python. I already talked about how a lot of the CoH data parsing is done in Rust. I also use a very fast Python library written in C++ to write out the JSON data. The three things the program really does is load the data, turn it into Python objects, and save those objects as JSON. The first and third things probably can't get much faster. That leaves turning the data into Python objects, and this is the place where tons of temporary objects get created and discarded.

I could speed up my extract code by stripping out the step that turns the raw data spit out by Rust into Python objects. That step is not strictly needed to emit the CoD data JSON. If that was removed, the code would basically be a variation on Ruby's tool. I suspect it would shave a good 3-4 minutes off the run time. But the Python transformation step gives me something that I can dump out and easily load back into an interactive Python interpreter later, and this is much easier (and faster) to work with than the final JSON. I use this setup all the time to find answers to questions for things you can't find on CoD itself, or would be very hard to generate. A semi-common example is to spit out a report like "what powers use attribmod X?") Adding 3-4 mins to my build time whenever there's a new CoH release is worth it if it does something that saves me tons of time every time I want to access the data.

This just shows that what's "best" (let alone "fastest") in a given scenario is highly dependent on use cases.

February 28, 2022

Not much new progress yet, but letting y'all know I haven't re-vanished. Still plenty of real-life stuff keeping my hobby coding time low for now.

February 21, 2022

Working on a data map makes clear I don't have explanations for all the struct fields. I'll need to go through and add (and in some cases, dig up) explanations for them. It's not a ton, but it's more than I want to leave unexplained, if I can avoid it.

Edit: OK, this is done. All the fields for all the Rust data structures (what gets turned into the "raw data") have doc comments that explain them to the best of my knowledge. Now I just need to turn that into something I can shove into the site and archives.

February 21, 2022

Thanks for the offers of assistance, gang. I think if I need help with paying for this I'm in deep trouble and would probably need help with other things first, so I am not expecting it, but I still appreciate it.

I just uploaded a couple of minor tweaks.

The raw data archive filename is now dependent on the data set viewed on the current page. For "live" data it's called "raw_data_homecoming.zip" and for "alpha" data, it's "raw_data_cryptic.zip". Those names align with the "revision" name up in the top right of the site ("Homecoming_xxx" for live and "Cryptic_xxx" for alpha), because those are the "internal" data set names I produce.
I discovered that, at least on some browsers, the vertical scrollbar could flicker on and off rapidly during dynamic page changes, like the main "hamburger" menu fly-out when clicked, or while the "busy" icon was spinning. This only happens at some window sizes, so I'm guessing some people never see this, even on browsers subject to it. Anyway, there is scrollbar style that avoids this effect, and I added it to the site's styling. The vertical scrollbar is now an "overlay" scrollbar, meaning it appears on top of content when it's visible, rather than taking up some of the space on the side of the page.

February 20, 2022

So a long-ago requested feature is now live.

This lets you download a zip archive containing the "raw" JSON representing the data that comes out of the Rust library, before I do any additional massaging of it as used in the CoD site. Right now this file is about 83MB.

This is really only helpful for either tool makers, like folks working on something like Mid's, or for the terminally curious. The data in the archive is pretty close to what Ruby's powers project code would emit in "raw data" mode. My code has diverged from Ruby's, but the basics are still very close.

I took some time today to make sure the Rust side doesn't export things that are not "actual" powers data info. This means book-keeping fields added to powers/powersets/powercats that are used while doing things like building power/AT relationships. They aren't actually used any other way, so they always should have been filtered out anyway.

Right now there's no "data map" explaining what all the fields are. That might take a while to produce, but the good news is that the Rust code's own comments can be used to automatically generate most of it. I plan to work on this later this week, and to add it to either the site help, in the archive, or both.

~~PS: If you switch to the Alpha data, you'll get that server's data. The file name is the same either way, currently (I may change that), so be sure to rename it if you download both.~~ (Fixed. See my next post below.)

February 19, 2022

14 hours ago, Uun said:

Don't get old?

Doing my best without considering any extreme measures.

February 18, 2022

I thought you folks might enjoy some details about how CoD is hosted. It's lives in AWS (Amazon Web Services), basically using just two services there.

The first is AWS S3 (Simple Storage Service), which (for the non-techies) is basically a place to store files. Not quite the same as the file system on your computer, but kind of similar, functionally. This is where all the files for the CoD site go. This includes all the data files for powers, critters, etc.

The next is called CloudFront (CF) which is basically AWS "content delivery network" or CDN. A CDN is something you can put between your web server and your end users. The CDN keeps copies of web files geographically closer to the end users, which reduces the time it takes them to download the files if they're far away from your web server. Basically a CDN is a big, geographically distributed file cache for the web. CDNs can do (a lot) more than this, but that's their main function.

CloudFront can serve files out of S3, and that's CoD in a nutshell. I put files in my S3 "bucket", point CF at the same bucket, set up DNS to point cod.uberguy.net at my CloudFront distro, and voila: CoD lives. (AWS provides DNS services too, but uberguy.net isn't there. I had that hosted somewhere else already.

Ever wonder what it costs to run CoD, in terms of actual bills?

AWS billing is suuuuper complicated, but I'm not doing much, so it's simple for me.

S3 charges by amount of data stored and number of uploads and downloads you make to it. AWS is scaled for giant companies like Netflix - I pay fractions of a penny for the data CoD uses. Thankfully I don't have to pay for files that CF reads from S3 to then serve to the web, or if I do, it's so small it doesn't even show in my bills. To limit the number of files I have to upload to S3 (which mostly saves me time), I have software that only uploads files that changed locally. So if I change something in the data extract code I run on my home computer, and it only changes 1000 CoD data files, only those 1000 files will be uploaded to S3 (out of ~50,000 total).
Cloudfront charges for a bunch of things, but the two main ones I see in my bill is data downloaded by y'all and number of "SSL terminations" - basically number of times you folks' browsers establish an SSL connection to the site. Interestingly, the SSL thing was always larger for CoD, partly because I pay more for people connecting from Europe. It probably makes sense - I set the site up to cache intelligently in our browsers, but it's also set to check and see CF has newer files semi-often. Once you've seen a file, you probably don't download it again for days unless I updated it, but if you load it several times a day, you'll reconnect to make sure there isn't a new copy.

This setup is pretty cheap for me to run. Until recently, it was in the low US dollar per month range. Recently, it's started costing way less than a dollar. Sometimes pennies a month.

It turns out that at the end of last year (2021), AWS greatly reduced the cost of Cloudfront and also took a chunk out of S3's pricing. A lot of AWS' services have a "free tier", which is basically a "freemium" style system, where you pay nothing if you use only certain features within certain limits. AWS massively extended the free tier thresholds for Cloudfront, basically causing all my CF costs to disappear. Like, literally. They went to 0. My S3 costs are usually like 10 cents a month. They went to a whopping 63 cents in December when I uploaded 10s of thousands of files for Page 3 updates.

CoD takes time to keep up to date and/or add features to, but at least it really doesn't cost anything to host. I'm not close to broke, but even if I was, I think I could afford this!

As an aside, I can only host CoD this way because it was designed to run completely in browsers. CoD has no server-side logic, no database, etc. A site like this forum can't be hosted fully the way CoD is, because it needs to run server-side software and store dynamically changing data (posts, user accounts, etc.). I'm sure these forums cost the homecoming team way more than CoD cost me even before AWS' prices crashed.

February 17, 2022

15 hours ago, Uun said:

Those exclamation marks are hard to see.

That's good to know. Any suggestions on how to improve it? Bigger font? Something else?

February 17, 2022

Regarding the original question: You have to pay attention to the nesting of the sections in the power, as indicated by the lines on the left-hand side of each row of information.

See how there is a line on the left that runs from that first "100% chance" (above where your picture starts) row down to the first "if target>enttype eq 'critter'".
Then, indented a bit inside of that line there's another line that runs from the 2nd "100% chance" (the first one in your picture) to "if target.HasTag?(Electronic)"? And then there's another, separate line at the 2nd indentation level running from the third "100% chance" down to "if !target.HasTag?(Electronic)". Those nested or split lines show you how the effects are grouped, with the controlling condition at the bottom.

So it reads like this:

If the target is a critter:

The target loses -40% Endurance
The target suffers -500% Regeneration for 45s
The target suffers -500% Regenerationfor 15s
If the target has has the "Electronic" tag:

The target takes 68.... points of Energy damage

The target suffers suffers a 17.88s mag 3 hold

There's a 50% chance the target suffers another mag 1 of the same duration.

If the target lacks the "Electronic tag:

The target suffers suffers a 8.334s mag 3 hold

There's a 50% chance the target suffers another mag 1 of the same duration.

If the target is a player:

etc....

So basically the Electronic tagged things get a longer hold and damage, while others get a shorter hold.

PS: In case you hadn't seen it, the nesting stuff is actually outlined in the help pages.

Edit: Fixing my own misreading of the blocks!

February 17, 2022

10 hours ago, Linea said:

Are you looking at the old EMP Arrow on the out of date CoD?

https://cod.uberguy.net/html/power.html?power=defender_buff.trick_arrow.emp_arrow

Er. CoD isn't out of date. It has been current since about a week after Page 3 landed.

February 13, 2022

On 2/7/2022 at 2:41 PM, UberGuy said:

Yeah, that should be doable.

This is done.

February 7, 2022

1 hour ago, BillyMailman said:

Turns out, from looking at the raw data, that the second and third ones are exclusive, but also directly give two or three copies at once. I have no damn clue why they didn't just have each of the three give one stack, and the distance criteria overlap, but I ain't on the powers team, so not my place to judge.

Anyway, in terms of CoD, I think that this, Blood Thirst (the Build Up replacement in Savage Melee), Tidal Power (the Aim variant in Water Blast) and others like them make a strong case that the display should show the count of stacks of a power being granted. It might only be needed if it's 2+ stacks, to keep from adding unnecessary noise to other powers? Dunno, design's not really my thing.

Yeah, that should be doable. It took a bit for me to understand what you meant, but looking at the raw data, I get it. I honestly don't think I realized this was a thing anywhere, as I don't much play either of those two powersets. I think your suggestion of displaying it only if it's more than one copy makes sense.

It constantly breaks my head how much information is packed into powers definitions, and it's only likely to grow with time as our devs come up with new ideas. It's quite the (ongoing) challenge to display all it in a way that's reasonably compact. It always makes me appreciate what the game itself is dealing with, parsing all this stuff on the fly (in a much more compact format to be sure) to know what to do with our powers. When you think about how many patches, auras and other long-lasting effects can be in play at something like a raid or iTrial, plus all the players and critters activating click powers, it's kind of amazing to consider what all the engine is up.

February 6, 2022

OK, I didn't get to it Saturday, but I did get to it today.

I hadn't realized Page 3 added "Running & Sprints" set category. (That name seems super redundant to me, but 🤷‍♂️.) There is actually still one plain "Running" set, just as there is still one plain "Leaping" set, so powers need to both plain versions and the "& Sprints" versions.

I've fixed this.

Also, I had not updated "Enhance Taunt" to "Enhance Threat Duration" in the code that builds enhancement icons when displaying set icons, which was why the site was saying "not found". That's a generic default message for "I could not load this data", which was happening because of an exception being hit when not finding "Enhanced Threat Duration".

Because I have to remember to update the code to deal with name changes / additions like these, errors like this are likely to happen again in the future. There's no good reason for this to so thoroughly break functionality when it happens. I updated the page logic to display "unknown" icons instead of barfing. Adding the logic to display something special was easy, but I had to add a bit of logic to the game data extract code to explicitly generate these icons, since this red question mark icon doesn't exist in the game files. (The borders and backgrounds do, however.)

They looks like this for "generic" boost types and set types, respectively.

Likewise, the only thing the boostset info pages use the boostset category name for is to look up the background color for the icon. I've changed those pages to simply ignore rendering the background if the category name isn't recognized. This looks weird, but prevents the pages from completely breaking due to unrecognized names the way they did until now.

February 4, 2022

Hm, just noticed these posts.

Ugh, the threat set one is, once again, a bug caused by the name change from "Taunt Duration" to "Threat Duration". It's very likely an easy fix, but I won't get to it tomorrow, as it's already well past my bedtime here. Look for it tomorrow afternoon.

The Kuji-In Rin issue is basically the same issue with a different name. I'm pretty sure it's not handling "Jump & Sprint" sets. Every power with a leaping or running component shows no enhancements because of this. How has no one else noticed that?

Edit: Couldn't get to it today, but tomorrow for sure.

Sign In

Profiles

Forums

Events

Store

Articles

Patch Notes

Posts posted by UberGuy