Suggestion: Using Neural Network Upscaling Algorithms to Improve Texture Quality

[Quinnocent]

These techniques use machine learning-trained algorithms to generate upscaled textures at a much higher quality level than traditional upscaling and sharpening techniques.

 

This could be used to dramatically improve the quality of all of the game's textures while remaining more or less 100% vanilla-friendly and true to the original design.  I believe City of Heroes is right in the sweet spot to make good use of this.

 

There are open source algorithms available which work very well.  Doing this would be compute intensive, but it would be free.  All you would need is one developer with a fairly recent nvidia GPU.  The results are mostly good enough to not really require after the fact editing, in my opinion, or you would need to do only minimal cleaning of artifacts.  So it wouldn't really be labor intensive.

 

Modders have already made extensive use of these techniques, and lots of older games have AI upscaled texture packs which look really good.

 

r/gameupscale's Getting Started guide describes better than I can, so I'll just post it here.  This subreddit focuses on this kind of conversion.  This guide lists the most commonly used techniques, and it gives examples of such techniques in action in existing game mods.

 

Edited December 23, 2020 by Quinnocent

[SeraphimKensai]

I'm all for improved textures, models, remastered audio, and general VR support.

[Grindingsucks]

How does it affect the end user, playing on an older rig?  I've got a rebuilt desktop using an integrated graphics card, for example.

 

[Glacier Peak]

Can you show an example of your suggestion?

[Quinnocent]

On 12/23/2020 at 4:59 PM, Glacier Peak said:

Can you show an example of your suggestion?

Sure.  Look at the before/after pictures in the examples shown in the article I linked above.  Here are some.

 

Morrowind upscaled with two different neural network algorithms:

 

https://www.nexusmods.com/morrowind/mods/45247

 

https://www.nexusmods.com/morrowind/mods/46221

 

The original version of Final Fantasy 7 with neural network upscaling of some of the background art and textures:

 

https://captrobau.blogspot.com/2019/01/ff7-remako-hd-graphics-mod-beta-released.html

 

Doom:

 

https://www.doomworld.com/forum/topic/99021-doom-neural-upscale-2x-v-10/

 

You can honestly find lots of mods like this.  Like most Bethesda or Bethesda engine games have these mods.  A bunch of other classic games have mods like this too.

[Quinnocent]

On 12/23/2020 at 8:59 AM, Grindingsucks said:

How does it affect the end user, playing on an older rig?  I've got a rebuilt desktop using an integrated graphics card, for example.

 

Ideally, there would be no real negative effects.  Basically, this can be used to create higher resolution textures with a much greater degree of clarity than you'd get with existing non-AI upscaling algorithms.  This is really useful for people with a little extra performance headroom who want better visuals.  Since CoX is such an old game, most people with remotely modern GPU's likely have enough VRAM headroom to enable higher resolution textures.

 

If this were to be implemented, we'd most likely see existing texture options left in place.  So for people who don't enable this mode, they'd get the existing textures and existing performance.

 

The real downside would be a decently larger install size, which is a problem with adding any high-resolution texture pack.  Some people in areas with strict bandwidth controls might not appreciate that.

[Grindingsucks]

Okay, good to know.  Thanks, Quinnocent.  👍

[Naomi]

My "tldr" opinion on this is that I think AI upscaling is more of a novelty that isn't quite there yet rather than something you would realistically want to pursue as a real feature to support and maintain.

 

I can't remember off hand if texture mods let you play with changing the image dimensions as well, but if they do, something like upscaling would be (just my opinion) better released as an optional and occasionally updated pack of some kind from the community, rather than distributed alongside patches or as piggs (texture_library itself is currently around 5 GB.. so depending on how many and how large we're talking, that's a lot for some folks to carry around if they happen to not want to use the feature.)

 

The biggest downside to upscaling is that no matter your algorithm, you aren't actually adding any new detail to the extra resolution being utilized. It's a cheap and effective way of imitating quality by allowing you to see the same existing detail closer to the camera / at higher resolution(s), which is really only half of the equation when it comes to visual quality.

 

It's kind of like re-sizing an image and applying sharpening filters.. unless you add new unique detail, it becomes exponentially wasteful, or in other words, you get diminishing returns. Granted the algorithms in question (some better than others) try to add this detail, it's done purely based on the information already present in the image rather than something unique. Skin, fur, hair, straight crisp lines and details tend to suffer the most. The photos shown on post #894 in this link highlight the effect pretty well. What should be a floor grate is clearly a vague blurry guess at its higher resolution shape.

 

So in conclusion, while it still might look better than nothing, you're also not getting the most bang for your buck v.s. hand authored.

 

Doing this for legacy games like Doom and others (CoH probably included if we're going based off age) is interesting, but it will always carry some burden of extra memory bloat (or CPU/GPU processing if we're talking about doing it in real time for every texture).

 

Most textures (according to NVIDIA) can be as high as ~30% wasteful already, and with upscaling, we're also not just talking about the (e.g.) 512x512 main surface texture, but all of the other supporting ones, in most cases the normal map that would also be doubled in resolution.

 

And so at this point we would have to scratch our heads at which supporting textures make any noticeable difference to include if we didn't outright decide on doing them all in bulk because we decide doing it selectively is taking up too much time (and it would take an enormous amount of time doing it selectively, because as far as CoH's engine is concerned, there's no instant preview of inspecting different texture changes (which you can occasionally get away with, but by and large, most of the time it requires a client restart if it doesn't crash the client or corrupt the surface.)

 

If we ignore the supporting maps entirely because we decide just the main texture is "good enough" then the same selective problem exists, since not all textures in the game are equal in terms of their fidelity or the way they're used and mapped. Some are mirrored and rely on a perfect seam that could become distorted (most costume parts) and tons of objects in the game. Some are also low resolution but are designed to tile favorably (in fact most textures designed to tile don't need to be upscaled or would benefit the worst since they are designed to repeat).

 

I'd say the time in selecting and deciding which get treatment alone is significant enough that it could be better spent elsewhere (hence it being better suited as a community crowd sourced thing), though in hypothetical terms, there is enough memory bloat and change in visuals alone that would very much point to this being a new setting entirely (as you mentioned), requiring code changes to support referencing upscaled versions over the originals.

 

[krj12]

Might be worth doing an example of this, maybe on a small scale on a particular mission or maybe just on a small number of mobs, before it's summarily dismissed.  Sounds interesting and might be worth looking into...  vs revamping the entire graphical system of the game and redoing all assets.

[Flashtoo]

I hate to be a wet blanket, but I am pretty sure I heard somewhere that this game's graphics engine has some kind of issue that makes it handle VRAM really poorly, or something - I don't remember exactly what it was. But the end result was implied to be that swapping out more than a few of the textures for new ones at double the size would cause crashing and incredible slowness.

 

If someone more knowledgeable about the CoH graphics engine has a correction on this, I would love for it to be refuted.

[Quinnocent]

On 1/2/2021 at 10:17 AM, Naomi said:

My "tldr" opinion on this is that I think AI upscaling is more of a novelty that isn't quite there yet rather than something you would realistically want to pursue as a real feature to support and maintain.

 

I can't remember off hand if texture mods let you play with changing the image dimensions as well, but if they do, something like upscaling would be (just my opinion) better released as an optional and occasionally updated pack of some kind from the community, rather than distributed alongside patches or as piggs (texture_library itself is currently around 5 GB.. so depending on how many and how large we're talking, that's a lot for some folks to carry around if they happen to not want to use the feature.)

 

The biggest downside to upscaling is that no matter your algorithm, you aren't actually adding any new detail to the extra resolution being utilized. It's a cheap and effective way of imitating quality by allowing you to see the same existing detail closer to the camera / at higher resolution(s), which is really only half of the equation when it comes to visual quality.

 

It's kind of like re-sizing an image and applying sharpening filters.. unless you add new unique detail, it becomes exponentially wasteful, or in other words, you get diminishing returns. Granted the algorithms in question (some better than others) try to add this detail, it's done purely based on the information already present in the image rather than something unique. Skin, fur, hair, straight crisp lines and details tend to suffer the most. The photos shown on post #894 in this link highlight the effect pretty well. What should be a floor grate is clearly a vague blurry guess at its higher resolution shape.

 

So in conclusion, while it still might look better than nothing, you're also not getting the most bang for your buck v.s. hand authored.

 

Doing this for legacy games like Doom and others (CoH probably included if we're going based off age) is interesting, but it will always carry some burden of extra memory bloat (or CPU/GPU processing if we're talking about doing it in real time for every texture).

 

Most textures (according to NVIDIA) can be as high as ~30% wasteful already, and with upscaling, we're also not just talking about the (e.g.) 512x512 main surface texture, but all of the other supporting ones, in most cases the normal map that would also be doubled in resolution.

 

And so at this point we would have to scratch our heads at which supporting textures make any noticeable difference to include if we didn't outright decide on doing them all in bulk because we decide doing it selectively is taking up too much time (and it would take an enormous amount of time doing it selectively, because as far as CoH's engine is concerned, there's no instant preview of inspecting different texture changes (which you can occasionally get away with, but by and large, most of the time it requires a client restart if it doesn't crash the client or corrupt the surface.)

 

If we ignore the supporting maps entirely because we decide just the main texture is "good enough" then the same selective problem exists, since not all textures in the game are equal in terms of their fidelity or the way they're used and mapped. Some are mirrored and rely on a perfect seam that could become distorted (most costume parts) and tons of objects in the game. Some are also low resolution but are designed to tile favorably (in fact most textures designed to tile don't need to be upscaled or would benefit the worst since they are designed to repeat).

 

I'd say the time in selecting and deciding which get treatment alone is significant enough that it could be better spent elsewhere (hence it being better suited as a community crowd sourced thing), though in hypothetical terms, there is enough memory bloat and change in visuals alone that would very much point to this being a new setting entirely (as you mentioned), requiring code changes to support referencing upscaled versions over the originals.

 

Thanks for the reply!

 

I guess I agree with most of what you're saying.  Maybe it's better if it's something done by the community, at least as a proof of concept.  And yeah, I actually had the same thought about ground textures and the like that are designed for tiling.

 

I do feel like these techniques have sorta "arrived" visually, at least to the extent that they're a noticeable visual upgrade without excessive noise or need for hand curation.  I suggested this specifically after being impressed by the real world results of similar projects for games like Morrowind.  Even HL1 isn't bad, I don't think?  This is the full video that the still you linked is drawn from, right?  Left is stock and right is ESRGAN?  I'm not sure how legit this is, but it certainly looks consistent with ESRGAN results I've seen from other games with a similar age.  The upscaled side looks way better, to my eyes.

 

 

Maybe I can figure it out myself, to test it out?  Would that be possible for an individual to do?  ESRGAN is open source, and there are guides out there to help newbies use it, and my computer is powerful enough to handle doing the crunching necessary, if I'm willing to let it sit there and work for a long time.

 

I also know there are already user-made texture mods for CoX, and we now have a mod manager in development to make replacement easier, and there are tools to unpack the textures (not really sure how to repack them).  There are even guides like this one: https://wiki.ourodev.com/view/Texture_modding.

 

I guess the biggest question I have is this: can you even substitute a texture for a higher-resolution one, via the simple replacement that texture mods use?  Like is that possible at all?  I've been Googling it, and some people say you can't.  Those are all super outdated posts, though, and I have no idea what the Homecoming client is capable of.

[arthurh35353]

I thought there was a problem with the fact that texture scale was baked into the file format here? You can't upscale some of the body textures, because the format-pointer doesn't support it?

[Flashtoo]

On 1/15/2021 at 8:03 PM, arthurh35353 said:

I thought there was a problem with the fact that texture scale was baked into the file format here? You can't upscale some of the body textures, because the format-pointer doesn't support it?

There are ways to hack around that. It's been done.

 

Source: Have done it.

[arthurh35353]

Ah, nice. I know I tried to help upscale the texture/mask for one of the straps, but the format wouldn't let me fix and upload it. Something in the piggs file. 😕

[Flashtoo]

It involves hex editing, unfortunately. In other words, not something you'd want to do often, but goddang those emblems were WAY too low rez...

[arthurh35353]

5 hours ago, Flashtoo said:

It involves hex editing, unfortunately. In other words, not something you'd want to do often, but goddang those emblems were WAY too low rez...

I would love to be able to hand fix the face textures up to at least the later add-on faces. This is something well in my capabilities, but if you can't upscale them due to those settings, it's a nonstarter.

[Sunsette]

This is an old topic, but I thought I'd bump this to see what opinions were on this given the recent rerelease of Mass Effect (in large part based on the strength of AI upscaling with hand touches afterwards). The process has come a long way in a short time.

[GraspingVileTerror]

I think you may need to do some convincing of @Naomi, @Sunsette, as I'm not sure Mass Effect is enough positive proof to quell all those concerns up there.

[Sunsette]

I'm not the expert here -- I won't presume that I can (or should) convince anyone of this who is. If that big rerelease done with a relatively small art crew is insufficient, then I presume there's a good reason. But I figured it was enough of a splash in the scene to make it worth checking if it had altered that opinion any... because goodness there are some pixelated things in the costume creator! Old faces especially.

Edited June 29, 2021 by Sunsette

[SwitchFade]

Or....

 

Use DOWNscaling! Let's turn this mofo into 8 bit goodness.

[Luminara]

2 hours ago, SwitchFade said:

Or....

 

Use DOWNscaling! Let's turn this mofo into 8 bit goodness.

 

See, this is why no-one likes the wealthy elite.  You and your 8 bits.  Did you ever stop to think that some of us couldn't afford that much?  Some of us have to settle for 6 bits, and be grateful to have that much.  Think of the bit impoverished!

[Zewks]

Bit of a necro, but I think some of the reasoning given as to why this wouldnt happen might be not as much of a roadblock today.

 

Many "AI upscales" today, dont actually have to change the resolution of the images. They simply improve on the detail and the aliasing to eliminate blurry and pixelated ones. This would keep the resolution size issue from being a problem while providing an improvement to detail and quality. There are also processes that do increase the resolution, but then use better, more modern techniques to downscale it back to the original resolution for a better image overall.

 

My question is, similar to the games sound files, are the textures something that we can modify on the client side to test this in the data folder?

Edited May 12, 2022 by Zewks

[Herotu]

On 1/18/2021 at 8:04 AM, Flashtoo said:

It involves hex editing, unfortunately. In other words, not something you'd want to do often, but goddang those emblems were WAY too low rez...

But, but... uh...  Hex editing can be automated.

 

Also, here is a reddit thread with an example upscaled image.

... and here it is: 

Spoiler

 

Edited May 9, 2023 by Herotu

[Zewks]

I think "upscaling" might be the wrong term being used.

 

yes, there are ways to take a smaller resolution image and "upscale" it to a much higher resolution image, like the above one. From what has been discussed, the game doesnt support this because its got a limit to the sizes for existing textures.

 

There is, however, a way people are using AI to take a low res image, and having the AI "reprocess" the texture to look better, at the same resolution. No, its not the same thing as upscaling, but it could possibly enhance some of the really low quality ones that were just poorly done

[Communistpenguin]

Man, I would love if some of the most basic stuff could get tweaked a bit. Like the dirt texture, or the grass texture.