Jump to content

DeTexturizer


The Philotic Knight
 Share

Recommended Posts

After I got done with primary work on my CoH Modder, and finished up my Rosetta Stone to help give people information to assist them in creating audio mods, I'm now starting to turn my attention towards visual mods. In this area, I will be substantially more limited, as I am NOT a graphic artist by ANY means.

 

However, in starting research on this area of CoH modding, I discovered .texture files, and became frustrated with trying to do anything with them. I spent the better part of three days seeking out help from OuroDev's website and the Discord, without much success. There's a perl script out there that I wasn't able to get to work, as well as some sort of application called GetTex that I couldn't even manage to compile!

 

There HAD to be a better way to at LEAST get a .texture file into a format that I can open up in GIMP, even if I have no hopes of being able to make any good images. I can at least make something that can get the files into and out of the .texture format. So, I cracked open a Hex Editor, and started looking at a few .texture files byte-by-byte.

 

The result of this work is what I'm calling the "DeTexturizer". ALL that this program does is to turn .texture files (hopefully!) back into their source image files and then back into the .texture files again. It does this by scraping the .texture files for the ByteArray that contains the actual image file, and then writing that ByteArray back to the original filename. It also generates an XML file that allows the program to then "Texturize" the image file back INTO a .texture file.

 

You can find the application at the top of my "PK's Tools" pages here:

 

http://www.cityofplayers.com/pks-tools/

 

Here's a screenshot of the program:

 

image.png.63f5a74e6b08c3bd803dd330ef24f629.png

 

I have a fairly detailed description of how to work the program on that page, but needless to say, you can either "Texturize" or "DeTexturize" the files either one at a time, or do an entire directory at once!

 

This has only been tested on a few types of texture files, so it's still in the BETA phase. If you run across any errors, or any specific files that this program doesn't work on, please let me know and attach the file to your post in this thread so that I can investigate the issue!

 

It's my hopes that with this tool (and an upcoming "Visual Rosetta Stone" for graphical file references) the mod community will have some easy to use tools and references to REALLY start changing the game to whatever they want it to be!

 

Now, after you get the file back to its original format (usually .dds from what I've seen), it's up to you to modify the file yourself. I can't help you with that, as I'm NOT a graphics designer! But, for a start, I suggest you try out GIMP, my preferred image manipulation program. Here you can see a DeTexturized inspiration opened up in GIMP:

 

image.png.5907bed86b6cf3114e20ff992b794815.png

 

I plan to keep improving this program over time to work with more .texture files that it doesn't work with right now, as well as add extra functionality, such as allowing you to create brand new .texture files that haven't before existed. But before I can do that, I have to spend some time reading up on graphics technology, and trying to figure out what the heck all of these "unknown" databits are for that I found in the .texture files!

 

image.thumb.png.25732a19e2985a5f99d9193ce037dd7b.png

 

Please let me know if you have any comments, questions, or concerns about this application!

 

Edited by The Philotic Knight
  • Like 8
  • Thanks 4
Link to comment
Share on other sites

One hell of a community member is all I have to say. I'm no graphic artist either, but I have appreciated Rosetta Stone and the CoH Modder Tool and I can see this program will help out graphic artists greatly. With this tool hopefully graphic artists will step in and we will start seeing some really cool visual additions to the game more often. Great work, @The Philotic Knight! As always you have my support!

Edited by Solarverse
  • Like 1
Link to comment
Share on other sites

I have just ran some tests on random .texture files in my source directory, and found a bunch that didn't work. I don't know why, but they seemed to contain some sort of additional header information as well as another set of binary data (almost like an image inside the image!). I don't know what this stuff is, but I found that if I ignore it, and find where the actual DDS file starts (luckily it starts with the ASCII string "DDS", so that was my key), then what I can do is tear that stuff out, and write the rest of the bytes to a file, which makes the DDS file open correctly. Such as, for example, this semi truck from the game (Semi_Cab01.texture):

 

image.thumb.png.f7e5dc544010509cbebfadb353919fe4.png

 

And this boring old concrete looking wall pattern (P_Tnl_B_C_Wall_front_det.texture), which happens to be (from what I can see), the largest texture file in the game (to the tune of 5,462 KB):

image.thumb.png.dec8369fd9af8bf54f2030d575423247.png

 

Now the "extra junk" I am NOT throwing away, because it's probably needed somehow by the game to make the file work. So, since I don't know what to do with it... what am I doing with it? I'm encoding the raw unknown bytes that I strip out into a Base64 encoded string, and storing that string in the XML file as well. That way, when you go back to re-encode the file back into a .texture file, all I have to do is yank that data out of my dataset, convert it back from a Base64 encoded string into a byte array, and then write that byte array to disk, between the filename string and the actual data for the image file itself, right back where I found it at.

 

Since this appears to work flawlessly so far with all of the dozen or so random .texture files that I've tested this on, I'm going to consider this program now out of beta, and up the version number to 1.0. So, the compiled program and source code can be found on my website if anyone wants to give it a try.

 

I might just make my own version of the Inspirations as a sample mod to put up in CoH Modder. It's always bugged me that the Tier2 and Tier3 inspirations look almost identical aside from the slightly different sizes of icons, so I might just open them all up and throw in a number in the bottom left corner of all inspirations that actually explicitly TELLS you the tier, rather than you having to guess at the size.

 

I think next I'll turn my attention to the "Visual Rosetta Stone" now to collect and categorize all of these texture files, so that people have some sort of context about what these things are, and where they are used.

 

But with this tool at least, regular users can now start to "extract" the files, modify them, and then convert them back into the format that the game needs to run.

 

As always, please let me know if you run across any texture files that this program cannot process. After this latest update, I haven't found one yet!

 

  • Like 2
Link to comment
Share on other sites

I tried this out, and it seems to work. Nice one, PK. It would be a little easier to use, however, if you could separately select an image file and a .texture file and have it copy the header and name from the .texture file, rather than just choosing an image file and having the program look in the same folder for an .xml file with the exact same name.

 

Edit: Also, it'd be nice if we could name the resulting .texture file anything we want. CoH will refuse to start if even a single .texture file's name doesn't match what the header says it should be. I don't know if that would be as easy to get around as changing a string in the .texture header, and I don't have the programming know-how to find out.

Edited by Vanden
Link to comment
Share on other sites

I'm not sure what you mean by that Vanden. The program is designed to use its own XML files that it creates itself when it DeTexturizes.

 

Are you suggesting that when Texturizing, I point to an existing Texture file to extract its header information and then overwrite the file itself with the graphics data from the image file chosen?

 

If so, I hadn't thought of that, and I'd consider it. But the hopes here with the XML file was that at SOME point, someone could help me make sense of the header data that I've extracted, and the XML data could then also feed into the ability to modify other settings in the texture file, like the height and width and whatever else the header is storing.

Link to comment
Share on other sites

On 10/2/2020 at 4:14 PM, The Philotic Knight said:

But before I can do that, I have to spend some time reading up on graphics technology, and trying to figure out what the heck all of these "unknown" databits are for that I found in the .texture files!

 

image.thumb.png.25732a19e2985a5f99d9193ce037dd7b.png

 

The extra texture header data there just gives the game some extra info on how the texture is encoded, and you probably don't actually need them if you're already extracting the entire .dds file. But for the record:

unknownInt = a set of flags with various info on the texture format, I can send you the values if you want

unkownFloat1 & unknownFloat2 = "fade" values -- not sure how to explain this one but it's used in some sort of texture fading/blending, first value is 'close' mix and second is 'far' mix

unknownByte = a boolean indicating whether the texture has an alpha channel

unknownString = "TEX" for version 1 and "TX2" for version 2, I think they should all be TX2 at this point

  • Thanks 1

"We're out of options, I'll have to use the jetpack," I said, strapping on the jetpack and ignoring the many non-jetpack options still left.

Having trouble deciding your next alt? Just need a cool name? Try out City Suggests

Looking for powers data? Try the Powers API

Link to comment
Share on other sites

1 hour ago, The Philotic Knight said:

I'm not sure what you mean by that Vanden. The program is designed to use its own XML files that it creates itself when it DeTexturizes.

 

Are you suggesting that when Texturizing, I point to an existing Texture file to extract its header information and then overwrite the file itself with the graphics data from the image file chosen?

 

If so, I hadn't thought of that, and I'd consider it. But the hopes here with the XML file was that at SOME point, someone could help me make sense of the header data that I've extracted, and the XML data could then also feed into the ability to modify other settings in the texture file, like the height and width and whatever else the header is storing.

Yeah, it sounds like you got the idea. The old PiggViewer, when it worked, would let you just pick a .texture file and create a new one to replace it from a .dds or .bmp file, which I assume it did by copying the header right then and there.

 

And the ability to give .texture files any arbitrary name is just me wanting to know if the /macro_image command would be able to use custom-made icons that don't replace existing ones.

Link to comment
Share on other sites

28 minutes ago, RubyRed said:

The extra texture header data there just gives the game some extra info on how the texture is encoded, and you probably don't actually need them if you're already extracting the entire .dds file. But for the record:

unknownInt = a set of flags with various info on the texture format, I can send you the values if you want

unkownFloat1 & unknownFloat2 = "fade" values -- not sure how to explain this one but it's used in some sort of texture fading/blending, first value is 'close' mix and second is 'far' mix

unknownByte = a boolean indicating whether the texture has an alpha channel

unknownString = "TEX" for version 1 and "TX2" for version 2, I think they should all be TX2 at this point

Where did you get all this info Ruby??? Digging through how the code handles texture files???

Link to comment
Share on other sites

16 hours ago, Vanden said:

Yeah, it sounds like you got the idea. The old PiggViewer, when it worked, would let you just pick a .texture file and create a new one to replace it from a .dds or .bmp file, which I assume it did by copying the header right then and there.

 

And the ability to give .texture files any arbitrary name is just me wanting to know if the /macro_image command would be able to use custom-made icons that don't replace existing ones.

I've given it some thought, and my greatest fear is that doing something like that can cause users to corrupt their files. If, for example, they take an inspiration icon DDS and try to point it at a wall texture file... I don't know what that would do exactly if I did as you suggested, but it probably wouldn't be good, and it'd probably make their game break. Now, I'm willing for a compromise here, since I don't really like the extra files hanging around anyways, so how about this - the program will be rewritten for Texturization to ask you to point to a DDS file. Then it just finds the appropriate texture file of the same name and pull the texture header from there and ONLY swaps the image bytes. No XML file, all of the data is transferred live in memory.

 

What do you think of that as a sort of compromise position?

Link to comment
Share on other sites

1 hour ago, The Philotic Knight said:

I've given it some thought, and my greatest fear is that doing something like that can cause users to corrupt their files. If, for example, they take an inspiration icon DDS and try to point it at a wall texture file... I don't know what that would do exactly if I did as you suggested, but it probably wouldn't be good, and it'd probably make their game break.

Can't you already do this, by just naming a mismatched image file the same as one of the .xml files the program produces? And wouldn't that just result in an invalid .texture file anyway? It's not like your program overwrites any actual source files.

 

1 hour ago, The Philotic Knight said:

Now, I'm willing for a compromise here, since I don't really like the extra files hanging around anyways, so how about this - the program will be rewritten for Texturization to ask you to point to a DDS file. Then it just finds the appropriate texture file of the same name and pull the texture header from there and ONLY swaps the image bytes. No XML file, all of the data is transferred live in memory.

 

What do you think of that as a sort of compromise position?

I suppose that does help cut down the file bloat. We still wouldn't be able to give .texture files any arbitrary name, though...

Link to comment
Share on other sites

17 hours ago, Vanden said:

I suppose that does help cut down the file bloat. We still wouldn't be able to give .texture files any arbitrary name, though...

Well you always can rename the texture file any time you want to... but if you rename it something that's not listed in the source code, the game won't find it. So I guess I'm trying to figure out what you're trying to accomplish by renaming things. You'll just end up creating useless files that the game won't load, and that may also crash your game instance if you have them in the data directory.

Link to comment
Share on other sites

4 hours ago, The Philotic Knight said:

Well you always can rename the texture file any time you want to... but if you rename it something that's not listed in the source code, the game won't find it. So I guess I'm trying to figure out what you're trying to accomplish by renaming things. You'll just end up creating useless files that the game won't load, and that may also crash your game instance if you have them in the data directory.

Are you saying that the DeTexturizer already writes the file names of the .xml and .dds files it uses into the .texture header instead of just copying the header wholesale? Because I know you can't just rename a .texture file, the game won't work if you do that. I want to at least know for sure if the game will have a problem with new .texture files even if the file name and header match. My concern is that if I were to update the erroneous icon pack to include i25 or later icons, anyone not playing on Homecoming that tries to use it won't be able to launch the game.

Link to comment
Share on other sites

Well @Vanden this is how I think it works from what I can understand of the code, though of course I may be wrong:

  1. There's an internal file name listed in the header that's a reference to what the final filename SHOULD be when CoH extracts the file from the texture.
  2. There's the actual filename of the file that was inserted into the texture file. The path doesn't matter, but the filename itself needs to match the internal filename referenced.
  3. There's the filename of the texture itself. That ALSO needs to match the filename (minus the extension) of the image file. I believe there's code in there somewhere that strips off  the extension and neutralizes the case for the sake of this comparison. 
  4. The game, when seeking out resources, seeks out the texture filename that's listed in the source DEF files for the resource.
  5. Once it finds this, it extracts the contents of the file and does a check to make sure that the names all line up properly.
  6. If they do, then it loads the resource.
  7. If they don't, then the game throws an error to the user. 

However, if for example there is a file that you're wanting to put in the game like "Astonishing_Excitement", I don't think the code will even know to LOOK for the file if it's not listed in the def file source code. If you have "extra" files in the directory that aren't referenced by code, the game won't care, because the game won't see them, they'll be essentially invisible to the game.

 

So your worries of putting in Homecoming files, and having them cause errors on non-Homecoming installations, I don't think it's anything to worry about.

 

However, for the files that ARE referenced in the game, all the filenames have to match up exactly from what I can see, you can't have the texture file named "Astonishing_Excitement" and have the internal reference say "Astonishing_Excitement2" and have the actual internal file have a name of "Ast-Exc2". That WOULD cause an error, I think they all have to match. 

Edited by The Philotic Knight
Link to comment
Share on other sites

8 hours ago, The Philotic Knight said:

However, if for example there is a file that you're wanting to put in the game like "Astonishing_Excitement", I don't think the code will even know to LOOK for the file if it's not listed in the def file source code. If you have "extra" files in the directory that aren't referenced by code, the game won't care, because the game won't see them, they'll be essentially invisible to the game.

I just tested this out, giving a .texture file a random name and the game refused to start. Then, just in case the game normally uses a .texture file named Rumspringa, I tried again with a file named after the opening lyrics for Your Love by Outfield, with the same result. So it looks like there's no list of .texture files the game expects to find, it just looks at every .texture file it can find in the data folder at startup.

Link to comment
Share on other sites

Well, that's unexpected. Must be some sort of safeguard to make sure nobody messes with an install.

 

However, the game DOES contain def files that reference specific resources files like textures, so it looks like it's both. So probably best if we always replace "the same" with "the same" @Vanden.

Edited by The Philotic Knight
Link to comment
Share on other sites

  • 3 weeks later
2 hours ago, therealtitanman said:

sorry, tested every ways on the source, no way i can convert the neither .dds nor .texture file back to piggs.

 

i already did some clean up for the hurricane blind spot. 

 

i cant find a way sent the file back.

You shouldn't have to push them back into piggs, just put the de-pigged files in the data directory in the "correct path", and it should show up in your game. That's how client side mods work.

Link to comment
Share on other sites

41 minutes ago, ShardWarrior said:

Can this be used to do costume mods of existing costume textures?

As far as I'm aware, I don't see why not. ALL the tool does it take an image out of a texture file, and then put the image file back into the texture file. What you do with the image file in between those two steps is up to you. I can't advise you as to that, as I'm not a graphics expert, I'm just a code monkey. But I've already used the tool to modify Inspirations in-game, so theoretically it should be possible to modify any image file.

  • Thanks 1
Link to comment
Share on other sites

Just now, The Philotic Knight said:

As far as I'm aware, I don't see why not. ALL the tool does it take an image out of a texture file, and then put the image file back into the texture file. What you do with the image file in between those two steps is up to you. I can't advise you as to that, as I'm not a graphics expert, I'm just a code monkey. But I've already used the tool to modify Inspirations in-game, so theoretically it should be possible to modify any image file.

yes is possible if you able to find the .textures files.

the textures aka graphic scatter all over the piggs files.

Link to comment
Share on other sites

27 minutes ago, ShardWarrior said:

Does anyone know where costume texture files are located?  I would not mind taking a stab at a mod or two.

Looks like perhaps they are all under data\texture_library\V_PLAYERS\AVATAR\ and data\texture_library\PLAYERS\AVATAR\?

 

image.png.a9919ba0cdfe80e14c6cdd9933517520.png

 

image.thumb.png.1399a14371a46b2e0f4c8a469be7ab99.png

 

I found this using the "Everything" app, which is freaking AWESOME by the way:

 

image.png.5b45796904b845fe4ef674257e6585ce.png

 

Edited by The Philotic Knight
  • Like 1
  • Thanks 1
Link to comment
Share on other sites

  • 2 weeks later
On 10/8/2020 at 6:17 PM, Vanden said:

I tried this out, and it seems to work. Nice one, PK. It would be a little easier to use, however, if you could separately select an image file and a .texture file and have it copy the header and name from the .texture file, rather than just choosing an image file and having the program look in the same folder for an .xml file with the exact same name.

 

Edit: Also, it'd be nice if we could name the resulting .texture file anything we want. CoH will refuse to start if even a single .texture file's name doesn't match what the header says it should be. I don't know if that would be as easy to get around as changing a string in the .texture header, and I don't have the programming know-how to find out.

I've just updated the DeTexturizer to have a slightly new interface:

 

It now has an extra checkbox:

image.png.f34c8f452db00f2d72b8ef39a4873c6a.png

 

The default behavior of the program is now as follows:

  • If you choose an image file, it'll "Texturize" it into a texture file - The Export XML checkbox does nothing
    • If I can find one of my XML files, I'm using that first to fold into the new texture file
    • If I can't find one of my XML files, then I'm searching for an existing texture file of the same name, and pulling the neccesary data from there. If I can find it, I use it, if I can't, then I error out, since I don't know what to do with a regular image file without having the texture file to tell me what to do with it. As in the past, I am making a backup copy of the texture file before replacing it.
  • If you choose an XML file, then I'm searching for an image file of the same name, if I find it, I Texturize them. If I don't find it, I error out. The checkbox does nothing
  • If you choose a Texture file
    • If the "Export XML" checkbox is checked, then I go back to the "old" behavior of exporting both an XML file and the image file
    • If it's not checked, then I just export the image file and the rest of the data disappears.

Good enough, @Vanden?

 

You can get the latest version here: http://www.cityofplayers.com/pks-tools/

Link to comment
Share on other sites

 Share

×
×
  • Create New...