UNOBB: Add support for KotOR2 Android obb archives

[DrMcCoy]

KotOR2 for Android from Aspyr is out. Our unobb tool doesn't handle those obb data files.

To recap: obb files are big data files used by Android apps. They don't have a set format; what's inside them is handled by the apk itself.

For the first KotOR game, Aspyr just used normal PKZIP files. For Jade Empire, they made them a virtual filesystem, with zlib compressed blocks of 4096 bytes.

Looking at the files in KotOR2, they seem to be still that virtual filesystem Aspyr introduced with Jade Empire. With a key difference: some files are not zlib compressed, but uncompressed. Might have been that Jade Empire already supports that too in principle, but that just no file there was uncompressed.

This has two implications for our unobb reader:

• The file identification fails, because we depend on there being a zlib header (78 9C)
• The compressed resource index list can't be found, since we depend on there being a compressed chunk beforehand

Manually ripping a resource list block from the KotOR2 obb files yields a zlib compressed resource list of the usual format.

This means, I need to revise our resource index list searching for obb files. Instead of searching backwards for a zlib header and then checking that the previous block metadata looks okay, why not just go by the metadata right at the end of the file (the last 16 bytes)? That should contain the offset of the resource list. Throw on inconsistent data there, and we don't need to check for the a zlib header at the start either.

Another thing I need to fix: check if compressed size == uncompressed size. If so, it's an uncompressed file. Read it directly instead of running it through zlib. Open question: what if the file is larger than 4096 bytes? Is there a block split somewhere or can we just read the whole thing as is?

There's also liblzma in the apk. At first I suspected Aspyr uses that in this obb format version, but I might be wrong. Open question: are all files in the obb either uncompressed or zlib? Or is there lzma too?

In either case, that looks like a nice little puzzle to solve/fix over the holidays for me :)

[Dhamp1r]

Thanks for working on this. But will it be possible in the future to import files back to .obb (all or some)?. This would be very useful, otherwise unpacking is practically meaningless, because the files on the iOS are identical and are not packed to a virtual file system.

[DrMcCoy]

Well, the problem is that I'm not yet sure about all of the features of the format. Each compressed block has 16 byte of metadata (not there on uncompressed blocks) that I don't know the contents of (well, the first 4 bytes are the offset to the start of the block, that I know).

I know enough to extract files, but I'm not sure I can create files yet. That needs a lot of experimenting. But I take pull requests, so feel free to write an obb tool, test it to heck and back and throw us a PR! :)

otherwise unpacking is practically meaningless, because the files on the iOS are identical and are not packed to a virtual file system

Except that you need an iOS device to buy the game on iOS and transfer the files away. I haven't even looked at what KotOR2 and Jade Empire iOS do, to be honest.

[DrMcCoy]

As for the 16 byte of metadata, this could be:

• Offset to the block, as an uint64_t
• Size of the uncompressed block, as an uint64_t

All Aspyr OBB files I've seen are < 2GB, and Android itself only allows files up to 4GB. In all blocks I've inspected, those upper 32 bits are 0, which fits. On the other hand, it might actually be some flags or some other information.

[DrMcCoy]

I made some changed to my local checkout, using the last 16 bytes at the end of the file to find the offset and uncompressed size of the resource index. With that, I'm able to correctly extract the two KotOR2 obb files.

...However, that doesn't match up with Jade Empire. The metadata in the obb files in Jade Empire behaves differently. First of all, the last 16 bytes there to not go to the offset of the start of the resource index. In fact, the values doesn't fit anything; it might just not be a metadatablock at all, but the end of the zlib stream.

Looking at the 16 bytes preceeding the resource index in Jade Empire, the first 4 bytes are an offset th obb file, and that is the start of a block, since I always see the zlib header (78 9C) there. But the 16 bytes preceeding that don't match at all, so that might not be the first block of the file at all.

Not sure what to make of this now. I can extract both types (though I'm unsure anything I do is strictly correct and would work with potential other files, like in new updates), but the path of one doesn't work with the other. I could just add a command line switch to make it user selectable and call it a day. Or try to rig some autodection heurstics. Or both, autodection + manual override. Hmmpf.

I'm not sure if this failure is because they changed the format (and if so, is there even a version marker anywhere in the file?) or if I'm just taking shortcuts in the code that fail to work for all possible valid Aspyr Obb VFS files.

[DrMcCoy]

Oh, also, according to my comments in unobb.cpp, there's also a list of (all?) blocks in the obb file, or at least in Jade Empire's version of it. I don't even remember what I found there, but that would be another thing one needs to understand first before an obb writer is feasible.

[DrMcCoy]

Also found a bug in our Archive extractor code. It normalizes file names through the FileType map but doesn't catch the case of an unknown FileType, which leads to files with an empty extension being created (and in the case of the .DS_Store file Aspyr accidentally packed too, even an extraction failure). That's obviously wrong, and possible impacts the existing Jade Empire unobb as well.

EDIT: Even worse, in the case of files with the same name but different extension, like vertex + fragment shader, this leads in one file overwriting another outright.

[shadowslasher410]

I made some changed to my local checkout, using the last 16 bytes at the end of the file to find the offset and uncompressed size of the resource index. With that, I'm able to correctly extract the two KotOR2 obb files.

I don't suppose you would be willing to push your local commit to your fork? I want to get into modding KOTOR 2, but to do that I need the contents of the OBB files...

[DrMcCoy]

It's too messy to really push it into a branch, but here's a patch:
https://gist.github.com/DrMcCoy/39d81320fda6ebffce1830f654bb2375

Note that this is makes the unobb tool work with the new KotOR2 OBB files, but will break the tool for the old style Jade Empire OBB files. I hadn't really found any time to properly unify the code to work with both, unfortunately.

And also, I of course give no warranties. If you break something, you get to keep all the pieces ;)

And also also, my previous comment still stands, that our archive extractor code (in src/archives/util.cpp, though fixing will ripple upwards) is broken. I had started fixing that by making all the archives classes, at least in xoreos-tools here, keep the full path, if they know it (some don't). But it looks like I've inadvertently thrown those changes away, because I can't find them anymore in my local repo.

Feel free to fix xoreos-tools up in those respects and send a PR our way. Or alternatively, just take the code more like a study on how the new format works, and write your own tool to extract it.

[bleau2000]

i cant install your patch itried multiple method but it does nothing the only thing that happen is that the command start but only to stop immediatly the only thing that appear in the window is a _ can you help me please

[DrMcCoy]

It's a patch you apply to the source code before compiling xoreos-tools, you are aware of that, right?

You need to apply it with patch(1) and then compile xoreos-tools. I.e. this is not really something a user is supposed to do, this is a developer's workflow.

If you are a dev, try these steps and if they fail, describe in detail what you did and how it failed, including any error messages.

If you just want to use xoreos-tools to extract files and have no experience compiling C++, then I'm afraid this won't help you. You need to wait until I (or someone else) find the time to clean up this patch so that it works alongside the old format, and then I can push a new release.

[bleau2000]

thanks for the quick answer i did not know that

[bleau2000]

I just try applying the patch to the source code but
It does the same thing it doesn't give me error the only thing appearing is a "_" by the way im on windows i have tried with gnuwin32 and the official linux on windows 10

[bleau2000]

Do you know any tool i can use to extract kotor 2 obb