There's still some file bloat. Some measures:
edit: the way the .git updater works has an inherent 1.7gb overhead. setting up autobuild to package the difference since last month as an update might be an option, I don't know.
Lossless sprite compression
I took the standard machop spritesheet and embedded it before realizing it's too large to do that with.
Running it through pngoptimizer with bit depth/colors/greyscale/etc turned off leads to a 586kb file, while pngwolf pops out a 553kb file. Comparing it with a xor filter in paint.net shows no difference whatsoever. 65535 or 256 colors instead of 32bpp saves a small amount of space, but you can expect a spritesheet to drop in size by 40~%. It's probably best to stick with 65535 because there should not be any autogen sprite that would need more.
There are likely other tools that can be used. Imagemagick + Imageoptim?
Test on Battlers folder:
Before optimization: 648mb, size on disk 649mb
After optimization: 352mb, size on disk 354mb
Test on CustomBattlers:
Before: 502mb, 509mb on disk.
After: 284mb, 385mb on disk. (these will come up later)
Minor gains:
Battlebacks isn't worth running it on, there are rarely files that gain over a percent.
Battle Animations had a few files with 30% changes in size.
Animations is 24mb by default, but some of the files have already been run through pngcrush or something similar.
Icons is taking up 5mb to do 2.56mb of files.
Characters is a 17mb folder taking up 26mb on an ntfs filesystem. optimizing pngs... did not help much here. The third step helps in this situation.
DELETING UNUSED FILES
tossing the graphics\battlebacks\unused folder is probably a good idea. the entire playerbase_orig folder is replaced already (?).
Is there a difference between \Graphics\Characters\Overworld and \Graphics\Characters\Followers?
There's a spare copy of trainer body 1 in the folder with sprites for biking/overworld.
There's a spare copie of Underwater.png in graphics\title.
There's a copy of itemback in graphics/icons. you could probably look for - copy or - copie for these
We have two versions of the rgss. I'm not sure why the 104E one is there. We also have two copies of the ruby runtime dll.
COMPACTING:
If you've been following along, you'll now have a 3.3gb folder down from something like 3.5. That's like 8 copies of Ocarina of Time or something, making PIF an 80/10 Famitsu game.
Let's use a little-known Windows 10/11 feature. Linux has its own ways, including deduplication and transparent filesystem compression.
Downsides: if you are on a single core laptop from 2008 or so, you should not do this. if you are updating the game it might take a little longer to extract? Kind of bad for loading lots of files, but spritesheets helped.
Upsides: we are running the bare minimum and it is hard to even meaningfully benchmark the effects. using it will flag the folder and new sprites will get compacted (but not pngcrushed or whatnot)
compact /C /S /F /EXE:XPRESS4k /I
If you're reading this on win10, you can do this right now! it's a built in transparent compression.
It can also be done with Compactor or CompactGUI. The /F is there because it's not used to some of these files being compressible. See up there where I mentioned icons taking up double the space it should? Now it's 700kb (from compression) stored in 1.79mb (not optimal, but an improvement.)
Let's see how the whole game does:
3.04 GB (3,272,220,672 bytes) on disk, 3.30 GB (3,551,740,168 bytes) in file size.
SHENANIGANS THAT TAKE EFFORT
- Storing individual autogen sprites as the actual difference between the base sprite and the autogen sprite would be neat.
I don't know if there's any delta encoding library out there that's easy to use atm. I tried doing one by hand, and... it didn't go great. I'm not sure how it'd be handling parts of the original sprite that aren't there on the autogen.
- Loading zips from a sprite update folder with the mkxp-z physfs would allow monthly updates to autoload from a folder and be iterated/mounted into the same filesystem as the rest of the game. Supports zip/7z/some weird options.
final
I don't expect much/any of this to get implemented, but it's good info for users to have. Smarter people than me can find better ways to losslessly crush .png files.
There's still some file bloat. Some measures:
edit: the way the .git updater works has an inherent 1.7gb overhead. setting up autobuild to package the difference since last month as an update might be an option, I don't know.
Lossless sprite compression
I took the standard machop spritesheet and embedded it before realizing it's too large to do that with.
Running it through pngoptimizer with bit depth/colors/greyscale/etc turned off leads to a 586kb file, while pngwolf pops out a 553kb file. Comparing it with a xor filter in paint.net shows no difference whatsoever. 65535 or 256 colors instead of 32bpp saves a small amount of space, but you can expect a spritesheet to drop in size by 40~%. It's probably best to stick with 65535 because there should not be any autogen sprite that would need more.
There are likely other tools that can be used. Imagemagick + Imageoptim?
Test on Battlers folder:
Before optimization: 648mb, size on disk 649mb
After optimization: 352mb, size on disk 354mb
Test on CustomBattlers:
Before: 502mb, 509mb on disk.
After: 284mb, 385mb on disk. (these will come up later)
Minor gains:
Battlebacks isn't worth running it on, there are rarely files that gain over a percent.
Battle Animations had a few files with 30% changes in size.
Animations is 24mb by default, but some of the files have already been run through pngcrush or something similar.
Icons is taking up 5mb to do 2.56mb of files.
Characters is a 17mb folder taking up 26mb on an ntfs filesystem. optimizing pngs... did not help much here. The third step helps in this situation.
DELETING UNUSED FILES
tossing the graphics\battlebacks\unused folder is probably a good idea. the entire playerbase_orig folder is replaced already (?).
Is there a difference between \Graphics\Characters\Overworld and \Graphics\Characters\Followers?
There's a spare copy of trainer body 1 in the folder with sprites for biking/overworld.
There's a spare copie of Underwater.png in graphics\title.
There's a copy of itemback in graphics/icons. you could probably look for - copy or - copie for these
We have two versions of the rgss. I'm not sure why the 104E one is there. We also have two copies of the ruby runtime dll.
COMPACTING:
If you've been following along, you'll now have a 3.3gb folder down from something like 3.5. That's like 8 copies of Ocarina of Time or something, making PIF an 80/10 Famitsu game.
Let's use a little-known Windows 10/11 feature. Linux has its own ways, including deduplication and transparent filesystem compression.
Downsides: if you are on a single core laptop from 2008 or so, you should not do this. if you are updating the game it might take a little longer to extract? Kind of bad for loading lots of files, but spritesheets helped.
Upsides: we are running the bare minimum and it is hard to even meaningfully benchmark the effects. using it will flag the folder and new sprites will get compacted (but not pngcrushed or whatnot)
compact /C /S /F /EXE:XPRESS4k /I
If you're reading this on win10, you can do this right now! it's a built in transparent compression.
It can also be done with Compactor or CompactGUI. The /F is there because it's not used to some of these files being compressible. See up there where I mentioned icons taking up double the space it should? Now it's 700kb (from compression) stored in 1.79mb (not optimal, but an improvement.)
Let's see how the whole game does:
3.04 GB (3,272,220,672 bytes) on disk, 3.30 GB (3,551,740,168 bytes) in file size.
SHENANIGANS THAT TAKE EFFORT
I don't know if there's any delta encoding library out there that's easy to use atm. I tried doing one by hand, and... it didn't go great. I'm not sure how it'd be handling parts of the original sprite that aren't there on the autogen.
final
I don't expect much/any of this to get implemented, but it's good info for users to have. Smarter people than me can find better ways to losslessly crush .png files.