Add CCI (compressed xbox disc) support#2808
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the emulator's capabilities by adding native support for the CCI Xbox disc container format. This allows users to load and play games from Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for the CCI Xbox disc container format, adding a new block driver (cci.c) and integrating LZ4 decompression. The UI has been updated to recognize and load .cci files. Feedback includes addressing an issue where the cci driver's child permissions might incorrectly allow write/resize operations, refactoring duplicated path suffix checking logic, simplifying a redundant condition in cci_read_one_sector, making debug logging in cci_co_preadv fully conditional, using g_path_get_basename() for portability in cci_probe, and relocating the xemu_cci_blockdev_change_dvd_medium function to ui/xemu.c for better modularity.
|
Copying original comments from older PR ========= I do think that some of the issues that Gemini is throwing a tantrum about are valid, specially the OOB error related issue in particular. Secondly, are there alternate tools to make a CCI file, other than just Repackinator, similar to Antangelos XISO tool or similar? Also, I presume that CCI still only works with Cerbios, if the proposed PR is decided to be merged, it would be nice to have a write-up of the format on the xemu-website github repository for end-users to read, and some of the benefits over an xiso (compression, etc) Also, the format will need to be tested with various games to ensure there is no degradation in emulation that isn't experienced on real HW etc... @Triticum0 would probably be a good person to maybe help get some headway on that, as they've been an invaluable resource when it comes to managing issues. ========== There's no PR description and I'm not familiar with this file format. Before this patch is reviewed or considered for merge, a few questions: What's the motivation? Saving some disk space? Anything else? |
|
CCI is a format used by repackinator as noted and supported by Cerbios, the format is used on Xbox Original as well as Xbox 360 Other than Repackinator there is (and i think others)... https://github.com/Team-Resurgent/XboxToolkit CCI format is documented here... https://en.everybodywiki.com/CerBios_Compressed_ISO To address questions... What's the motivation? Saving some disk space? Anything else? Who manages the CCI format spec? What's the maintenance plan for this code? Do you plan to continue maintaining it? What's the existing tooling ecosystem around this file format? What does existing file format adoption look like? How does this format compare to others, like CHD? Why is CCI better than other options (in terms of Xbox preservation, compatibility, usability, etc)? Are there any docs on the CCI format? e.g. https://en.everybodywiki.com/CerBios_Compressed_ISO Does it support multi-partition images? Does it embed additional disc security/manufacturing data? Does the format embed any other metadata? |
This is surprising for sector compression. Is this 8TB number an estimate that includes the 'random' data on disc + video partition? Does the 2TB compressed version also include these data? I saw much more modest (but still useful) compression ratios when gzipping xisos
Any data to support this? Is this format used only by those using Cerbios? You've mentioned Xbox 360--Is the format adopted by any other projects?
How is CHD more complex than CCI? To be clear, I'm not very familiar with either CCI or CHD. Until I know more, it's hard to make a judgement about which might be most suitable for xemu. |
This is with Scrubbing security sectors and the fact CCI compresses sector by sector, which is where you get the small sizes. If you scrubbed a ISO then the size would still be the same obviously due to no compression.
As to how popular I cant quantify the number of users, however this is based on users feedback / questions on the Xbox-Scene discord. Aurora now supports CCI as well as the Net ISO (currently not publicly released yet). I am also in progress adding the format to Xenia.
Compressed Hunks of Data (CHD) is more complex than CCI because it is designed as a full archival container rather than a simple compressed ISO. CHD supports multiple compression codecs, stores detailed metadata (such as track layouts, sector sizes, and hashes), and can represent many storage types like hard disks and CD-ROMs. It also supports features like parent/clone images and strong data integrity checks. In contrast, CCI is a much simpler format created solely for ISO files, using a single compression method with a basic block table and minimal metadata, making it easier to implement and with advantages i explained earlier. Due to CHD's complexity it would be very unlikely for a BIOS to support it. Thus Xemu supporting CCI as well as ISO can only see as a benefit to users as they are currently using CCI as well as ISO.
|
|
Will this/does this have any performance impacts on hardware/ emulator or is entire disk decompress on disk before the disk is read? |
On real hardware not very noticeable delays, however there is one or two games it causes stutter on intro video. There is plans to enable users to set a compression threshold (e.g. don't compress sector if only saving 10 bytes) and a alternative setting to say don't compress sectors containing video files as an example. Emulator wise i would not expect to see any performance issues and one of the reasons LZ4 was chosen due to its very fast decompression. |
|
/gemini review |
|
Cool, just wondering as seen some game just run very poorly if cpu bottlenecked. so didn't know if would have a noticeable impact. Does it have any effects on hdd cache as don't know if cache done on fly or only cached during boot. Examples would be half life 2 or advent rising which look cpu bottleneck. Matt benchmark done so tests for them both on hardware an can both drop to mid teens so don't know it decompression would make it worse. https://www.youtube.com/channel/UCK0UufoB0meRNyzGFIj65Bg |
There was a problem hiding this comment.
Code Review
This pull request introduces support for CCI Xbox disc containers by adding a new block driver (block/cci.c) that handles header parsing, index loading, and LZ4 decompression of sectors. It also integrates this new format into the UI for disc loading, allowing .cci files to be recognized and used. The strisend utility function was added to qemu/cutils.h and util/cutils.c to facilitate case-insensitive file extension checks. Feedback includes a critical issue regarding the introduction of a new LZ4 implementation instead of utilizing the existing vendored one, a missing newline at the end of block/qapi-system.c, and a case-sensitivity problem in the UI's file extension checks for disc images.
…nt/xemu-cci into cci_support" This reverts commit b45c8c4, reversing changes made to f41ca87.
The largest data point I have is for repacked images vs. LZ4 + repacked images, yielding a ~10% savings. In my opinion, this would be better as an external fuse driver instead of xemu aligning with tools focused on piracy. 1 |
re stats: TIL about CSO format. Looks very similar to CCI format described in above link. The compression ratio you've presented seems much more realistic to me for LZ4 sector compression re FUSE driver: I'm inclined to agree. Some filesystems even support transparent compression of files, which would probably outperform this userspace decompression when factoring in caching. As for the footnotes, that is very concerning. I'm not sure if the mentioned tool project is 'focused' on piracy, but including obfuscated links to disc backups is prohibited by xemu project standards. |
🤷 |
My apologies for the link. My intention was to show that the code still exists on GitHub (parentless/wo ref) and that the tool that would be pushed to create these files has nefarious intentions, is still being shared, and would have to be monitored closely.
We added CSO support to Stellar in early 2022, with the idea that there was a large number of titles that required the full disc image to function properly and the best way to store those was through compression. We went with the CSO format since it's been around for ages (since the early days of the PSP), has a healthy ecosystem of tools, and is used in various communities (PSP, GameCube, Wii, etc). Reality is different though and there are only a couple of titles that have sector verification. So, we just implemented sector emulation for those titles instead, because even with a non-blocking drive, some titles still perform poorly (due to having to monitor and purge write-through pages when DMA is bypassed for in-place decompression). Since the end of December 2022, we've been pushing against compressed images, simply out of a desire to have fewer formats (which ultimately only confuse, offer little space savings, and potentially impose performance penalties on users). I don't believe it's xemu's responsibility to manage and maintain various compressed formats. We could debate CSO, CCI, CHD, etc all day, but at the end of it, it's pretty simple. Most users will use a full-disk-image-style backup or a repacked ISO of their own making and won't worry about it once it works for them. Power users who want to save a little storage are more likely to use a fuse driver or use antangelo's (amazing) web tool to repack or simply remove a title or two from their collection. |
5 participants
Add CCI (compressed Xbox disc) support
This adds support for CCI (Team-Resurgent / XboxToolkit-style compressed Xbox disc images) as a QEMU block format so xemu can mount single
.ccifiles on the DVD drive like ISO/XISO.User-facing
.ccilike other disc images (e.g. alongside.iso/.xiso).qmp_blockdev_change_mediumwith formatcciwhen the path ends in.cci(case-insensitive), andrawotherwise—same tray/medium flow as existing images.Implementation
cciblock driver (block/cci.c): CCIM header, sector index, raw or LZ4-compressed 2048-byte sectors; read-only (pwritevreturns-EROFS).CCISectorDecoder: first byte is trailer length; LZ4 data starts atbuf + 1with lengthspan - 1 - trail.strisend()inutil/cutils.c/include/qemu/cutils.hfor case-insensitive suffix checks (driver probe + UI).g_path_get_basename()for portable basename handling.cci_child_perm: child stays read-only (write/resize not restored from parent).DEBUG_XEMU_CCIor build with-DDEBUG_XEMU_CCI=1forCCI_DPRINTFtraces (compile-time).References
Testing
.ccivia the UI and confirm the title boots..iso/.xisobehavior unchanged.-DDEBUG_XEMU_CCI=1and spot-check logs during disc access.