[Question] What ZIP standards does `rc-zip` follow?

[justin13888]

What's the scope of the ZIP standards does/should rc-zip follow? Based on my knowledge, there are two big ones at least:

• PKWARE APPNOTE
• ISO/IEC 21320-1:2015

Everything? The subset that is the ISO spec? The key bits that are most commonly used?

As of writing, the "writing" part of rc-zip has yet to be completed. But what sorts of things would be within scope?

• Character encoding (e.g. CP437, UTF-8)
• Time representations?
• Permissions

[fasterthanlime]

I would honestly bother with the ISO spec when it comes to writing. I think other crates might have scopes where they want to be able to read and write a zip file as is. Personally, I would be happy with just the ISO part.

There's a saying about being liberal in what you accept, but conservative in what you emit, and I think we should follow it here.

[justin13888]

I've been brainstorming more details about the scope for the writing part.

I've collected some questions after analyzing the ISO spec:

1. ISO spec encourages "to support bit 11 of the “general purpose bit flag”
   (Appnote, 4.4.4) being set". Should we start support writing with CP437 (where bit 11 is unset)? (Since Rust strings are valid UTF-8 by default, less testing could be done by skipping supporting CP437 alongside UTF-8 at first.)
2. I think supporting Store and DEFLATE only (as per ISO) is good personally but are there any strong opinions to support any other compression algorithms?
3. For UTF-8 filenames, should we do Unicode normalization? If so, NFC (supported by IRIs) or NFKC ("safer")? It's okay to not support other normalization forms for writing right?
4. We should have at least some functions to automatically detect whether ZIP64 is necessary and dynamically write as appropriate? My use cases will likely be explicit (mostly using ZIP64) but this question is a matter of convenience and runtime behaviour.
5. For storing timestamps in ZIP64, should we populate the extra fields on a best-effort basis (given as much file metadata is provided) or based on the library's target (e.g. NTFS extra field for Windows, UNIX extra field for Linux/macOS)?

[justin13888]

I've been brainstorming more details about the scope for the writing part.

I've collected some questions after analyzing the ISO spec:

1. ISO spec encourages "to support bit 11 of the “general purpose bit flag”
   (Appnote, 4.4.4) being set". Should we start support writing with CP437 (where bit 11 is unset)? (Since Rust strings are valid UTF-8 by default, less testing could be done by skipping supporting CP437 alongside UTF-8 at first.)

2. I think supporting Store and DEFLATE only (as per ISO) is good personally but are there any strong opinions to support any other compression algorithms?

3. For UTF-8 filenames, should we do Unicode normalization? If so, NFC (supported by IRIs) or NFKC ("safer")? It's okay to not support other normalization forms for writing right?

4. We should have at least some functions to automatically detect whether ZIP64 is necessary and dynamically write as appropriate? My use cases will likely be explicit (mostly using ZIP64) but this question is a matter of convenience and runtime behaviour.

5. For storing timestamps in ZIP64, should we populate the extra fields on a best-effort basis (given as much file metadata is provided) or based on the library's target (e.g. NTFS extra field for Windows, UNIX extra field for Linux/macOS)?

Side note, I could see the likely answers based on correctness and compatibility which I'm all for but I raise these questions for everybody to pitch in.

[fasterthanlime]

1. ISO spec encourages "to support bit 11 of the “general purpose bit flag”
   (Appnote, 4.4.4) being set". Should we start support writing with CP437 (where bit 11 is unset)? (Since Rust strings are valid UTF-8 by default, less testing could be done by skipping supporting CP437 alongside UTF-8 at first.)

When writing, I would only write UTF-8.

2. I think supporting Store and DEFLATE only (as per ISO) is good personally but are there any strong opinions to support any other compression algorithms?

I think it's a good starting point. No argument there.

3. For UTF-8 filenames, should we do Unicode normalization? If so, NFC (supported by IRIs) or NFKC ("safer")? It's okay to not support other normalization forms for writing right?

I had never given that any thought. What does APFS do? And yes, I think it's okay to pick one and stick to it until someone complains about it.

4. We should have at least some functions to automatically detect whether ZIP64 is necessary and dynamically write as appropriate? My use cases will likely be explicit (mostly using ZIP64) but this question is a matter of convenience and runtime behaviour.

I think that's what most zip implementations would do, yes. So I agree that we should try to detect it. However, it might restrict the APIs that we can expose. I'm not sure.

5. For storing timestamps in ZIP64, should we populate the extra fields on a best-effort basis (given as much file metadata is provided) or based on the library's target (e.g. NTFS extra field for Windows, UNIX extra field for Linux/macOS)?

I think we should write as many fields as possible there. Also, the output of rc-zip should be the same, no matter what platform it's built for or ran on.

[justin13888]

3. For UTF-8 filenames, should we do Unicode normalization? If so, NFC (supported by IRIs) or NFKC ("safer")? It's okay to not support other normalization forms for writing right?

I had never given that any thought. What does APFS do? And yes, I think it's okay to pick one and stick to it until someone complains about it.

Based on some searching, some sources claimed to use NFD but I can't find any evidence that APFS does normalization on filenames itself besides Finder doing the normalization itself (see my findings in next post).

4. We should have at least some functions to automatically detect whether ZIP64 is necessary and dynamically write as appropriate? My use cases will likely be explicit (mostly using ZIP64) but this question is a matter of convenience and runtime behaviour.

I think that's what most zip implementations would do, yes. So I agree that we should try to detect it. However, it might restrict the APIs that we can expose. I'm not sure.

Yeah, I might refer to some C implementations later because we'll likely need to be able to determine from the get-go whether ZIP64 is required or not before we start writing? Something to be careful of.

5. For storing timestamps in ZIP64, should we populate the extra fields on a best-effort basis (given as much file metadata is provided) or based on the library's target (e.g. NTFS extra field for Windows, UNIX extra field for Linux/macOS)?

I think we should write as many fields as possible there. Also, the output of rc-zip should be the same, no matter what platform it's built for or ran on.

Agreed, although one caveat is if we were writing metadata based on what's available (dependent on platform and filesystem), we may have different behaviours based on what fs library we use. Perhaps rc-zip should expose functions in a way that require explicitness and leave this behaviour up to each IO-specific package (rc-zip-sync, rc-zip-tokio, etc.)?

Additionally, I'm hoping to make it possible to experiment with a rc-zip-monoio package to see whether I could find additional throughput gains for my own use cases (writing many 100GB+ zips).

[justin13888]

Some of my findings with APFS (macOS):

First, I wrote some Rust code which wrote files with different normalizations and see what the std::fs returns. It seems it does no normalization at all.

use std::{fs, path::PathBuf};
use unicode_normalization::UnicodeNormalization;

// Usage
// 1. `touch "tch Å Å ① 1"`
// 2. `echo "Å Å ① 1" > "ech Å Å ① 1"`
// 3.
// ```
// cat << EOF > "cat Å Å ① 1"
// Å Å ① 1
// EOF
// ```
// 4. `cargo run` and read the output

fn d(s: &str) -> String {
    format!("\"{}\" === {:?} ({} bytes)", s, s.as_bytes(), s.len())
}

fn main() -> std::io::Result<()> {
    let original = "Å Å ① 1";
    let original_nfc = original.nfc().collect::<String>();
    let original_nfd = original.nfd().collect::<String>();
    let original_nfkc = original.nfkc().collect::<String>();
    let original_nfkd = original.nfkd().collect::<String>();
    let original_path = PathBuf::from(format!("org {}", original));
    let original_nfc_path = PathBuf::from(format!("nfc {}", original_nfc));
    let original_nfd_path = PathBuf::from(format!("nfd {}", original_nfd));
    let original_nfkc_path = PathBuf::from(format!("nkc {}", original_nfkc));
    let original_nfkd_path = PathBuf::from(format!("nkd {}", original_nfkd));
    
    println!("Unicode normalizations of \"{}\":", original);
    println!("Original: {}", d(original));
    println!("Original path: {}", d(&original_path.to_string_lossy()));
    println!("NFC: {}", d(&original_nfc));
    println!("NFC path: {}", d(&original_nfc_path.to_string_lossy()));
    println!("NFD: {}", d(&original_nfd));
    println!("NFD path: {}", d(&original_nfd_path.to_string_lossy()));
    println!("NFKC: {}", d(&original_nfkc));
    println!("NFKC path: {}", d(&original_nfkc_path.to_string_lossy()));
    println!("NFKD: {}", d(&original_nfkd));
    println!("NFKD path: {}", d(&original_nfkd_path.to_string_lossy()));

    // create some files with std::fs
    fs::write(original_path, original)?;
    fs::write(original_nfc_path, original_nfc)?;
    fs::write(original_nfd_path, original_nfd)?;
    fs::write(original_nfkc_path, original_nfkc)?;
    fs::write(original_nfkd_path, original_nfkd)?;
    
    println!("\nFiles in current directory:");
    let files: Vec<String> = fs::read_dir(".")?
        .filter_map(Result::ok)
        .map(|entry| entry.path())
        .filter(|path| path.is_file())
        .map(|path| d(&path.file_name().unwrap().to_string_lossy()))
        .collect();
    
    for file in files {
        println!("{}", file);
    }
    
    Ok(())
}

Next, I zipped and unzipped a file using the shell script and checked what it returns with ls. Both the pre-zipped file and unzipped file have the same names. Obtaining the filenames via ls seems to not normalize anything. Meanwhile, the exact same filenames copied through Finder appears in NFD form. (TL;DR I cannot conclude what is actually stored in the underlying filesystem.)

Below are the results of my Rust code on different platforms...

macOS (APFS):

Unicode normalizations of "Å Å ① 1":
Original: "Å Å ① 1" === [226, 132, 171, 32, 65, 204, 138, 32, 226, 145, 160, 32, 49] (13 bytes)
Original path: "org Å Å ① 1" === [111, 114, 103, 32, 226, 132, 171, 32, 65, 204, 138, 32, 226, 145, 160, 32, 49] (17 bytes)
NFC: "Å Å ① 1" === [195, 133, 32, 195, 133, 32, 226, 145, 160, 32, 49] (11 bytes)
NFC path: "nfc Å Å ① 1" === [110, 102, 99, 32, 195, 133, 32, 195, 133, 32, 226, 145, 160, 32, 49] (15 bytes)
NFD: "Å Å ① 1" === [65, 204, 138, 32, 65, 204, 138, 32, 226, 145, 160, 32, 49] (13 bytes)
NFD path: "nfd Å Å ① 1" === [110, 102, 100, 32, 65, 204, 138, 32, 65, 204, 138, 32, 226, 145, 160, 32, 49] (17 bytes)
NFKC: "Å Å 1 1" === [195, 133, 32, 195, 133, 32, 49, 32, 49] (9 bytes)
NFKC path: "nkc Å Å 1 1" === [110, 107, 99, 32, 195, 133, 32, 195, 133, 32, 49, 32, 49] (13 bytes)
NFKD: "Å Å 1 1" === [65, 204, 138, 32, 65, 204, 138, 32, 49, 32, 49] (11 bytes)
NFKD path: "nkd Å Å 1 1" === [110, 107, 100, 32, 65, 204, 138, 32, 65, 204, 138, 32, 49, 32, 49] (15 bytes)

Files in current directory:
"tch Å Å ① 1" === [116, 99, 104, 32, 226, 132, 171, 32, 65, 204, 138, 32, 226, 145, 160, 32, 49] (17 bytes)
"Cargo.toml" === [67, 97, 114, 103, 111, 46, 116, 111, 109, 108] (10 bytes)
"nkc Å Å 1 1" === [110, 107, 99, 32, 195, 133, 32, 195, 133, 32, 49, 32, 49] (13 bytes)
"nfc Å Å ① 1" === [110, 102, 99, 32, 195, 133, 32, 195, 133, 32, 226, 145, 160, 32, 49] (15 bytes)
"nkd Å Å 1 1" === [110, 107, 100, 32, 65, 204, 138, 32, 65, 204, 138, 32, 49, 32, 49] (15 bytes)
"org Å Å ① 1" === [111, 114, 103, 32, 226, 132, 171, 32, 65, 204, 138, 32, 226, 145, 160, 32, 49] (17 bytes)
"cat Å Å ① 1" === [99, 97, 116, 32, 226, 132, 171, 32, 65, 204, 138, 32, 226, 145, 160, 32, 49] (17 bytes)
"nfd Å Å ① 1" === [110, 102, 100, 32, 65, 204, 138, 32, 65, 204, 138, 32, 226, 145, 160, 32, 49] (17 bytes)
"Cargo.lock" === [67, 97, 114, 103, 111, 46, 108, 111, 99, 107] (10 bytes)
".gitignore" === [46, 103, 105, 116, 105, 103, 110, 111, 114, 101] (10 bytes)
"ech Å Å ① 1" === [101, 99, 104, 32, 226, 132, 171, 32, 65, 204, 138, 32, 226, 145, 160, 32, 49] (17 bytes)

Linux (XFS):

Unicode normalizations of "Å Å ① 1":
Original: "Å Å ① 1" === [226, 132, 171, 32, 65, 204, 138, 32, 226, 145, 160, 32, 49] (13 bytes)
Original path: "org Å Å ① 1" === [111, 114, 103, 32, 226, 132, 171, 32, 65, 204, 138, 32, 226, 145, 160, 32, 49] (17 bytes)
NFC: "Å Å ① 1" === [195, 133, 32, 195, 133, 32, 226, 145, 160, 32, 49] (11 bytes)
NFC path: "nfc Å Å ① 1" === [110, 102, 99, 32, 195, 133, 32, 195, 133, 32, 226, 145, 160, 32, 49] (15 bytes)
NFD: "Å Å ① 1" === [65, 204, 138, 32, 65, 204, 138, 32, 226, 145, 160, 32, 49] (13 bytes)
NFD path: "nfd Å Å ① 1" === [110, 102, 100, 32, 65, 204, 138, 32, 65, 204, 138, 32, 226, 145, 160, 32, 49] (17 bytes)
NFKC: "Å Å 1 1" === [195, 133, 32, 195, 133, 32, 49, 32, 49] (9 bytes)
NFKC path: "nkc Å Å 1 1" === [110, 107, 99, 32, 195, 133, 32, 195, 133, 32, 49, 32, 49] (13 bytes)
NFKD: "Å Å 1 1" === [65, 204, 138, 32, 65, 204, 138, 32, 49, 32, 49] (11 bytes)
NFKD path: "nkd Å Å 1 1" === [110, 107, 100, 32, 65, 204, 138, 32, 65, 204, 138, 32, 49, 32, 49] (15 bytes)

Files in current directory:
"tch Å Å ① 1" === [116, 99, 104, 32, 226, 132, 171, 32, 65, 204, 138, 32, 226, 145, 160, 32, 49] (17 bytes)
"Cargo.toml" === [67, 97, 114, 103, 111, 46, 116, 111, 109, 108] (10 bytes)
"cat Å Å ① 1" === [99, 97, 116, 32, 226, 132, 171, 32, 65, 204, 138, 32, 226, 145, 160, 32, 49] (17 bytes)
".gitignore" === [46, 103, 105, 116, 105, 103, 110, 111, 114, 101] (10 bytes)
"nkc Å Å 1 1" === [110, 107, 99, 32, 195, 133, 32, 195, 133, 32, 49, 32, 49] (13 bytes)
"org Å Å ① 1" === [111, 114, 103, 32, 226, 132, 171, 32, 65, 204, 138, 32, 226, 145, 160, 32, 49] (17 bytes)
"Cargo.lock" === [67, 97, 114, 103, 111, 46, 108, 111, 99, 107] (10 bytes)
"ech Å Å ① 1" === [101, 99, 104, 32, 226, 132, 171, 32, 65, 204, 138, 32, 226, 145, 160, 32, 49] (17 bytes)
"nfd Å Å ① 1" === [110, 102, 100, 32, 65, 204, 138, 32, 65, 204, 138, 32, 226, 145, 160, 32, 49] (17 bytes)
"nkd Å Å 1 1" === [110, 107, 100, 32, 65, 204, 138, 32, 65, 204, 138, 32, 49, 32, 49] (15 bytes)
"nfc Å Å ① 1" === [110, 102, 99, 32, 195, 133, 32, 195, 133, 32, 226, 145, 160, 32, 49] (15 bytes)

Windows (NTFS) (shouldn't apply since it is uses UTF-16 internally):

Unicode normalizations of "Å Å ① 1":
Original: "Å Å ① 1" === [226, 132, 171, 32, 65, 204, 138, 32, 226, 145, 160, 32, 49] (13 bytes)
Original path: "org Å Å ① 1" === [111, 114, 103, 32, 226, 132, 171, 32, 65, 204, 138, 32, 226, 145, 160, 32, 49] (17 bytes)
NFC: "Å Å ① 1" === [195, 133, 32, 195, 133, 32, 226, 145, 160, 32, 49] (11 bytes)
NFC path: "nfc Å Å ① 1" === [110, 102, 99, 32, 195, 133, 32, 195, 133, 32, 226, 145, 160, 32, 49] (15 bytes)
NFD: "Å Å ① 1" === [65, 204, 138, 32, 65, 204, 138, 32, 226, 145, 160, 32, 49] (13 bytes)
NFD path: "nfd Å Å ① 1" === [110, 102, 100, 32, 65, 204, 138, 32, 65, 204, 138, 32, 226, 145, 160, 32, 49] (17 bytes)
NFKC: "Å Å 1 1" === [195, 133, 32, 195, 133, 32, 49, 32, 49] (9 bytes)
NFKC path: "nkc Å Å 1 1" === [110, 107, 99, 32, 195, 133, 32, 195, 133, 32, 49, 32, 49] (13 bytes)
NFKD: "Å Å 1 1" === [65, 204, 138, 32, 65, 204, 138, 32, 49, 32, 49] (11 bytes)
NFKD path: "nkd Å Å 1 1" === [110, 107, 100, 32, 65, 204, 138, 32, 65, 204, 138, 32, 49, 32, 49] (15 bytes)

Files in current directory:
".gitignore" === [46, 103, 105, 116, 105, 103, 110, 111, 114, 101] (10 bytes)
"Cargo.lock" === [67, 97, 114, 103, 111, 46, 108, 111, 99, 107] (10 bytes)
"Cargo.toml" === [67, 97, 114, 103, 111, 46, 116, 111, 109, 108] (10 bytes)
"ech Å Å ① 1" === [101, 99, 104, 32, 226, 132, 171, 32, 65, 204, 138, 32, 226, 145, 160, 32, 49] (17 bytes)
"nfc Å Å ① 1" === [110, 102, 99, 32, 195, 133, 32, 195, 133, 32, 226, 145, 160, 32, 49] (15 bytes)
"nfd Å Å ① 1" === [110, 102, 100, 32, 65, 204, 138, 32, 65, 204, 138, 32, 226, 145, 160, 32, 49] (17 bytes)
"nkc Å Å 1 1" === [110, 107, 99, 32, 195, 133, 32, 195, 133, 32, 49, 32, 49] (13 bytes)
"nkd Å Å 1 1" === [110, 107, 100, 32, 65, 204, 138, 32, 65, 204, 138, 32, 49, 32, 49] (15 bytes)
"org Å Å ① 1" === [111, 114, 103, 32, 226, 132, 171, 32, 65, 204, 138, 32, 226, 145, 160, 32, 49] (17 bytes)

In all these cases, the filesystems themselves does indeed appear to not do the UTF-8 normalization itself.

If anybody thinks I'm missing something, please let me know. Next observations might be to check the behaviours of the various zip applications across platforms.

[justin13888]

Also my suspicion is that we could very well create multiple UTF-8 encoded filenames with same display but different byte sequences on all the filesystems. In such cases, I'd be inclined to respect that during writing.