Align with GNU Tar when a file name is too long

[gdh1995]

Bug report

Bug description:

Recently I found tarfile may generate a file slightly different with the one made by GNU Tar (https://www.gnu.org/software/tar/), especially when a path name is longer than 100 bytes.

Here's the test code:

# py-tar.py
import tarfile, io
memory_file = io.BytesIO()
tar_obj = tarfile.open(name=None, mode="w", fileobj=memory_file, format=tarfile.GNU_FORMAT)
tar_info = tarfile.TarInfo("abcdef" * 20)
tar_info.type = tarfile.DIRTYPE
tar_info.mode = 0o755
tar_info.mtime = 1609459200  # UTC 2021-01-01
tar_info.uid = 1000
tar_info.gid = 1000
tar_info.uname = "ubuntu"
tar_info.gname = "ubuntu"
tar_obj.addfile(tar_info, None)
tar_obj.close()
memory_file.seek(0)
binary_data = memory_file.read()
# import binascii
# hex_data = binascii.hexlify(binary_data)
# sep = 16
# for i in range(0, len(hex_data), sep * 2):
#     part = hex_data[i:i + sep * 2]
#     print(*(part[i:i+2].decode() for i in range(0, len(part), 2)), binary_data[i//2:][:sep], sep=" ")
with open("py.tar", "wb") as fp:
    fp.write(binary_data)

mkdir -m 755 abcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdef
tar cf gnu.tar --sort=name --owner=ubuntu:1000 --group=ubuntu:1000 --mtime='UTC 2021-01-01' abcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdef/
python ./py-tar.py

As a result, when comparing the generated py.tar and gnu.tar, we may get such a difference:

So I wonder will python might align such a detail on tarfile.GNU_FORMAT with the one of GNU tar?

BTW, here's my environment (I'm on Ubuntu 24.04), and I find the main branch of CPython has a similar Lib/tarfile.py and should have a same behavior difference:

$ LANG=C tar --version
tar (GNU tar) 1.35
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.
$ LANG=C python --version
Python 3.12.3

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs

• gh-130819: Update tarfile.py#_create_gnu_long_header to align with GNU Tar #130820

[picnixz]

Reposting the comment from the PR:

Can you motivate the choice for this? namely is there a real benefit between having an explicit user+mode rather than letting the "defaults"? And more importantly, can you cite the relevant manpage / specs where we can find this?

Note: whether this is accpeted or not, this should be treated as a feature request and not a bug IMO. As such, a What's New entry will need to be created, unless the motivation behind this change is not sufficient (in which case we would close the issue as "not planned")

[gdh1995]

Why did I file this issue?

I once met such a use case:

1. user A packs a folder into a .tar file and then upload it to a cloud server
2. user B, C, and so on will incrementally sync their local data folder with the cloud server
3. Information like file mode, owner, group are not useless in this case
4. To make network traffic minimal, we want those packed tars to keep same on all environments

Then, an old way which does work is:

1. copy files from the local data folder ./foo to /tmp/xxx-foo
2. chmod() on /tmp/xxx-foo and mark files 0o644 and folders 0o755
3. run tar -c --sort=name --owner=root:0 --group=root:0 --mtime='UTC 2001-01-01' ... to generate a .tar file
4. compare file hashes of the local .tar and the one on our cloud server, and skip updating if they're same

Apparently this process contains too much work and is quite a bit slow. So I want to improve it by:

1. use the tarfile module of Python3 to pack ./foo into a .tar file directly
2. save the tar data into a in-memory bytes and then calculate its MD5 hash
3. do comparison and skip updating if possible

However, I found sometimes the generated .tar file buffer has a different MD5, and the root cause is:

1. in Python 3.8, tar headers of files and folders have devmajor = "0" and devminor = "0", while the GNU tar (https://www.gnu.org/software/tar/) generates devmajor = "\0" and devminor = "\0" for common files and folders.
2. this difference has been "fixed" in Python 3.9 on Feb 13, 2020 by bpo-18819: tarfile: only set device fields for device files #18080

When I turned to Python 3.12, I still met a difference, and then I found it's because of this issue:

1. mode, uname and gname in the GNU long header are different with those filled by the GNU tar

Where does 0o100644 / root / root come from?

Copied directly from the source code of the GNU Tar (https://www.gnu.org/software/tar/):

1. Its official Git repo is https://git.savannah.gnu.org/cgit/tar.git ;
2. I find the code about GNU long header is in https://git.savannah.gnu.org/cgit/tar.git/tree/src/create.c#n516
3. The related C functions are: write_gnu_long_link and start_private_header

union block * start_private_header (const char *name, idx_t size, time_t t)
{
  // ...
  MODE_TO_CHARS (S_IFREG|S_IRUSR|S_IWUSR|S_IRGRP|S_IROTH, header->header.mode);
  // ...
}

static void write_gnu_long_link (struct tar_stat_info *st, const char *p, char type)
{
  union block *header = start_private_header ("././@LongLink", size, 0);
  if (! numeric_owner_option)
    {
      static char *uname, *gname;
      if (!uname)
	{
	  uid_to_uname (0, &uname);
	  gid_to_gname (0, &gname);
	}
      UNAME_TO_CHARS (uname, header->header.uname);
      GNAME_TO_CHARS (gname, header->header.gname);
    }
  // ...
}

When git blame the file, I find the first version of this "feature" was imported by: (though without the numeric_owner feature)

• https://git.savannah.gnu.org/cgit/tar.git/commit/?id=c50565d9655329deb983c3df5b151d1fe2032953
• its commit datetime is 2004-08-08 14:20:17 +0000

Why did I named this PR "a bug"?

Um, I don't prefer any of the options "bug" and "feature". My old work is just because:

1. I searched on this repo and noticed the devmajor issue (tarfile fills devmajor and devminor fields even for non-devices #63019), and it was marked type-bug
2. its corresponding PR https://github.com/python/cpython/pull/18080/files didn't include a What's New entry.

If this PR should be a "feature", then how might I write a What's New?

1. Add a line into https://github.com/python/cpython/blob/main/Misc/NEWS.d/3.14.0a5.rst, or any other file?

[picnixz]

If this PR should be a "feature", then how might I write a What's New?

Open Doc/whatsnew/3.14.rst and add an entry for the tarfile module under "Improved modules" and then add a bullet point summarizing your change. The NEWS entry should remain and usually duplicates what's been said in 3.14.rst, but it should be a single paragraph without empty lines as it goes into changelog.html.

[picnixz]

Um, I don't prefer any of the options "bug" and "feature"

Your PR is an improvement and doesn't necessarily fix something (namely there isn't really an issue with not having an explicit owner and mode). Therefore it falls under a "feature" and won't be backported to previous Python versions.

Bugfixes are backported and usually fix faulty behaviours without changing working ones. As such they are more "backwards compatible" (there are of course exceptions but bug fixes should not introduce new features or new behaviours that are backwards incompatible with existing ones to avoid breaking existing code)

[picnixz]

As for the GNU tar, AFAICT, note that we are not necessarily using the root user but the user whose GID/UID is 0 so this should stay the same as well.

For the future of the PR, I'll need to be on my laptop to check some other stuff (being on mobile makes it hard). I'll be able to reply around Wednesday/Thursday

[gdh1995]

I find a TarFile instance has cache members of self._unames: Dict[uid, uname] and self._gnames: Dict[gid, gname], while TarInfo._create_gnu_long_header has no TarFile in its arguments.

To cache the result of pwd.getpwuid and grp.getgrgid, may I add a static member to cache them? Is there any privacy risk if I only cache names for uid=0 and gid=0 (since a Python process may work as a server and live much longer than a tar process) ?

If it's suitable to cache the 2 names, I'll update my PR to implement it.

[gdh1995]

Here's my naive implementation: gdh1995@775e0d3