List (RFC): what kind of file metadata is there?

[arxanas]

Please add/suggest your own.

Without making any particular design suggestions or decisions, I think it would be good to try to enumerate all the kinds of state/context/configuration/metadata/etc. that are relevant for VCSes on a per-file basis, both physically and logically. Ideas from other VCSes are also useful to consider. We can refer to this list in other design work.

• Operating system kind/mode
  • File
  • Executable file
    • Note that Windows doesn't have executable files
  • Directory
  • Symlink
• Conflicts
  • Git uses file-level conflicts
  • Mercurial probably uses out-of-band state? Not familiar
  • jj (now) uses tree-level conflicts
  • Pijul stores cool data structures, but I am unfortunately unfamiliar with where they live in the implementation data model/store
  • Regardless, the per-file mental model is still useful
• Copied/renamed info
  • Both where the file is copied/renamed from and where it might have been copied/renamed to.
  • Git always infers this
  • Mercurial stores this explicitly (interesting discussion on actual usage)
  • jj does neither currently
  • Pijul tracks copy information at the byte-range level, not just the file level, which is then used for conflict tracking and resolution
• Sub-repositories
  • Git has submodules (which are a big UI problem in practice)
  • Git also has git-subtree to kind of "flatten" sub-repositories into the normal commit graph
  • Interesting design in josh
  • Mercurial probably has something? Not familiar
  • Not natively materialized or tracked by jj, just left untouched as part of working-copy snapshotting
• Is checked out
  • In Git and jj, via the sparse-checkout configuration
  • Mercurial probably has an equivalent; not familiar
  • Some designs, like josh, generalize this to configuring files to be materialized at various locations, not just configured via a boolean.
  • I think some other version control systems (Perforce? Clearcase?) let you check out files to arbitrary locations, but I can't find supporting documentation right now
• Is tracked
  • Git and Mercurial allow untracked files
  • jj tries very hard to force you track or ignore them
• Is dirty
  • That is, whether the file is "dirty" in the working copy
  • Git detects via the index
  • Mercurial detects via dirstate
  • jj detects via working-copy snapshotting and then diffing against the parent commit
  • Filesystem monitors (like watchman) also become relevant when calculating this
• Is ignored
  • Git has .gitignore
  • Mercurial has .hgignore
  • jj respects .gitignores
• Is precious
  • Git started https://lore.kernel.org/git/[email protected]/#Z31Documentation:technical:precious-files.txt
  • I'm assuming Mercurial doesn't have this; not familiar
  • Not currently handled by jj; this is a common workflow complaint
• Is locked
  • At least for writes; locking for reads would be unusual? (except as part of an ACL strategy, see later)
  • Primarily useful for centralized version control, particularly for organizations that are working on binary assets that can't be automatically merged
  • Perforce has p4 lock
• Is encrypted
  • Git has systems like git crypt.
  • No jj support.
• Owning commit
  • In some workflows (absorb, squash, "mega-merge"), a file's changes are ultimately meant to go into a certain commit, even though they're not there currently
  • Perforce might support this?
• Blame/annotation info
  • Git has git blame
  • Mercurial has hg annotate
  • I'm sure Pijul has something super cool since all the byte-ranges are tracked
  • No jj support
• Content merge strategy
  • Git has .gitattributes
  • No jj support
• Read/write preprocessors
  • Git has .gitattributes to defined clean/smudge filters
    • Logically subsumes features like end-of-line configuration
  • No jj support
• Large file handle
  • Git typically handles via git-lfs or git-annex, which in turn may be backed by clean/smudge filters as an implementation detail
  • Mercurial might implement? Not familiar
  • No jj support
• Lazy file handle
  • That is, don't materialize the file unless you really need it
    • Sometimes this is because the files are too big; sometimes this is because the files are too many
    • Sometimes you can't easily build a sparse profile because it would be too difficult to directly figure out what files you need
  • Git has "partial clones"
  • Mercurial probably implements? Not familiar
  • No jj support
  • Often related to sparse checkouts and large file handles
  • Often implemented via virtual filesystem (VFS)
• Deleted because of file size
  • In particular to address problems of repository bloat, you may want to delete/purge a file from the history, even from collaborators' copies of the repository
  • This can also be addressed by other mechanisms, such as "partial clones"
• Deleted because of sensitive contents
  • If secrets were accidentally checked into the repository, you may want to delete/purge them, even from collaborators' copies of the repository
  • If the secrets have been pushed somewhere, this is technically only "best-effort", but still valuable in a "forward secrecy" sense
• Access controls and permissioning (ACLs)
  • No Git support
  • Not aware that Mercurial has support
  • No jj support (at least in open-source)
    • Is this implementable with the jj backend traits at present?
  • I recall that Piper has ACLs
  • This is often implemented in terms of sub-repositories, where, if you don't have access to the sub-repository (such as via submodule) as determined by out-of-band permissioning, then you automatically can't access the contents (neither for reads nor writes)
  • Read access: useful for securing sensitive contents and data
  • Write access: useful when implementing "code ownership" systems; at present, this information usually lives in the code review tool instead

[arxanas]

@joyously You always have insightful thoughts/comments, and experience with other version control systems, so I'd be interested to hear if anything comes to mind for you 🙂.

[joyously]

Not sure where it would go in your list

• text versus binary (and then for text: line endings and character encoding)
• file system time stamps
• is encryption considered to be at the file system level?
• for copy/rename info, Breezy tracks both because of using branches as folders
• Pijul tracking bytewise allows to handle binary files as well as text, however, it bloats the database for no additional benefit
• can we assume signing is done per commit, not per file?

A lot of your list can be found at the Wikipedia VCS Comparison page.
One thing about defining "precious" files is that if a file is truly ignored (never touched), you wouldn't need precious. I suggest this is a hack jj should not take part in. If jj continues to use .gitignore even when there is no Git backend, there is something a bit "off".

The VCS named Grace is deliberately centralized and so handles some of these things differently than distributed VCS. It does auto-saves.