lib: substantially reduce syscalls when attempting to create dirs

[landaire]

After filing #7685 I ran some perf traces to try to understand just what was taking so long during these slow operations. The changes in this PR reduces clone time for my large repo from about 10 minutes to 4m30s.

You can see my thought process in the comments of the above task but to summarize:

During checkout we check files/directories being created to ensure that we are not attempting to write to a reserved directory (.jj/, .git/). same_file::is_same_file() is an expensive check that invokes at least 4 syscalls when called in a naive manner (open() and close() for each path -- plus possibly more for getting file info? I haven't counted).

There are a few optimization gaps here that are causing significant slowdowns. The following checklist reflects what I've optimized in this PR, and what still remains:

• create_parent_dirs will be called for each file/directory and for each parent dir in a path try to create it and check if the dir is an illegal name via reject_reserved_existing_path(). There is no caching of directories which have already been created.
• reject_reserved_existing calls same_file::is_same_file() in a loop for all reserved names, but the path which has maybe been created isn't going to change, so its handle could be cached.
• can_create_new_file attempts to create the file then just uses the result as an indicator of whether or not the file is created. However, since we have a File that File can be directly converted to a same_file::Handle and avoid a syscall that currently occurs when converting the Path to a same_file::Handle.
• can_create_new_file deletes the file immediately after. There's probably an opportunity here to not delete the file and re-use it for file write operations.
• Say we have 1000 files in foo/. For each file that's written, reject_reserved_existing is going to make at least RESERVED_DIR_NAMES.len() * 1000 syscalls constructing foo/{reserved_dir_name} paths, testing their existence, etc. Maybe jj might create this dir? But I don't think that should ever happen -- so why not cache the handle if it's created and use a lookup table in reject_reserved_existing to only conduct these types of checks if the handle is resolved? Or alternatively cache that the file does not exist after the first check.

Here are some perf traces of running a jj git clone of my large repo before:

Release: https://share.firefox.dev/4oiSTBw
Debug: https://share.firefox.dev/4qmJBX1

And after:

Release: https://share.firefox.dev/4nK66mH
Debug: https://share.firefox.dev/470W1ed

Checklist

If applicable:

• I have updated CHANGELOG.md
• I have updated the documentation (README.md, docs/, demos/)
• I have updated the config schema (cli/src/config-schema.json)
• I have added/updated tests to cover my changes

[martinvonz]

Thanks! Could you move/copy the PR description to the commit message? See https://jj-vcs.github.io/jj/latest/contributing/

[landaire]

Done. Sorry about that.

[landaire]

I wasn't planning on it, but I addressed the last two optimization opportunities. The changes in this PR right now make my original scenario go from 10 minutes to 4.5, and with the changes applied in this commit I've shaved off another ~30s: landaire-contrib/jj@checkout-perf-improvements...landaire-contrib:jj:push-uvunuzyowqrw

Please let me know if you would like me to merge it into this commit or in a separate PR. I think I need to sleep on it and clean up the code a little bit, I'm unsure if I'm happy with how that commit is right now.

New debug trace: https://share.firefox.dev/4hfSltF.
Release: https://share.firefox.dev/46MxHxO

You'll notice in particular that reject_reserved_existing* is basically gone from the debug flamegraph, and can_create_new_file never calls CloseHandle.

I'll also note that this is still slower than a git clone which runs at 2 min 1s, which is probably expected since the repo is colocated.

[landaire]

Please advise on these .expect() calls. Happy to convert to a proper error if you can recommend an error type, but I don't think these will ever panic in practice.

[yuja]

Since this PR includes two separate changes and touches security code which we should review carefully, I suggest splitting it into "reuse open file" and "reuse previously-created directory" patches. Thanks.

[yuja]

Maybe we can instead add same_file_handle_from_file()/same_file_handle_from_path() helpers? Since "from_file" shouldn't fail with NotFound error, we can make it a bit stricter.

[landaire]

Let's continue this discussion on #7742