Rsync workflow instead of git remotes

[WolfgangSteiner]

In the docu it is stated that it should be possible to use rsync to merge work from several computers on one central server. I have tried this out with a test repo and it seems to work well enough. But I am still unsure what the best command line flags for rsync would be. Does anyone with more experience have a good rsync command line that they could share?

I have used something like:
rsync -avpuz -e ssh my_repo/ rsync_storage:archive/my_repo/ --exclude="**/build" --exclude="**/.cache"

This would ensure that the .jj store is always only updated by adding to the database, like we would probably want. But in the case of deleting a source file from the repo, it would not be deleted on the remote, and this is probably not what we want?

I guess my workflow will work (probably with the --delete option) for syncing from a single computer. I can not overstate how freeing this workflow already is in comparison to working with git+github and also jj+git+github!

[hotsphink]

I haven't tried this, I've only been intrigued by the model that allows this. So I will give some uninformed advice.

There's a distinction between the VCS state and the working copy. There's no such thing as deleting a source file from the VCS state, that's all append-only. Well... ok, mostly append-only. But garbage collection is the only thing that should remove anything like a source file from the VCS state.

As for the working copy, I'm not sure there's any use in syncing that over to the server? It should be updated by jj based on the op_head you end up with, and setting it to anything else is just likely to make an unwanted new commit.

So I would probably try rsyncing only .jj and .git, without --delete. And if you have multiple computers sending data to the server, think about what to do with the .jj/repo/op_heads/heads/ directory. (I had more specific advice, but realized I didn't know what I was talking about.) If your rsyncs put multiple heads in there, they're going to end up getting merged when you pull it back down. Which might be what you want? (Is it ok if it contains every op head from when any rsync ran? Hopefully it's fine merging operations with their ancestors.)

I would be tempted to do the rsync in two parts, first skipping the op heads, and then only doing the op heads. If the first rsync dies in the middle, you haven't really "committed the transaction"; you just have some extra unused stuff lying around.

Again, this is based on my theoretical and very probably flawed understanding of the concurrency model. Hopefully someone knowledgeable can chime in.

[emilazy]

FWIW my understanding is that Git repositories themselves are not safe to rsync in the way Jujutsu ones are meant to be, and that the full concurrency‐safety applies only to the proof‐of‐concept native backend. In the face of concurrent modifications, I think you need to exchange Git objects through native Git protocols and then rsync the Jujutsu stuff. But I don’t know how common the cases where things would blow up from plain rsync are.

[WolfgangSteiner]

Thank you for your suggestions! I have created the repo using jj git init without the --colocate part. So I presume, the git backend data is stored inside of the .jj directory and thus there is not really a git state that has to held consistent? But I just guessing here.

The point about only syncing .jj does not seem to work (or I misunderstood). Cloning .jj into an empty directory, jj diff --stat will list all files as deleted in the current changeset. So to me it seems that the files of the working copy also need to be synced.

[martinvonz]

jj git init will still be backed by a Git repo. That's stored in .jj/repo/store/git/. As Emily said, Git repos cannot be rsynced safely, and there's also #2193 to keep in mind. I don't think there will be data loss but you may have to run jj debug reindex after rsyncing.

Cloning .jj into an empty directory, jj diff --stat will list all files as deleted in the current changeset. So to me it seems that the files of the working copy also need to be synced.

The working copy is basically derived information. You can avoid rsyncing it and instead rebuild it from the repo.

[WolfgangSteiner]

Yes, there is of course still a git backend. I was just hoping that it would be more resilient against rsyncing when it is only ever updated through jj itself.

What would be the correct command for restoring the working copy? I tired jj restore, but this results in an empty top changeset, effectively restoring the state to @- instead of just restoring the working copy files.

[martinvonz]

You should be able to run jj evolog to see previous versions of the working-copy commit. The latest one should be the one that deletes all files. You should be able to restore to the previous commit from jj evolog.