RFC: Extend tracking to new bookmarks

[emilazy]

This is a slightly edited and expanded version of my proposal from #4871, intended to replace --allow-new, git.push-new-bookmarks, and git.auto-local-bookmark with a single mechanism that generalizes the existing tracking system and that I hope should be simpler, easier to learn, and more flexible at elegantly adapting to various workflows around remote bookmarks.

I don’t think this solution is perfect because bookmark tracking in general has dodgy semantics, IMO; being used for both the fetch and push end leads to weirdness that @arxanas has pointed out in the past and that we work around with things like trunk() in the default immutable heads. Ideally, it should probably be split into multiple separate concepts for the “direction of travel”. But I feel this proposal would be better than the status quo, the configuration knobs should make it comparably convenient for the important use cases, and hopefully the result would feel less inelegant than the current situation does to me and seemingly many others.

(I’m open to discussion of more fundamental reworks of the tracking system along these lines, but kindly request we do so in another issue, because I feel it’s a can of worms with a lot of UX questions that would need resolving, so I worry about it blocking an incremental improvement.)

Outline

1. We extend the concept of tracking a bookmark on a remote to apply to ones that don’t yet exist on that remote. jj git push will only push bookmarks that are tracked, whether they are new or pre‐existing. This means that the default behaviour will remain as currently, i.e. no --allow-new by default.

2. --allow-new is eliminated and replaced by the more orthogonal jj bookmark track ⟨bookmark⟩@⟨remote⟩ before pushing. If you jj git push and the lack of tracking is the only thing preventing a push, it can print a hint to this effct.

3. It’s fairly likely we’d want to add a --track-on ⟨remote⟩ option to jj bookmark create that sets up the tracking for you.

4. To make the UX more convenient for cases like the single‐remote one where people are usually creating new bookmarks to push to one obvious remote, we introduce an auto-track-bookmarks option, which takes a (list of) pattern(s) like glob:*@origin, or possibly even a revset. This would affect both fetched bookmarks and newly‐created ones.

5. git.auto-local-bookmark = true and git.push-new-bookmarks = true can both be deprecated and replaced with something like auto-track-bookmarks = "glob:*@origin" or auto-track-bookmarks = "glob:*@*". I think this matches the semantics people want for a “one‐remote repository”.

   In a shared repository where you only create bookmarks to push your work but you don’t want everyone else’s work to be mutable, you could do auto-track-bookmarks = "glob:emily/*@origin" based on your templates.git_push_bookmark or manual branch naming. In this scenario, if you don’t have a consistent format (everyone just uses the same namespace but only some bookmarks are owned by you), then your options fall back to the more explicit jj bookmark create --track-on or jj bookmark track or jj git push --named. This would be slightly less convenient than the current case where you can set git.push-new-bookmarks = true but git.auto-local-bookmark = false. (However, it behaves more consistently when cloning a new copy of a repository.)

Advantages

1. This eliminates the weird thing where new local bookmarks “want to be pushed to every remote by default” but are held back by a flag/setting, whereas bookmarks that come down from a remote are associated with it and you need to opt in to syncing them with a local one. I find the current model very strange, and believe the mental model here is much simpler: newly‐created bookmarks are local (…because you’re creating them locally), and tracking is the sole thing that determines whether a local bookmark is synced with a remote on fetch and push.

2. @senekor’s tutorial can introduce bookmark tracking up front as the way we tell Jujutsu which bookmarks we want to be synced with which remotes, which is important to learn and a common source of confusion for new users. This would still mean telling them to set a configuration setting if he doesn’t want people to have to explicitly specify --track-on origin for new bookmarks, but hopefully the setting makes a lot more sense than the current “disable some safety behaviour to work around (what I see as) an exception to the general tracking model” one.

3. Automatic tracking of a subset of bookmarks is useful in other cases, too. In Nixpkgs, I often have to push manual merges to staging or staging-next to resolve conflicts. However, we also create versions of these for each stable release, i.e. staging-25.05 and staging-next-25.05. Being able to set auto-track-bookmarks = "glob:staging*@origin" would be handy for this.

4. This would help pave the way to the request to support tracking multiple bookmarks on different remotes that have the same name but are otherwise unrelated, because the model would be “local bookmarks have a set of remote bookmarks they sync with” and so you could add UI to track a remote bookmark under a different local name.

Unresolved questions

1. How exactly should auto-track-bookmarks behave? Some potential options:

   1. The first time we consider a bookmark – whether new locally or fetched from a remote – we store its tracking status according to the current auto-track-bookmarks.
   2. If a bookmark has not been explicitly tracked or untracked, then operations that care about its tracking status check auto-track-bookmarks.
   3. auto-track-bookmarks takes precedence over manual tracking; you cannot manually untrack a bookmark it matches.

   (iii) seems like the most conceptually elegant option, but I’m mostly ambivalent. (i) and (ii) are only distinguished by what happens when you change your auto-track-bookmarks setting. With (ii) you can’t explicitly “forget” having expressed an opinion about whether a bookmark should be tracked, which seems kind of weird.

2. How exactly should jj bookmark (un)track behave, especially in interaction with auto-track-bookmarks? Do we want the setting to be “timeless” but the command to care about what bookmarks or remotes exist, or should jj bookmark track 'glob:*@*' be equivalent to setting auto-track-bookmarks = "glob:*@*"?

   This basically comes down to what we want the data model to look like, I think. Clearly this proposal is moving in the direction of “local bookmarks have an associated set of remote bookmarks that may or may not yet exist”. Two obvious models:

   1. Store a map from local bookmarks to remote bookmark names; use auto-track-bookmarks as a fallback. This would be compatible with all of the possible auto-track-bookmarks behaviours, I think, but jj bookmark (un)track would behave differently to the setting.
   2. Store a list of every bookmark (un)track command in chronological order; whenever considering a remote operation, start with the assessment of auto-track-bookmarks, then amend it based on each bookmark (un)track in sequence. I believe this is how @senekor was imagining it would work, and it would correspond to the (1.ii) behaviour for auto-track-bookmarks. (Obviously this would be a ridiculous way to actually store it and you’d have some fancy normalized glob intersection structure, but it’s just about the actual end result.)

3. jj bookmark create --track-on has overlap with jj git push --named; it’s unclear to me whether we’d want to eliminate it in favour of this functionality or if we value having a one‐shot create‐and‐push command that takes custom names and sets up the tracking. If we want to keep it, jj git push --named would presumably set up the tracking according to the remotes you’re pushing the new bookmark to, so jj git push --remote ⟨remote⟩ --named ⟨bookmark⟩=⟨rev⟩ would be short for jj bookmark create --track-on ⟨remote⟩ -r ⟨rev⟩ ⟨bookmark⟩ && jj git push -b ⟨bookmark⟩, which would be short for jj bookmark create -r ⟨rev⟩ ⟨bookmark⟩ && jj bookmark track ⟨bookmark⟩@⟨remote⟩ && jj git push -b ⟨bookmark⟩.

4. Maybe we’d want to move away from ⟨bookmark⟩@⟨remote⟩ syntax for tracking if people find the idea of tracking a bookmark that doesn’t yet exist on a remote to be confusing (I personally don’t really find it confusing). We would also probably want to adjust it if we start allowing tracking bookmarks under different names.

5. Is the workflow compromise in (5) of the outline acceptable for the simpler model and increased flexibility? If many people don’t want to track fetched bookmarks, really want all new bookmarks to push with jj bookmark create ⟨bookmark⟩ && jj git push without any additional flags, don’t want to use any of the listed alternatives, and don’t want to define an alias for jj bookmark create --track-on ⟨remote⟩, then this could be a deal‐breaker for the proposal

   I personally think that most use cases for automatic tracking fall under the “single‐person repository, track everything” or “shared repository with a consistent bookmark naming scheme, track only my bookmarks” cases that this proposal handles well, and the available options for this awkward case still seem okay to me. We already make such users pass --allow-new or configure a setting already.

[joyously]

I can relate to number 5, where it all seems like too much hassle.
In my opinion, the "bookmark to branch" bridge is simply for interfacing with Git, and so the implementation should hide all the gory details from the user, and simply behave like Git does, because the user chose Git on init. If this proposal actually makes more sense in the Google world where there is no Git, it needs a rethink, because the tool should be doing more tracking than the human.

So, Git has a staging area where you prepare what to push. JJ got rid of that, but still needs it to push to Git. But the user shouldn't have to work too hard telling the tool what to track where, since that's what the tool is for.

[emilazy]

There’s no Google context here; I don’t know if Google’s setup even has remote bookmarks.

The issue tracking solves is that you don’t necessarily “own” every branch on every remote, that assigning a local bookmark name doesn’t necessarily mean you want to push it out to the world, and that the same branch name can exist on multiple Git remotes. Tracking lets you tell Jujutsu what remote bookmarks you want to keep in sync with a local bookmark. That’s also how we ensure that you don’t accidentally modify commits on another person’s branch.

The proposal here reads complex because it talks about a lot of different approaches we could take and compares it to the status quo, but I think that the eventual command surface and documentation would be quite simple: “you can tell Jujutsu what local bookmarks should be synced with which remotes when fetching and pushing, either with an explicit tracking command or a setting to do it automatically; commits on untracked remote bookmarks will be immutable by default; you don’t have to think about any of this if you use jj git push -c to automatically create bookmarks and never push to existing bookmarks; if you’re the only user of a repository and manually create bookmarks to push, you probably want to tell Jujutsu to automatically track all bookmarks”.

Fundamentally there are multiple common workflows here with conflicting requirements, so I believe that some amount of telling the tool what you want is likely unavoidable. The aim of this proposal is that it should be easy to tell Jujutsu about several of the most common workflows with a single simple setting that has a coherent model.

[martinvonz]

There’s no Google context here; I don’t know if Google’s setup even has remote bookmarks.

It does. We use bookmarks to link a change to a "pending changelist" in Piper. We have a single remote called "piper". When the user runs something like jj piper export xyz, we create, say, cl/123 and point that to the xyz change. We also point the remote bookmark cl/123@piper to that commit to indicate that that's the last seen state. Then the user can update their change as usual and the local bookmark moves along as usual and the remote bookmark stays as usual (no Google-specific handling here). The next time the user exports to Piper, we pass the target of the local bookmark and the remote bookmark to the server. That gives the server enough context to figure out how to update the state in Piper, including doing trivial conflict resolutions if the description changed only locally or only remotely, for example.

[yuja]

I like the idea in general, thanks. Some random comments:

4. To make the UX more convenient for cases like the single‐remote one where people are usually creating new bookmarks to push to one obvious remote, we introduce an auto-track-bookmarks option, which takes a (list of) pattern(s) like glob:*@origin, or possibly even a revset. This would affect both fetched bookmarks and newly‐created ones.

I thought we could add two separate config options for "fetch" and "fetch+push" so we can introduce fetch/fetch+push tracking modes later. (I assumed "push" implies "fetch" because the local bookmark should be kept in sync with the remote to push to.) However, in order to replace the current git.push-new-bookmarks=true behavior, the user would need to set auto-track-bookmarks-for-fetch+push="glob:*@*", which basically means auto-track-bookmarks-for-fetch-only is useless. So I think it's good to add a single option for now.

Unresolved questions

1. How exactly should auto-track-bookmarks behave? Some potential options:

   1. The first time we consider a bookmark – whether new locally or fetched from a remote – we store its tracking status according to the current auto-track-bookmarks.
   2. If a bookmark has not been explicitly tracked or untracked, then operations that care about its tracking status check auto-track-bookmarks.
   3. auto-track-bookmarks takes precedence over manual tracking; you cannot manually untrack a bookmark it matches.

   (iii) seems like the most conceptually elegant option, but I’m mostly ambivalent. (i) and (ii) are only distinguished by what happens when you change your auto-track-bookmarks setting. With (ii) you can’t explicitly “forget” having expressed an opinion about whether a bookmark should be tracked, which seems kind of weird.

I think explicit choice should have higher precedence, so (i) or (ii) seems better.

[senekor]

Thank you, I think this proposal is great. Some thoughts about the unresolved questions:

1. How exactly should auto-track-bookmarks behave?

I think both (i) and (ii) make sense, I would go for the one that's easier to implement, which is probably (i).

2. How exactly should jj bookmark (un)track behave, especially in interaction with auto-track-bookmarks? Do we want the setting to be “timeless” but the command to care about what bookmarks or remotes exist, or should jj bookmark track 'glob:*@*' be equivalent to setting auto-track-bookmarks = "glob:*@*"?

I think (i) is the way to go. I must've expressed myself poorly if I came across as supporting (ii). With (ii), I don't understand what even the difference is between setting auto-track-bookmarks and running jj bookmark track. jj bookmark track seems like just a convenient way to build a complicated function to select bookmarks to track incrementally. I would expect jj bookmark track 'glob:*@*' to apply only to bookmarks and remotes that exist at the time the command is run.

3. jj bookmark create --track-on has overlap with jj git push --named

I think this question should be low priority. The proposal is already much more convenient what we have now with git.push-new-bookmarks. We could decide later whether we want even more convenience with jj bookmark create --track-on, based on user feedback.

4. Maybe we’d want to move away from ⟨bookmark⟩@⟨remote⟩ syntax for tracking if people find the idea of tracking a bookmark that doesn’t yet exist on a remote to be confusing (I personally don’t really find it confusing).

Just another data point, I also don't find it confusing.

5. Is the workflow compromise in (5) of the outline acceptable for the simpler model and increased flexibility?

I think so, yes. Today, people who want to push bookmarks conveniently have to seek out the config option git.push-new-bookmarks. With the proposal, it would be auto-track-bookmarks. Wherever that is documented, there should also be a note recommending the auto-track-bookmarks = "glob:emily/*@origin" combined with templates.git_push_bookmark approach. So then the only people "left out" would be those who are forced into a rigid branch naming scheme by their workplace. Those people are probably best served with templates.git_push_bookmark = '<complicated template involving Jira ticket numbers or whatever>'. Then they can use jj git push --change, which is very convenient, and tracking of remote bookmarks stays the same as it is now.

[emilazy]

I thought we could add two separate config options for "fetch" and "fetch+push" so we can introduce fetch/fetch+push tracking modes later. (I assumed "push" implies "fetch" because the local bookmark should be kept in sync with the remote to push to.) However, in order to replace the current git.push-new-bookmarks=true behavior, the user would need to set auto-track-bookmarks-for-fetch+push="glob:*@*", which basically means auto-track-bookmarks-for-fetch-only is useless. So I think it's good to add a single option for now.

Yeah, my feeling is that splitting fetch and push directions is probably a good idea, but it should be part of splitting up the actual tracking concept rather than adding it on to the automatic stuff while there’s still only one jj bookmark track. For the time being, keeping with one concept seems simpler and better to me, since I think it can still reasonably cover the major use cases.

IMO, tracking conflates a bunch of stuff and seems to be confusing for new users (it’s not clear why “tracking” should affect pushing and mutability, I think). But I also think it’s hard to disentangle in a way that produces a nice UX. I know @arxanas has talked about this.

I think explicit choice should have higher precedence, so (i) or (ii) seems better.

I think both (i) and (ii) make sense, I would go for the one that's easier to implement, which is probably (i).

That’s reasonable.

I think I would prefer (1.ii) over (1.i), as I think it is more elegant and the mental model is simpler to understand – you do not have to worry about “when things appear” and adjusting the configuration setting does not require a bunch of fiddling with manual tracking states. In terms of implementation complexity, I’m not too worried: you can have an abstraction to look up explicitly‐stored tracking information in the data structure and then fall back to assessing the current value of the setting if it’s not present. That is probably even simpler than explicitly storing the status at that point. We could also, perhaps, have jj bookmark list show some kind of indicator of whether the tracking status is from an explicit command or the automatic rules. (It will already need adjusting to be able to show things like @origin (absent) for new bookmarks.)

However, there is a slight question about the data model; see below.

I think (i) is the way to go. I must've expressed myself poorly if I came across as supporting (ii). With (ii), I don't understand what even the difference is between setting auto-track-bookmarks and running jj bookmark track. jj bookmark track seems like just a convenient way to build a complicated function to select bookmarks to track incrementally. I would expect jj bookmark track 'glob:*@*' to apply only to bookmarks and remotes that exist at the time the command is run.

Right. That’s a sensible judgement, I think. The thing is that we are moving away from a track operating solely on remote bookmarks that already exist, so we are in some way making some kind of decision about how much we “defer”, because you can jj bookmark track ⟨bookmark⟩@⟨remote⟩ when jj show ⟨bookmark⟩@⟨remote⟩ will tell you that it doesn’t exist.

Restricting it to existing bookmark names and existing remotes seems sensible to me, though. The model can be:

1. The repository state stores a map of local bookmark name → remote bookmark name × {tracked, untracked}.
2. jj bookmark (un)track ⟨pattern⟩ matches against the pattern against all existing remote bookmarks in addition to ⟨bookmark⟩@⟨remote⟩ for the cartesian product of local bookmark names × remote names. It then stores the appropriate entries in the map.
3. jj git fetch, jj git push, and untracked_remote_bookmarks() look at the map first, and fall back to auto-track-bookmarks if there is no entry.

This implements (1.ii). (I think using a remote bookmark name rather than a remote name is better, because even if we don’t solve the UI issues around tracking a bookmark under another name immediately, it’s a reasonable and commonly‐requested feature and it’s cheap to prepare the data model for it.)

To implement (1.i), you can drop {tracked, untracked} from the map: everything listed is tracked. When a new bookmark is fetched or created, it’s assessed against auto-track-bookmarks, and the relevant entries are stored. jj bookmark untrack deletes entries from the map.

This does mean that if you explicitly jj bookmark untrack and then adjust auto-track-bookmarks to include the bookmark, it will remain untracked, in line with giving explicit choices higher precedence. The map will never get smaller, other than by deletion of bookmarks, unless we added an obscure jj bookmark forget-tracked-status or similar. Removing {tracked, untracked} with the same meaning, and having jj bookmark untrack delete entries, but still assessing things at fetch/push time, would yield (1.iii).

Actually I no longer feel confident which semantics I desire :) I’d probably have to play with it to see how I feel. (1.i) might be fine, but it seems like it would make it difficult to change auto-track-bookmarks to me, if you decide it’s not working well for your workflow: changing it won’t re‐apply all the tracking rules and Jujutsu has no idea which ones were explicit decisions you made and which ones are from auto-track-bookmarks, so we can’t even really have a jj bookmark reassess-tracking-but-please-remember-my-explicit-choices, only something to clobber it all and set tracking status for everything according to auto-track-bookmarks.

(BTW, if we do end up allowing tracking bookmarks under another name, maybe we’d want a template, or pair of templates, to map between local and remote bookmark names for automatic tracking. That could potentially subsume templates.git_push_bookmark. But it’s probably not worth considering this now.)

[senekor]

I don't think there should be a distinction between bookmarks that were tracked explicitly and ones that were tracked "implicitly" by the setting. The auto-track-bookmarks is mostly a convenience, not something that should introduce additional complexity. What if a bookmark is "implicitly" tracked and then I move & push it, something that can only be done with a tracked bookmark. Have I with this act also expressed an explicit intent for that bookmark to be tracked? In that case, changing auto-track-bookmarks shouldn't untrack such bookmarks that were "touched" by the user, even if initially, they were tracked implicitly. I find this very confusing for no apparant benefit.

[emilazy]

Fair enough. Then perhaps we just implement the (1.i) plan where we keep a map of local bookmark name → remote bookmark name, and creation of a new local or remote bookmark stores entries as appropriate.

Edit: Well… actually, the problem is that this is workflow‐dependent so you are likely to want to set the configuration per‐repository, and doing it after cloning doesn’t work. But I guess you can jj config set --repo auto-track-bookmarks ⟨pattern⟩ && jj bookmark track ⟨pattern⟩, which isn’t so bad. It seems worth trying.

[emilazy]

Ah, and I forgot to respond to this:

I think this question should be low priority. The proposal is already much more convenient what we have now with git.push-new-bookmarks. We could decide later whether we want even more convenience with jj bookmark create --track-on, based on user feedback.

I think it’s more that it may be a more orthogonal interface than jj git push --named (I brought it up originally during the discussions of that feature, actually). If you primarily make bookmarks to push, but don’t want auto‐tracking for various reasons, then in the absence of jj git push --named, it’s inconvenient to not have --track-on. But as long as we do have jj git push --named, I agree that the workflows where this change makes things less convenient but you have no other option are very marginal.

[senekor]

the problem is that this is workflow‐dependent so you are likely to want to set the configuration per‐repository, and doing it after cloning doesn’t work. But I guess you can jj config set --repo auto-track-bookmarks ⟨pattern⟩ && jj bookmark track ⟨pattern⟩, which isn’t so bad

Yes 👍 another option is to use scope tables to configure different auto-track-bookmark settings for different (groups of) repositories. I organize my repos "by url", i.e. my clone of the jj repo lives in ~/repos/github/jj-vcs/jj. Work repositories are in ~/repos/<employer>/.... With such a setup, it's very likely that workflows match the directory structure. But I'm sure there are other people who organize their repos in ways where that doesn't work as well, so that's not a silver bullet.

[yuja]

1. How exactly should auto-track-bookmarks behave? Some potential options:

   1. The first time we consider a bookmark – whether new locally or fetched from a remote – we store its tracking status according to the current auto-track-bookmarks.
   2. If a bookmark has not been explicitly tracked or untracked, then operations that care about its tracking status check auto-track-bookmarks.
   3. auto-track-bookmarks takes precedence over manual tracking; you cannot manually untrack a bookmark it matches.

I think I would prefer (1.ii) over (1.i), as I think it is more elegant and the mental model is simpler to understand – you do not have to worry about “when things appear” and adjusting the configuration setting does not require a bunch of fiddling with manual tracking states. In terms of implementation complexity, I’m not too worried: you can have an abstraction to look up explicitly‐stored tracking information in the data structure and then fall back to assessing the current value of the setting if it’s not present. That is probably even simpler than explicitly storing the status at that point.

Based on the current data model, I agree (1.ii) (or some variant of 1.i and 1.ii) is easier to implement. We don't have a trusted list of remotes at jj side. In order to achieve (1.i), we'll need to diff the list of remotes and record tracking states accordingly. I would choose whatever easy/simple in implementation-wise.