lock file missing

[MarkoMin]

To reproduce:

rebar3 new lib abc
cd abc
rebar3 ct

rebar.lock is missing even tho compile is a dependency of ct and it is executed before running tests. If you explicitly run rebar3 compile then lock file is created. I think that lock file should be created in both scenarios.

Lock file is created when running other commands that run in the default profile so probably test profile (ct) is the problem here.

Open to make a contribution.

[ferd]

Only runs for the default builds are locked. The ct command runs with the test profile in each run, which precludes creating the same lock file.

[MarkoMin]

Makes sense. Are profile-based lock-files something to consider for rebar4 project maybe?

[ferd]

They might be. I think we're looking for plugins first. The bit of a challenge with profile-based locks is that people can make any "profile" they want: rebar3 as local,prod,test,deps compile will possibly fetch deps in a different order than rebar3 as prod,test,deps,local compile, so the lock file can grow endlessly based on profile combinations.

Breaking that pattern would require a weirder approach of always downloading all profiles' deps, and then doing a second pass that fetches locked entries from all of them in the new combination, maybe. It could make all builds slower by having to do resolution for all potential profiles on the first invocation. We haven't designed it, but it would definitely require going to rebar4.

[MarkoMin]

I think we're looking for plugins first.

Sure, but it'd be hard to change lockfile format once rebar4 is out. I might put some work into this, but scope of it is pretty large I think, so I won't promise anything. I'll track development of rebar4 and might jump in with this.

Breaking that pattern would require a weirder approach of always downloading all profiles' deps, and then doing a second pass that fetches locked entries from all of them in the new combination, maybe. It could make all builds slower by having to do resolution for all potential profiles on the first invocation. We haven't designed it, but it would definitely require going to rebar4.

Slowing down builds in general would be bad, but slowing down builds only for profile combinations would be acceptable. Lockfile could have "lock section" per profile and one "reference section" per profile combination. Yes, it can grow endlessly, but entries would be of acceptable size so I don't think it will become cluttered.

Example:

#{profiles=>
   #{<<"default">> => [{1, LockEntryDepX},
                       {2, LockEntryDepY}],
     test => [{1, LockEntryTestDep}]},
 combinations =>
     <<"default+test">> => [{default, 1}, {test, 2}]
}.

where LockEntry... is lock entry of one dependency as it is now.

[ferd]

yeah. Currently every pass traverses the deps structure to make sure they're right, since we must assume that both the lock file or what's on disk are potentially incomplete (e.g. you switched branches, or just added a new dep that is not yet locked) so that re-resolution happens every time.

The implication there is that what you lock for a profile will also be a function of which profiles preempt its own values, so there's a very real possibility that if you do something like:

default: a1, b1, c1
test: b2
    transitive: d3 (from b2), e2 (from b2->d3)
other: f1 

default+other+test: a1,b2,d3,e2,f1

then there's a real possibility that running a new build where other adds g1. If g1 has a transitive dep on d1:

default: a1, b1, c1
test: b2
    transitive: d3 (from b2), e2 (from b2->d3)
other: f1, g1 
    transitive: d1 (from g1)

default+test+other: a1,b2,d1,e2,f1,g1

then we must do a mixed lock hierarchy where the other value takes over the test value for lib d because the profile priority order matters (later specs are more specific and act like overrides).

Right now we do a top-down ordering of "just decide as you traverse level-order", but an expanded lock file that uses references like that would need to keep its references constant across all potential fetches and lock values, so you'd need to at least do one full independent dep pass per library (and we need to fetch them and load their configs because deps can override other deps' own deps) to rebuild the local lock index, then do the level-order resolution to pick what wins.

It's doable, it's just more complex and still possibly a bit costlier because building an index shares the entries with other potential profile runs.