Should we re-consider zinc's incremental compilation?

[ittaiz]

Given zinc 1.0 (http://scala-lang.org/blog/2017/11/03/zinc-blog-1.0.html)
I'm wondering if we should take another look at if/how to integrate zinc's incremental compiler into rules_scala.
@johnynek @gkossakowski wdyt?

[johnynek]

I think this is an interesting question. I think the problem is we would need to basically implement a build system in memory to do it.

If we hashed all the jar inputs to a rule, you could use that as a key to a mutable in memory filesystem that keeps all the state for zinc. Then zinc could run statefully with that Key.

That circumvents the intent of bazel: stateless compilers with all state managed by bazel.

Alternatively we could push to add a way for bazel to track compiler state. I have contemplated this in the past where rules have optional state with invalidation policies. When the rule runs bazel passes it the state. The rule can say things like a dependency change invalidates state, etc...

[ittaiz]

Let's try and discuss this at the conference with Damien/Ulf/Dmitry

…

On Sat, Nov 4, 2017 at 5:14 PM P. Oscar Boykin ***@***.***> wrote: I think this is an interesting question. I think the problem is we would need to basically implement a build system in memory to do it. If we hashed all the jar inputs to a rule, you could use that as a key to a mutable in memory filesystem that keeps all the state for zinc. Then zinc could run statefully with that Key. That circumvents the intent of bazel: stateless compilers with all state managed by bazel. Alternatively we could push to add a way for bazel to track compiler state. I have contemplated this in the past where rules have optional state with invalidation policies. When the rule runs bazel passes it the state. The rule can say things like a dependency change invalidates state, etc... — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#328 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIF3eBDL9FclgzGqIYwScr9oKYEsBqks5szH8_gaJpZM4QR_9h> .

[gkossakowski]

I agree that rules (or actions to be precise) not being able to carry an internal state is problematic. However, one could use zinc purely within one target and in-memory (in worker mode). See this discussion about bazel+zinc https://gitter.im/sbt/zinc-contrib?at=59f725fdb20c6424296673a1

…

On 4 November 2017 at 09:12, Ittai Zeidman ***@***.***> wrote: Let's try and discuss this at the conference with Damien/Ulf/Dmitry On Sat, Nov 4, 2017 at 5:14 PM P. Oscar Boykin ***@***.***> wrote: > I think this is an interesting question. I think the problem is we would > need to basically implement a build system in memory to do it. > > If we hashed all the jar inputs to a rule, you could use that as a key to > a mutable in memory filesystem that keeps all the state for zinc. Then zinc > could run statefully with that Key. > > That circumvents the intent of bazel: stateless compilers with all state > managed by bazel. > > Alternatively we could push to add a way for bazel to track compiler > state. I have contemplated this in the past where rules have optional state > with invalidation policies. When the rule runs bazel passes it the state. > The rule can say things like a dependency change invalidates state, etc... > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <https://github.com/bazelbuild/rules_scala/issues/ 328#issuecomment-341904423>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/ ABUIF3eBDL9FclgzGqIYwScr9oKYEsBqks5szH8_gaJpZM4QR_9h> > . > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#328 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAQeUzrQnfo5LbpRDmt7K6ETQ_Fw_2vks5szIzrgaJpZM4QR_9h> .

-- gkk

[pauldraper]

@lihaoyi you had been looking into this...did anything come of it?

I plan on getting Zinc incremental working in the next month or so, and don't want to duplicate any work.

The plan (which has been expressed elsewhere): use zinc in a worker, which is not sandboxed and can save state. Use zinc incremental strategy for local dev, and non-incremental for CI.

[pauldraper]

Current progress: Zinc is undocumented, besides example usage in zinc + pants.

I have 100 line example of zinc compilation + analysis file persistence (outside Bazel) https://github.com/pauldraper/zinc-example/blob/master/src/main/scala/Main.scala

I plan to add this Bazel as an exprimental worker strategy. I'll keep the current interface to scalac as is, since the API to zinc is difference and AFAIK there is no facade for that.

[johnynek]

we are pretty happy with how well caching is working for us currently, so I still stand by the feeling that getting hermetic builds + caching and remote execution is the scalable way forward.

If zinc can be a part of that, great, but I personally don't want to give up hermetic builds.

[andyscott]

FWIW I have a prototype of Zinc based rules with the intention of applying learnings back to rules_scala. I'm not currently concerned with incremental compilation and the primary goal was to get access to the analysis output of Zinc for downstream analysis.

[pauldraper]

we are pretty happy with how well caching is working for us currently,

Good to know. My belief remains that we cannot get near-sbt build times, without creating significant developer burden on creating fine grained DAGs.

If zinc can be a part of that, great, but I personally don't want to give up hermetic builds.

I assume that adding a opt-in worker strategy that persists and re-uses analysis is acceptable.

Worker strategies in general are trust-me with respect to hermeticity. This is a riskier (but acceptable, IMO) trust-me approach.

FWIW I have a prototype of Zinc based rules

@andyscott exciting. What's the status of that?

[andyscott]

It works, but it's early stage. And full disclosure: It's a separate repository written mostly from scratch. The intention is to use the latest and greatest Bazel patterns, experiment with solving some of the key issues faced in this repo, and then incrementally apply the learnings back here. I also needed a way to really dive into Bazel without being encumbered with lots of existing code. If you're curious to dig through my mess... https://github.com/andyscott/rules_scala_annex. https://github.com/andyscott/rules_scala_annex/blob/master/tests/compat/0/BUILD tests some code ported over from this repo.

[sdtwigg]

Is there a way (or is it the default) for Zinc to guarantee that the incremental compile result is identical to a non-incremental compile result? I've speculated pushing for Bazel to allow 'principled use' of an incremental compiler if Bazel had flags to run non-incremental variants of the job as a 'check' on the incremental compile.

Scala (with Zinc) was going to be my test case; however, I've never really had a chance to look deeply at the reliability of the incremental compile.

[andyscott]

@gkossakowski might be able to weigh in here

[pauldraper]

Is there a way (or is it the default) for Zinc to guarantee that the incremental compile result is identical to a non-incremental compile result?

Not generally, because there are many possible inputs to an "incremental compile". So, you could check a select number of state transitions but not all. And the zinc project has tests for that purpose.

In any worker strategy, it's possible to be nonhermetic. In fact, I believe scalac 2.12 had precisely this problem; statics were polluting subsequent compilation runs. The difference here is that we are purposefully "polluting" runs with past state rather than accidentally doing it.

Experience will tell whether zinc 1.0 has correct incremental compiles or not.

[ittaiz]

Actually I think Gregg turned it off for 2.12 because he found that it’s problematic

…

On Sat, 7 Apr 2018 at 3:51 Paul Draper ***@***.***> wrote: Is there a way (or is it the default) for Zinc to guarantee that the incremental compile result is identical to a non-incremental compile result? Not generally, because there are many possible inputs to an "incremental compile". So, you could check a select number of state transitions but not all. And certainly zinc has tests for that purpose. In any worker strategy, it's possible to be nonhermetic. In fact, I believe scalac 2.12 had precisely this problem; statics were polluting subsequent compilation runs. The difference here is that we are *purposefully* "polluting" runs with past state rather than accidentally doing it. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#328 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIFwh8O2m8bkVveONXZdUZCSAGfVUJks5tmA14gaJpZM4QR_9h> .

[bpatters]

I've been following this for a while, and have a question. Is Zinc a possibility to use, at least for developer only iterative development builds, in the near term? We have several units of compilation that if changed, trigger 100+ second rebuilds for our developers that are non-trivial to break into smaller DAG's. Momentum is building to switch our build back to sbt, because allowing developers to iterate several orders of magnitude faster is more important than the faster centralized builds that occur as part of our release process. We get a lot of benefit from bazel and I'd like to keep it, but the developer productivity is really suffering because of these long build times.