Special handling for parts: `buildStep.writePart`

[davidmorgan]

A new feature that could address a number of outstanding requests / issues.

• Give BuildStep a method writePart(String content), for use when the primary input is a library
• A bit like the source_gen shared part builder, it handles allowing multiple builders to write to one part file
• It does the formatting, so it can be done efficiently / in parallel
• Parts are not the same as outputs: they don't trigger other builders, they are not inputs. Builders only run on libraries. Builders do see the partial state of the part file as it accumulates source.
• build_runner decides on the location of the part file, and is responsible for updating (or prompting to update) the manually-maintained file with the correct part statement; a bit like the "multi-root packages" idea, but without any SDK feature needed, this means build_runner can e.g. put all the parts under one folder

Re: folder layout, we could do for example: lib/_parts/..., test/_parts/...; note that parts can't be used from anywhere else so there is no need to put them under src.

[natebosch]

• Builders do see the partial state of the part file as it accumulates source.

Will we need new ordering guarantees or will we track multiple revisions of the part file so we can expose only earlier phases to readers?

• is responsible for updating (or prompting to update) the manually-maintained file with the correct part statement;

Will we break the compatibility with bazel, or will that be a special case where the part statements aren't rewritten because multi-root URIs handle it?

• allowing multiple builders to write to one part file

Will we need a special interface to support import statements in part files to avoid interleave imports and code? This is probably also an oncoming concern for source_gen shared parts as is, but if we're adding a new API we can be ready for language changes.

[davidmorgan]

• Builders do see the partial state of the part file as it accumulates source.

Will we need new ordering guarantees or will we track multiple revisions of the part file so we can expose only earlier phases to readers?

Yes, the changing part handling would need to go by phases.

• is responsible for updating (or prompting to update) the manually-maintained file with the correct part statement;

Will we break the compatibility with bazel, or will that be a special case where the part statements aren't rewritten because multi-root URIs handle it?

I think there can be a "no source changes possible" mode for bazel whereby the changes are output as messages (please fix ...) instead of the changes being made.

• allowing multiple builders to write to one part file

Will we need a special interface to support import statements in part files to avoid interleave imports and code? This is probably also an oncoming concern for source_gen shared parts as is, but if we're adding a new API we can be ready for language changes.

Hmm, you mean with enhanced parts I guess? Since parts can't have imports today.

The rough idea I have here is that because the builder just says "output part" and not which file it ends up in, build_runner can be flexible. In particular, with enhanced parts it can output a tree of parts, so that any builder output that uses imports can go in its own file. That way there is no need to worry about merging namespaces.

[natebosch]

the changes are output as messages

👍

Hmm, you mean with enhanced parts I guess? Since parts can't have imports today.

Yes I mean in a future language version. When we designed the API for shared parts in source_gen we relied on the fact that there aren't imports in parts - but for new APIs we define today we should think about future-proofing.

[Milad-Akarie]

I like the idea of something like buildStep.writePart, or maybe creating a dedicated PartBuildStep that could be passed to PartBuilder and SharedPartBuilder. This would remove the responsibility from the user and enforce API usage.

One issue we currently have is that, although almost all SharedPartBuilders used by known packages write to cache, it’s not something that’s enforced. I believe that’s why we can’t reduce the scope of a glob lookup to just .dart_tool/....

Regarding buildStep.writePart, my expectation would be that it writes to a buffer in memory, and then a final call like buildStep.flushParts would handle the actual writing, including pre-formatting, adding the part header, and any other necessary headers for the final accumulated file.

If my understanding is correct and you agree, at which phase could we call something like flushParts? I’m still trying to fully understand the workflow of build_runner, but I haven’t found a clear cue for when a single input is processed by all applicable builders. If we could do something at that stage, I think we wouldn’t need something like the CombiningBuilder.

On the other hand, there’s the option of appending writes directly to a physical file, e.g.:

Future<void> writePart(String allPartsPath, String content) async {
  final raf = await File(allPartsPath).open(mode: FileMode.append);
  await raf.lock(FileLock.exclusive);
  await raf.writeString(content);
  await raf.writeString('\n'); // separate blocks
  await raf.unlock();
  await raf.close();
}

All parts would be appended to a single file (e.g., lib0001.parts.dart) in cache. No globbing would be required, and the CombiningBuilder could just read input.changeExtension('.parts.dart').

I’m not fully sure about this approach, though, because the behavior of file locks differs across platforms, which might complicate things.

[davidmorgan]

It's actually not possible to do using accumulating writes; this is the most complex part of build_runner, I will see if I can at least explain it without adding unnecessary complexity ;)

build_runner maintains one instance of the analyzer, and supplies it with an in-memory filesystem.

For each build step, it transforms that in-memory filesystem into the appropriate view of the filesystem at the phase the builder runs at. These can happen out of order, because of optional builders: we don't know that all build steps at phase 1 will run, so during phase 2 we might discover we need an optional output from phase 1 and have to go run that build step.

So when "going back in time" to phase 1, build_runner will remove outputs from the in-memory view of the filesystem that the analyzer will use.

That happens here:

build/build_resolvers/lib/src/analysis_driver_model.dart

Line 144 in bcbbcb8

if (!buildStep.phasedReader.hasChanged(

So phasedReader is a view of the filesystem at a particular phase, it returns missing if the generation will run at the current phase or a later phase, or the generation output if it was at an earlier phase.

That should extend naturally to changes at multiple phases. For part files that accumulate data over multiple phases, it needs to be able to set the part to its value at any phase, triggering any necessary build steps that haven't run yet. "Any phase" might include a past phase, which is why we can't just accumulate: it needs to be able to restore to an earlier state if needed.

There is already support for representing data that changes at multiple phases, because import graphs do that:

build/build/lib/src/library_cycle_graph/phased_value.dart

Line 13 in bcbbcb8

/// A value that changes during the build, according to the `int` build phase.

So pretty much it'll be a matter of changing lots of places that currently assume zero or one changes, to zero, one or many changes :)

[Milad-Akarie]

Yup, this is more complicated than I initially thought. I guess the best we can do here is try to optimize globbing or narrow its scope to gain at least a few milliseconds.

I also see now what you meant earlier about creating exclusive part folders like lib/_parts. With the possibility of writing parts to both source and cache, it looks like we’d probably need two _parts folders, unless forcing all parts to write to cache is an option.

Just curious though — wasn’t it possible to create a more streamlined approach with no backtracking? Like adding the outputs of phase one to the analysis session before phase two triggers. Of course, you’d have to sort your phases accordingly: if builder A depends on B’s outputs, then A belongs in the next phase.

[davidmorgan]

Yup, this is more complicated than I initially thought. I guess the best we can do here is try to optimize globbing or narrow its scope to gain at least a few milliseconds.

I also see now what you meant earlier about creating exclusive part folders like lib/_parts. With the possibility of writing parts to both source and cache, it looks like we’d probably need two _parts folders, unless forcing all parts to write to cache is an option.

I'm not quite sure I follow.

There's no need to write parts to intermediate files for buildStep.writePart, build_runner can just keep them in memory. It anyway holds all writes until the build is done.

The appropriate status of part files needs providing to the analyzer for each builds step; but this is also in memory.

Just curious though — wasn’t it possible to create a more streamlined approach with no backtracking? Like adding the outputs of phase one to the analysis session before phase two triggers. Of course, you’d have to sort your phases accordingly: if builder A depends on B’s outputs, then A belongs in the next phase.

That's a good question, it would be nice to make the phases run in order.

But, that's not possible, because of optional builders. You can have an optional builder that runs in phase 1 and outputs foo.o.dart. Because it's optional, phase 1 does not run at all as a standalone phase. Later in phase 2 if a builder tries to read foo.o.dart then it runs just that one build step from phase 1.

Optional builders are useful for intermediate files that take some work to produce but aren't actually a final product the user wants, so you can completely skip building anything that isn't needed by something the user does want, for example that isn't reachable from an entrypoint.

[Milad-Akarie]

It anyway holds all writes until the build is done.

See, I learned that just now.

also I'm not really familiar with optional builders, is there an actual package that uses it? what's a real life use-case for an optional builder?

[davidmorgan]

also I'm not really familiar with optional builders, is there an actual package that uses it? what's a real life use-case for an optional builder?

Yeah, they're a bit surprising :)

They're used in build_web_compilers

build/build_web_compilers/build.yaml

Line 27 in b3818f0

is_optional: True

otherwise it would try to compile everything instead of just the parts you're using.