Integrate BAST module into language and passes, add auto-generation

Reorganized the standalone bast/ module:
- Moved utility code (BASTWriter, BASTReader, BASTUtils, etc.) to language/bast/
- Moved BASTWriterPass to passes/ module
- Removed bast/ module from build.sbt

Added automatic BAST loading during parsing:
- TopLevelParser checks for newer .bast files and loads them automatically
- Supports BAST imports referenced in parsed files via BASTLoader

Added --auto-generate-bast / -B CLI option:
- Generates .bast files next to .riddl files after parsing
- Works like Python's .pyc caching mechanism
- Also available as 'auto-generate-bast' in config files

Fixed bug where BAST header was not being written:
- BASTWriterPass.finalize() return value was being ignored
- Added finalizedBytes field to store and use the result

Moved tests from bast/ to passes/:
- BASTLoaderTest, BASTRoundTripTest, DeepASTComparison

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Author: reidspencer

bast/NOTEBOOK.md

@@ -2,9 +2,13 @@
 
 ## Current Status
 
-**BAST (Binary AST) serialization and import integration is complete through Phase 4.**
+**BAST (Binary AST) serialization is fully integrated into the language and passes modules.**
 
-The `bast/` module provides efficient binary serialization/deserialization of RIDDL AST nodes, enabling fast `import "module.bast"` functionality. All 19 tests pass.
+As of Jan 15, 2026, the standalone `bast/` module has been **removed from build.sbt** and the code has been reorganized:
+- **Utility code** → `language/shared/src/main/scala/com/ossuminc/riddl/language/bast/`
+- **BASTWriterPass** → `passes/shared/src/main/scala/com/ossuminc/riddl/passes/`
+
+This enables BAST to work like Python's `.pyc` files - automatic loading from cache when available.
 
 ## Work Completed (Recent)
 
@@ -16,12 +20,21 @@ The `bast/` module provides efficient binary serialization/deserialization of RI
   - [x] Support imports at root level AND inside domains
   - [x] BASTLoader utility to load and populate contents
   - [x] Path resolution verified: `ImportedDomain.SomeType` resolves correctly
-
-## In Progress
-
-**Phase 5: CLI & Testing**
+- [x] Phase 5: Module Reorganization & Auto-Generation (Jan 15, 2026)
+  - [x] Removed standalone `bast/` module from build.sbt
+  - [x] Split BASTWriter: utility class in `language/bast/`, Pass wrapper in `passes/`
+  - [x] Added `BASTUtils.scala` with `checkForBastFile()`, `loadBAST()`, `tryLoadBastOrParseRiddl()`
+  - [x] Integrated automatic BAST loading into `TopLevelParser.parseURL()` and `parseInput()`
+  - [x] Added `--auto-generate-bast` / `-B` CLI option to riddlc
+  - [x] Added `auto-generate-bast` config file option
+  - [x] Implemented auto-generation in `Riddl.parse()` when option enabled
+
+## Completed
+
+**Phase 5: CLI & Auto-Generation**
 - [x] Add `riddlc bast-gen` command - 77ba7544
-- [ ] Add `--use-bast-cache` flag (auto-generate BAST during parsing)
+- [x] Add `--auto-generate-bast` flag (auto-generate BAST during parsing)
+- [x] Automatic BAST loading when .bast file exists and is newer than .riddl
 - [ ] Performance benchmarks (parse RIDDL vs load BAST)
 - [ ] Cross-platform tests (JS, Native)
 
@@ -46,7 +59,7 @@ The `bast/` module provides efficient binary serialization/deserialization of RI
 
 ## Open Questions
 
-- Should BAST files be auto-generated as a build cache, or explicitly created by users?
+- ~~Should BAST files be auto-generated as a build cache, or explicitly created by users?~~ **RESOLVED**: Both options available - explicit via `bast-gen` command, automatic via `--auto-generate-bast` flag
 - How should BAST versioning handle breaking format changes?
 
 ## Test Status
@@ -61,12 +74,23 @@ The `bast/` module provides efficient binary serialization/deserialization of RI
 
 ## Key Code Locations
 
+**Language Module** (`language/shared/src/main/scala/com/ossuminc/riddl/language/bast/`):
 - `package.scala` - Constants, node type tags (1-255)
 - `BinaryFormat.scala` - Header spec, file structure
-- `BASTWriter.scala` - Serialization pass
+- `BASTWriter.scala` - Serialization utility class (non-Pass)
 - `BASTReader.scala` - Deserialization
 - `BASTLoader.scala` - Import loading utility
+- `BASTUtils.scala` - File checking, BAST loading helpers
+
+**Passes Module** (`passes/shared/src/main/scala/com/ossuminc/riddl/passes/`):
+- `BASTWriterPass.scala` - Pass wrapper using AST traversal framework
+
+**Commands Module** (`commands/jvm/src/main/scala/com/ossuminc/riddl/commands/`):
+- `BastGenCommand.scala` - `riddlc bast-gen` command
+
+**Tests** (`passes/jvm/src/test/scala/com/ossuminc/riddl/passes/`):
 - `BASTLoaderTest.scala` - Import integration tests
+- `BASTRoundTripTest.scala` - Serialization round-trip tests
 
 ## Git Information
 

build.sbt

@@ -32,9 +32,6 @@ lazy val riddl: Project = Root("riddl", startYr = startYear, spdx ="Apache-2.0")
     language,
     languageNative,
     languageJS,
-    bast,
-    bastNative,
-    bastJS,
     passes,
     passesNative,
     passesJS,
@@ -208,26 +205,6 @@ lazy val language = language_cp.jvm.dependsOn(utils)
 lazy val languageJS = language_cp.js.dependsOn(utilsJS)
 lazy val languageNative = language_cp.native.dependsOn(utilsNative)
 
-val BAST = config("bast")
-lazy val bast_cp = CrossModule("bast", "riddl-bast")(JVM, JS, Native)
-  .dependsOn(cpDep(language_cp), cpDep(passes_cp))
-  .configure(With.typical, With.GithubPublishing)
-  .settings(
-    Test / parallelExecution := false,
-    scalacOptions ++= Seq("-explain"),
-    description := "Binary AST (BAST) serialization for fast import of RIDDL modules"
-  )
-  .jvmConfigure(With.coverage(70))
-  .jvmConfigure(With.noMiMa)
-  .jsConfigure(With.ScalaJS("RIDDL: bast", withCommonJSModule = true))
-  .jsConfigure(With.noMiMa)
-  .nativeConfigure(With.Native(mode = "fast"))
-  .nativeConfigure(With.noMiMa)
-
-lazy val bast = bast_cp.jvm.dependsOn(language, passes)
-lazy val bastJS = bast_cp.js.dependsOn(languageJS, passesJS)
-lazy val bastNative = bast_cp.native.dependsOn(languageNative, passesNative)
-
 val Passes = config("passes")
 lazy val passes_cp = CrossModule("passes", "riddl-passes")(JVM, JS, Native)
   .dependsOn(cpDep(utils_cp), cpDep(language_cp))
@@ -354,7 +331,7 @@ val riddlLibNative = riddlLib_cp.native
 
 val Commands = config("commands")
 lazy val commands_cp: CrossProject = CrossModule("commands", "riddl-commands")(JVM, Native)
-  .dependsOn(cpDep(utils_cp), cpDep(language_cp), cpDep(passes_cp), cpDep(diagrams_cp), cpDep(bast_cp))
+  .dependsOn(cpDep(utils_cp), cpDep(language_cp), cpDep(passes_cp), cpDep(diagrams_cp))
   .configure(With.typical, With.GithubPublishing)
   .settings(
     scalacOptions ++= Seq("-explain", "--explain-types", "--explain-cyclic", "--no-warnings"),

commands/jvm/src/main/scala/com/ossuminc/riddl/commands/BastGenCommand.scala

@@ -6,11 +6,10 @@
 
 package com.ossuminc.riddl.commands
 
-import com.ossuminc.riddl.bast.{BASTOutput, BASTWriter}
 import com.ossuminc.riddl.command.{CommandOptions, PassCommand, PassCommandOptions}
 import com.ossuminc.riddl.language.Messages
 import com.ossuminc.riddl.language.Messages.Messages
-import com.ossuminc.riddl.passes.{PassCreators, PassesResult}
+import com.ossuminc.riddl.passes.{BASTOutput, BASTWriterPass, PassCreators, PassesResult}
 import com.ossuminc.riddl.utils.PlatformContext
 import org.ekrich.config.Config
 import scopt.OParser
@@ -72,7 +71,7 @@ class BastGenCommand(using pc: PlatformContext) extends PassCommand[BastGenComma
   }
 
   override def getPasses(options: Options): PassCreators = {
-    Seq(BASTWriter.creator())
+    Seq(BASTWriterPass.creator())
   }
 
   override def replaceInputFile(opts: Options, inputFile: Path): Options = {
@@ -88,7 +87,7 @@ class BastGenCommand(using pc: PlatformContext) extends PassCommand[BastGenComma
       case Left(errors) => Left(errors)
       case Right(result) =>
         // Get the BAST output
-        result.outputOf[BASTOutput](BASTWriter.name) match {
+        result.outputOf[BASTOutput](BASTWriterPass.name) match {
           case None =>
             Left(Messages.errors("BASTWriter did not produce output"))
           case Some(bastOutput) =>

commands/jvm/src/test/scala/com/ossuminc/riddl/commands/BastGenCommandTest.scala

@@ -6,7 +6,7 @@
 
 package com.ossuminc.riddl.commands
 
-import com.ossuminc.riddl.bast.{BASTReader, BASTOutput, HEADER_SIZE}
+import com.ossuminc.riddl.language.bast.HEADER_SIZE
 import com.ossuminc.riddl.language.AST.Domain
 import com.ossuminc.riddl.utils.{pc, PlatformContext}
 import org.scalatest.matchers.must.Matchers

commands/shared/src/main/scala/com/ossuminc/riddl/command/CommonOptionsHelper.scala

@@ -53,6 +53,7 @@ object CommonOptionsHelper:
   private inline def max_parallel_parsing = "max-parallel-parsing"
   private inline def max_include_wait = "max-include-wait"
   private inline def warnings_are_fatal = "warnings-are-fatal"
+  private inline def auto_generate_bast = "auto-generate-bast"
 
   lazy val commonOptionsParser: OParser[Unit, CommonOptions] = {
     val builder: OParserBuilder[CommonOptions] = OParser.builder[CommonOptions]
@@ -144,6 +145,12 @@ object CommonOptionsHelper:
         .action((_, c) => c.copy(warningsAreFatal = true))
         .text(
           "Makes validation warnings fatal to encourage code perfection"
+        ),
+      opt[Unit]('B', auto_generate_bast)
+        .optional()
+        .action((_, c) => c.copy(autoGenerateBAST = true))
+        .text(
+          "Automatically generate .bast files next to .riddl files after parsing (like Python's .pyc)"
         )
     )
   }
@@ -226,6 +233,8 @@ object CommonOptionsHelper:
       else default.maxIncludeWait
     val warningsAreFatal =
       if obj.hasPath(warnings_are_fatal) then obj.getBoolean(warnings_are_fatal) else default.warningsAreFatal
+    val autoGenerateBAST =
+      if obj.hasPath(auto_generate_bast) then obj.getBoolean(auto_generate_bast) else default.autoGenerateBAST
 
     CommonOptions(
       showTimes,
@@ -244,7 +253,8 @@ object CommonOptionsHelper:
       noANSIMessages,
       maxParallelParsing,
       maxIncludeWait,
-      warningsAreFatal
+      warningsAreFatal,
+      autoGenerateBAST
     )
   end commonOptionsReader
 

language/shared/src/main/scala/com/ossuminc/riddl/language/bast/BASTLoader.scala

@@ -0,0 +1,151 @@
+/*
+ * Copyright 2019-2026 Ossum, Inc.
+ *
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+package com.ossuminc.riddl.language.bast
+
+import com.ossuminc.riddl.language.AST.*
+import com.ossuminc.riddl.language.{At, Messages}
+import com.ossuminc.riddl.language.Messages.Messages
+import com.ossuminc.riddl.utils.{PlatformContext, URL}
+
+import scala.collection.mutable
+import scala.concurrent.{Await, ExecutionContext}
+import scala.concurrent.duration.*
+import scala.util.{Failure, Success, Try}
+
+/** Utility for loading BAST imports.
+  *
+  * This loads BAST files referenced by BASTImport nodes and populates their
+  * contents field with the loaded Nebula contents. Imports can appear at the
+  * root level or inside domains.
+  */
+object BASTLoader {
+
+  /** Result of loading BAST imports */
+  case class LoadResult(
+    loadedCount: Int,
+    failedCount: Int,
+    messages: Messages
+  )
+
+  /** Load all BAST imports in a Root, including those inside domains.
+    *
+    * Finds all BASTImport nodes in the Root and its domains, loads the referenced
+    * BAST files, and populates each BASTImport's contents field with the loaded
+    * Nebula contents.
+    *
+    * @param root The Root containing BASTImport nodes to load
+    * @param baseURL The base URL for resolving relative BAST file paths
+    * @param pc The platform context for file loading
+    * @return LoadResult with counts and any error messages
+    */
+  def loadImports(root: Root, baseURL: URL)(using pc: PlatformContext): LoadResult = {
+    val msgs = mutable.ListBuffer.empty[Messages.Message]
+    var loaded = 0
+    var failed = 0
+
+    def loadImport(bi: BASTImport): Unit = {
+      loadSingleImport(bi, baseURL) match {
+        case Right(nebula) =>
+          // Copy Nebula contents into BASTImport contents
+          nebula.contents.foreach { item =>
+            bi.contents.append(item)
+          }
+          loaded += 1
+        case Left(error) =>
+          msgs += Messages.Message(
+            bi.loc,
+            s"Failed to load BAST import '${bi.path.s}': $error",
+            Messages.Error
+          )
+          failed += 1
+      }
+    }
+
+    def processContents[T <: RiddlValue](contents: Contents[T]): Unit = {
+      contents.foreach {
+        case bi: BASTImport => loadImport(bi)
+        case d: Domain => processContents(d.contents)
+        case _ => () // Not a BASTImport or Domain, skip
+      }
+    }
+
+    processContents(root.contents)
+    LoadResult(loaded, failed, msgs.toList)
+  }
+
+  /** Load a single BAST import.
+    *
+    * @param bi The BASTImport to load
+    * @param baseURL The base URL for resolving relative paths
+    * @param pc The platform context
+    * @return Either an error message or the loaded Nebula
+    */
+  private def loadSingleImport(bi: BASTImport, baseURL: URL)(using pc: PlatformContext): Either[String, Nebula] = {
+    Try {
+      // Resolve the path relative to the base URL
+      val bastURL = if URL.isValid(bi.path.s) then {
+        URL(bi.path.s)
+      } else {
+        baseURL.parent.resolve(bi.path.s)
+      }
+
+      // Load the BAST file bytes
+      implicit val ec: ExecutionContext = pc.ec
+      val future = pc.load(bastURL).map { data =>
+        // Parse as BAST - note: data is loaded as String, convert to bytes
+        val bytes = data.getBytes("ISO-8859-1") // Binary data preserved
+        val reader = BASTReader(bytes)
+        reader.read() // Returns Either[Messages, Nebula]
+      }
+
+      // Wait for the result (with timeout)
+      Await.result(future, 30.seconds)
+    } match {
+      case Success(Right(nebula)) => Right(nebula)
+      case Success(Left(msgs)) => Left(msgs.map(_.format).mkString("; "))
+      case Failure(ex) => Left(ex.getMessage)
+    }
+  }
+
+  /** Check if a Root has any unloaded BASTImport nodes.
+    *
+    * @param root The Root to check
+    * @return true if there are BASTImport nodes with empty contents
+    */
+  def hasUnloadedImports(root: Root): Boolean = {
+    var found = false
+    def checkContents[T <: RiddlValue](contents: Contents[T]): Unit = {
+      if !found then
+        contents.foreach {
+          case bi: BASTImport => if bi.contents.isEmpty then found = true
+          case d: Domain => checkContents(d.contents)
+          case _ => ()
+        }
+      end if
+    }
+    checkContents(root.contents)
+    found
+  }
+
+  /** Get all BASTImport nodes from a Root, including those inside domains.
+    *
+    * @param root The Root to search
+    * @return Sequence of BASTImport nodes
+    */
+  def getImports(root: Root): Seq[BASTImport] = {
+    val result = mutable.ListBuffer.empty[BASTImport]
+    def collectFromContents[T <: RiddlValue](contents: Contents[T]): Unit = {
+      contents.foreach {
+        case bi: BASTImport => result += bi
+        case d: Domain => collectFromContents(d.contents)
+        case _ => ()
+      }
+    }
+    collectFromContents(root.contents)
+    result.toSeq
+  }
+}