[Feature] Packages & Modules

[faiface]

Currently, Par only supports single-file programs. Being able to split a program into multiple files, and manage a project and its dependencies is absolutely crucial, and here's my idea about what it could look like in Par.

My priorities for the design:

• Simple to understand
• Scalable to projects of any size
• Easy to find what you're looking for in a codebase
• Easy to add and manage dependencies
• Seamless support for Rust interop
• Fits well with the pattern we're seeing in Builtin.par:
  • String (main module type),
  • String.Builder (associated type in a module),
  • String.Equals (a def in a module).

So, here's what I came up with! I'm curious to hear your thoughts, and criticisms :)

Bullet-point summary

• Package = project
  Every Par program or library is a package with this layout:
  Par.toml, src/, dependencies/, external/.

• Par.toml
  Defines the package (name, url) and its dependencies:

  [dependencies]
  web = "github.com/author/par-web"
  

  The left-hand name (web) is what you use in imports.

• Modules under src/
  Arbitrary directory structure; each module is usually one file named after it, e.g. handlers/api/Posts.par starts with:

  module Posts
  

• Imports are absolute and path-based
  From the src/ root, no relative paths:

  import data/Post
  import handlers/api/Posts
  

  Importing from dependencies uses @name:

  import @web/http/Server
  

• Self = main module type/def
  A type or def named Self inside a module is imported under the module’s name:

  import data/Post
  -- gives: Post (from Self), Post.FetchAllFromDB, etc.
  

  Other names are accessed as Module.Name.

• Aliasing imports
  To avoid clashes:

  import @dep1/blah/Data as Data1
  import @dep2/bleh/Data as Data2
  

• Visibility & exports
  Inside a package, everything sees everything.
  To expose things to other packages:

  export module Iterator
  export type Self<a> = ...
  

  Only exported modules may contain exported types/defs.

• Multi-file modules
  Module.par, Module.foo.par, Module.bar.par in the same directory form one module; all start with the same module Module declaration and share a single namespace.

• Dependencies & Rust interop

  • dependencies/ is managed by the tool and stores all fetched packages (with URL-encoded suffixes; versions later).
  • external/ holds an optional Rust crate compiled together with the Par runtime to implement external definitions.

• No cycles
  Cyclic imports between modules and cyclic dependencies between packages are disallowed.

Packages

A Par project — be it a program or a library — is called a package. Each package has this general file structure:

projectName/
  Par.toml
  src/
  dependencies/
  external/

Let's take it one by one.

Par.toml

A configuration file that specifies the properties and dependencies of a package. These are:

name = "projectName"
url = "github.com/faiface/projectName"

And then a list of dependencies:

[dependencies]
libraryName = "sourcecontrol.com/author/libraryName"

The name assigned to the dependency in the [dependencies] list does not have to match the name specified in the dependency's Par.toml file. Instead, the assigned name will be used inside the package for referring to the dependency.

This proposal doesn't address versioning, we'll address it in a future version. Go managed to go years without versioning, we can go some time too. Of course, they eventually realized it was a mistake at scale, but it worked well at the beginning. And we are at the beginning!

src/

Here, the Par source code of the package lives, organized into directories. These directories contain modules. I'll explain modules in depth right after finishing explaining packages.

dependencies/

This directory is not to be managed by the programmer, but the package manager instead. Par will automatically download dependencies here, and their dependencies, transitively.

The dependencies downloaded here will not create their own dependencies/ directories. Instead, if the dependencies share another dependency, it will be only downloaded once, into this top-level dependencies/ directory.

Each dependency will be stored in a directory named after the name of its package specified in its Par.toml file — we can't go off of the assigned names, those may differ among different dependents — plus a suffix created by some encoding of the package's URL, to avoid conflicts between packages with the same names.

For example: dependencies/wonderfulgui_1ujr191u13i1i93139/....

This way, we avoid conflicts, and the programmer is also able to easily locate the source code of the dependencies.

Once versioning exists, the dependencies/ layout will likely include version in the directory encoding as well, so different versions can coexist if we decide to allow that.

external/

If the package has a part of it implemented in Rust, here goes a crate implementing those definitions. This crate will then be compiled together will the general Par runtime to create a specialized binary capable of executing all external code of all the dependencies of the program.

Modules

Now on to the contents of the src/ directory!

It can contain any directory structure. For example, here's some ad-hoc web server, not really adhering to any methodology:

src/
  data/
  handlers/
    api/
  util/

And the directories are populated with modules! A module is (usually) a single file with the same name as the module. Let's put some modules in the above structure:

src/
  Main.par
  data/
    User.par
    Post.par
  handlers/
    api/
      Posts.par
      Profile.par
    Index.par
  util/
    DateTimeUtils.par

Read until the end to find out how to have multiple files per module.

The start of a module file looks like this:

module Posts

This is the src/handlers/api/Posts.par file. It defines a module called Posts.

Now what if it wants to use another module in the same package? A Posts handler certainly wants to use the Post type:

module Posts

import data/Post

The import statement specifies the whole path (from src/) of a module to import. Relative import paths are not supported!

Now before I go on to say how Posts will use the Post module, let's take a look at the Post module.

module Post

type Self = box choice {
  .title => String,
  .content => String,
}

dec Self : [String, String] Self

dec FetchAllFromDB : [!] List<Self>

The Post module specifies a Self type, as well as a Self definition, a constructor.

A type or a def with the name Self is special. It becomes available as the name of the module to the importer.

Self is not imported as Self in the importing module; it is only visible under the module name (or alias).

Other types and definitions will be usable as Post.*.

For example, in a String module, the String type itself would be defined as Self inside the module.

So inside the Posts handler:

module Posts

import data/Post

// we've imported:
// - type Post
// - def Post
// - def Post.FetchAllFromDB

The handler will be able to use these for implementation!

And for example the Main.par module will start like this:

module Main

import handlers/Index
import handlers/api/Posts
import handlers/api/Profile

Importing from dependencies

Let's say the little web server example above has some dependencies. Here's its Par.toml file:

name = "postify"
url = "github.com/faiface/postify"

[dependencies]
web = "github.com/afjwie/par-web"

We're depending on a web library!

Importing from a dependency is the same as importing from within a package itself, the only difference is we prefix the path with @<name>, like so:

import @web/http/Server

Aliasing

Module name conflicts can occur, especially between dependencies, and so this will be supported:

import @dep1/blah/Data as Data1
import @dep2/bleh/Data as Data2

Simply renaming a module via as Alias. The Self type or a def defined inside the module will also carry this name.

Visibility

Everything is visible between definitions inside the same package. Encapsulation within a package is only achieved by the means it is alraedy today: choices, existential types, and so on.

However, to make a module, a type or a definition visible outside a package, we use the keyword export:

export module Iterator

export type Self<a> = recursive choice {
  .close => !,
  .next => either {
    .end!,
    .item(a) self,
  }
}

Modules and definitions marked with export will be available to packages that depend on this package.

Exports only affect visibility to other packages. Inside a package, all modules, types, and defs are visible regardless of export.

For a program that's not a library, nothing really needs to be exported.

A module can only contain exported types and definitions if it is itself exported.

Splitting a module into multiple files

Sometimes one file may not be enough for a module, and so in addition to Module.par, all files within the same directory that have this form:

Module.*.par

will be considered a part of the same module. All files of the module must start with the same module Module declaration.

All top-level definitions across these files share one module namespace. But, each file has its own import section, which only applies within that file.

Restrictions

No cyclic imports between modules allowed.

No cyclic dependencies between packages allowed.