Import

[byorgey]

This PR adds a real, honest-to-goodness import construct, towards #495. This PR will not remove run; I will leave that for a second, follow-up PR.

Highlights of this PR:

• Add import "xyz" in ... syntax which makes definitions from xyz available in ..., with import "xyz"; ... as shorthand
  • Imports are loaded recursively, with no import being processed more than once
  • Import cycles are detected
  • Imports can be local, relative to the home dir, an absolute path, or a URL
  • Most interesting code is in Swarm.Language.Syntax.Import and Swarm.Language.Load.
• SrcLoc now contains an import location in addition to line/column numbers, to be able to report error locations from recursively processed imports
• processTerm and friends (in Swarm.Language.Pipeline) changed quite a bit:
  • many of them now require IO, since they may need to go read imports from the filesystem/network
  • many of them now return a SourceMap, which is a map from import locations to ASTs
• Add type-level parameters indicating various phases in the term processing pipeline (Raw, Resolved, Inferred, Typed, Elaborated ...) and use them to enforce invariants.
  • See Swarm.Language.Phase.
  • Syntax and Term got type annotations, along with pretty much anything that transitively contains them.
  • See also Swarm.Language.Elaborate, where we need to use unsafeCoerce.
  • Without this I would have been very unsure of my ability to get everything right (and indeed, the type parameters caught several bugs where I e.g. forgot to call elaborate and so on).
    • As one example, previously we had FromJSON instances that did everything from parsing through typechecking + elaboration. Now we can't do that since parseJSON can't use IO, so it can't load and check imports. To ensure we process everything correctly, we now have instances like FromJSON (Syntax Raw) and the type parameter ensures that anywhere such an instance was used must be updated to do all the other steps in order to end up with a Syntax Elaborated.
  • A lot of the changes just consist of adding phase parameters to various things.

Downsides:

• Need for a bunch of explicit type arguments on robot lenses

To do:

• Improve responsiveness while typing imports (don't load + typecheck import on every keypress; perhaps don't bother typechecking imports at all until hitting Enter)
• fix readValue
• fix processParsedTermNoImports (and decide what it's really needed for)
• load URL imports
• Enforce that imports contain only imports + def (+ write documentation explaining why)
• Fix parseCodeFile to take imports into account
• Fix unit tests
• add more tests
• Properly collect tydefs in imports
• Make sure the very first import is processed relative to the location of its containing file
• Provide a way to import relative to the standard swarm data files
• Solve or make issues
  • LSP hover does not have info about imports (e.g. types of imported things)
  • print SrcLoc in PrettyPrec LocatedTCFrame instance
  • Track set of evaluated imports so we don't re-evaluate
  • Add last modified timestamp / hash to import location
  • Add global import cache
  • Don't put transitive imports in scope at runtime
  • Re-export

[byorgey]

@xsebek this is definitely ready for you to try out, if you like. Let me know if you have questions, and definitely let me know all the horrible bugs you find. 😄

[xsebek]

I tried to test it on the move tutorial:

   - name: 1P wall
     system: true
     program: |
-      def main = \a. pure noop end
       instant {
-        run "scenarios/Tutorials/move_surveil.sw";
+        import "scenarios/Tutorials/move_surveil.sw" in
         main [entity="wall", room=1]
       }

But the robots fail to run:

0:00:00.0: Fatal error: Import not found:
/Users/xsebek/Git/swarm-import/scenarios/Tutorials/move_surveil.sw
Please report this as a bug at <https://github.com/swarm-game/swarm/issues/new>.

Also, could you please fix the Swarm/Doc/Pedagogy.hs? Since it fails to build I cannot run integration tests or cabal install. So I run this:

cabal run swarm:exe:swarm -- --scenario Tutorials/Move --run <(echo 'move;move') --debug=all

[byorgey]

Ah, sorry, I guess the way I was building it didn't actually end up building every component. swarm-doc should be fixed now.

And thanks for the bug report! I will take a look. By the way, you can also write a semicolon after the import instead of in, if you like.

[byorgey]

But the robots fail to run:

@xsebek what did you do to get that crash? When I try it, the system robots in the move tutorial do not work properly, but at least it doesn't crash. But I'm not sure if that's because I'm doing something differently than you, or if I incidentally fixed the crash in the meantime.

[xsebek]

@byorgey the robots still fail to run for me. I used import in Move tutorial thus:

Click to expand diff

diff --git a/data/scenarios/Tutorials/move.yaml b/data/scenarios/Tutorials/move.yaml
index 5a20eb6e..ea7f35cc 100644
--- a/data/scenarios/Tutorials/move.yaml
+++ b/data/scenarios/Tutorials/move.yaml
@@ -149,54 +149,44 @@ robots:
   - name: 1P wall
     system: true
     program: |
-      def main = \a. pure noop end
       instant {
-        run "scenarios/Tutorials/move_surveil.sw";
+        import "scenarios/Tutorials/move_surveil.sw" in
         main [entity="wall", room=1]
       }
   - name: 2P wall
     system: true
     program: |
-      def main = \a. noop end
       instant {
-        run "scenarios/Tutorials/move_surveil.sw";
+        import "scenarios/Tutorials/move_surveil.sw" in
         main [entity="wall", room=2]
       }
   - name: 3P wall
     system: true
     program: |
-      def main = \a. noop end
-      instant {
-        run "scenarios/Tutorials/move_surveil.sw";
-        main [entity="wall", room=3]
-      }
+      import "scenarios/Tutorials/move_surveil.sw" in
+      main [entity="wall", room=3]
   #################
   ## GARDENERS   ##
   #################
   - name: 1P flower
     system: true
     program: |
-      def main = \a. noop end
       instant {
-        run "scenarios/Tutorials/move_surveil.sw";
+        import "scenarios/Tutorials/move_surveil.sw" in
         main [entity="flower", room=1]
       }
   - name: 2P flower
     system: true
     program: |
-      def main = \a. noop end
       instant {
-        run "scenarios/Tutorials/move_surveil.sw";
+        import "scenarios/Tutorials/move_surveil.sw" in
         main [entity="flower", room=2]
       }
   - name: 3P flower
     system: true
     program: |
-      def main = \a. noop end
-      instant {
-        run "scenarios/Tutorials/move_surveil.sw";
-        main [entity="flower", room=3]
-      }
+      import "scenarios/Tutorials/move_surveil.sw" in
+      main [entity="flower", room=3]
 entities:
   - name: door
     display:

Then run:

cabal run swarm:exe:swarm -- --scenario Tutorials/Move --run <(echo 'move;move') --debug=all --paused

After one step, you should see the robots all turn idle in the robot panel. Select any robot to view their log and you should see the "crash".

[byorgey]

@xsebek I think I have now fixed the bug you ran into with the Move tutorial (actually it was two bugs 😄 ). Feel free to poke at this more if you have a chance, I'm sure there are yet more bugs. But it feels like it is slowly converging. Down to only five XXX comments!

[xsebek]

@byorgey great! It seems to take a bit to load, I guess because "Track set of evaluated imports" is not yet done. But after it loads it runs and is responsive. 👍

I can try converting other scenarios with robots to get more test coverage, but the tests do not build now. 🙁
I tried to make them compile at least and it seems the Template Haskell seed program broke:

      Tutorials/place:                                 FAIL (0.02s)
        test/integration/Main.hs:570:
        Fatal error: Antiquoted variable found at runtime: $int:_randTime

Enforce that imports contain only imports + def

I tried it and it does go into the unexpected branch in step. 🔥

Fatal error: Bad machine state in
  stepRobot: FImport frame in
  non-suspended state
E· import ◀__res__ <- forever (
  n <- random 20;
  wait (10 + n);
  d <- randdir;
  turn d;
  dist <- random 10;
  repeat dist move;
  r <- random 5;
  if (r == 0) {say "meow"} {}
);
suspend __res__▶ in suspend ()

[byorgey]

It seems to take a bit to load, I guess because "Track set of evaluated imports" is not yet done.

Yeah, there's definitely some optimization that still needs to happen. Shouldn't be too hard though.

I can try converting other scenarios with robots to get more test coverage, but the tests do not build now. 🙁 I tried to make them compile at least and it seems the Template Haskell seed program broke:

      Tutorials/place:                                 FAIL (0.02s)
        test/integration/Main.hs:570:
        Fatal error: Antiquoted variable found at runtime: $int:_randTime

Hmm, probably just didn't rename something consistently...

I tried it and it does go into the unexpected branch in step. 🔥

Hmm, thanks, I'll look into this more.