Infrastructure tracking: compilation, source mapping, and TypeScript

[aschampion]

This is an internal tracking ticket for infrastructure for compiling client code through Django-pipeline @tomka and I have been discussing for some time.

Motivation

• We would like to take advantage of the type safety and productivity benefits of TypeScript for central libraries and APIs in CATMAID, while allowing widgets and experimental features to still be prototyped quickly in plain JS as before.
• We would like to factor more components into modules to decrease complexity, better delegate responsibilities to components, make composition easier, and allow for reuse outside of the full CATMAID client.

Requirements

These are the capabilities any compiled pipeline must maintain:

• The client must recompile when edited in a development environment without any explicit action.
• Source mapping in the browser console.
• Likewise, no mangling or breaking of the API in the browser console for scripting use.

NOTE: The below plan is no longer valid. See the discussion beginning in 2019 below for a webpack-based approach.

Draft Plan

• TS Compilation
  • Pipeline plugin
  • Pipeline's development mode should recompile on page load, but if this is too slow there are alternatives, e.g, using gulp
• Source mapping
  • The Atlantic has a PR adding source mapping support to pipeline (finally)
  • Since we bundle multiple source files together, we need a compressor that is also source-map aware. The PR above uses closure's compressor. To use a compressor like closure we'd need to go ahead and annotate all our modules and exports so that they're still callable from the console. Non-mangling compressors like JSMin produce source maps but don't consume them from earlier steps, or if they do consume only one (like UglifyJS). This may be a non-issue if we can just compile a single TypeScript entry point file, so then there is only a single source map to pass through. Also annoying is that pipeline can't apply different compilers and compressors to different sets of files.
  • Closure can take advantage of TypeScript's typing through JSDoc annotations, for example google has a project that does just that
• Dependencies
  • We prefer to keep our dependencies in-repo rather than packaged from NPM or bower, but would also want matching TS definitions through Typings

Drawbacks

• Requires installation of TS and possibly closure compilers for non-docker servers
• No in-browser editing of compiled files
• Potentially more latency in the edit > test > edit cycle for developers

Alternatives

• Django pipeline is unergonomic for modern Javascript ecosystems for SPAs, feeling more oriented towards classic HTML + JS + CSS full page delivery. It might be better to adopt a different asset management strategy.

[aschampion]

@tomka I was looking at the webpack feature branch again curious how much work it would be so that we could move dependencies out-of-tree and be able to support TypeScript, etc. It actually seems somewhat straightforward now to be able to avoid name mangling and still separate libraries into separate bundles, and we're most of the way there if we wanted to incrementally migrate libraries or code out of pipeline.

1. If I PR'd the branch with changes doing this, do you feel likely to merge it, or would the overhead of needing to npm install and webpack outweigh the benefits at this stage?
2. If yes to 1, as a MVP, would it be better to migrate a dependency (i.e., using NPM instead of an in-tree copy) or to move some small, isolated part of CATMAID-lib to a webpack module?
3. If no to 1, is there some more elaborate MVP that would make the benefits more compelling? Something like converting some part of CATMAID-lib to TypeScript, or changing a cumbersome form somewhere to React with hot loading?

[clbarnes]

For testing and benchmarking alternative implementations of Arbor.js and friends, I wrote https://github.com/clbarnes/arbor-harness / https://www.npmjs.com/package/arbor-harness, which fetches and mangles the original implementation from raw.github . If those bits of catmaid-lib (arbor, arbor-parser, synapse-clustering) were on npm, that would get easier. Hardly isolated, though!

Would we run into issues surrounding the order in which everything is imported, IIFEs executed, and stuff added to the CATMAID object?

[aschampion]

If those bits of catmaid-lib (arbor, arbor-parser, synapse-clustering) were on npm, that would get easier

Making components available elsewhere as their own libraries would be a goal (besides better tooling and organization within CATMAID). First we'd separate them into modules in-tree to sort things out, then later move them out.

Would we run into issues surrounding the order in which everything is imported

It's a consideration, but my plan for incremental conversion would be to start from the "top" of the import order, and have a pseudo webpack entry point before the real legacy CATMAID scripts that would just import whatever is ES6 moduled/webpackified. It's similar to how we've done several other modernization/big cleanup sweeps in the codebase.

If everything were converted at once, it wouldn't be a problem. But I think that would be too high of an activation energy.

[tomka]

Yes, I would merge such a change. The features/webpack branch looks indeed pretty clean. Being able to use TypeScript would be great!

npm install

During regular development, I suppose, that would only be needed if NPM dependencies have changed (or initial setup), i.e. I don't need to call it normally?

webpack

This would also only be need to be called if if the NPM dependencies or our own TypeScript and webpack bundles change, wouldn't it? If so, unless, work is done in TypeScript or bundles, there is no need to call this a lot either.

Both would of course be part of a regular production server update.

but my plan for incremental conversion would be to start from the "top" of the import order

👍

If everything were converted at once, it wouldn't be a problem. But I think that would be too high of an activation energy.

I believe so too and think we should try to do this as an incremental update.

[aschampion]

TypeScript would be great!

I was doing something yesterday that required me editing on one computer then reloading and debugging very slowly on another, and waiting a minute to refresh a page and setup some widgets just to find out I'd typoed a property name was very motivating to want to use TS.

During regular development, I suppose, that would only be needed if NPM dependencies have changed (or initial setup), i.e. I don't need to call it normally?

Correct, basically pip install for our JS dependencies.

This would also only be need to be called if if the NPM dependencies or our own TypeScript and webpack bundles change, wouldn't it?

Also correct, although there is also a server/watch mode that will watch for changes and automatically do incremental updates to the bundles, which would be useful if frequently edited code is bundled. The "neat" deep end of this (not that we'd use it) is being able to do thing like React component hot reloading, where you edit React components, and they live-update in the browser because the state of the component is separate from the UI/controls that are getting reloaded.

[clbarnes]

I think it would be nice to move towards using the webpack dev server; by constraining usage of TS / modern JS to a small set of new files to avoid having to manually recompile all the time, we're obviously losing out on a lot of the incentive to switch.

Related, I've been fumbling with catmaid-lib in this context today and have got it to a stage where webpack can bundle it and it doesn't immediately error when loaded in a browser. I haven't really touched any of the IIFEs, so the modules don't really add any value, but it's there as an experiment. But there's still bits of catmaid-lib which depend on globals or CATMAID members which are defined in the other catmaid code, which doesn't seem like it should be the point of catmaid-lib. requestQueue probably won't work because the global needs to be overwritten to set up CSRF.

[aschampion]

I've been fumbling with catmaid-lib in this context today

Would it be easier to factor out smaller libraries (like arbor) first and bundle those, or do you think it's best to bundle all of CATMAID-lib at once, then factor it into focused libraries later?

But there's still bits of catmaid-lib which depend on globals or CATMAID members which are defined in the other catmaid code, which doesn't seem like it should be the point of catmaid-lib. requestQueue probably won't work because the global needs to be overwritten to set up CSRF.

Agreed, it makes sense for CATMAID-lib to define the RequestQueue type, but the global default queue instance should be in the non-lib CATMAID, and any requests issued from the lib should take a queue explicitly. Beyond just organizing into libraries, making these coupling points explicit is useful now that we want to talk to multiple backends from one client as part of the federation features.

[aschampion]

To add, a concern with bundling all of CATMAID-lib at once and moving it to an NPM package is that some parts of CATMAID-lib still have high churn. The update process is a bit more cumbersome for those if it requires editing and publishing in a separate repo, publishing to NPM, and updating in CATMAID. That would make sense for stable parts like arbor.

However, there may be nothing stopping us from have multiple packages in-tree here. So CATMAID-lib can stay in this repo, with local dependencies from the main CATMAID front-end package to the CATMAID-lib packages, and each of the individual CATMAID-lib packages get published separately to NPM as part of the release process. I like that option if we can get it set up, as there's no new impedance to our existing workflow but all the benefits of separate, published libraries.

[clbarnes]

Would it be easier to factor out smaller libraries (like arbor) first and bundle those, or do you think it's best to bundle all of CATMAID-lib at once, then factor it into focused libraries later?

Yes, probably. My current factoring doesn't really benefit from the modularisation because it's still dependent on the import order, and most of the imports are only being done for the side-effect of adding stuff to the global CATMAID object. Long-term I think the goal would be to completely decouple CATMAID-lib from that global object, and just let the caller populate it.

The exports are:

• CATMAID
• requestQueue (global instance)
• Arbor
• CircuitGraphAnalysis
• SynapseClustering
• InstanceRegistry
• fetchSkeletons

Those in bold are good targets for excising first.

Beyond just organizing into libraries, making these coupling points explicit is useful now that we want to talk to multiple backends from one client as part of the federation features.

Agreed; this is an opportunity to take a more systematic approach and separate them properly. The other such point I found is that active_skeleton.js needs the global SkeletonAnnotations from catmaid, and access to CATMAID.TracingTool which is also defined in catmaid.

Minor point, here for documentation: d3 will need to remain an in-tree dependency because node 7+ can't build d3 <4. There may be similar issues with other dependencies; I haven't looked too closely at their version constraints yet.

there may be nothing stopping us from have multiple packages in-tree here

https://github.com/lerna/lerna seems to be popular for managing interdependent JS projects in a monorepo, and handles symlinking to each other as local dependencies, deduplicating shared external dependencies, synchronising version updates and publishing changes.

I've seen a fair bit of advice recommending that frontend and backend are treated essentially as separate projects, which took me a while to get my head around given our current repo structure. If we're considering making "real" JS libraries and building the frontend, it might be worth separating the concerns a bit by giving them (closer to) top-level directories rather than keeping them buried in django/applications/catmaid/static/thekitchensink/etc ; especially if we were to move towards not having them managed by django at all. The compiled bundles could always go into those django/etc paths: that could make our lives a bit simpler when it comes to not serving source files.

So our repo could look more like

django/
docs/
scripts/
sphinx-doc/
frontend/
  catmaid/
    package.json
    webpack.config.json
    src/
  catmaid-lib/
    package.json
    src/
  arbor/
    package.json
    src/
  vendor/
    d3/
package.json
lerna.json
README.md
LICENSE
.travis.yml
...etc

For reference: https://github.com/dan-kez/lerna-webpack-example

[clbarnes]

Removing the IIFE could (/should) mean unindenting most of the file, thus trashing the git blame anyway, thus removing the block against using an autoformatter.

[aschampion]

While I'm a fan of autoformatting, because of the peculiarities of JS, especially the heavy use of anon functions, I'm initially skeptical that one that wouldn't hurt readability exists. Happy to be persuaded, though.

OTOH we should autoformat and static analyze the hell out of the backend. I aspirationally put flake8 in Travis years ago and for awhile played the minigame of reducing warnings.

[aschampion]

Also with any formatting tools, they have to be compatible with all of our dev environments, which can be a tall order:

dev	editors
Tom	vim
Andrew	Sublime, vim
Chris	PyCharm
Will	VS Code

Even getting jshint working for everyone wasn't painless.

[aschampion]

Relatedly, if we ES6ify a lot of classes at the same time we remove the IIFE, not that much indentation should change.