Idea: collaboration with pnpm

[zkochan]

I am thinking about how could we collaborate (maybe even merge).

I already see many people that are creating issues and commenting in both repos and pnpm was based on the ideas of ied (or maybe even a fork? i don't know the whole story). So why don't we unite efforts?

I really like the new issue created by @alexanderGugel about rewriting ied to Go - #181. It is something that I was thinking about as well.

I might be wrong, but I think currently pnpm has less issues than ied. It runs good on Windows, can install packages globally, works mostly well when running with the --preserve-symlinks option. What if we will recommend to use pnpm and will try to solve all the issues there. In the meantime we will be working on rewriting ied to Go. The two projects can use the same store structure, CLI and config files in order to guarantee a smooth switch once the rewrite is ready.

[zkochan]

cc @iamstarkov, @andreypopp, @rstacruz

[billiegoose]

Ooh ooh! If you guys are writing a new package manager & thinking about a common store directory structure, I want in. I started on a (highly) experimental package manager a little while ago. One of my goals is to eventually be able to require() dependency tarballs directly, rather than have to extract them first. This adds some safety from having immutable dependencies, and allows you to verify the tarball checksum at any point in the future (say, just before loading) to make sure it hasn't been altered. (My initial prototyping utilized archivemount to mount the tarballs in the FUSE file system in read-only mode. ) BUT it has the interesting effect that you cannot put symlinks inside the package's node_modules folder, because the package folder is read-only. So I had to invent an entirely new store directory structure that is different from pnpm and ied 1.x. (Actually it looks similar to the directory structure used by ied 2.x... although I can't tell because ied 2.x has never worked on my machine.) Any opportunity to collaborate would be great for motivating me to continue working on the project. :)

[zkochan]

I think a common store directory structure is a good idea. What do you think about creating a spec-project that would describe the store structure. It can be called something like Modules Store specs?

[alexanderGugel]

Yes, that sounds like a good idea. Sorry for the delayed response, I just checked out @wmhilton's module loader... looks really nice. I like it.

But back to the spec: Yes, I think formalising the symlink-based installation procedure is a good idea. On the one hand it makes a stronger case for not "deprecating" the preserve-symlink module resolution, on the other way it would provide some common ground for ied, pnpm and whatever comes next.

I would suggest we keep this issue open as a discussion forum for ongoing work on this spec while writing some technical documentation on how the resolution is supposed to work (including the currently available module resolution in Node, since we might have to patch Node if everything goes wrong).

I'm going to come up with an initial draft this weekend, if anyone else wants to own this (@zkochan you seem exited about this 😄 ) and / or already has some docs, I would suggest we work from there. Otherwise I'm going to write some really serious specifications docs on this and we collaborate on that? 🎉 🎈

Thanks @zkochan for opening this issue!

[zkochan]

That sounds great! And I totally would like to be a collaborator there!

I am not really familiar with the ied store structure but let me describe the pnpm one. And I think there are many good ideas in the pnpm store structure.

.modules.yaml

First of all we have a .modules.yaml in the root of every node_modules folder. It contains currently just one property with the store location. E.g.:

storePath: 'C:\Users\zoltan.kochan\.pnpm\.store\nested'

We use this like a checksum. If someone tries to install a new package using a diferent store, pnpm throws an exception. One node_modules should use one store. (On the other hand, one store can be used by many node_modules)

.store.yaml

In store.yaml we store the dependency graph of the store. We need it mostly for uninstall, as far as I remember. It is in the root of the store and looks like this:

pnpm: 0.45.0
type: nested
preserveSymlinks: true
packages:
  'C:/Users/zoltan.kochan/.pnpm':
    dependencies:
      npmrc: npmrc@1.1.1
      gulp: gulp@3.9.1
  npmrc@1.1.1:
    dependents:
      - 'C:/Users/zoltan.kochan/.pnpm'
  'C:/src/resources/foo':
    dependencies:
      debug: debug@2.3.2
      compression: compression@1.6.2
      require-from-string: require-from-string@1.2.1
      express-cluster: express-cluster@0.0.4
      safe-eval: safe-eval@0.3.0
      snabbdom-to-html: snabbdom-to-html@3.0.0
      node-fetch: node-fetch@1.6.3
      express-status-monitor: express-status-monitor@0.1.5
      tap-spec: tap-spec@4.1.1
      snabbdom: snabbdom@0.5.4
      tape: tape@4.6.2
      request: request@2.78.0
      express: express@4.14.0
  debug@2.3.2:
    dependents:
      - 'C:/src/resources/foo'
      - precinct@3.5.0
      - babel-core@6.18.2
      - send@0.1.4
      - babel-traverse@6.18.0
      - array-index@1.0.0
      - resp-modifier@6.0.2
      - connect-injector@0.4.4
      - eslint@2.13.1
      - eslint@3.0.1
      - gulp-livereload@3.8.1
      - mini-lr@0.1.9
      - babel-traverse@6.19.0
      - eslint@3.10.2
      - 'E:/src/resources/foo'
    dependencies:
      ms: ms@0.7.2
...

Store directory structure

pnpm tries to make user friendly names for directories in the store. So for foo@1.0.0 there will be a directory called like that in the store. For a package resolved via github, there would be a directory called as the repo URL (escaped to be a valid folder name).

I think the store structure should be human friendly, when possible. I would suggest to make the structure something like this: <package source>/<package name>/<package version>.

So for a package resolved from the official npm registry, the path to the package would look like: .store/registry.npmjs.org/gulp/2.1.0/

to a git-hosted package mayby something like: .store/github/alexGugel/ied/b246270b53e43f1dc469df0c9b9ce19bb881e932/

[billiegoose]

@alexanderGugel why thank you! it's not really much more than experiment at the moment.

You know, that .store.yaml file looks a lot like a yarn.lock file...

# yarn lockfile v1
abbrev@1:
  version "1.0.9"
  resolved "https://registry.yarnpkg.com/abbrev/-/abbrev-1.0.9.tgz#91b4792588a7738c25f35dd6f63752a2f8776135"

accepts@~1.3.3, accepts@1.3.3:
  version "1.3.3"
  resolved "https://registry.node-modules.io/tarballs/accepts/1.3.3.tgz#c3ca7434938648c3e0d9c1e328dd68b622c284ca"
  dependencies:
    mime-types "~2.1.11"
    negotiator "0.6.1"

accord@^0.26.0:
  version "0.26.3"
  resolved "https://registry.node-modules.io/tarballs/accord/0.26.3.tgz#7a6f07dfe204f67f0d48ee6c7b0fe40723183010"
  dependencies:
    convert-source-map "^1.2.0"
    glob "^7.0.5"
    indx "^0.2.3"
    lodash.clone "^4.3.2"
    lodash.defaults "^4.0.1"
    lodash.flatten "^4.2.0"
    lodash.merge "^4.4.0"
    lodash.partialright "^4.1.4"
    lodash.pick "^4.2.1"
    lodash.uniq "^4.3.0"
    resolve "^1.1.7"
    semver "^5.3.0"
    uglify-js "^2.7.0"
    when "^3.7.7"
...

except yours is a doubly-linked list because it has dependents too. I suppose that makes uninstalling packages simpler? Because you'd be doing reference counting rather than garbage collection? I kinda like that.

So for a package resolved from the official npm registry, the path to the package would look like: .store/registry.npmjs.org/gulp/2.1.0/

I like that idea! But I'd say we make the second example .store/github.com/... for consistency. Namespace using domains. Almost everything has a website. Even if it's non-traditional (ipfs, gittorrent, dat) you can still namespace by domain (ipfs.io, datproject.org) even if the content isn't physically stored on that domain.

This is really interesting!

[iamstarkov]

@wmhilton

You know, that .store.yaml file looks a lot like a yarn.lock file...

that was intention

[iamstarkov]

I suppose that makes uninstalling packages simpler?

yep

[billiegoose]

I was focused on other projects this weekend, did I miss any spec-writing action?

[zkochan]

I've just created a repo and added you guys as collaborators: https://github.com/zkochan/package-store-spec

We probably can create PRs there with suggestions and discuss each part of the spec

[alexanderGugel]

Nice! I'm still at Overview and Motivation though 😄
@zkochan Do you want to move your repo to the ied wiki or keep it separate?

[alexanderGugel]

@zkochan I added what I have so far to the Wiki. I would suggest we move the discussion to there, so anyone can edit. Then we can move it to a common organisation or sth. Thoughts?

https://github.com/alexanderGugel/ied/wiki/Spec

[zkochan]

@alexanderGugel yeah, sure, I have no objections.

[alexanderGugel]

Also I referenced a talk I recently gave at Node Live about the installation mechanism, directory structure etc. (https://www.youtube.com/watch?v=mNhZrd1VgPs) Might be useful, since it explains the directory struct quite well etc.