.NET Core paket and dotnet/sdk

[enricosada]

Paket depends now on .NET framework (win) and mono (unix/mac).

This is really good because the normal paket.exe console app can run xplat if mono/.net is installed.

For some scenario, like .net core development (or fable) with new dotnet/sdk (new csproj/fsproj), the mono/.net shouldnt be needed.

To remove that deps of paket in these scenario there are some points to address:

• xplat paket.exe console app, based on .net core
• boostrapping process (from clean repo)
  • minimal set of files (and size) needed to be committed on repo
  • self updating of boostrapping process
  • binaries sources: nuget/github
  • chicken/egg. how to boostrap the boostrapper
• execute paket from command line, xplat. with good dev UX
  • magic mode
• auto-update of paket
  • frequent release schedule -> minimize size/download
  • binaries sources: nuget/github
• minimize UX changes for users
• maintenance.
  • try to avoid two version (net/netcore) because old sdk project can also use .net core based paket
• tradeoff downloadsize vs dependencies

Atm ihmo boostrapping and auto-updates are the real issue

Current Plan

• make paket console app to .net core so works the same as now if published as SCD (wip [WIP] .net core paket #2918)
• implement dotnetcli tools, per repo ( ref Repo tools #2919 )
• modify build.fsx to put in Paket nuget both paket .net core FDD (in tools/netcoreapp2.0) and paket .net fw (in tools/net451)
• reuse paket SCD itself as boostrapper, or use a dotnetclitool, to add paket in PATH

Notes

notes about console deployment options on .net core

Options are:

• SCD, self contained deployment
  • bigger, one release for platform (unix/win/macosx), require just .net core runtime 2.0 DEPS installed
• FDD, framework dependent deployment
  • smaller size
  • require .net core 2.0 runtime installed
  • execution is dotnet paket.dll, so require a wrapper script to execute for UX (paket.cmd/paket.sh)
  • where is dotnet? is the right version installed?
• DotnetCliTool
  • just a proj (few lines of text) is required
  • update based on just nuget.org
  • require restore
  • require .net core sdk installed
  • run as dotnet paket
  • require execution from same dir of proj, so need wrapper scripts (paket.cmd/paket.sh)

File size:

	DotnetCliTool	FDD	SCD
File Size	0,1 MB	1MB	60 MB
Requirements	.net sdk + runtime + runtime deps (120MB)	.net core runtime + runtime deps (60mb)	.net core runtime deps (native pkg)

notes about paket, dotnet/sdk, .net core

paket is a strange tool, because atm for restore process, need to be there BEFORE the restore.
or at least a target file to override default nuget implementation in dotnet/sdk
instead old sdk require the exe to restore/update projs

But lots of challenges are shared with normal tools in .net ecosystem

There is no builtin per repo tools. or a way to script (like an fsx).
AFAIK the most similar to per repo tools are the cli tools or the FDD.
AFAIK the most similar to script is an msbuild target + optional task in assembly

How now .net sdk support external tools? or custom execute code xplat.

• Dotnet Cli tools (per project)
• msbuild targets
• built it on machine. dotnet new + override Program.cs + dotnet publish

Other package manager expect the dev sdk installed, and use that to boostrap:

• npm -> node
• pip -> https://bootstrap.pypa.io/get-pip.py -> python
• bundler -> gems -> ruby

Possible solutions

My current ideas, not in preference order.

plan A

support tools at repo level natively in dotnet/sdk. ref https://github.com/dotnet/cli/issues/4723

PRO:

• native support in dotnet/sdk
• less maintenance by paket team
• no boostrapper needed. the sdk is the boostrapper.

CONS:

• native support in dotnet/sdk
  • require new version of dotnet/sdk. probably at least 2.2 or 3.x
  • will not work on older VS version or just msbuild (but VS now bundle .net core sdk...)
• depends on dotnet/sdk team
  • slower to patch/workaround, based on .net team schedule not paket team
• require .net sdk installed, but native paket app can be download if needed, that's just for boostrap
• cannot replace current paket for older sdk

plan B

Why: paket deps doesnt change a lot. paket does.

like now, using native app (SCD) but smarter update mechanism

• paket published as SCD and FDD
• current paket.boostrapper.exe as SCD
• SCD are big, need a smart update mechanism
  1. first usage download latest native console binaries (per OS)
  2. updating is just xplat paket FDD, so paket.dll and non-bcl managed deps (paket.core.dll, etc)
  3. if not possible the minimal update, do step 1
• boostrapping process start with a dotnet msbuild (target + dll), to just download paket.boostrapper.exe

PRO:

• pretty much like now
• can replace current boostrapping
  • new deps will be just .net core 2 runtime deps, not mono/.net
• boostrapping can also download the net version like now, if want to support that too

CONS:

• maintenance on paket team
• paket specific :( a missed opportunity
• download paket native boostrapper is big. but can be cached.

plan C

Why: support and enhance .net cli tools. use that support for paket itself.

Add support of repo level tools to paket and .net cli tools. Use a built paket native (SCD) to boostrap paket (replace previous paket.boostrapper.exe).
The chicked/egg solution.

• paket as packaged console app (like a .net cli tool, ex dotnet-fable), so small and easy to pack/version
• boostrapper (paket native):
  • is paket, with a paket.depencies who contains paket-cli as tool
• add .net cli tools support to paket
  • the .net cli tool (like dotnet fable) just invoke dotnet with args and a file with deps. paket generate these args/files needed
  • generate a shims/script wrapper, so a dotnet hello tool can be invoked as hello. and paket as paket
• boostrapping based on dotnet msbuild to just download native paket per OS

PRO:

• cli tools can be in also used paket.deps/lock. per repo.
• boostrapper is not a special case, is just paket built as native console app
• minimal size.
• enable other tools (good). like dotnet fable as fable. will work in whole repo not just the proj dir.

CONS:

• download paket native boostrapper is big. but can be cached.
• boostrapper paket schedule <> paket schedule, so doesnt need to update often. more maintance.
• atm cli tools are from nuget feed. github fallback need to be implemented (paket download nupkg from github to a dir -> add that dir as feed -> restore)

Conclusions

• plan A: .net oss way, but slower to implement and require at least .net sdk vNext version as deps.
• plan B: upgrade current boostrapping mechanism
• plan C: remove current bootstrapper, replace with paket who support .net cli tools

Atm i prefer plan C, while is longer than B to implement, add a lot more value with less maintenance long term.
Plan A is ok, but doesnt help paket in medium term, because cannot replace current way.

TODO

as a note, i volounteer to implement the choosen solution. if someone want to help, will be nice, because tasks can be split

[forki]

Is there a way to put bootstrapper directly into sdk?

[enricosada]

Is there a way to put bootstrapper directly into sdk?

yes that's another way. two options afaik:

bundled in sdk, as app/package

Technically, yes. we have done that for fsharp.net.sdk, and put inside sdk is just a packaging thing (cli/sdk build script restore a pinned package and unzip in release dir).
But depends if .net cli/sdk team want to do that.
For example with fsharp.net.sdk, was just possibile to put in dotnet cli, but not in nuget packages distribuited in the sdk installed with VS because of policy. so it will had worked with dotnet sdk (the cli) but not in msbuild bundled with VS. This was strange because some package were there, like newtonsoft or others by aspnet deps, but that it.

Issue is, if there is an error in boostrapper, you need another release of sdk to fix that (and just sdk team can schedule that).

as Sdk attribute

Another possibility, the real issue, is if Sdk attribute can be extended, like by restored packages.
that is the real extensibility paket need afaik.
So will be possibile to use like Sdk="Paket.Sdk;Microsoft.NET.Sdk".
Is somewhat tracked in dotnet/msbuild#1493 but is now stalled afaik, because noone in MS need that (they can just continue to bundle things..)

Both mean will work only with new msbuild/cli/sdks versions.. and given the churn in sdk tooling and the need for stability, ihmo is not going to be implemented soon..

[enricosada]

ok, dotnet cli is adding tools (sdk scooped) in v2.2 (ref https://github.com/dotnet/cli/issues/8067 )

we can use that later for boostrapper. meanwhile need another options for that

And paket can manage tools too, because should be easy. adding per repo tools (who can be used to self-update itself, if the boostrapper is just paket)

the tools are just .net core fdd app published in the tools/netcoreapp2.1 dir of a nupkg.

ref a nuspec

<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2010/07/nuspec.xsd">
    <metadata>
        <id>global.tool.console.demo</id>
        <version>1.0.4</version>
        <description>test app</description>
        <authors>testauthor</authors>
    </metadata>
    <files>
        <file src="bin\Release\netcoreapp2.1\publish\*.*" target="tools\netcoreapp2.1\" /> 
        <file src="DotnetToolsConfig.xml" target="tools\DotnetToolsConfig.xml" /> 
    </files>
</package>

and an xmlfile to read the entry points, so commands can be renamed and multiple

<?xml version="1.0" encoding="utf-8" ?>
<DotNetCliTool>
  <Commands>
    <Command Name="demo" EntryPoint="consoledemo.dll" Runner="dotnet" />
  </Commands>
</DotNetCliTool>

[enricosada]

For interested ppl, there is a demo version on a ready to use repository ( more info #2918 (comment) ).
is ok to beta test (just the boostrapper require mono atm, but is temporary)

based on the PR wip, will be updated

[enricosada]

By @0x53A in #2418 (comment)

MSBuild will add a package-based SDK resolving mechanism.

MSBuild spec: dotnet/msbuild#1493
NuGet work item: NuGet/Home#5220

There is nothing to do at the moment - but as soon as NuGet actually implements this stuff, we should also start so we don't hang behind.

From the empty checklists in both issues, I would expect this to land in 16.0 maybe.

The important question is how the resolving works when we don't control the host (VS or Rider, or VS Code, ...).
Can we pass-in our paket resolver, or will VS always hardcoded use the NuGet resolver?