Improve dev loop for libraries contributors

[alefranz]

Hello,

I would like to bring to your attention my experience in trying to work on the libraries, in the hope that it could be valuable feedback.

Since the moved to the monorepo, contributing or simply building libraries like Microsoft.Extensions.* has become more challenging and time consuming due to the need to build the clr.
I understand the benefits for the core team to work with a monorepo and that probably the core devs use powerfull workstations or VMs, however I believe the dev loop is quite taxing and if we could improve it, it would make easier to contributors to join the project.

---

To get started building a library, it now requires to install more SDKs and tools and a bit of setup, however that part is well document and although it requires more disk space, it is a one-off operation.

The troubles start when you actually want to run the build.

The recommended way to build to be able to work on the libraries is:

git clean -xfd
.\build.cmd -subset clr+libs -runtimeConfiguration Release

This command on a well specced current gen laptop would take over 10 mins. During this time the laptop would be totally unresponsive (not even able to type) and it will overheat (so not be able to type on it for a while once completed)

Once you managed to build it, you can use a helper to open the solution in Visual Studio
./build.cmd -vs SolutionName
however, if you do this as first thing, as recommended in the guide, if will generally fail to build

You could build from the command line, however the steps described in the guide do not specify that you have to modify your enviroments variables to use the local setup of the dotnet sdks. To do that I use:

    $env:DOTNET_INSTALL_DIR = "$pwd\.dotnet"
    $env:DOTNET_MULTILEVEL_LOOKUP = 0
    $env:DOTNET_SKIP_FIRST_TIME_EXPERIENCE = 1
    $env:Path = $env:DOTNET_INSTALL_DIR + ";" + $env:Path

it would be nice to have a helper in place like the activate script in the AspNetCore repo.
If you forget to set the environment variables the build will fail.
You also need to kill every dotnet.exe process with taskkill /IM dotnet.exe /F after changing the environment or it will not build correctly.

The most problematic thing is that if you build in Visual Studio too early or with a wrong dotnet SDK version, you build will break irreparably and the only solution I found to recover it is to do a git cleanand full build .

On top of that, there are sometimes commit not properly building, so you generally end up having to git clean and then build 2-3 times, assuming you did something wrong, then trying to move a few commits back and repeat until I get a working version.
This means every time you rebase a branch, even if you are one day behind, you would easily spend an our just to have a working build of the library again.

---

I think it worth discussing what could be done to improve the dev loop for libraries contributors.
Ideally, it would be nice to get a lightweight build that get a prebuilt CLR from CI and maybe even prebuilt shared libraries, to get to the same experience it was before the monorepo move.

Maybe I'm just really unlucky or the documentation is out of date and there are already tricks to simplify the process.
I'm happy to test all your suggestion and contribute to the docs if that is the case.

Thank you for your time,
Alessio

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

Tagging subscribers to this area: @safern, @ViktorHofer
See info in area-owners.md if you want to be subscribed.

[safern]

Thanks for the great valuable feedback, we really appreciate it and want to hear this to improve your experience.

it would be nice to get a lightweight build that get a prebuilt CLR from CI and maybe even prebuilt shared libraries,

I'm currently on my phone and can't reply to all the concerns.

FWIW given the timeline and that we're trying to burn down 5.0 issues down, I haven't been able to fund much time for this, but I did start looking into this as a prove of concept to see how feasible that would be, so stay tuned. 😃

[davidfowl]

@alefranz agreed. I was trying to make a change to dependency injection and I couldn't make anything work inside of VS so I just gave up and built from the command line while using VS code

[ericstj]

We have done some recent changes to make it easier to approach the repo and just build libraries.

Today the build of the CLR+LIbs is still a requirement to run tests since we don't want folks running on an LKG framework.

I've assigned @ViktorHofer to take a look at this as he has recently made a ton of changes to improve this experience and I want to make sure he has a chance to look at your feedback.

[ViktorHofer]

Hello @alefranz, thanks for the detailed feedback. I'll try to respond to all of it.

This command on a well specced current gen laptop would take over 10 mins. During this time the laptop would be totally unresponsive (not even able to type) and it will overheat (so not be able to type on it for a while once completed)

You are right with building the runtime part of the repository takes a considerable amount of time. During that, we fan out the CPU so that it builds as fast as possible but with the caveat of making the machine unresponsive during the build. @jkoritzinsky and others were looking into speeding up the coreclr build without fanning out the CPU in such an extreme manner (ie with Ninja). We should continue this discussion in #41637.

Once you managed to build it, you can use a helper to open the solution in Visual Studio
./build.cmd -vs SolutionName

The -vs switch is no requirement, it's a convenience wrapper to open Visual Studio with the repo local SDK. If you have a matching SDK installed globally, you don't need the -vs switch at all (make sure that you restored/built the solution once with the CLI to avoid NuGet issues).

however, if you do this as first thing, as recommended in the guide, if will generally fail to build

You are right that building inside Visual Studio isn't supported until you have built at least once from the repo root. Usually a build.cmd libs should be sufficient to just build the libraries part. Note that you might want to build coreclr as well if you intend to run tests: build.cmd clr+libs -rc Release. I definitely see room for improvement here, possibly by supporting one of all of these features:

• Represent the Microsoft.NETCore.App shared framework as a single msbuild project file which can be referenced from OOB libraries (with incremental build support and low evaluation time) and use ProjectReference items to define dependencies between shared framework libraries instead of current named references to allow building transitive dependencies as well.
• Layer the runtime (coreclr/mono) dependency as a ProjectReference with incremental build support so that building a test app also stages the test environment.
• Allow to download and stage prebuilts from a last known good (LKG) so that the coreclr/mono/libraries build can be skipped.

You could build from the command line, however the steps described in the guide do not specify that you have to modify your enviroments variables to use the local setup of the dotnet sdks. To do that I use:
it would be nice to have a helper in place like the activate script in the AspNetCore repo.
If you forget to set the environment variables the build will fail.
You also need to kill every dotnet.exe process with taskkill /IM dotnet.exe /F after changing the environment or it will not build correctly.

Thanks again for the feedback here. Our goal is to make building/testing possible without any environment changes (like what aspnetcore does with their active scripts). As mentioned above, using Visual Studio should just work after an intial root build. Maybe you can provide more data on why this isn't working for you.

The most problematic thing is that if you build in Visual Studio too early or with a wrong dotnet SDK version, you build will break irreparably and the only solution I found to recover it is to do a git cleanand full build .

I'm interested in how this breaks irreparably. Can you please provide more data so that I can follow-up?

On top of that, there are sometimes commit not properly building, so you generally end up having to git clean and then build 2-3 times, assuming you did something wrong, then trying to move a few commits back and repeat until I get a working version.
This means every time you rebase a branch, even if you are one day behind, you would easily spend an our just to have a working build of the library again.

100% agree. Note that this recently started to happen and is tracked in #40778. I'm working on a fix right now.

I think it worth discussing what could be done to improve the dev loop for libraries contributors.
Ideally, it would be nice to get a lightweight build that get a prebuilt CLR from CI and maybe even prebuilt shared libraries, to get to the same experience it was before the monorepo move.
Maybe I'm just really unlucky or the documentation is out of date and there are already tricks to simplify the process.
I'm happy to test all your suggestion and contribute to the docs if that is the case.

@alefranz thanks again for the feedback. Let's make sure that we follow-up on each individual issue that you reported. Note that we did some recent work that should allow building and testing a library for a non NetCoreAppCurrent (currently net6.0 in master) configuration from a fresh clone.

Building netstandard2.0: dotnet.cmd build -f netstandard2.0 src\libraries\Microsoft.Extensions.Configuration.FileExtensions\tests\.
Testing against net461: dotnet.cmd test -f net461 src\libraries\Microsoft.Extensions.Configuration.FileExtensions\tests\.

With the option to use prebuilts (ie as an option in dotnet.cmd/dotnet.sh) we should be able to make building and testing for any configuration work from a fresh clone in a reasonable amount of time. That said, even without the prebuilt option we should be able to sequence the runtime and shared framework dependency into the right place to remove any needed upfront restore/build steps.

//edit: Moving to 6.0 as any optimizations would target master.