Exploration of ideas for supporting multiple files in dotnet run file

[DamianEdwards]

Draft for discussion

[jjonescz]

That would exclude *.cs files too. We would also need to add <Compile Include="entrypoint.cs" /> for dotnet run entrypoint.cs to do something useful. I guess the CLI could recognize you set EnableDefaultItems=false and add the <Compile Include> automatically. Or we could have a special directive for that like #:single-file.

[DamianEdwards]

Yeah, I think it would be the former (have the CLI recognize and then add the compile item for the entry-point). It could be combined with #:include to get control over what files are included.

[jjonescz]

I don't think EnableDefaultItems is related to depth of included directory trees.

[DamianEdwards]

That's not what I meant. I meant in general for a normal project. Does it have a depth limit, does it stop in if it finds another project file, does it not search at all if in a drive root, etc.

[jjonescz]

Yeah, it doesn't search at all if in a drive root, but otherwise there are no limits I think.

[jjonescz]

I don't think we would need Exclude for normal programs, especially since we start with "exclude all". If one needs a more complex pattern, they should convert to a project-based program.

[DamianEdwards]

Yeah that's where I'm at too.

[jjonescz]

We could do this even now without compiler changes - just let users write Main methods instead of top-level statements until compiler supports multiple entry point files in one compilation.

[DamianEdwards]

just let users write Main methods instead of top-level statements

Sure, but I absolutely don't want that to be the trade-off. Top-level statements are the modern, idiomatic approach for C# and for the two main cohorts run file is aimed at.

[jjonescz]

Definitely, I was just thinking we could choose this approach now without waiting for compiler changes, to experiment with it.

[jjonescz]

I don't think we need to support this scenario. Even in your example, I don't see cat.cs and cat.benchmarks.cs referencing each other, instead they are both referencing cat.helpers.cs. That seems like a way this kind of problem could always be solved in.

[DamianEdwards]

In my example I did that because that's the only way it could work without us doing more to support it. I think this scenario is interesting though.

[jjonescz]

Might be better to generate the entry-point in a namespace derived from the file name. E.g., App.Program, App2.Program.

[jjonescz]

It is not clear how the converted project-based app would work. Since we want the converted project to be equivalent to the virtual project, that means the converted project should have all entry-point files included. But how are you going to invoke different entry points? I guess you could do something like dotnet run -p:Main=Entrypoint (assuming MSBuild property Main would be passed to the compiler's -main option).

[DamianEdwards]

The converted project would still be based on which entry-point file was specified so the property would be set in there too, but I agree there's an impedance mismatch between the capability of run file vs. explicit project in this case. This alone might be enough reason to not pursue it, at least for now, but I still think it's an interesting and appealing scenario (multi-entry-point files).

[jjonescz]

We should also think about the effect of #: directives. Currently your approaches seem to be based on one project for all entry points. That means that it is not possible for each entry point to have a different #:sdk directive.

[DamianEdwards]

Currently your approaches seem to be based on one project for all entry points.

In the case of default items being applied, yeah you're right, and in that case different SDKs (or other directives potentially) don't work well. But when default items are disabled and files are included manually by the user, they get control over what's included. Even in that case though, if you include another entry-point file that specifies a different SDK, it's not going to work well.

Do you have thoughts on alternative approaches that aren't just one project for all entry points? My kneejerk reaction to multi-project projections is that it's too complex and thus pushes against the goals of run file (simplicity, clear mapping to explicit project).

[jjonescz]

Do you have thoughts on alternative approaches that aren't just one project for all entry points?

Some possibilities for multiple entry points I've been thinking about:

1. dotnet project convert app.cs generates just a single project for the app.cs entry-point (there is some property in the generated project specifying the entry point) as you suggested, i.e., the other entry points are not invokable anymore basically (seems like a bug almost).
2. Similar to (1) but you get all the files copied during migration. Effectively you also end up with multiple projects (if you convert all your entry points), but they are independent, so they can stay simple.
3. Make changes so it's possible to have a single project (which is as simple as dotnet new console) that doesn't mind multiple entry points. That is, the compiler and SDK are updated, and you can do something like dotnet run app1.cs to distinguish which entry point to run. Under the hood the compiler could simply generate an entry point like:

return args[0] switch
{
  "app1.cs" => App1.Program.Main(args[1...]),
  "app2.cs" => App2.Program.Main(args[1...]),
};

4. Multiple projects. I don't think they are "too complex". Below I suggested to have Directory.Build.props to share the converted #: directives but maybe it would be simpler to just copy those to each project. Then you would have app1.csproj, app2.csproj whose contents are just dotnet new console (+ any converted #: directives). Then compiler would need to know which entry point each project file executes. Previously I suggested each project would have a property specifying the entry point but that's not needed either, the SDK props could derive that impliclity from project name and pass that to the csc task.

[jjonescz]

I would suggest an approach similar to this one but which

• has one project per entry point,
• the projects are all in one folder with the original files (so they automatically include all files in that folder),
• each entrypoint can have a different #:sdk (it's part of its project),
• has a Directory.Build.props in the target folder which
  • has elements from non-sdk directives (#:property, #:package),
  • includes the parent Directory.Build.props,
  • excludes the entry points via <Compile Exclude="..." />,
• each project includes its entry point via <Compile Include="..." />.

If there is just one entry point, the model could be simplified (no Directory.Build.props, and no <Compile Include>).

If/when compiler supports multiple entry points in a compilation, we could get rid of the explicit <Compile> elements and just let each project have some property specifying the entry point (-main).

If multiple projects in one folder are a problem for some reason, I think this approach could be modified in some way to fix that (e.g., changing DefaultItems.props infra to be able to include files from the parent folder easily if opted-in; or having multiple projects with InternalsVisibleTo; or copying the shared files to each project folder).

[DamianEdwards]

So this is basically what we had before in your original PR? Each entry-point file is its own virtual project, and we discover and exclude other entry-point files in each virtual project.

[jjonescz]

Yeah, I guess it's similar. But I've just started from your proposal, trying to fix issues I saw with it (multiple #:sdks, ability to execute multiple entry points in the grown up project, etc).

[jjonescz]

An alternative opt out is simply moving the .cs file to its own folder. dotnet run folder would still work (not implemented yet, but planned) and be just as concise as the original dotnet run file.cs.

[DamianEdwards]

dotnet run folder would still work

I'm not sure I want us to support that. I think the semantics of "run this specific file" are much clearer than updating the existing run folder semantic to include not just "find the project file in this folder and run it" but also "find the implicit project in this folder and run it".

Moving the file into its own folder would still work to isolate a file though of course, just that it would still be run with dotnet run file.cs.

[jjonescz]

include not just "find the project file in this folder and run it" but also "find the implicit project in this folder and run it".

Hm, to me that just sounds consistent - implicit projects behaving more like explicit projects. Also simple - if dotnet run doesn't find the project file, it then switches to the implicit project mode (just like for dotnet run file).

But I guess it would need to behave in a confusing way to avoid a break: If there is a project file in the current working directory, dotnet run ./folder/ should continue to run the current folder project file and pass ./folder/ to it as the first argument.

Anyway, imagine we have some cs binary that only can run files and doesn't have any project-based support nor backcompat baggage. Would you still want cs folder to not work?

[DamianEdwards]

Anyway, imagine we have some cs binary that only can run files and doesn't have any project-based support nor backcompat baggage. Would you still want cs folder to not work?

Nope. In that world, I imagine you must pass the entry-point file as the first argument.

[jaredpar]

Curious: do you have any concrete examples of an app that you see built this way? The one case I can definitely see is infra. Basically I see dirs today with a bunch of test-blah.ps1, build-tests.ps1, etc ... that have a simple lift and shift to .cs entry point files. It's harder for me to see the case in a web app.

[DamianEdwards]

What I had in mind was mostly learning scenarios where one might have a folder of very simple apps with potentially some shared types.

[jjonescz]

Btw, to bring back support for this, I think we need a change in Roslyn so it's possible to pass (via a feature flag for example) the entry-point file path. Then Roslyn itself excludes other entry-point files from the compilation. The alternative is detecting entry points in SDK itself but that's problematic because we cannot 100% correctly parse files when we don't have access to DefineConstants. Imagine a file that has top-level statements under #if SOMETHING and there is <DefineConstants>SOMETHING</> in a Directory.Build.props. The SDK cannot easily know that when deciding which files to pass down to the compilation, only csc.exe knows the set of effective DefineConstants. Perhaps this inconsistency would be fine for starters if we don't want to change Roslyn like that though.

[jasonmalinowski]

It's a bit of a catch-22 though -- csc.exe doesn't really know the DefineConstants either -- those just come from MSBuild. Which is only known once we've already created the project. Imagine the truly evil case of your Directory.Build.props making the DefineProps conditional on the existence of a Compile item! We obviously don't care about such a scenario, but fundamentally we'll have to decide how we break the dependency loop of:

1. To build the project file, we have to know entry point files.
2. To know entry point files, we have to parse files.
3. To parse files, we have to know the compiler settings.
4. To know the compiler settings, we must build the project file.

[jjonescz]

Imagine the truly evil case of your Directory.Build.props making the DefineProps conditional on the existence of a Compile item

I don't see a problem there, we don't need to filter the Compile items, we always include all of them, then csc.exe filters them (after all MSBuild logic has completed) according to the entry point that the user specified (via dotnet run entry-point-specified-here.cs).

I.e., there is no loop:

1. User specifies the entry point on the command line of dotnet run file.cs
2. We let MSBuild build a virtual project that contains all the files in the directory as usual. We don't consider the entry point here at all (except that it is passed via MSBuild as a compiler feature flag or some other input).
3. The compiler knows the entry point should be file.cs (can filter others or report errors if it doesn't have top level statements nor Main method) and it also has all the info it needs from MSBuild to perform the compilation.

[jasonmalinowski]

Ah, so csc might get file paths that ultimately get ignored later?

[jjonescz]

Yes, because the logic of which file has an entry point is complicated (especially with different variants of Main) and should be left to the compiler to decide. So it needs to know all paths and then decide which to ignore. It might not even ignore the other entry point files completely, just decide which entry point to set in the dll as Main (so classes from the other entry point files are still usable for example).

[DamianEdwards]

It might not even ignore the other entry point files completely, just decide which entry point to set in the dll as Main (so classes from the other entry point files are still usable for example).

This is where I'd love to land.

[jaredpar]

Not a blocker, but these will end up dove tailing into language design issues. Think we'd need to dive into general top level functions if end up including multiple entry point files.

Did you consider an option where we just have the compiler exclude them? Basically if you execute dotnet run app1.cs that we implicitly exclude app2.cs?

[DamianEdwards]

Yeah it seems the natural best-fit here is still to default to including default items, except for other entry-point files.

[jjonescz]

I see like three stages here we could do incrementally:

1. Start with excluding other entry-point files.
2. Change compiler to generate Programs from top-level statements in namespaces (could be implementation-only feature flag for starters). This will simplify the generated project files - no need to have "excludes", just specify an entry-point.
3. Change language to also make local functions visible across top-level statement files.

[jaredpar]

The other issue is that it means more work to have a single file app look like the dotnet new console experience.

[dsplaisted]

As growing up into project files: Ideally, you could have Program1.cs and Program2.cs in the folder, they could be run separately with dotnet run, and they could both depend on shared source code in the folder. Then you could convert them both to projects, and end up with Program1.csproj and Program2.csproj.

However, we don't have good support for different project files in the same directory. By default they'll share the obj folder and try to use the same project.assets.json file. Maybe we could have the grown up projects specify UseArtifactsOutput=true so that each project file gets separate output files. But that ends up being different from the default you get with dotnet new console.

[DamianEdwards]

As growing up into project files: Ideally, you could have Program1.cs and Program2.cs in the folder, they could be run separately with dotnet run, and they could both depend on shared source code in the folder. Then you could convert them both to projects, and end up with Program1.csproj and Program2.csproj.

For this scenario (which I very much want to support) I think the output of the grow-up should be a new directory for each project. There's questions to be answered about whether the grow-up command requires the entry-point file to be specified (so it's clear which is being upgraded) or if you can upgrade a whole directory of file-based apps to a bunch of projects, each in their own directories.

I think we need to default the experience so that you can't ever end up with multiple projects in the same directory as part of the grow-up story, as that's a pit of failure. Perhaps that means that the only time you can ever do an in-place project upgrade is when you have a single entry-point file in the directory.

[jjonescz]

I imagine IDEs can work something like this in multi file scenarios:

• If a single file is opened (like code C:/script.cs), consider just that single file.
• If a folder is opened (like code C:/scripts/) and then in the editor, one of the folder's files is opened (e.g., C:/scripts/a/b/c.cs) and it doesn't belong to any project, search for the nearest entry point file (that's where the implicit project will come from) but do not go above the opened workspace folder (i.e., search up to C:/scripts/ but not in C:/).

[jjonescz]

I'm thinking about an approach where:

• only the single entrypoint file is included in the program by default,
• you can reference other projects via something like #:reference ../shared (where there is ../shared/shared.csproj),
• non-.cs files from the directory could be included (like appsettings.json or even .razor depending on the #:sdk),
• implicit build files would also be considered (directory.build.props).

Then:

• No surprises for users (dotnet run file.cs includes only the file.cs unless there are directives saying otherwise).
  • That is, you can put your .cs file-based program anywhere without worrying about implicit dependencies.
• Simple IDE implementation (I think? no need to discover other files unless there is a directive but that should be then similar to standard project references).
• Allows for nicer organization where the entry point .cs files can be sprinkled around a codebase but still reference common utilities (unlike folder-based approaches where all entry point files need to be in one directory with the utilities).
• Straightforward grow up (create new folder if necessary and project for the entry point, migrate #:reference to <ProjectReference/>).
  • If there are other files like appsettings.json or .razor and multiple entry points in one folder, then we could copy the non-.cs files or explicitly include them via MSBuild items. But that should be rare. I would expect users to either have a folder with just .cs files (scripts) or a folder with .razor files and a single .cs, and in either case there would be no grow up problems.
• Users start with single file programs. If they want to share some code among scripts, that's a bit more advanced scenario, so they need to create a project to put the shared code in.
  • An IDE gesture like "move this class to separate file" could do that for them.
  • The CLI could also have a command to help here.

[DamianEdwards]

So multiple files or sharing files requires a project in this proposal? I don't think we should have an approach that "crosses the streams" between file-based apps and projects. I'd rather us add a directive to include other files, e.g. #:include **/*.cs, than support referencing projects. But ideally we could land on an approach where other files are simply included by default as they would be in a project, and we sort out the multiple-entry points issue somehow.

[jjonescz]

Agree that would be nice but allowing other utility files and multiple entry points in the same directory will result in complex project files on grow up.

From scripting perspective, supporting just single files seems perfect. Ideally you would only ever import nuget libraries into them. But if you want some custom utilities, you import "local nuget library", i.e., a project (likely a project your repo already contains).

From "lower entry barrier" perspective, I see why you would want to support multiple files in a folder without a project file. Perhaps single file is enough to lower the entry barrier though? You always start with a single file anyway and then you are just one dotnet project convert invocation away from multi file support.

[DamianEdwards]

As discussed today, I think the extra complexity in the converted projects that comes out of having multiple entry-point files is a reasonable trade-off. If you convert a file-based app to a project and there's only a single entry-point, we can do it in-place. If there's multiple entry-points, we have to produce multiple projects which means it has to be done in a separate (user-provided) directory. Common/shared files in the multi-entry-point file scenario get put into a class library project and referenced by the app projects that were created from the entry-point files. This mimics what one would reasonably do if they'd simply started with projects in the first place.

[jjonescz]

This mimics what one would reasonably do if they'd simply started with projects in the first place.

Note that the app would need to have InternalsVisibleTo to the shared project.

Also if we want to change compiler in the future to allow entrypoints see each other's code, the converted projects would all need to reference each other which I'm not sure is even possible.

[DamianEdwards]

When you say "allow entrypoints see each other's code" do you mean non-top-level types that are defined in the entrypoint file being able to be seen by another entrypoint file? If so, then yeah, that doesn't translate once you move to multiple projects, and so should be a sign for us to not pursue that. Making the stipulation that shared code must be in separate files seems reasonable though.