Add new files directly to CMakeLists.txt (#2132)

[malsyned]

This change addresses item #2132

This changes visible behavior

The following changes are proposed:

Add two new commands that edit CMake List files to append or delete source file names from command invocations.

• CMake: Add File to CMake Lists cmake.addFileToCMakeLists
  Present the user with options for which command invocation to modify, and then edit that command invocation to add a new source file to its arguments.
• CMake: Remove File from CMake Lists cmake.removeFileFromCMakeLists
  Edit any command invocations that include a source file, removing it from their argument lists.

These commands can optionally be triggered when a source file is created or deleted.

Quick Pick items for targets and invocations are carefully sorted to make the first option very likely to be the desired one, and settings exist to automatically choose the best option rather than asking.

Add options for controlling the behavior of these commands.

Optionally, present proposed edits to the user to Apply or Discard.

The purpose of this change

• To make more of the workflow for CMake-based projects possible without having to hand-edit CMake lists.
• To remove a manual step, and thus a possible source of user error, from the workflow for CMake-based projects.
• To bring vscode-cmake-tools closer to feature parity with other IDEs like Visual Studio, CLion and QtCreator.

Other Notes/Information

Phew. This one wound up being a lot bigger of a bite than I expected when I decided to try to tackle it. Apologies for the surprise 1.5 kloc pull request. I'm assuming/hoping that it's work you're interested in, since it's in response to a 3-year-old feature ticket that had a lot of momentum and maintainer buy-in when it was first introduced.

It's still a draft and needs some polishing before it's merge-ready, but I don't want to let it get any bigger before getting some feedback from the maintainers about whether I'm on the right track. I'd consider it just about feature-complete though, and ready for testing.

Here's an explanation of how this feature works. Consider this a rough draft of the user documentation.

Terminology

I have tried to use this terminology consistently throughout the implementation and documentation.

• CMake List file: Any file written in the CMake language, including both CMakeLists.txt files and other *.cmake files used via include() or similar mechanisms.
• CMakeLists.txt: A CMake List file with this exact file name, usually intended to be used by calling add_subdirectory on the directory that contains it. (This distinction matters because the Backtrace Graph code path can find command invocations in any CMake List file that contributes to the build system, but the parser-based legacy fallback path, and the variable invocation search, only take into account files named CMakeLists.txt.)
• Target: A CMake target defined with add_executable, add_library, or add_custom_target.
• Target Source Command Invocation: A line of CMake code that invokes a command which adds source files to a target. For example, target_sources, add_executable, add_library, or a user-defined function or macro that in turn invokes one of those commands.
• Variable Command Invocation: A line of CMake code that invokes set(), list(APPEND), list(PREPEND), or list(INSERT).
• Source List: A subset of the arguments of a command invocation representing a list of source files. Some commands, such as target_sources, can contain more than one source list with different scopes or file set types.

Selecting targets, command invocations, and source lists

If only one option is identified at any of these steps, it is used without prompting the user, to reduce the amount of user interaction to only what is necessary. Additionally, every step has a corresponding auto setting to choose the highest-sorted option automatically instead of prompting the user.

Targets

Identifying

1. The code model is walked to find targets.
2. Only targets defined for the current project.currentBuildType() are considered.
3. Only targets whose sourceDirectory contains the new file are considered.
4. If cmake.modifyLists.targetSelection is askNearestSourceDir, only targets defined in the closest CMake List file are considered. Otherwise, all parent directories up to the project root are considered.

Sorting

1. Within a project, all non-UTILITY targets are grouped before any UTILITY projects.
2. Within those groups, targets with longer path components sort earliest, on the assumption that a file is most likely to be part of the target defined closest to it.
3. Within a list file, targets with more source files are prioritized over targets with fewer. idea here is that if a single list file defines multiple targets, one is the "main" target and the rest are probably small supplemental targets.
4. Ties are broken alphabetically by target name.

Target Source Command Invocations

Identifying

Command invocations are identified using different methods depending on whether the file API, which provides a backtraceGraph for each target, is available.

From Backtrace Graph

1. The backtrace for the selected target is searched for frames that invoke one of the commands in cmake.modifyLists.targetSourceCommands. The backtraces for those frames are then walked to find the outermost line of code that caused that command to be invoked.
2. Invocations in list files that are outside of the project root are filtered out.
3. Invocations in list files in sub-directories that don't include the new source file are filtered out.

Fallback: parse CMakeLists.txt files

1. CMakeLists.txt in the same directory as the new file and every parent directory up to the project root are considered.
   • If cmake.modifyLists.targetCommandInvocationSelection is askFirstParentDir, only the first one encountered will be examined.
2. command invocations matching the commands listed in cmake.modifyLists.targetSourceCommands are identified.
3. Invocations are filtered for those whose first argument matches the selected target. Target arguments are compared exactly as they appear in the CMakeLists.txt. Variable substitution is not performed on their names. (This is a disadvantage compared to the Backtrace Graph method, which is operating on data produced by CMake after all variable substitution has been performed and all generator expressions have been evaluated.)

Sorting

1. If a target has source command invocations that span list files in multiple directories, invocations are grouped by file with the longest path first, on the assumption that a target may be created in a parent directory but have source files added to it in a subdirectory.
2. Within a directory, invocations in CMakeLists.txt are sorted before other list files.
3. Lines indented less sort before lines indented more -- the idea here is that commands in if(), for(), or while() blocks are less likely to be "default dumping grounds" for source lists
4. Within a file, invocations at the same indentation level are sorted with the longest argument list first. This is based on the assumption that one of those invocations is considered the "default" place to add new source files, and it will already be the longest.
5. Although ties are unlikely by this point, they are broken by the order of commands in cmake.modifyLists.targetSourceCommands, then alphabetically by file name, then by line number.

Source Lists

Every target source command invocation produces at least one source list for consideration, but some can produce more than one, such as:

• A target_sources command with multiple scopes (e.g. a PUBLIC HEADERS file set and a PRIVATE-scoped set of .c / .cxx files).
• A user-defined command that uses parse_arguments to have more than one multi-value source list.

Identifying

1. If the selected command invocation's arguments contain any of the keywords PUBLIC, PRIVATE, or INTERFACE, then its argument list is parsed as for a target_sources command (regardless of the actual command name), and any identified scopes or file sets are produced.
   
2. Otherwise, the argument list for the selected command is searched for matching keywords, with subsequent non-keyword arguments treated as a list of source files.
   
   if cmake.modifyLists.sourceListKeywords has any entries, then keyword arguments are only considered to introduce source lists if they match an element of that array. Otherwise, any argument matching /^[A-Z_]+$/ is assumed to introduce a multi-value source list.
3. Any other invocation is assumed to have a single source list which continues up to the closing parenthesis.

Sorting

1. FILE_SET lists are only shown for header files and C++ module files. When they are, they appear before non-FILE_SET scopes.
2. File sets in the same scope are sorted with user-defined names sorting earlier than sets with default names (e.g. HEADERS, CXX_MODULES).
3. Scopes sort differently depending on the file type of the file being added.
   • For source files, scopes are sorted with PRIVATE first, then INTERFACE, then PUBLIC.
   • For header files and C++ modules, scopes are sorted PUBLIC first, then INTERFACE, then PRIVATE.
4. Non-scope lists, such as those in user-defined commands that use parse_arguments, are sorted by the order their keywords appear in cmake.modifyLists.sourceListKeywords.

Variable set() and list() invocations

If cmake.modifyLists.variableSelection is never (the default), then variable command invocations will not be modified by any command, and the user will never be offered to select variable command invocations to modify.

Conversely, if that setting has any other value, any eligible variable command invocations are found in the tree above a new source file, and the user cancels instead of selecting one, no other command invocation options will be offered.

Identifying

1. CMakeLists.txt in the same directory as the new file and every parent directory up to the project root are considered.
   • If cmake.modifyLists.variableSelection is askFirstParentDir, only the first one encountered will be examined.
2. set(), list(APPEND), list(PREPEND), and list(INSERT) commands are identified.
3. Only set() invocations whose <variable> arguments, and list() invocations whose <list> arguments, match against the patterns in cmake.modifyLists.sourceVariables are considered. Arguments are compared exactly as they appear in the CMakeLists.txt. Variable substitution is not performed on their names.

Sorting

1. Invocations are grouped by the number of path components of the file they occur in, with more path components sorting earlier than shorter ones.
2. Within a file, invocations are first sorted by variable name, in the order defined in cmake.modifyLists.sourceVariables.
3. If multiple invocations modify the same variable, they are ordered with longer argument lists first. (This is measured in characters at the moment, but I could be convinced to count argument tokens instead.)
4. In the event of a tie, they are ordered by line number.

Remaining Work

• Automated test suite.
• Localization.
• Finalize documentation and add to docs/.
• Glob matching for targetSourceCommands.
• Don't run if the code model isn't valid due to cmake configure errors.
• Fallback operations modes for a null Code Model.
• Cope better with out-of-date code models due to edited or broken CMakeLists.txt.
• Offer to create new CMakeLists.txt when the first source file is created in an empty directory.
• Offer to add file sets or scopes to target_sources.

Questions

• Do we want to skip the Invocations Quick Pick and combine all source lists for all invocations into a single Quick Pick? The advantage is one fewer user interaction. The biggest disadvantage that I can see is that the Quick Pick list is a lot busier and harder to visually parse quickly if there are a lot of options.
  
• Should I be using ConfigReader instead of reading directly from a WorkSpaceConfiguration? If so, how is that meant to be used?
• The new settings for this command have an ordering among them that in my opinion makes them easiest to understand. The rest of the configuration contribution point for this extension don't provide any ordering for settings. Providing an ordering on these new options makes them appear at the top of the list, which is too prominent a place for them I think. Does it make sense to add a new category for these settings? If so, does it make sense to add categories for other groups of settings, like "cmake", "cpack", "presets", "variants", "basic", "advanced", etc.?
• This is the first time the backtraceGraph is exported from the File API, is there more I should be doing to post-process it?
• Is project.sourceDir the right thing to use for the "top-level source directory" as defined by the File API?
• Is it possible, or can it be made possible, to know whether the code model is up to date or not? I don't see an event emitted when a configure fails, but when they do the code model isn't reliable anymore.
• Command Invocation filtering could be made a little more robust if I had access to the isGenerated property from the File API's cmakeFiles object. What's the best way to make that available in the code model?

[malsyned]

@gvcallen
@andreeis
@benmcmorran
@gcampbell-msft

I would love to get any or all of your feedback on this, since you were involved in one way or another in the original discussions on #2132.

Thanks!

[malsyned]

There's a little corner case in this code. The issue is with figuring out how far to indent the new line with the new source file on it. I don't think the VSCode API gives access to enough information to actually do this perfectly every time without unpleasant UI disruptions.

Every time a new file is added to a source list in a CMake list file, it is preceded by a newline followed by an indent:

 target_sources(my_target
     PUBLIC
     include/i.h
+    include/foo.h
 
     INTERFACE
     FILE_SET other_headers

In order to do this perfectly, we would need to know exactly what vscode thinks indentSize, tabSize, and indentSpaces should be for the CMake list file. This is partially determined by global, workspace, and language settings, which can be queried through workspace.getConfiguration(). However, it can also be overridden on open files, and it can be guessed by vscode when opening a file, overriding anything in the accessible configuration.

The only way to get the per-editor overrides or guessed values with the vscode API is through a TextEditor object's options property, but in order to get a TextEditor an extension has to either find it in window.visibleTextEditors (which only contains editors that are visible on-screen) or make it visible on-screen by calling window.showTextEditor().

So I guess my question is -- how hard do we want to work to get a "correct" indentation, and what kinds of UI wonkiness are we willing to put up with in order to do it?

There's a simple way to do a pretty good job most of the time, which is to just use the indentation of the previous line. This works unless the previous line is the first line of a command invocation, in which case you probably want to add an additional indentation level.

At time of writing, this PR is using the window.showTextEditor() technique, but I am about to push a commit that switches to a less perfect, but less disruptive, guessing technique. It can be reverted if correctness in corner cases is more desired than a smooth UI experience.

[malsyned]

I did some playing with Visual Studio, to see how its functionality compares to what I've implemented in this PR. It looks like when it doesn't feel confident modifying CMakeLists.txt automatically, it will present its equivalent of a Refactor Preview to allow the user to select which modifications to apply.

I made a quick toy version of this PR that combines all possible list file edits into a single WorkspaceEdit with needsConfirmation === true for all but the highest-sorted one, to get behavior similar to Visual Studio's.

Pros

• Fewer discrete linear steps of user interaction in complicated cases.
• UI that would be familiar to a Visual Studio user.

Cons

• Loses a lot of helpful guidance and context that is provided by the QuickInput items in this PR.
• Loses all of the attempts at helpful sorting, aside from leaving the most likely candidate checked and unchecking the rest.
• Isn't modal, so it's easier to miss the fact that edits are being proposed.

Limitations

Under many circumstances, Visual Studio will initialize its edit selection dialog with every item checked. That isn't possible with the VSCode WorkspaceEdit API -- if every insert has needsConfirmation === false, they'll all be performed automatically without user interaction, which would defeat the purpose of the feature. So if this PR were to go in the same direction, it would have to leave n-1 items checked at most -- at which point, I'd rather leave just one checked, to give the user some guidance on where the most likely "best" place is to add the new file name.

Other differences

Visual Studio is also less aware in some other respects than this PR is.

• It looks like it only ever considers editing the closest parent CMakeLists.txt file.
• It doesn't try to identify different scopes or file sets in target_sources commands, it just always adds the new source file to the end.
• It doesn't even try to worry about indentation conventions, as far as I can tell

I wouldn't want to take those capabilities out of this PR, though -- I think they're unobtrusive "nice to haves".

Final thoughts

I don't really like using the Refactor Preview for this, personally. It's a situation that only comes up with more complicated CMake configurations, and in those cases I think the extra hand-holding and context that's available in my QuickInput-based implementation makes it worth deviating from a Visual Studio-style implementation.

It's an easy enough change to make to this PR, though, so if the maintainers prefer it, I can turn my toy implementation into a real one.

[Copilot]

Pull Request Overview

This PR introduces commands and underlying support for automatically appending or removing source files in CMakeLists, along with new configuration options and a custom CMake parser.

• Registers two new commands (addFileToCMakeLists, removeFileFromCMakeLists) and hooks them into the ExtensionManager
• Extends the File API data model to include backtraceGraph and sourceFileExtensions
• Adds a standalone CMakeParser for parsing command invocations and a suite of new settings in package.json

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/extension.ts	Imports and wires up the new CMakeListsModifier, registers commands, and updates code model callbacks
src/drivers/codeModel.ts	Adds backtraceGraph and sourceFileExtensions to the CodeModel types and defines shared interfaces
src/drivers/cmakeFileApi.ts	Populates backtraceGraph and sourceFileExtensions from the File API
src/cmakeParser.ts	Implements a new CMake command‐invocation parser (CMakeParser)
package.json	Adds the two new commands to contributes.commands and defines a new cmake.modifyLists settings section

Comments suppressed due to low confidence (1)

src/cmakeParser.ts:1

• The new CMakeParser has no accompanying unit tests. Consider adding tests for parsing different invocation styles, comments, and nested parentheses.

import * as vscode from "vscode";

[Copilot]

Updating the code model for the list file modifier on every invocation may introduce overhead. Consider debouncing these updates or checking for relevant changes before running.

[malsyned]

@gcampbell-msft is this a legit suggestion from Copilot? The much longer invokeAsync block just below it, "Update code model for cpptools", doesn't do any kind of debouncing.

This is a bit of the PR that I'm not totally confident in, so I would like feedback on it, but I'm not sure about Copilot's take here.

[gcampbell-msft]

@malsyned I'm not sure either, I have it on my backlog to review this PR, but in full transparency it may not be until next week or the week after when I get to it.

Thank you for your patience.

[malsyned]

No worries! This was a big PR to just drop on you out of nowhere, I appreciate you considering it. By the time I realized how big the patch was going to be, it was already way bigger than I had imagined.

It's far enough along that I don't want to take it any further without some feedback from a maintainer about whether it's headed in the right direction. It's not done yet, though, there's plenty of polish it would need before being ready for merging. Should I convert it to a draft?