Produce and Publish the set of vscode built in extensions

The "builtin extensions" - What are they?

The "builtins", "built-ins" or "vscode builtin extensions" are a set of extensions that whose code resides in the public vscode repository. Some contain no code as such and contribute the means to perform static rule-based syntax highlighting (through contributing a TextMate grammar) and sometimes also some code snippets, for a particular programming language. E.g. vscode.bat, vscode.javascript. A few built-in extensions contribute advanced language support for a given language, including a Language Server, a debug adapter, command/tasks contributions and more. e.g. vscode.typescript-language-features, vscode.markdown-language-features, vscode.html-language-features, 'vscode.json-language-features'. Finally, some built-ins contribute misc potentially useful features, like various themes, integration to a 3PP services like vscode.github* and more.

Every sub-folder of vscode/extensions/ is one built-in vscode extension (with a couple of exceptions like node_modules after a build:

https://github.com/microsoft/vscode/tree/main/extensions

They are built along and packaged as a group (not individual .vsix), in the Visual Studio Code product as well as in products based on Code OSS and derivatives such as VSCodium. The built-in extensions are distributed as a part of their IDE product and are not made available separately, e.g. on the Visual Studio Marketplace.

Using the built-ins in Theia-based applications

See also: Consuming Builtin and External VS Code Extensions

In Theia-based application, there is often a need for the features provided by various built-ins. To obtain some or all of the builtin extensions, the IDE author can include all built-in extensions by adding to the app's package.json, the URL of one of our builtins extension pack - open-vsx.org, produced and published alongside the builtin extensions. It's possible to exclude unwanted extensions, referenced by the pack, by adding them to the theiaPluginsExcludeIds list. It's also possible to curate the exact list of wanted built-ins and add them individually to the application's package.json and let the user install more later if needed. In theory they could decide to bundle no built-ins at all, if that fits their purpose.

From the example Theia application, in section "theiaPlugins":

theiaPlugins": {
    "vscode.markdown-language-features": "https://open-vsx.org/api/vscode/markdown-language-features/1.64.2/file/vscode.markdown-language-features-1.64.2.vsix",
    "eclipse-theia.builtin-extension-pack": "https://open-vsx.org/api/eclipse-theia/builtin-extension-pack/1.70.2/file/eclipse-theia.builtin-extension-pack-1.70.2.vsix",
    "EditorConfig.EditorConfig": "https://open-vsx.org/api/EditorConfig/EditorConfig/0.14.4/file/EditorConfig.EditorConfig-0.14.4.vsix",
    "dbaeumer.vscode-eslint": "https://open-vsx.org/api/dbaeumer/vscode-eslint/2.1.20/file/dbaeumer.vscode-eslint-2.1.20.vsix"
  },

How to exclude certain pack extensions:

"theiaPluginsExcludeIds": [
    "ms-vscode.js-debug-companion",
    "vscode.extension-editing",
    "vscode.git",
    "vscode.git-base",
    "vscode.github",
    "vscode.github-authentication",
    "vscode.ipynb",
    "vscode.microsoft-authentication",
    "ms-vscode.references-view"
  ]

Producing

Start with the latest master state:

cd vscode-builtin-extensions
git clean -ffdx
git checkout master
git pull origin master

Create a branch for the vscode builtins you intend to build/publish:

BIVERSION="1.72.2" # for example
git branch $BIVERSION
git checkout $BIVERSION

Checkout vscode git submodule at wanted version

git submodule init
git submodule update
cd vscode
git chean -ffdx
git checkout $BIVERSION
cd ..

Build the built-ins, package them as .vsix and generate an extension pack:

yarn && yarn build:extensions 

# proceed if everything built correctly
yarn package-vsix:latest && yarn create-extension-pack:latest

The extensions, including the pack, will be under folder dist, each packaged as a .vsix file.

vscode-builtin-extensions$ ls dist/
bat-1.72.1.vsix                         html-language-features-1.72.1.vsix      references-view-1.72.1.vsix
builtin-extension-pack-1.72.1.vsix      ini-1.72.1.vsix                         restructuredtext-1.72.1.vsix
builtin-notebook-renderers-1.72.1.vsix  ipynb-1.72.1.vsix                       ruby-1.72.1.vsix
clojure-1.72.1.vsix                     jake-1.72.1.vsix                        rust-1.72.1.vsix
coffeescript-1.72.1.vsix                java-1.72.1.vsix                        scss-1.72.1.vsix
configuration-editing-1.72.1.vsix       javascript-1.72.1.vsix                  search-result-1.72.1.vsix
cpp-1.72.1.vsix                         json-1.72.1.vsix                        shaderlab-1.72.1.vsix
csharp-1.72.1.vsix                      json-language-features-1.72.1.vsix      shellscript-1.72.1.vsix
css-1.72.1.vsix                         julia-1.72.1.vsix                       simple-browser-1.72.1.vsix
css-language-features-1.72.1.vsix       latex-1.72.1.vsix                       sql-1.72.1.vsix
dart-1.72.1.vsix                        less-1.72.1.vsix                        swift-1.72.1.vsix
debug-auto-launch-1.72.1.vsix           log-1.72.1.vsix                         theme-abyss-1.72.1.vsix
debug-server-ready-1.72.1.vsix          lua-1.72.1.vsix                         theme-defaults-1.72.1.vsix
diff-1.72.1.vsix                        make-1.72.1.vsix                        theme-kimbie-dark-1.72.1.vsix
docker-1.72.1.vsix                      markdown-1.72.1.vsix                    theme-monokai-1.72.1.vsix
emmet-1.72.1.vsix                       markdown-language-features-1.72.1.vsix  theme-monokai-dimmed-1.72.1.vsix
extension-editing-1.72.1.vsix           markdown-math-1.72.1.vsix               theme-quietlight-1.72.1.vsix
fsharp-1.72.1.vsix                      media-preview-1.72.1.vsix               theme-red-1.72.1.vsix
git-1.72.1.vsix                         merge-conflict-1.72.1.vsix              theme-solarized-dark-1.72.1.vsix
git-base-1.72.1.vsix                    npm-1.72.1.vsix                         theme-solarized-light-1.72.1.vsix
github-1.72.1.vsix                      objective-c-1.72.1.vsix                 theme-tomorrow-night-blue-1.72.1.vsix
github-authentication-1.72.1.vsix       perl-1.72.1.vsix                        typescript-1.72.1.vsix
go-1.72.1.vsix                          php-1.72.1.vsix                         typescript-language-features-1.72.1.vsix
groovy-1.72.1.vsix                      php-language-features-1.72.1.vsix       vb-1.72.1.vsix
grunt-1.72.1.vsix                       powershell-1.72.1.vsix                  vscode-theme-seti-1.72.1.vsix
gulp-1.72.1.vsix                        pug-1.72.1.vsix                         xml-1.72.1.vsix
handlebars-1.72.1.vsix                  python-1.72.1.vsix                      yaml-1.72.1.vsix
hlsl-1.72.1.vsix                        r-1.72.1.vsix
html-1.72.1.vsix                        razor-1.72.1.vsix

Local testing

This section assumes you have a local clone of the main Theia repo. If required, step the supported vscode API in file dev-packages/application-package/src/api.ts. Luckyly I did nmot need to do so in this case:

/**
 * The default supported API version the framework supports.
 * The version should be in the format `x.y.z`.
 */
export const DEFAULT_SUPPORTED_API_VERSION = '1.72.2';

Build the example application:

yarn && yarn browser build

If already present, delete folder plugins. We will use the builtins we built

rm -rf plugins
mkdir plugins

Copy the builtin extensions in Theia's plugins folder.

cd vscode-builtin-extensions # back to vscode-builtin0-extensions repo
cp -a dist/* ../theia/plugins   # adjust according to where your theia folder resides

Get rid of a few that will interfere with testing (note: we keep these extensions where they were generated, but remove them from our test Theia application):

cd theia  # back to theia repo
rm -rf plugins/ipynb-*
rm -rf plugins/extension-editing-*
rm -rf plugins/github-*

To test vscode builtin git, we need to remove the Theia-specific git extension from the example application:

theia$ git diff
diff --git a/examples/browser/package.json b/examples/browser/package.json
index da42a2b6004..f5f52d7ffcd 100644
--- a/examples/browser/package.json
+++ b/examples/browser/package.json
@@ -25,7 +25,6 @@
     "@theia/file-search": "1.36.0",
     "@theia/filesystem": "1.36.0",
     "@theia/getting-started": "1.36.0",
-    "@theia/git": "1.36.0",
     "@theia/keymaps": "1.36.0",
     "@theia/markers": "1.36.0",
     "@theia/memory-inspector": "1.36.0",
diff --git a/examples/electron/package.json b/examples/electron/package.json
index 1f81d6076c6..35d1b6d927c 100644
--- a/examples/electron/package.json
+++ b/examples/electron/package.json
@@ -27,7 +27,6 @@
     "@theia/file-search": "1.36.0",
     "@theia/filesystem": "1.36.0",
     "@theia/getting-started": "1.36.0",
-    "@theia/git": "1.36.0",
     "@theia/keymaps": "1.36.0",
     "@theia/markers": "1.36.0",
     "@theia/memory-inspector": "1.36.0",

Re-build and start the example application (e.g. the browser version):

yarn && yarn browser build
yarn browser start

• Connect to localhost:3000 with a browser
• Observe backend log for new exceptions, specially during activation of builtin extensions
• quick TypeScript test
• quick JSON test
• quick X test

Once confident that the new set of builtins do not have obvious issues, you can proceed with publishing them to open-vsx.org

Publishing to openvsx.org

3PP IP / Security

3PP License Check

Eclipse Theia is an Eclipse Foundation project, we follow the Foundation's IP process regarding 3PPs (runtime dependencies as well as "build and test" dependencies, obtained from the public npm registry). See the Eclipse Foundation Project Handbook IP section for more details. We use dash-licenses to obtain (mostly) automated approval for the 3PP dependencies used to build the built-in extensions. This is done manually ATM, documented in details here

3PP Vulnerability Audit

At the moment this is only informational. This check uncovers known vulnerabilities in 3PP dependencies, in specific versions of dependencies, used by the current set of builtins. There used to be a lot when we were a couple of years late in producing a set of older built-ins. The portrait has recently improved a lot, because we have made rapid progress in supporting newer vscode extensions API versions. This means the vscode built-in extensions we produced recently are based on more recent packages, that have accumulated less vulnerability reports. When we catch-up and stay on top of new APIs as they are released, the built-in extensions we build will have close to zero vulnerabilities, in the moment when we produce and publish them, just like their twins that are part of the most recently released version of Visual Studio Code product.

See here for how to run yarn audit, for the full set of built-in extensions.