[RFC] OpenSearch Dashboards: Decoupling the rigid dependency

[AMoo-Miki]

OpenSearch Dashboards: Decoupling the rigid dependency

Background

By definition, decoupling refers to running activities separate from each other or allowing them to not develop in the same way (Cambridge Business English Dictionary). The OpenSearch Dashboards: Decoupling initiative wishes to target both of those definitions.

• Allow OpenSearch Dashboards to be the user interface to visualize data from other data storage, search, and analytics engines.
• Remove the rigid version dependency between OpenSearch, OpenSearch Dashboards, plugins, extensions and the SDK.

Situation

When OpenSearch Dashboards is run, it compares its version with those of the configured OpenSearch instance and all plugins.

OpenSearch Dashboards: Version matching

When OpenSearch Dashboards’ CLI is triggered via src/cli/dist.js with no sub-commands

• Control is passed to src/cli/cli.js where Commander is asked to run the serve sub-command.
• The serve sub-command is defined in src/cli/serve/serve.js and its action is instructed to call bootstrap.
• The bootstrap method, defined in src/core/server/bootstrap.ts, then sets up and starts the top-level entry point for the server.
• The setup method of src/core/server/root/index.ts calls setup on an instance of Server.

With Engine

• The setup method of src/core/server/server.ts calls setup on an instance of OpenSearchService.
• The setup method of src/core/server/opensearch/opensearch_service.ts makes a call pollOpenSearchNodesVersion.
• pollOpenSearchNodesVersion of src/core/server/opensearch/version_check/ensure_opensearch_version.ts gets the version of the configured OpenSearch engine and passes it along with the version of OpenSearch Dashboards to mapNodesVersionCompatibility: so they can be compared by opensearchVersionCompatibleWithOpenSearchDashboards.
• opensearchVersionCompatibleWithOpenSearchDashboards of src/core/server/opensearch/version_check/opensearch_opensearch_dashboards_version_compatability.ts considers any OpenSearch Engine and OpenSearch Dashboards incompatible:
  • if they are not within the same major version, or
  • when the major versions match, if Engine has a lower minor version than Dashboard.
• The result is passed back to the setup of OpenSearchService to calculateStatus$ and set the status level to ServiceStatusLevels.critical.

With Plugins

• The setup method of src/core/server/server.ts calls discover on an instance of PluginsService.
• The discover method of src/core/server/plugins/plugins_service.ts makes a call to an imported discover method.
• The discover method of src/core/server/plugins/discovery/plugins_discovery.ts locates plugins and calls createPlugin$ for each one.
• createPlugin$ then calls parseManifest on the plugin’s manifest file.
  • A plugin manifest file is always named opensearch_dashboards.json at the root of a plugin and contains information such as its id, version, and ... .
• parseManifest of src/core/server/plugins/discovery/plugin_manifest_parser.ts reads in opensearchDashboardsVersion , or in its absence version, from the manifest file and compares it to the version of OpenSearch Dashboards using isVersionCompatible.
• isVersionCompatible normalizes the versions to drop any possible suffixes and evaluates them as incompatible:
  • if the versions are not identical
• The result is passed back to the parseManifest of src/core/server/plugins/discovery/plugin_manifest_parser.ts to throw an exception if needed.

• If OpenSearch and OpenSearch Dashboards are not within the same major version, Dashboards is marked Red.
• If OpenSearch and OpenSearch Dashboards are within the same major version but OpenSearch has a minor version less than that of OpenSearch Dashboards, OpenSearch Dashboards is marked Red.
• If a plugin’s manifest has a value for opensearchDashboardsVersion and it is not identical to the version of Dashboards, an exception is thrown and the process exits.
• If a plugin’s manifest does not have a value for opensearchDashboardsVersion and the plugin’s version it is not identical to the version of Dashboards, an exception is thrown and the process exits.

Problems

The self-imposed rules are based on the idea of minimizing the compatibility matrix to reduce the amount of surface to test and maintain. This not taking advantage of semantic versioning has complicated development and releases, and is potentially an inhibitor to rapid adoption of Dashboards and plugins.

1. This compatibility check logic does not match the promises we have made by adopting semantic versioning:

   • OpenSearch Dashboards 1.3.5 and OpenSearch 1.3.5 ✅
   • OpenSearch Dashboards 1.3.5 and OpenSearch 1.3.0 ✅
   • OpenSearch Dashboards 1.3.0 and OpenSearch 1.3.5 ✅
   • OpenSearch Dashboards 1.3.0 and OpenSearch 1.2.5 ❌
   • OpenSearch Dashboards 1.3.5 and Index Management plugin 1.3.5 ✅
   • OpenSearch Dashboards 1.3.5 and Index management plugin 1.3.4 ❌

2. As OpenSearch, releasing a new version of OpenSearch and OpenSearch Dashboards requires updating the manifests for all plugins and consequently bumping their versions, even if they have no other changes. In fact, the release automation blindly overwrites the opensearchDashboardsVersion value in the manifest file of the plugins (ref).

3. For a contributor, it is an annoyance to have to edit the manifests of plugins to make them employable for testing during development of OpenSearch Dashboards and other plugins. If a contributor rebases the upstream main branch of Dashboards on to their development branch and it includes a patch version bump, they are forced to update all the plugins before bootstrapping. This is substantially more irritating for someone maintaining a staging instance since they need to individually build all the plugins and install them again, even those that had no code changes.

4. An administrator of an OpenSearch instance might put off upgrading their instances of OpenSearch due their delicate data but OpenSearch Dashboards, being just a UI, could be updated without an impact on their OpenSearch; while on OpenSearch 2.2.0, one might want to allow their users to enjoy the new Wizard functionality which was stabilized in Dashboards 2.3.0. This potential upgrade path is blocked by the current compatibility checks.

5. OpenSearch, OpenSearch Dashboards, and plugin repositories bump their versions as soon as their development cycles start after a release. Since we follow the train model, if a repository is not ready for a release, a previous release is pulled into the release pipeline and its version is bumped. Even though a tag is cut, the git history is contaminated with conflicting histories where the same version is used for completely different codebases. While the conflicting histories isn’t a big detractor of our reputation, this process is an unwelcome complication for our release process.

Decoupling OpenSearch Dashboards and OpenSearch

Various components of OpenSearch Dashboards interact with OpenSearch using one of three unrelated methods: the legacy client, the OpenSearch client, and direct HTTP calls.

• [email protected] (npm)
  • via core/server/opensearch/legacy
  • via plugins/data_source/server/legacy
  • directly by @osd/opensearch-archiver
  • to provide type definitions in
    • plugins/data/server/search
    • plugins/data/public/search
    • plugins/data/common/search
    • plugins/data_source/server/client
    • plugins/discover/public/application
    • plugins/vis_type_vega/server
    • plugins/visualizations/server
  • some tests
• @opensearch-project/opensearch (npm)
  • via core/server/opensearch/client
  • via plugins/data_source/server/client
  • to provide type definitions in
    • plugins/data/server/search
    • plugins/data/common/search
• OsdClientRequester (github) making HTTP calls
  • via @osd/dev-utils

Proposed Solution

Make OpenSearch Dashboards rely solely on the Node.js client, @opensearch-project/opensearch, and delegate the responsibility of interacting with OpenSearch to it. The Node.js client is a low-level library that could be compatible with more that one major version of OpenSearch and any variations would be normalized by a wrapper (introduced below). OpenSearch Dashboards will be instructed to use the latest compatible major version of the Node.js client available. This involves replacing the components that consume the legacy elasticsearch client and the direct HTTP channel with method that call the official Node.js client.

Addendum

The legacy elasticsearch@^16.7.0 has been used by all releases of OpenSearch Dashboards so far. Since OpenSearch Dashboards never depended on the previously deprecated and now removed type mapping, its can use @opensearch-project/opensearch@^2 to communicate with any of the existing releases of OpenSearch and since both, the engine and the client, follow semantic versioning, if there is a breaking change that @opensearch-project/opensearch needs to adapt to in the future, it will not cause a breaking change for OpenSearch Dashboards v2. In fact, the Node.js client, as of v2.1.0, only checks if the distribution name is “opensearch” and does not care for the version (ref). As such, theoretically, any version of OpenSearch Dashboards can function against any version of OpenSearch.

Decoupling OpenSearch Dashboards and Plugins

OpenSearch Dashboards plugins directly import public objects exposed by Dashboards’ source code to build and function. While the Extensions SDK promises to eliminate a direct dependency on OpenSearch Dashboards, some of these plugins might choose not to switch to extensions due their need for certain privileges or APIs not exposed by the extensions SDK. Since the majority of the plugins import directly from source files of OpenSearch Dashboards, no promises can be made that they will not be impacted adversely by changes in OpenSearch Dashboards’ code.

Proposed Solution

If OpenSearch Dashboards would want to continue to support external core plugins, a mechanism would be created to expose the type definitions of all public objects of OpenSearch Dashboards that the developers would like to make accessible. During development of plugins, the consumption of any objects not deliberately exposed by OpenSearch Dashboards would be prevented by linting rules. At runtime and during their initialization, plugins will access the type definitions exposed by OpenSearch Dashboards to verify that they are compatible with that version of OpenSearch Dashboards and would silently fail if they are not; this will allow for OpenSearch Dashboards to gracefully degrade.

Decoupling the Node.js Client and OpenSearch

The Node.js client, @opensearch-project/opensearch (npm), is a low-level communication facilitator between applications running with JavaScript and the OpenSearch. The current version (v2.1.0) of the Node.js client does not discriminate based on the version of OpenSearch and as long as the engine advertises itself as “opensearch,” it is deemed compatible; in other words, it is pretty much decoupled already.

Proposed Solution

The Node.js client would continue to be the low-level client that it is and will continue to expose all OpenSearch endpoints. In addition, a mechanism would be built to expose the non-core capabilities of the OpenSearch that are made available via plugins and extensions. As a result, as long as the OpenSearch’s APIs or the client dependencies don’t introduce breaking changes, there would be no need for a major version bump.

Decoupling OpenSearch Dashboards and the Node.js Client

At the time of building or installing an instance of OpenSearch Dashboards, the package manager, Yarn, pulls in the highest version of the Node.js client that satisfies the requirements of OpenSearch Dashboards; this is trusting that the Node.js client is following semantic versioning. OpenSearch Dashboards would expect the basic index creation/removal and document retrieval APIs never change within a major version of the client.

Proposed Solution

With an eye on making OpenSearch Dashboards an independent product, a new wrapper would be created to take on the job of interacting with the low-level OpenSearch Node.js client. The wrapper would be a high-level abstraction layer that would service the needs of OpenSearch Dashboards with normalized responses. The wrapper will recognize the features made available to it by the OpenSearch instance, in order to allow OpenSearch Dashboards to gracefully degrade. The wrapper will also be responsible for adjusting its communication recipes for variations of a feature across versions of OpenSearch.

When OpenSearch Dashboard starts up, it would instruct the wrapper to establish a connection with OpenSearch via the client. The wrapper would make a few requests to ascertain that it is indeed capable of understanding what it gets back from the OpenSearch instance and would then attempt to execute a few more requests to determine the variations in the APIs and feature unavailability.

This design would allow wrappers to be created for engines other than OpenSearch to facilitate visualization and analysis of a wider collection of data sources using OpenSearch Dashboards (image below).

Decoupling OpenSearch Dashboards and the Extension SDKs

The Extensions SDK could possibly have internal mechanisms to normalize calls against older versions of OpenSearch Dashboards APIs known to the SDK. However, this will not enable interactions between an extension and the APIs exposed by a later version of OpenSearch Dashboards which increases the burden on extensions developers to keep up with newer releases of the Extensions SDK.

Proposed Solution

At runtime and during the initialization of an extension, if the Extensions SDK it was built with does not recognize the version of OpenSearch Dashboards, it would access the type definitions exposed by OpenSearch Dashboards to verify that the APIs consumed by the extensions match those the Extensions SDK knows how to handle; an initialization error would be triggered if the Extensions SDK does not know how to communicate with any of the APIs required by the extension. The Extensions SDK might also choose to facilitate optional consumption of APIs by the extension to allow the extension to detect missing functionality and degrade gracefully.

Discussion

Version Detection

A technique frequently employed for managing dependencies is versioning and the same can technically be employed for detecting the compatibility of APIs exposed by OpenSearch Dashboards and data storage/search engines. When breaking changes are introduced in any API, a major version bump would have to be introduced, marking everything as incompatible. Versioning APIs will only help reduce the frequency of version bumps for plugins, SDK, and extensions; it will not help decouple them from OpenSearch Dashboards.

let’s consider an example where OpenSearch Dashboards provides APIs A, B, and C at v1 and Plugin Y uses APIs A and B at v1. If API C is deprecated or if its signature changes enough to make it not backwards-compatible, OpenSearch Dashboards will bump the APIs to v2, making Plugin Y incompatible even though the APIs that it used never changed. If the Extensions SDK relies on versioned APIs, an Extension X, also only consuming APIs A and B, and all other extensions will have to be recompiled with a new version of the SDK to continue to function on the newer OpenSearch Dashboards release. Instead, if the SDK depends on detecting the capabilities of OpenSearch Dashboards and the signature of its APIs, Extension X would continue to work as expected even though it was compiled with an older SDK.

Feature Detection

The main component of decoupling OpenSearch Dashboards and data storage/search engines is the introduction of the wrapper that serves both the targets of decoupling by allowing OpenSearch Dashboards to function independent of a specific data storage/search engine (where each engine will get its own wrapper) and to not be locked into a specific version of an engine, by the wrapper taking on the job of modifying calls and normalizing responses. The key to both of these is for the wrapper to recognize the capabilities and limitations of the specific version of the engine it is communicating with. If the OSD-OS-wrapper relied solely on detecting the version of the engine, older releases of OpenSearch Dashboards will not be compatible with newer releases of OpenSearch simply because the wrapper wouldn’t know how to handle the newer version of OpenSearch. Relying on feature detection, the older OpenSearch Dashboards will continue to function until the wrapper is unable to effectively communicate or understand the engine, or when OpenSearch Dashboards is told that an API it relies on is not normalizable by the current wrapper.

While we could simply enrich the current low-level OpenSearch Node.js client with the OpenSearch Dashboards wrapper, that choice could limit the involvement of the open-source community in creating other OpenSearch Dashboards wrappers using existing clients of other data storage/search engines.

Decoupling Shared Dependencies

OpenSearch Dashboards and plugin/extensions could have common dependencies. For core plugins, during bootstrapping and build, a mechanism is enforced to have all shared dependencies be employed with a common version; this undesirable enforcement is not sustainable with decoupling. For dependencies that are imported and used by Node.js runtime, NPM and Yarn don’t hoist the different versions of shared dependencies and instead place them in the node_modules folder of the component. However, for those dependencies that are bundled by webpack and sent to the browser, since their inclusion is only by name, conflicts are inevitable. One technique to avoid this is to use NPM/Yarn aliases for the dependencies of each extension while building them. This could be further enhanced by merging dependencies that have matching or compatible signatures. The negative impact of being able to enforce common dependencies is that the bundle payload size increases.

Design

[dbwiddis]

The Node.js client would continue to be the low-level client that it is and will continue to expose all OpenSearch endpoints. In addition, a mechanism would be built to expose the non-core capabilities of the OpenSearch that are made available via plugins and extensions.

I really like this approach, not just for Dashboards, but as a standard that Extensions should adhere to for compatibility with languages beyond Java / js.

[dblock]

My biggest question is why do we need a separate wrapper? That feels like another way to extend Dashboards. I'd want connectors to data sources to be implemented on top of the SDK instead, and I would prefer that the SDK be the only way to extend any functionality of dashboards, including storage engines.

• Decoupling OpenSearch Dashboards and OpenSearch: +1 on using the opensearch-js client and relying on semver.
• Decoupling OpenSearch Dashboards and Plugins: I would not advertise public interfaces at all, and rely on the SDK that would act as the middle with support for multiple OpenSearch Dashboards versions (this it would be the only supported consumer of interfaces marked public in OpenSearch Dashboards, deprecating support for plugins).
• Decoupling OpenSearch Dashboards and the Node.js Client: why is this a special case? can't connectors to other data sources be extensions? So, instead of a wrapper I think you want to build extension points, expose those in the SDK, and author connectors on top of the SDK that are themselves extensions that work within OpenSearch Dashboards.
• Decoupling OpenSearch Dashboards and the Extension SDKs: makes sense, however maybe the implementation of extensions SDK can cleanly implement support for multiple versions of OpenSearch Dashboards by carrying complete separate (and potentially duplicate) implementations? Basically for every new major version of OpenSearch the tasks of adding support to the extensions SDK would be to 1) clone existing code into v3/..., 2) make whatever changes you need. Version selection of the implementation could happen at runtime.
• Version Detection: instead of complicated version negotiation and signature verifications I would 1) add support for Dashboards v.next in the SDK (by doing what I described above), then rebuild the extension with that SDK. Major version bums happen very, very rarely (once every couple of years), so this is not a big deal.
• Feature Detection: This shouldn't be a wrapper, but the SDK, otherwise makes sense.
• Decoupling Shared Dependencies: makes sense.

[joshuarrrr]

What about plugins that consist of both an Opensearch plugin and an OpenSearch Dashboards plugin? Is there a way for OpenSearch plugins to expose APIs that the node client can discover and make available to both OpenSearch Dashboards and OpenSearch Dashboard plugins?

In the discussion section - are you saying that version detection is insufficient, and that we need feature detection instead? And is feature detection and capability detection synonymous in this doc?

[AMoo-Miki]

Thanks for sharing your thoughts.

My biggest question is why do we need a separate wrapper? That feels like another way to extend Dashboards. I'd want connectors to data sources to be implemented on top of the SDK instead, and I would prefer that the SDK be the only way to extend any functionality of dashboards, including storage engines.

Decoupling OpenSearch Dashboards and the Node.js Client: why is this a special case? can't connectors to other data sources be extensions? So, instead of a wrapper I think you want to build extension points, expose those in the SDK, and author connectors on top of the SDK that are themselves extensions that work within OpenSearch Dashboards.

The wrappers (or maybe a better term would be "connectors") were envisioned to do normalization of requests and responses which is similar to one of the many things the Extensions SDK will be doing; during implementation, we might end up deciding they are way too similar to have different implementations/packaging/documentation.

The main reason for separating them was that the Extensions SDK might choose not to provide all the functionality that a wrapper/connector needs to facilitate the data fetching; this choice would be driven by privileges the Extensions SDK should not expose, self-imposed ease-of-use boundaries, or ease of maintenance of the Extensions SDK. For example, the Extensions SDK may decide that they need to prohibit the extensions from directly consuming the low-level client and force them to use the data retrieval APIs expose by the SDK itself.

It is only logical for the Dashboards developers to build the very first wrapper/connector for interacting with OpenSearch and it is quite possible that they will end up building the first version as an extension, using the Extensions SDK, if no blockers are found.

Decoupling OpenSearch Dashboards and Plugins: I would not advertise public interfaces at all, and rely on the SDK that would act as the middle with support for multiple OpenSearch Dashboards versions (this it would be the only supported consumer of interfaces marked public in OpenSearch Dashboards, deprecating support for plugins).

While a decision hasn't been made to block discovery of plugins, we certainly will not be advertising them as a public interface; we might even choose to stop facilitating their installation but not only no such decision has been made, but also being JS, an administrator can effortlessly work around lack of facilitating scripts.

Decoupling OpenSearch Dashboards and the Extension SDKs: makes sense, however maybe the implementation of extensions SDK can cleanly implement support for multiple versions of OpenSearch Dashboards by carrying complete separate (and potentially duplicate) implementations? Basically for every new major version of OpenSearch the tasks of adding support to the extensions SDK would be to 1) clone existing code into v3/..., 2) make whatever changes you need. Version selection of the implementation could happen at runtime.

I trust the Extension SDK will have a less involved and more self-adapting mechanism for dealing with changes. That said, there certainly will be instances where adaption is not possible and nothing but a brand new set of instruction will be needed for a specific version onward of OpenSearch or OpenSearch Dashboards.

Version Detection: instead of complicated version negotiation and signature verifications I would 1) add support for Dashboards v.next in the SDK (by doing what I described above), then rebuild the extension with that SDK. Major version bums happen very, very rarely (once every couple of years), so this is not a big deal.

Feature Detection: This shouldn't be a wrapper, but the SDK, otherwise makes sense.

While during implementation we might find a smarter way, the signature of the APIs will really be treated as versions; an optional parameter added would be a minor difference and a parameter added/moved/removed is a major. Automating signature extraction is an uncomplicated task due to the code being in TypeScript. An alternate would have been to version each individual API manually which would be inferior when compared to the automatic signature extraction.

Feature Detection is not specifically built into or as a wrapper; it will be a holistic approach throughout the Dashboards ecosystem. While it is true that major versions will be released far apart, when thinking of OpenSearch or OpenSearch Dashboards, but the compatibility check plays a role in inter-extension communication as well, and they would have independent release cycles.

[AMoo-Miki]

What about plugins that consist of both an Opensearch plugin and an OpenSearch Dashboards plugin?

As an OpenSearch extension developer, I would love to be able to feed the paired Dashboards with parts of my extension that I would like them to expose. A simple implementation would be for OpenSearch extensions to have OpenSearch ask for Dashboards to activate certain extensions during the handshake. That way, extensions developers would build two extensions: one for OpenSearch and one for Dashboards, but force Dashboards to install the paired extension, if it isn't already installed.

We could discuss this more if the community has an appetite for it.

Is there a way for OpenSearch plugins to expose APIs that the node client can discover and make available to both OpenSearch Dashboards and OpenSearch Dashboard plugins?

Called out in Decoupling the Node.js Client and OpenSearch, I propose that the low-level client should discover and expose APIs made available by extensions. A less-intelligent approach would be for the low-level client to blindly translate the invocation name of a method to an OpenSearch API call. We might actually start without the discovery and add it in a future version of the low-level client.

In the discussion section - are you saying that version detection is insufficient, and that we need feature detection instead?

I am calling out that version detection can solve the problem with a guarantee that compatibility will break with the next major release of OpenSearch, Dashboards, SDK, or even an extension, but feature detection has the potential to allow for continued compatibility across major releases of any of them.

And is feature detection and capability detection synonymous in this doc?

I should have been clearer in my choice of words:

• feature detection will check that an API exists and has a known signature,
• capability detection might use feature detection or a version-based matrix to only check if an API exists.

PS, to avoid confusion, I will change the example that references detecting capabilities.

[dblock]

Thanks @AMoo-Miki. As famous last words go, I'd say that if we build connectors on top of the extensions SDK we will be improving the extensions SDK along the way of one of the most technically demanding features. That would give me more confidence in the success of the extensions SDK and our approach.

[wbeckler]

Is it possible to separate the node version issue from the core<->dashboard coupling issue?

[AMoo-Miki]

Is it possible to separate the node version issue from the core<->dashboard coupling issue?

This RFC has run its course. Next step is to create individual issues for the decoupling items that haven't been already relaxed. I will close this as soon as individual issues are created.

[Haitham-Shalaby]

Is it possible to separate the node version issue from the core<->dashboard coupling issue?

This RFC has run its course. Next step is to create individual issues for the decoupling items that haven't been already relaxed. I will close this as soon as individual issues are created.

is it possible to know what issues will be created and if there is a planned date for these issues to be done