-
Owners:
- @Swiftb0y, @acolombier
-
Implementation Status:
Not implemented -
Related Issues and PRs:
- Create a reactive programming API for controllers
- QML-based components API for controllers
- Respect the Midi timestamp when scratching
- make brake, soft start, and spinback part of the effects system
- Move the controller screen rendering feature away from
ControllerScriptEngineLegacy - Confusing Midi Sysex handling
- Ability for controller to share data at runtime
- Allow Controller Mappings to be located in their own directory
- support external displays on controllers
- Handle hot-plugging and graceful recovery of controllers
- Controller scripts need to be able to cross-communicate
This document is supposed to serve as an overview for a new controller engine for Mixxx. This does not only refer to the runtime "execution engine" of some mapping specific code, but almost all aspects from protocol IO, to the metadata schema, point-and-click mapping editor, and the scripting API.
- Mapping: a file or set of files which defines everything that’s needed to a controller to communicate with mixxx
- Controller: Physical self-contained piece of hardware, possibly consisting of multiple sinks and sources
- Source: a source of data from the controller (midi/HID messages from a particular port/endpoint)
- Sink: a port/endpoint of the controller that receives data from mixxx (e.g screens, HID output report, write bulk endpoint are example of different sink types)
- Manifest: Metadata about the controller definition. Execution Engine and Source/sink topology. (equivalent to what we do with the XML in the current legacy engine)
- Execution Engine: Scripting engine that evaluates the scripting source files of the engine (so essentially what makes the scripting interactive/smart)
- Module: ES6/QML Module that's part of a mapping, evaluated in an execution engine (its what you expect ;) )
- Capability: abstract definition of a concept that is not native to mixxx (eg. shift, controlling different decks with the same physical hardware (1/3, 2/4 deck switching)). This is needed for “open-ended” cross-mapping communication (avoiding “controller lock-in”).
- Point-and-click editor: A tool to easily customize a controller mapping by clicking a button on screen and on the hardware surface and those two corresponding to each other.
The current controller engine suffers from a number of issues primarily summarized as a lack of flexibility.
To summarize the issues linked in the Related Issues and PRs section:
- The current controller engine is not able to handle multiple controllers at once. This is important when a single piece of hardware is advertising multiple different control endpoints (eg HID + Bulk for control + screens such as Native Instruments Traktor devices) as well as when multiple pieces of hardware are supposed to present a single unified control surface (such as the modular Behringer CMD-MM1/-PL1/-DC1/-LC1 or modular NI Traktor Kontrol F1/X1/Z1). Being able to bundle multiple different endpoints into the same mapping is important for a good user experience and to allow for more complex mappings.
- The current controller engine (
ControllerScriptEngineLegacy) is not able to support modules of any kind. This is important for code organization and reusability. In order to increase mapping code quality and maintainability, it is important to be able to split the mapping code into multiple files and to be able to reuse code between different mappings. This is especially important now that mapping authors are starting to reuse functionality across different hardware devices (see Numark NS6II and Mixtrack variants or similarity between different pioneer controllers) - Current mappings modify global state and assume that they are the only mapping running. This needs to be fixed in order to support multiple mappings running at the same time in the same "execution engine".
- Device hotplug is currently not possible / hard to implement. This is primarily because of the underlying protocol IO layer not willing to support it. This document proposes an alternative approach that lets us more easily switch the underlying library.
- Allow changing the manifest format. Many people have expressed distaste in the XML manifest format. Careful implementation of the new format should allow experimentation with different formats.
- Changing out infrastructure like this is virtually impossible. The proposed architecture is designed to be modular and allows for all the required components to be implemented gradually.
Goals and use cases for the solution as proposed in How:
- Allow multiple controllers to be used at the same time.
- Mappings should be able to be partitioned / modularized within their subfolder.
- Code reuse across mappings should be easier and encouraged (complex hardware of the same vendor often shares functionality).
- IO protocols should easily be changeable and not not be tied to the mapping.
- Allow easy, gradual implementation of the system.
This is primarily targeted at developers and power users who are interested in the controller engine. End-users only interested in the point-and-click mapping editor should not experience any significant changes.
- Replace
ControllerScriptEnginein the short term until we are confident in the new design. - make changes to the scratching code as it is orthogonal to this proposal.
- Make changes to the effects system as it is orthogonal to this proposal.
Explain the full overview of the proposed solution. Some guidelines:
Built-in mappings reside in res/controllers/ where each mapping is a directory
containing at least a manifest file named manifest.xxx (where .xxx is the
language-specific file extension (eg. .xml, .yml, .json, etc)).
Additionally, res/controllers/lib contains shared modules that can be used by
multiple mappings (such as componentsJS). Mapping folders may be zipped during
export (see considerations regarding shared modules) for the users convenience.
The manifest is a declarative file that describes the mapping. It contains the following information:
- Sources and Sinks
- Execution Engines
- Used Mixxx APIs
- Capabilities
In order to reduce boilerplate markup in the manifest, syntactic shorthands may be introduced to cover the most common usecases. The most common usecase may be that only one source and sink of a particular protocol coupled to a single execution engine and/or protocol dispatcher.
Sources and Sinks are the endpoints of the controller. They are defined in the manifest and are used to define the IO protocol of the controller. They are characterized by their protocol (eg. MIDI, HID, Bulk, etc) and their direction (eg. input, output, bidirectional). They will also contain a heuristic description to map the source/sink to the physical controller (protocol specific, in the case of HID, usb vid&pid could be used for example).
Execution engines is yet another section of the manifest. It defines the scripting engine that will be used to evaluate the mapping. It is defined by its type (JS, QML, something else?) along with a specific entry point. In the case of JS, this would be a ES6 module that exports a controller class to be instantiated. In the case of QML, this would be a QML file that defines a controller object. Each Execution engine should be run in its own independent thread. Data exchange between threads should be non-blocking where possible, preferably using Qt Signal/Slots or Non-blocking pipes (since those two are used in mixxx already). Scheduling requirements are handled by each engine individually (eg refreshrate for a GUI engine or latency requirements of an engine handling IO).
Capabilities are the concept that allows for cross-mapping communication while
avoiding controller-lock-in. They are defined by mixxx and are used to define
concepts that can't be easily mapped to ControlObjects. For example, a
capability could be shift or deck-switching. Capabilities are defined in the
manifest and can be used by the execution engine to communicate with other
mappings. Capabilities are essentially "opt-in" APIs.
In order to still support the point and click style use-case, we add yet another
section that directly connects patterns of midi messages to a control object.
This section is again attached to a source/sink. There is no support for
interacting with an execution engine via this section. This essentially works
like the current mapping editor, but removes the
<key>path.to.some.JS.input.handler</key> feature as it is not compatible with
the new architecture. This method is favoured over code generation as that is
not always possible, nor would it likely produce a good result. Moreover, this
would let us recycle parts of the current codebase. No need to reinvent the
wheel. Execution engines can still be wired up explicitly to access the
dispatcher here as well. This is to avoid the need to have a separate dispatcher
defined in the mapping and for easier integration with "hybrid" mappings. the
API is essentially already implemented as Registering MIDI Input Handlers From
Javascript, though QML would need
a separate declarative API.
Since this flexible layout results in a many-to-many relationship sources/sinks and execution engines, we specify a separate section in the manifest that defines how sources and sinks are connected to execution engines.
In order to share mappings that access modules outside the mappings root folder,
the execution engine must be able to create a list of all files accessed and
export them along the controller files. This would be done by creating a copy of
all the required files in the root of the mapping during exporting. That tree is
then overlaid over the built-in libraries. This is currently only possible
within a QQmlEngine using QQmlEngine::addUrlInterceptor and
QQmlEngine::addImportPath.
In order to obtain optimal developer experience of mixxx APIs in the execution
engine, each execution engine should have its own implementation that maps
idiomatically to the strengths of the execution engine. This is especially
important for QML, as it is a reaktive declarative language that does have many
more features than JS. This is essentially already the case in our current
codebase, see *JSProxy and *QMLProxy classes as examples.
We can gradually transition to the new architecture by modelling the semantics
of ControllerScriptEngineLegacy as a separate execution engine. This would
allow us to gradually migrate old mappings to the new architecture.
How feasible is the heuristic detection of sources and sinks? How do we handle multiple sources/sinks of the same type originating from different controllers? Eg how do we avoid that the screen from one piece of hardware is connected to the wrong controller?
- Shoehorn the new architecture into the existing
ControllerScriptEngineclass. This would not eliminate the one-controller-per-mapping limitation and would not allow us to iterate on the design nor on the API.
flowchart LR
%% Define front-end web components
subgraph io["IO"]
co["Control Object"]
midiIO["Midi IO"]
hidIO["HID IO"]
mixxxLib["Mixxx Library"]
mixxxUi["Mixxx UI"]
end
style io fill:transparent,stroke:green,color:#fff
subgraph layers["Intermediary Layers"]
co---midiDispatcher["MIDI Dispatcher"]
midiIO---midiDispatcher
mixxxUi---uiShift["UI Shift Capability"]
end
style layers fill:transparent,stroke:yellow,color:#fff
subgraph proxies["API Proxies"]
co---coJSProxy["COJSProxy"]
co---coQMLProxy["COQmlProxy"]
midiIO---midiJSProxy["MIDI IO JSProxy"]
midiIO---midiQMLProxy["MIDI IO QmlProxy"]
hidIO---hidJSProxy["HID IO JSProxy"]
hidIO---hidQMLProxy["HID IO QMLProxy"]
midiDispatcher---midiDispatcherJSProxy["MIDI Dispatcher JSProxy"]
mixxxLib---libQmlProxy["Library QmlProxy"]
uiShift---CapJSProxy["Capability JSProxy"]
end
style proxies fill:transparent,stroke:cyan,color:#fff
subgraph engine["Execution Engine"]
coJSProxy---jsEngine["JSExecution Engine"]
midiJSProxy---jsEngine
hidJSProxy---jsEngine
hidQMLProxy---qmlEngine
midiDispatcherJSProxy---jsEngine
CapJSProxy---jsEngine
coJSProxy---jsLegacyEngine["LegacyJS Execution Engine"]
hidJSProxy---jsLegacyEngine
hidQMLProxy---jsLegacyEngine
midiJSProxy---jsLegacyEngine
CapJSProxy---jsLegacyEngine
coQMLProxy---qmlEngine["QmlExecutionEngine"]
midiQMLProxy---qmlEngine
libQmlProxy---qmlEngine
end
style engine fill:transparent,stroke:red,color:#fff
While the manifest is supposed to be implemented language-independent, the easiest implementation would likely be based on XML, as Qt already contains the necessary parsing infrastructure. So it makes sense to use XML for mockups. Inline XML comments serve as explanation to the reader of this proposal and are not intended to be part of real manifests.
<?xml version="1.0" encoding="UTF-8"?>
<!-- lets start with the same header are our current mapping, though we distinguish from the old format
using the schemaVersion -->
<MixxxControllerPreset mixxxVersion="2.6.0+" schemaVersion="2">
<!-- the info section matches the old schema for now -->
<info>
<name></name>
<author></author>
<description></description>
<manual></manual>
</info>
<sources>
<!-- the id is used to refer to this entry by name in the remaining document-->
<midi id="controlInput"/>
</sources>
<sinks>
<midi id="controlOutput"/>
<bulk id="screenOutput"/>
</sinks>
<engines>
<Javascript id="js" entry="path-to-main-module.mjs">
<!-- the handler is a property on the exported module object that receives all data from the sink -->
<sinkRef id="controlInput" handler="incomingData"/>
<sourceRef id="controlOutput"/>
<!-- if the sources and sinks are not referenced from multiple locations, the sinkref
could be replaced directly by the corresponding entry -->
</Javascript>
<QMLUI id="js">
<sinkRef id="screenOutput">
</QMLUI>
</engines>
<dispatchers>
<!-- should this midi tag be disambiguated from the ones in other sections? -->
<midi source="controlInput" sink="controlOutput">
<!-- this would contain the <control> entries from the legacy format -->
</midi>
</dispatchers>
<settings>
<!-- same as legacy format -->
</settings>The following action plan is very abstract because the architecture is designed to be implemented gradually. Once we have decided on priorities, we can decide on a more concrete plan.
- Define the abstract Manifest outline.
- Implement a concrete manifest parser of the basic outline.
- Choose any component from the architecture summary diagram and implement it using tests and making instanstiable by via the manifest.
- Once 2 connected components are implemented, implement their connection via the manifest.
- Repeat until all components are implemented.