UI: Chart/Plot-Dashboard API Design Refactoring

[RalphSteinhagen]

The current plotting system in the OpenDigitizer UI has become complex and less maintainable. The plotting components need to be refactored to support more specialised plots. This goal is to improve modularity, composability, and unit-testability.

Current Status

The UI design currently relies on two main components:

1. DashboardPage:

   • Manages both the dashboard layout and plot rendering.
   • Contains draw() and drawPlots() methods responsible for rendering.
   • The drawPlots() function:
     • Handles drag-and-drop for signals using ImPlot::BeginDragDropTargetPlot().
     • Calls ImPlot::BeginPlot().
     • Computes axis limits.
     • Calls ImPlotSink::draw() for each signal.
   • Issue: The drawPlots() method has become too long and complex, making the system less composable and harder to unit-test.

2. ImPlotSink<T>:

   • Stores incoming data in a local HistoryBuffer<T> (Y-values only).
   • The draw(property_map) method uses ImPlot::PlotLine() to render data.
   • Pros:
     • Consistent color scheme for all signals.
     • Handles any type T using type erasure.
     • Accepts property_map for custom styles.
   • Cons:
     • Does not handle proper X-axis (like time or frequency).
     • No synchronization between sinks in the same plot.
     • Cannot plot two Y-values against each other (e.g., correlation plots).
     • Limited data representation support.
     • Inflexibility with dynamic signal updates.

Additional Issues:

• Extending the current plotting system feels increasingly hackish and less composable.
• Difficult to extend functionality without significant code changes.
• Challenges in unit-testing due to tightly coupled components.
• Handling different typed inputs is challenging.
• Managing signals with varying sample rates is non-trivial.
• Copying data when changing charts or adding signals is cumbersome.
• Handling ImPlot's drag-and-drop targets is tricky due to semi-static port definitions.

Short-Term Needs

Implementation of more specialized plots is required to enhance visualization capabilities. Chart types to support include:

• XYChart:

  • Plots one or more Y-values against a common X-axis.
  • Signals with different units should be on separate axes.
  • X-axis could be:
    • UTC time (nanoseconds since 1970-01-01).
    • Time since a common start marker.
    • Sample index (if sample rate or timing info isn't available).

• XXChart (Correlation Plots):

  • Plots Y-values from one signal against Y-values from another.
  • Requires signal synchronization to align X-values.

• Other Chart Types:

  • Ridgeline plots, waterfall plots.
  • Custom annotated plots (e.g., accelerator geometry).
  • Specialized charts like eye diagrams, candlestick charts, etc.

Key Aspects Across All Chart Types:

• Proper axis and unit labels.
• Accurate time representation on axes.
• Correct handling of signal quantities and units to avoid mixing incompatible signals.
• Support for measurement errors (error bars, whiskers, etc.).

The following plots illustrate the minimum styles that should be replicated after the refactoring:

Further examples (to be implemented medium-term) can be seen here: https://github.com/fair-acc/chart-fx

Proposed Solution

To improve the modularity and extensibility of the plotting system, the following steps are proposed:

1. Refactor Dashboard Implementation

• Separate Concerns:

  • Create a dashboard component focused solely on layout management (if not already done).
  • Implement an abstract Chart class that handles plot/chart-internal drawing and layout.

• Abstract Chart Interface:

  • Implement a draw(const property_map&) method for custom drawing routines.
  • Manage signals using std::vector<std::shared_ptr<SignalSink>>.
  • Allow charts to be added, removed, moved or copied, enabling re-use of existing SignalSink lists.
  • Provide methods to obtain chart type names.

2. Abstract and Upgrade ImPlotSink<T>

• Develop a simplified SignalSink interface with:

  • Signal Metadata:

    • signal_name
    • signal_quantity
    • signal_unit

  • Axis Information:

    • signal_min
    • signal_max
    • sample_rate
    • time_first
    • time_last
    • sample_index

  • Additional Fields:

    • std::uint64_t lastSampleTimeStamp (in nanoseconds since 1970-01-01 UTC)
    • size_t total_sample_count (total number of samples processed)

  • History Management:

    • HistoryBuffer<T> for X, Y, and tags.
    • min_history mechanics implemented as a vector of sizes and timeouts.
    • History buffer automatically resizes based on the maximum of min_history sizes.
    • Entries removed from min_history if not updated within the specified timeout.
    • History buffer automatically shrinks/grows as needed, copying old applicable data to the new buffer.

  • Enhanced draw(const property_map&) Method:

    • Support various plot styles (lines, markers, bars, etc.).

  • Data Retrieval Methods:

    • getX(float t_min = default, float t_max = default)
    • getY(float t_min = default, float t_max = default)
    • getTags(float t_min = default, float t_max = default)
    • Consideration for returning ranges for lazy evaluation/conversion if underlying buffers require it.

3. Refactor into a Unit-Testable XYChart

• Testing:

  • Implement basic non-GR4 SignalSink test signals for unit testing.

• User Interaction:

  • Add context menus (right-click) for:
    • Adjusting signal plot styles.
    • Moving signals to new Y-axes.
    • Configuring plot settings on the fly.

• Extensibility:

  • Ensure new chart types can be added with minimal changes to existing code.

Proposed Interfaces

Below are the proposed interfaces with explicit virtual functions that need to be implemented by specific implementations. The min_history mechanics are elaborated as per the requirements.

SignalSink Interface

template <typename T>
struct SignalSink : SignalSinkBase { //TODO: split between templated and polymorphic interface
    // signal metadata
    uint32_t    signal_color = 0xff0000;
    uint32_t    signal_style; // drawing style (e.g. dashed, dotted, ..., width)
    std::string signal_name;
    std::string signal_quantity;
    std::string signal_unit;

    // axis information
    float         signal_min = std::numeric_limits<float>::lowest();
    float         signal_max = std::numeric_limits<float>::max();
    float         sample_rate = 0.0f;
    // first sample timestamp in nanoseconds since 1970-01-01 (UTC)
    std::uint64_t time_first;
    // last sample timestamp in nanoseconds since 1970-01-01 (UTC)
    std::uint64_t time_last;
    // total number of samples that have passed through the history buffer
    std::size_t   sample_index = 0;

    // history buffers for X, Y, and tags
    HistoryBuffer<T> x_values;
    HistoryBuffer<T> y_values;
    HistoryBuffer<Tag> tags;

    // minimum history requirements and timeouts
    struct MinHistoryEntry {
        std::size_t size;
        std::chrono::steady_clock::time_point last_update;
        std::chrono::milliseconds timeout;
    };
    std::vector<MinHistoryEntry> min_history;

    virtual ~SignalSink() = default;

    // drawing method to be implemented by derived classes -- handles most XY plotting scenarios 'properties' config what is being drawn
    virtual void draw(const property_map& properties) = 0;

    virtual std::span<const float> getX(float t_min = default_value, float t_max = default_value) = 0;
    virtual std::span<const float> getY(float t_min = default_value, float t_max = default_value) = 0;
    virtual std::vector<Tag> getTags(float t_min = default_value, float t_max = default_value) = 0;
    
    // alt-interface: lazy-evaluated ranges
    virtual std::ranges::view auto getX(float t_min = default_value, float t_max = default_value) = 0;
    virtual std::ranges::view auto getY(float t_min = default_value, float t_max = default_value) = 0;
    virtual std::ranges::view auto getTags(float t_min = default_value, float t_max = default_value) = 0;

    // needs to be invoked by every chart to update the timestamp -> expire old length requirements
    virtual void updateHistory(std::size_t requiredMinSize, std::chrono::milliseconds timeout = 30s) {
        // resize history buffers based on max of min_history sizes
        // remove entries from min_history if timeout has occurred
        // update x_values, y_values, and tags with new data
        // update lastSampleTimeStamp and total_sample_count
    }

protected:
    std::size_t getMinRequiredBufferSize() {
        std::size_t maxSize = 0UZ;
        for (const auto& entry : min_history) {
            if (entry.size > max_size) {
                maxSize = entry.size;
            }
        }
        return maxSize;
    }

    // Updates the min_history with a new size and timeout
    void updateMinHistory(size_t size, std::chrono::milliseconds timeout) {
        auto now = std::chrono::steady_clock::now();
        auto it = std::find_if(min_history.begin(), min_history.end(),
            [size](const MinHistoryEntry& entry) {
                return entry.size == size;
            });
        if (it != min_history.end()) {
            it->last_update = now;
            it->timeout = timeout;
        } else {
            min_history.push_back({size, now, timeout});
        }

        // remove expired entries
        auto now = std::chrono::steady_clock::now();
        min_history.erase(std::remove_if(min_history.begin(), min_history.end(),
                [now](const MinHistoryEntry& entry) {
                    return now - entry.last_update > entry.timeout;
                }), min_history.end());

        // adjust HistoryBuffer<T> based on getMinRequiredBufferSize()
    }
};

• Template on T: HistoryBuffer<T> is templated to handle different data types.
• Data Retrieval Methods: Consideration for returning ranges to allow lazy evaluation or conversion if necessary.

Chart Abstract Class

struct Chart {
    Chart() = default;
    virtual ~Chart() = default;

    // signal sinks associated with the chart
    std::vector<std::shared_ptr<SignalSinkBase>> signal_sinks;

    // drawing method to be implemented by derived classes - customised by user
    virtual void draw(const property_map& properties) = 0;

    virtual std::string_view getChartTypeName() const = 0;
    virtual std::string_view getUniqueChartTypeName() const = 0;

    // user interaction methods
    virtual void onContextMenu() = 0;

    // methods to add or remove signal sinks
    void addSignalSink(const std::shared_ptr<SignalSinkBase>& sink) {
        signal_sinks.push_back(sink);
    }

    void removeSignalSink(const std::shared_ptr<SignalSinkBase>& sink) {
        signal_sinks.erase(std::remove(signal_sinks.begin(), signal_sinks.end(), sink), signal_sinks.end());
    }

    // methods to move or copy signal sinks to another chart
    void moveSignalSinksTo(Chart& target_chart) {
        target_chart.signal_sinks = std::move(signal_sinks);
        signal_sinks.clear();
    }

    void copySignalSinksTo(Chart& target_chart) const {
        target_chart.signal_sinks = signal_sinks;
    }
};

• SignalSinkBase: An appropriate base class or interface for SignalSink<T> types.
• Chart Type Methods:
  • getChartTypeName(): Returns the name of the chart type.
  • getUniqueChartTypeName(): Returns a unique name for the chart type (if needed).

XYChart Implementation

class XYChart : public Chart {
public:
    XYChart() = default;
    virtual ~XYChart() = default;

    // Implement the draw method for XYChart
    void draw(const property_map& properties) override {
        // Synchronize X-axis based on signal metadata (e.g., timestamps)
        // Plot Y-values against the common X-axis
        // Handle units and axis labels appropriately
        // Respect min_history requirements of each SignalSink
        // Update history buffers if needed
        // Implement drawing logic using ImPlot or other plotting libraries
    }

    // Implement chart type methods
    std::string_view getChartTypeName() const override {
        return "XYChart";
    }

    std::string_view getUniqueChartTypeName() const override {
        return "XYChartUniqueIdentifier";
    }

    // Implement user interaction methods
    void onContextMenu() override {
        // provide options to adjust plot styles, axes, etc.
    }
};

Dashboard Component

class Dashboard {
public:
    Dashboard() = default;
    ~Dashboard() = default;

    // layout manager for the dashboard
    LayoutManager layout_manager;

    // collection of charts in the dashboard
    std::vector<std::unique_ptr<Chart>> charts;

    // methods to add, remove, and transmute charts
    void addChart(std::unique_ptr<Chart> chart) {
        charts.push_back(std::move(chart));
    }

    void removeChart(Chart* chart) {
        charts.erase(std::remove_if(charts.begin(), charts.end(),
            [chart](const std::unique_ptr<Chart>& c) { return c.get() == chart; }), charts.end());
    }

    void transmuteChart(Chart* old_chart, std::unique_ptr<Chart> new_chart) {
        old_chart->moveSignalSinksTo(*new_chart);
        // Replace old_chart with new_chart in the charts vector
        auto it = std::find_if(charts.begin(), charts.end(),
            [old_chart](const std::unique_ptr<Chart>& c) { return c.get() == old_chart; });
        if (it != charts.end()) {
            *it = std::move(new_chart);
        }
    }

    void copyChart(const Chart* source_chart, std::unique_ptr<Chart> new_chart) {
        // Copy signal_sinks from source_chart to new_chart
        source_chart->copySignalSinksTo(*new_chart);
        // Add new_chart to the charts vector
        charts.push_back(std::move(new_chart));
    }

    void draw() {
        for (const auto& chart : charts) {
            chart->draw(/* properties */);
        }
    }

    void handleUserInput() {
        // handle interactions like adding/removing charts or signals
    }
};

• Transmuting Charts: Allows re-using and moving an existing SignalSink list to create a new chart of a different type.
• Copying Charts: Enables re-using and copying a SignalSink list to create a new chart.