Merge #804: Improve debug log performance

fb87a8b qml: render debug log entries with a virtualized list (johnny9)
de76d84 qml: read debug log updates without reloading the file (johnny9)
d813155 qml: read debug log only while its page is selected (johnny9)

Pull request description:

  ## Summary

  - Load and watch the debug log only while its settings page is selected.
  - Read the bounded log tail on a worker thread and process ordinary appends incrementally instead of resetting every row.
  - Limit individual physical log lines in the viewer to 1 MiB; oversized lines are skipped until their terminating newline without modifying `debug.log`.
  - Initially load the newest 1,000 entries and expose older entries in 1,000-row increments through **Load more**.
  - Replace the fully instantiated output with a virtualized `ListView`.
  - Preserve the visible row and pixel offset across live prepends, tail pruning, and load-more appends.

  ## Motivation

  The previous debug-log view instantiated every displayed row and rebuilt the full output whenever the model reset. A new log write could therefore trigger another full file read, model reset, and delegate rebuild, including while the debug-log page was not selected.

  This change treats the model and view as one performance path: the model reads only the new bytes and publishes changed rows during normal appends, while the view instantiates only visible delegates and keeps its scroll anchor stable.

  Follow-up to the performance problem described in #784 and the original viewer in #532. Contributes to #576.

  Related: #768. Its targeted relative-time role update behavior is already present on `qt6` and remains intact after this refactor; this PR does not close that issue.

  ## Testing

  - Built every commit in the branch independently.
  - Built the final application with the matching non-OpenGL depends toolchain.
  - Ran Bitcoin commit-message and whitespace checks on every commit.
  - Ran Ruff 0.15 and `qmllint` on the changed Python and QML files.
  - Ran the complete C++ unit suite: 540 passed, including `DebugLogModelTests`: 19 passed.
  - Ran the complete QML suite: 391 passed; the remaining `Send::test_send_uri_import_schedules_fee_estimate` failure (`BitcoinUri is not defined`) reproduces on the PR base.
  - Ran `test/functional/qml_test_debug_log.py` against the real application.
  - Verified watcher-driven updates remain capped, offscreen rows are virtualized, and **Load more** adds older rows without moving the viewport.

ACKs for top commit:
  pseudoramdom:
    ACK fb87a8b

Tree-SHA512: 20162c60c882b41935ab6b45cef5a30e881fbad8b33175674c2cad555bcb17ea2293a3c0667fb555084300596853b61ca140cd59e28a8f1630e624662fb9daa7

Author: hebasto

qml/components/DebugLogOutputView.qml

@@ -8,6 +8,7 @@
 #include <util/fs.h>
 
 #include <QAbstractListModel>
+#include <QByteArray>
 #include <QFileSystemWatcher>
 #include <QList>
 #include <QString>
@@ -33,15 +34,16 @@ class QThread;
 //! Pagination: only the most recent `loadLimit` lines are kept in memory.
 //! Call loadMore() to increase the limit by 1000, up to kMaxLoadLimit.
 //!
-//! File watching: the model connects a QFileSystemWatcher to the log file and
-//! coalesces rapid writes with a 500 ms debounce timer before calling refresh().
-//! Reads themselves run on a dedicated worker thread so a busy, noisy node cannot
+//! File watching: the model only watches the log while active and coalesces
+//! rapid writes with a 500 ms debounce timer before calling refresh(). Reads
+//! themselves run on a dedicated worker thread so a busy, noisy node cannot
 //! stall the UI on every debug-log burst.
 class DebugLogModel : public QAbstractListModel
 {
     Q_OBJECT
 
     Q_PROPERTY(bool hasMoreLines READ hasMoreLines NOTIFY hasMoreLinesChanged)
+    Q_PROPERTY(bool active READ active WRITE setActive NOTIFY activeChanged)
     Q_PROPERTY(int  loadLimit   READ loadLimit    WRITE setLoadLimit   NOTIFY loadLimitChanged)
     Q_PROPERTY(QString filter   READ filter       WRITE setFilter      NOTIFY filterChanged)
     Q_PROPERTY(QString openError READ openError   NOTIFY openErrorChanged)
@@ -68,6 +70,10 @@ class DebugLogModel : public QAbstractListModel
     //! Hard ceiling on loadLimit to protect against unbounded memory growth.
     static constexpr int kMaxLoadLimit = 50'000;
 
+    //! Individual physical log lines larger than this are omitted from the
+    //! in-app viewer. The debug.log file itself is never modified.
+    static constexpr qsizetype kMaxLogLineBytes = 1024 * 1024;
+
     explicit DebugLogModel(const fs::path& log_path, QObject* parent = nullptr);
     ~DebugLogModel() override;
 
@@ -78,6 +84,9 @@ class DebugLogModel : public QAbstractListModel
 
     bool hasMoreLines() const { return m_has_more_lines; }
 
+    bool active() const { return m_active; }
+    void setActive(bool active);
+
     int loadLimit() const { return m_load_limit; }
     void setLoadLimit(int limit);
 
@@ -94,6 +103,7 @@ class DebugLogModel : public QAbstractListModel
 
 Q_SIGNALS:
     void hasMoreLinesChanged();
+    void activeChanged();
     void loadLimitChanged();
     void filterChanged();
     void openErrorChanged();
@@ -102,25 +112,23 @@ class DebugLogModel : public QAbstractListModel
 
 private:
     struct LogLine {
-        QString lineNumber;
         QString content;      // HTML-escaped full message text
         QString command;      // parsed message prefix, plain text
         QString message;      // parsed message body, plain text
-        QString identity;     // stable identity for incremental refresh matching
+        qint64  source_offset{-1}; // byte offset in debug.log (stable across appends)
         qint64  timestamp_ms; // epoch ms, -1 if not parseable
         QString relativeTime; // cached human-readable age
         Severity severity{InfoSeverity};
 
-        // Identity for change detection in buildDisplayLines(). relativeTime is
-        // derived (refreshed separately by the relative-time timer) and so is
-        // deliberately excluded.
+        // Identity for incremental display diffs. relativeTime is derived
+        // (refreshed separately by the relative-time timer) and deliberately
+        // excluded.
         bool operator==(const LogLine& o) const
         {
-            return lineNumber == o.lineNumber
+            return source_offset == o.source_offset
                 && content == o.content
                 && command == o.command
                 && message == o.message
-                && identity == o.identity
                 && severity == o.severity
                 && timestamp_ms == o.timestamp_ms;
         }
@@ -132,29 +140,55 @@ class DebugLogModel : public QAbstractListModel
     struct ReadResult {
         bool file_opened{false};
         QString error_message;
-        QList<LogLine> filtered; // oldest-first, already filtered of blank lines
+        //! A full snapshot is newest-first and contains at most loadLimit
+        //! entries. A delta contains only newly completed lines, newest-first.
+        QList<LogLine> lines;
+        bool full_snapshot{true};
+        bool continuity_lost{false};
+        bool has_more_lines{false};
+        int snapshot_limit{0};
+        qint64 file_size{-1};
+        QByteArray trailing_partial;
+        //! True after an unfinished line exceeds kMaxLogLineBytes. Subsequent
+        //! bytes are ignored until its terminating newline restores framing.
+        bool discarding_oversized_line{false};
+        QByteArray file_anchor;
     };
 
     //! File-reading worker. Pure function — no QObject / signal access —
     //! so it can safely run on the dedicated worker thread.
     static ReadResult ReadAndFilter(const fs::path& log_path,
                                     int load_limit,
                                     bool full_load,
+                                    qint64 previous_file_size,
+                                    const QByteArray& previous_partial,
+                                    bool previous_discarding_oversized_line,
+                                    const QByteArray& previous_anchor,
                                     const std::atomic_bool& cancelled);
 
-    //! Raw file read (one pass). Called from ReadAndFilter.
-    static QList<LogLine> ReadRawLines(const fs::path& log_path,
-                                       int max_lines,
-                                       const std::atomic_bool& cancelled);
+    //! Read a bounded tail snapshot by scanning backward in fixed-size blocks.
+    static ReadResult ReadTail(const fs::path& log_path,
+                               int load_limit,
+                               const std::atomic_bool& cancelled);
+
+    //! Parse complete newline-terminated records, returning newest first.
+    static QList<LogLine> ParseCompleteLines(const QByteArray& bytes,
+                                             qint64 base_offset,
+                                             int max_filtered_lines,
+                                             const std::atomic_bool& cancelled);
 
     //! Completion handler invoked on the GUI thread after ReadAndFilter
     //! returns. Applies the result to m_all_lines and rebuilds the display.
     void onReadCompleted(const ReadResult& result,
-                         const QString& prev_top_identity,
                          bool full_load);
 
     void connectFileWatcher();
-    void buildDisplayLines();
+    void watchLogPath();
+    void buildDisplayLines(bool force_reset = false);
+    void applyLines(QList<LogLine> lines, bool force_reset);
+    bool applyDelta(QList<LogLine> lines);
+    void applyDisplayLines(QList<LogLine> lines, bool force_reset);
+    QList<LogLine> filteredLines(const QList<LogLine>& lines) const;
     QString relativeTimeLabel(qint64 timestamp_ms, qint64 now_ms) const;
 
     static void PopulateParsedFields(LogLine& entry, const QString& raw_message);
@@ -171,8 +205,20 @@ class DebugLogModel : public QAbstractListModel
     QString m_filter;
     int  m_load_limit{1000};
     bool m_has_more_lines{false};
+    //! Tail capacity represented by m_all_lines. Kept separate from rowCount
+    //! so empty/short logs and interrupted loadMore requests are unambiguous.
+    int m_loaded_limit{0};
     QString m_open_error;
 
+    //! End-of-file state from the last successful worker read. Normal
+    //! refreshes validate the anchor, seek to file_size, and parse only bytes
+    //! appended since then. A bounded partial final line is carried across
+    //! reads; an oversized one is discarded through its terminating newline.
+    qint64 m_file_size{-1};
+    QByteArray m_trailing_partial;
+    bool m_discarding_oversized_line{false};
+    QByteArray m_file_anchor;
+
     QFileSystemWatcher m_watcher;
     QTimer m_debounce;
     QObject* m_reader{nullptr};
@@ -183,7 +229,9 @@ class DebugLogModel : public QAbstractListModel
     bool m_read_in_flight{false};
     bool m_refresh_pending{false};
     bool m_pending_full_load{false};
+    bool m_active{false};
     bool m_stopping{false};
+    quint64 m_activation_generation{0};
     std::atomic_bool m_read_cancelled{false};
 };
 

qml/models/debuglogmodel.cpp

@@ -280,7 +280,12 @@ Page {
                         sourceComponent: NetworkTraffic { showBackButton: false; showHeader: false }
                     }
                     MempoolInformationSettings { showBackButton: false }
-                    SettingsDebugLog { showBackButton: false }
+                    Loader {
+                        id: debugLogLoader
+                        objectName: "settingsDebugLogLoader"
+                        active: root.visible && root.currentSection === 8
+                        sourceComponent: SettingsDebugLog { showBackButton: false }
+                    }
                     PageStack {
                         id: aboutStack
                         initialItem: SettingsAbout {

qml/models/debuglogmodel.h

@@ -132,6 +132,11 @@ Page {
                 color: Theme.color.neutral9
                 placeholderTextColor: Theme.color.neutral5
                 placeholderText: qsTr("Search...")
+                // The page is unloaded whenever another Settings section is
+                // selected, while the C++ model intentionally retains its
+                // filter. Mirror that retained value on re-entry so the field
+                // and the rows cannot disagree.
+                text: debugLogModel.filter
                 verticalAlignment: TextInput.AlignVCenter
                 selectByMouse: true
                 Accessible.name: qsTr("Search debug log")
@@ -201,11 +206,13 @@ Page {
             topPadding: 10
             accessibleName: qsTr("Debug log entries")
             autoScrollToBottom: false
-            // DebugLogModel renders newest-first at the top, so y > 0 means
-            // "scrolled away from the newest entries" — which is exactly when
-            // the "N new entries" pill should be offered.
-            onScrolled: function(y) {
-                root.userIsScrolled = (y > 0)
+            // DebugLogModel renders newest-first at the top, so leaving the
+            // beginning is exactly when the "N new entries" pill applies.
+            onScrolled: function() {
+                // A ListView's content origin is not guaranteed to be zero,
+                // particularly with variable-height rows and incremental model
+                // changes. Its boundary state is the authoritative answer.
+                root.userIsScrolled = !logView.atTop
                 if (logView.atTop && root.pendingNewLines > 0) {
                     root.pendingNewLines = 0
                 }
@@ -220,6 +227,7 @@ Page {
             Layout.preferredHeight: 36
 
             TextButton {
+                objectName: "debugLogLoadMoreButton"
                 anchors.centerIn: parent
                 text: qsTr("Load more")
                 textSize: 13
@@ -320,5 +328,6 @@ Page {
         }
     }
 
-    Component.onCompleted: debugLogModel.refresh(true)
+    Component.onCompleted: debugLogModel.active = true
+    Component.onDestruction: debugLogModel.active = false
 }