Add encode-tensogram action for Tensogram message format

CLAUDE.md

@@ -0,0 +1,131 @@
+# Claude and Other Agents
+
+# Guidelines
+
+- NOTE: When the user's request matches an available skill:
+    - ALWAYS invoke it using the Skill tool as your FIRST action. 
+    - Do NOT answer directly, do NOT use other tools first. 
+    - The skill has specialized workflows that produce better results than ad-hoc answers.
+
+- CRITICAL: Always prefer the LSP tool over Grep/Read for code navigation. 
+    - Use it to find definitions, references, and workspace symbols.
+
+
+- IMPORTANT: when planing and before you do any work:
+  - ALWAYS mention how you would verify and validate that work is correct
+  - include TDD tests in your plan
+  - take a behaviour driven approach
+  - you are very much ENCOURAGED to ask questions to get the design correct
+  - ALWAYS seek clarifications to sort out ambiguities
+  - ALWAYS provide a summary of the Design and implementation Plan
+
+
+- NOTE: When the user asks for "second pass", "third pass" or "N-th pass" perform:
+  - simplification opportunities,
+  - naming/comments/docs quality review,
+  - scan for edge-cases and logical regression,
+  - in C/C++ NEVER produce undefined behavior and never segfault or stop executiong without returning error or exceptions
+  - all documentation up-to-date with changes,
+  - running required formatter/lint/tests
+
+- NOTE: when user asks for 'error handling' checks:
+  - verify no panic in rust code
+  - verify how errors are handled across-code base, all languages
+  - ensure all errors handled and reported correclty with enough information reaching users
+
+- NOTE: when user asks for 'edge cases':
+  - look specifically edge cases
+  - look for undefined behaviour or ambiguities
+  - if necesary, ask the user to clarify 
+
+- NOTE: when user asks for 'code coverage':
+    - explore all the code base looking for code that isn't yet tested. 
+    - Look specifically for testing edge cases.
+    - Aim to have at least 95% test coverage.
+
+- NOTE: When user asks for 'final prep' make:
+    - final check everything builds, all languages and all tests pass
+    - all examples in all languages Rust, Python and C++ compile and run
+    - all docs build
+    - if successful, carefully:
+        - select files and contributions to git add
+        - ignore the build files and artifacts, don't add hidden directories
+        - if not in a branch, create a new properly named branch
+        - git commit
+        - make a pull request to upstream github project
+
+- NOTE: when user asks to do 'pr reply' or 'pull request reply':
+    - check github pull request reviews
+    - consider them with respect to the phylosophy and aims of this software
+    - if in doubt seek user clarifications
+    - fix code and address the raised issues
+    - update the docs/
+    - make a summary and push your changes to update the PR
+    - poll to wait for the CI to finish running
+    - continue iterating until all recomentations and issues were addressed
+
+- NOTE: When user asks for 'make release' execute:
+    - check all changes are commited and pushed upstream
+    - final check everything builds, all languages and all tests pass
+    - all examples in all languages Rust, Python and C++ compile and run
+    - all docs build
+    - if any of the above fails STOP and prompt the user for action
+    - otherwise, proceed by check the latest version upstream and in VERSION file
+    - if needed, bump in VERSION file, commit and tag then push to upstream
+    - make a Github release
+
+# Design & Purpose
+
+- README.md -- entry level generic information
+- plans/MOTIVATION.md -- why Tensogram exists and what we're building
+- plans/DESIGN.md -- design rationale and key architectural decisions
+
+If they are not present is because we want to still make them.
+
+# Build / lint / test (required before marking done)
+
+## Languages
+This project contains C, C++, Fortran and Python code
+
+
+# Version control
+- Git project in github.com/ecmwf/tensogram
+- IMPORTANT: 
+    - versions are tagged using Semantic Versioning form 'MAJOR.MINOR.MICRO'
+    - NEVER update MAJOR unless users says so. 
+    - Increment MINOR for new features. MICRO for bugfixes and documentation updates.
+- NEVER prepend git tag or releases with 'v'
+- REMEBER on releases:
+    - check all is commited and pushed upstream, otherwise STOP and warn user
+    - update the VERSION file
+    - git tag with version
+    - push and create release in github
+
+- NOTE: SINGLE SOURCE OF TRUTH FOR VERSION — The `VERSION` file at the repo root is the
+  canonical version for the ENTIRE project. ALL version strings everywhere MUST match it.
+  When bumping the version (e.g. during a release), you MUST update ALL of these locations
+  to match the VERSION file:
+    - `VERSION` (the source of truth)
+    - `pyproject.toml`
+    - `CHANGELOG.md` (new release entry header)
+
+# Tracking Work Done
+
+Keep track of implementations in plans/DONE.md for all code changes.
+
+# Documentation
+
+Create and maintain documentation under docs/ 
+- Easy to follow by average tech person, with well separated topics.
+- Use mdbook
+- Add mermaid diagrams when necessary
+- Add examples when it becomes hard to follow
+- Especially note the edge cases
+
+# Examples
+
+Create and maintain a sub-dir examples/<lang> 
+- 1 sub-dir per supported language of the caller C++, Python, Fortran
+- Populate with examples of caller code showing how to use interfaces
+- examplify the most common cases
+- show how to use all API functions

CMakeLists.txt

@@ -87,6 +87,13 @@ if( HAVE_FORTRAN )
 endif()
 
 
+### Tensogram encoding
+
+ecbuild_add_option( FEATURE TENSOGRAM
+                    DEFAULT OFF
+                    DESCRIPTION "Encode data using Tensogram format"
+                    REQUIRED_PACKAGES "NAME tensogram" )
+
 ### Maestro plugin
 
 ecbuild_add_option( FEATURE MAESTRO

cmake/FindTensogram.cmake

@@ -0,0 +1,120 @@
+# (C) Copyright 2025- ECMWF.
+#
+# This software is licensed under the terms of the Apache Licence Version 2.0
+# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
+# In applying this licence, ECMWF does not waive the privileges and immunities
+# granted to it by virtue of its status as an intergovernmental organisation
+# nor does it submit to any jurisdiction.
+
+# Try to find the Tensogram library (N-dimensional tensor message format)
+#
+# Tensogram is a Rust-core library with a C FFI layer and a C++ header-only wrapper.
+# This module locates the pre-built static library and the required headers.
+#
+# The following paths will be searched, both in the environment and as CMake variables:
+#
+#   TENSOGRAM_ROOT
+#   TENSOGRAM_DIR
+#   TENSOGRAM_PATH
+#
+# If found, the tensogram::tensogram imported target will be created.
+#
+# Output variables:
+#   tensogram_FOUND           - True if tensogram was found
+#   TENSOGRAM_INCLUDE_DIRS    - Include directories (C++ wrapper + C FFI header)
+#   TENSOGRAM_LIBRARIES       - Libraries to link against
+
+# --- Locate the C++ header-only wrapper: tensogram.hpp ---
+
+find_path(TENSOGRAM_CPP_INCLUDE_DIR
+    NAMES tensogram.hpp
+    HINTS
+        ${TENSOGRAM_ROOT}
+        ${TENSOGRAM_DIR}
+        ${TENSOGRAM_PATH}
+        ENV TENSOGRAM_ROOT
+        ENV TENSOGRAM_DIR
+        ENV TENSOGRAM_PATH
+    PATH_SUFFIXES include
+)
+
+# --- Locate the C FFI header: tensogram.h ---
+# This is typically in a different include path (crates/tensogram-ffi/)
+
+find_path(TENSOGRAM_FFI_INCLUDE_DIR
+    NAMES tensogram.h
+    HINTS
+        ${TENSOGRAM_ROOT}
+        ${TENSOGRAM_DIR}
+        ${TENSOGRAM_PATH}
+        ENV TENSOGRAM_ROOT
+        ENV TENSOGRAM_DIR
+        ENV TENSOGRAM_PATH
+    PATH_SUFFIXES include crates/tensogram-ffi
+)
+
+# --- Locate the Rust static library: libtensogram_ffi.a ---
+
+find_library(TENSOGRAM_LIBRARY
+    NAMES tensogram_ffi
+    HINTS
+        ${TENSOGRAM_ROOT}
+        ${TENSOGRAM_DIR}
+        ${TENSOGRAM_PATH}
+        ENV TENSOGRAM_ROOT
+        ENV TENSOGRAM_DIR
+        ENV TENSOGRAM_PATH
+    PATH_SUFFIXES lib lib64 target/release
+)
+
+# --- Aggregate results ---
+
+set(TENSOGRAM_INCLUDE_DIRS ${TENSOGRAM_CPP_INCLUDE_DIR} ${TENSOGRAM_FFI_INCLUDE_DIR})
+set(TENSOGRAM_LIBRARIES    ${TENSOGRAM_LIBRARY})
+
+include(FindPackageHandleStandardArgs)
+find_package_handle_standard_args(tensogram
+    DEFAULT_MSG
+    TENSOGRAM_LIBRARY
+    TENSOGRAM_CPP_INCLUDE_DIR
+    TENSOGRAM_FFI_INCLUDE_DIR
+)
+
+mark_as_advanced(TENSOGRAM_CPP_INCLUDE_DIR TENSOGRAM_FFI_INCLUDE_DIR TENSOGRAM_LIBRARY)
+
+# --- Create imported target ---
+
+if(tensogram_FOUND AND NOT TARGET tensogram::tensogram)
+
+    # The Rust static library (imported)
+    add_library(tensogram_ffi STATIC IMPORTED GLOBAL)
+    set_target_properties(tensogram_ffi PROPERTIES
+        IMPORTED_LOCATION "${TENSOGRAM_LIBRARY}"
+    )
+
+    # Header-only C++ wrapper (INTERFACE) linking the Rust lib + platform libs
+    add_library(tensogram::tensogram INTERFACE IMPORTED GLOBAL)
+    set_target_properties(tensogram::tensogram PROPERTIES
+        INTERFACE_INCLUDE_DIRECTORIES "${TENSOGRAM_CPP_INCLUDE_DIR};${TENSOGRAM_FFI_INCLUDE_DIR}"
+    )
+    target_link_libraries(tensogram::tensogram INTERFACE tensogram_ffi)
+
+    # Platform-specific system libraries required by the Rust static library
+    if(APPLE)
+        target_link_libraries(tensogram::tensogram INTERFACE
+            "-framework CoreFoundation"
+            "-framework Security"
+            "-framework SystemConfiguration"
+            "-lc++"
+            "-lm"
+        )
+    elseif(UNIX)
+        target_link_libraries(tensogram::tensogram INTERFACE
+            dl
+            pthread
+            m
+            stdc++
+        )
+    endif()
+
+endif()

docs/content/processing-pipelines.rst

@@ -198,6 +198,129 @@ the template, so what GRIB template to use will depend on the types of data bein
          unstructured-grid-type : eORCA025
 
 
+Encode-Tensogram
+~~~~~~~~~~~~~~~~
+
+This action encodes raw field data into the `Tensogram`_ N-dimensional tensor message format,
+producing self-describing binary messages that preserve MARS metadata. This action is particularly
+useful for producing compact, portable output that can be processed by external analysis tools.
+
+MARS metadata from the input message is preserved in two ways:
+
+* On the output Message itself (for downstream routing within multio pipelines)
+* Embedded in the Tensogram payload under ``base[0].mars`` (for external tool interoperability)
+
+The action supports configurable encoding (simple_packing for lossy compression), multiple
+compression algorithms (szip, zstd, lz4), optional filtering (shuffle), and integrity
+verification (xxh3 hashing).
+
+**Note:** This action requires tensogram support to be enabled at build time with
+``-DENABLE_TENSOGRAM=ON``. The tensogram library must be installed and available
+(see `github.com/ecmwf/tensogram`_).
+
+Configuration options:
+
+====================  ========================  ====================  ============================================
+Key                   Allowed Values            Default               Description
+====================  ========================  ====================  ============================================
+``encoding``          ``none``,                 ``simple_packing``    Encoding method: ``none`` (raw float64) or
+                      ``simple_packing``                              ``simple_packing`` (quantized integers)
+``compression``       ``none``, ``szip``,       ``szip``              Compression algorithm applied after
+                      ``zstd``, ``lz4``                               encoding
+``filter``            ``none``, ``shuffle``     ``none``              Pre-compression filter (shuffle improves
+                                                                      compression ratio)
+``hash``              ``xxh3``, (empty string)  ``xxh3``              Hash algorithm for integrity checking
+                                                                      (empty string disables)
+``bits-per-value``    Integer (1-64)            ``16``                Bits per value for simple_packing
+                                                                      (higher = more precision)
+``decimal-scale-      Integer                   ``0``                 Decimal scale factor for simple_packing
+factor``                                                              (multiplier = 10^factor)
+====================  ========================  ====================  ============================================
+
+Example configurations:
+
+**High compression** (suitable for visual analysis):
+
+.. code-block:: yaml
+
+       - type : encode-tensogram
+         encoding : simple_packing
+         compression : szip
+         bits-per-value : 12
+         filter : shuffle
+
+**Lossless** (raw float64, no packing):
+
+.. code-block:: yaml
+
+       - type : encode-tensogram
+         encoding : none
+         compression : zstd
+
+**Balanced** (default settings):
+
+.. code-block:: yaml
+
+       - type : encode-tensogram
+         # Uses defaults: simple_packing, 16 bits, szip compression
+
+**Complete pipeline example** (select → encode → sink):
+
+.. code-block:: yaml
+
+       - name : surface-to-tensogram
+         actions :
+           - type : select
+             match :
+               - levtype : [sfc]
+
+           - type : encode-tensogram
+             encoding : simple_packing
+             compression : szip
+             bits-per-value : 16
+             hash : xxh3
+
+           - type : sink
+             sinks :
+               - type : file
+                 append : true
+                 path : output.tgm
+
+Output files can be validated and inspected using the tensogram command-line tools:
+
+.. code-block:: bash
+
+   # Validate message integrity
+   tensogram validate output.tgm
+
+   # Display metadata
+   tensogram info output.tgm
+
+   # List all messages
+   tensogram ls output.tgm
+
+   # Dump message contents
+   tensogram dump output.tgm
+
+Or processed in Python using the tensogram package:
+
+.. code-block:: python
+
+   import tensogram
+
+   with tensogram.TensogramFile.open("output.tgm") as f:
+       for msg in f:
+           meta, objects = msg
+           # Access MARS metadata
+           mars = meta.base[0].get('mars', {})
+           # Access data arrays
+           desc, data = objects[0]
+           print(f"Shape: {data.shape}, dtype: {data.dtype}")
+
+.. _`Tensogram`: https://github.com/ecmwf/tensogram
+.. _`github.com/ecmwf/tensogram`: https://github.com/ecmwf/tensogram
+
+
 Sink
 ~~~~