Merge pull request #3 from daphne-eu/main

update fork repository

Author: irhox

containers/daphne-dev.Dockerfile

@@ -34,6 +34,7 @@ RUN apt-get -qq -y update && apt-get -y upgrade && apt-get -y --no-install-recom
     vim nano rsync sudo iputils-ping virtualenv openssh-server iproute2 git htop gdb lldb lld gpg-agent net-tools \
     software-properties-common ca-certificates file unzip wget tar zstd \
     ccache python3-pip python3-networkx python3-dev graphviz-dev clang-format \
+    libbz2-dev libnsl-dev libncurses5-dev libncursesw5-dev \
     && apt-get clean && rm -rf /var/lib/apt/lists/*
 
 COPY --from=daphneeu/daphne-deps /usr/local/bin/ /usr/local/bin/

doc/DaphneLib/Overview.md

@@ -131,7 +131,7 @@ More precisely, DAPHNE matrices and frames can be obtained in the following ways
 - generating data in DAPHNE (e.g., random data, constants, or sequences)
 - reading files using DAPHNE's readers (e.g., CSV, Matrix Market, Parquet, DAPHNE binary format)
 
-A comprehensive list can be found in the [DaphneLib API reference](/doc/DaphneLib/APIRef.md#daphnecontext).
+A comprehensive list can be found in the [DaphneLib API reference](/doc/DaphneLib/APIRef.md#daphnecontext-api-reference).
 
 ## Building Complex Computations
 
@@ -241,6 +241,10 @@ If the result of the computation in DAPHNE is a matrix, `compute()` returns a `n
 
 So far, DaphneLib can exchange data with numpy, pandas, TensorFlow, and PyTorch.
 By default, the data transfer is via shared memory (and in many cases zero-copy).
+Numpy and pandas are *required* dependencies for DaphneLib, so they should anyway be installed.
+TensorFlow and PyTorch are *optional* for DaphneLib; if these libraries are not installed, DaphneLib cannot exchange data with them, but all remaining features still work.
+In case you run DAPHNE inside the [`daphne-dev` container](/doc/GettingStarted.md), please note that TensorFlow and PyTorch are *not* included in the `daphne-dev` container due to their large footprint.
+Please follow the [instructions](/doc/development/InstallPythonLibsInContainer.md) on installing Python libraries in the `daphne-dev` container if you need them.
 
 ### Data Exchange with numpy
 

doc/Extensions.md

@@ -173,9 +173,12 @@ make
 ### Step 3: Using a Kernel Extension
 
 The kernels in a kernel extension can be used either automatically by DAPHNE or manually by the user.
-Automatic use is currently restricted to the selection of the kernel based on result/argument data/value types, but in the future we plan to support custom cost models as well.
-Besides that, the manual employment of custom kernels is very useful for experimentation, e.g., to see the impact of the kernel in the context of a larger integrated data analysis pipeline.
-To this end, DaphneDSL [compiler hints](/doc/DaphneDSL/LanguageRef.md#compiler-hints) tell DAPHNE to use a specific kernel, even though DAPHNE's optimizing compiler may not choose the kernel, otherwise.
+The manual use has precedence over the automatic use.
+
+#### Manual Use of Custom Kernels
+
+The manual employment of custom kernels is very useful for experimentation, e.g., to see the impact of a particular kernel at a certain point of a larger integrated data analysis pipeline.
+To this end, DaphneDSL [compiler hints](/doc/DaphneDSL/LanguageRef.md#compiler-hints) tell DAPHNE to use a specific kernel in a specific place, even though DAPHNE's optimizing compiler may not choose the kernel, otherwise.
 
 *Running example:*
 
@@ -205,7 +208,7 @@ s = sum::mySumSeq(X);
 print(s);
 ```
 
-We execute this script with the following command, whereby the argument `--kernel-ext` specified the kernel catalog JSON file of the extension to use:
+We execute this script with the following command, whereby the argument `--kernel-ext` specifies the kernel catalog JSON file of the extension to use:
 ```bash
 bin/daphne --kernel-ext scripts/examples/extensions/myKernels/myKernels.json scripts/examples/extensions/myKernels/demoSeq.daphne
 ```
@@ -222,4 +225,27 @@ print(s);
 We execute this script by:
 ```bash
 bin/daphne --kernel-ext scripts/examples/extensions/myKernels/myKernels.json scripts/examples/extensions/myKernels/demoSIMD.daphne
+```
+
+#### Automatic Use of Custom Kernels
+
+The automatic use of custom kernels is currently restricted to the selection of a kernel based on its result/argument data/value types and its priority level.
+In the future we plan to support custom cost models as well.
+
+*Running example:*
+
+Continuing the running example from above, we can make DAPHNE use the custom kernels `mySumSeq()` or `mySumSIMD()` even without a manual kernel hint by specifying a suitable *priority* when registering the `myKernels` extension with DAPHNE.
+
+Priority levels can optionally be specified with the `--kernel-ext` command line argument by appending a colon (`:`) followed by the priority as an integer.
+The default priority of `0` is used for all built-in kernels and for extension kernels in case no priority is specified.
+When registering a kernel extension, the given priority is assigned to *all* kernels provided by the extension.
+When multiple kernels are applicable for an operation based on the combination of argument/result data/value types as well as the backend, DAPHNE chooses the kernel with the highest priority.
+If there are multiple kernels with the highest priority, it is not specified which of them is used.
+
+By registering a kernel extension with a priority greater than zero, one can enforce that the kernels provided by the extension are always preferred over the built-in ones whenever they are applicable.
+For instance, the following command registers the `myKernels` extension with a priority of `1`.
+As the `myKernels` extension provides two kernels for the same operation, argument/result types, and backend, we cannot tell, based on priorities, which of these kernels will be used, but we can be sure that the built-in kernel will not be employed.
+
+```bash
+bin/daphne --kernel-ext scripts/examples/extensions/myKernels/myKernels.json:1 scripts/examples/extensions/myKernels/demo.daphne
 ```

doc/GettingStarted.md

@@ -125,7 +125,7 @@ git clone https://github.com/daphne-eu/daphne.git
 
 <!-- TODO assumes X86-64 is correct -->
 
-The development container image already contains all necessary dependencies of a DAPHNE development environment as well as a useful initialization of environment variables etc., such that you don't need to worry about these things and can have a productive start.
+The development container image already contains all necessary (¹) dependencies of a DAPHNE development environment as well as a useful initialization of environment variables etc., such that you don't need to worry about these things and can have a productive start.
 
 **Get the container image**
 
@@ -136,6 +136,8 @@ docker pull daphneeu/daphne-dev:latest_X86-64_BASE
 
 ***Hint:** You may want to choose another image tag based on your platform and needs, e.g., `latest_X86-64_CUDA` (for GPU support) or `latest_ARMV8_BASE` (for ARM support).*
 
+***Hint:** (¹) TensorFlow and PyTorch are *optional* for DaphneLib and *not included* in the `daphne-dev`container due to their footprint of several gigabytes. Please follow the [instructions](/doc/development/InstallPythonLibsInContainer.md) on installing Python libraries in the `daphne-dev` container if you need these libraries.*
+
 **Enter the container**, which should finally print something like the following (where `xyz` is your user name on your system, and the password and IP address may vary):
 
 ```bash

doc/README.md

@@ -36,6 +36,7 @@
 - [Logging in DAPHNE](/doc/development/Logging.md)
 - [Profiling in DAPHNE](/doc/development/Profiling.md)
 - [Testing](/doc/development/Testing.md)
+- [Installing Python Libraries in the `daphne-dev` Container](/doc/development/InstallPythonLibsInContainer.md)
 
 ### Source Code Documentation
 

doc/development/InstallPythonLibsInContainer.md

@@ -0,0 +1,67 @@
+<!--
+Copyright 2025 The DAPHNE Consortium
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Installing Python Libraries in the `daphne-dev` Container
+
+The `daphne-dev` container (see [GettingStarted](/doc/GettingStarted.md)) already contains all *required* dependencies for running DAPHNE. However, there can be reasons to install additional Python libraries inside the container, e.g.:
+
+1. **To use/test DaphneLib's data exchange with TensorFlow and PyTorch.** 
+    DaphneLib, DAPHNE's Python API, supports the efficient data exchange with widely-used Python libraries like numpy, pandas, TensorFlow, and PyTorch.
+    Numpy and pandas are required for DaphneLib.
+    Thus, they are already installed in the `daphne-dev` container.
+    In contrast to that, TensorFlow and PyTorch are optional for DaphneLib; if these libraries are not installed on the system, DaphneLib cannot exchange data with them, but all remaining features still work.
+    Likewise, the test cases related to the data exchange with TensorFlow and PyTorch will only run if these libraries are installed.
+    As TensorFlow and PyTorch would increase the `daphne-dev` container size by several gigabytes, they are *not* included in the container.
+1. **To add support for additional Python libraries in DaphneLib.**
+    For instance, while implementing efficient data exchange with these additional libraries.
+1. **To build integrated data analysis pipelines involving additional Python libraries.**
+    For instance, for experiments.
+
+## Installing Additional Python Libraries
+
+Additional Python libraries are best installed in a *Python virtual environment* inside the `daphne-dev` container.
+To that end, execute the following commands inside the container:
+
+**Create a Python virtual environment and activate it:**
+
+```bash
+sudo apt update
+sudo apt install python3.12-venv
+python3 -m venv daphne-venv
+source daphne-venv/bin/activate
+```
+
+Here, we call the virtual environment `daphne-venv`.
+Feel free to choose a different name.
+
+**Install the desired Python libraries using `pip`:**
+
+For instance, if you want to use/test DaphneLib's efficient data transfer with widely-used Python libraries like numpy, pandas, TensorFlow, and PyTorch, install the following libraries.
+Feel free to install any library you like.
+
+```bash
+pip install numpy pandas tensorflow torch
+```
+
+The libraries you install that way will be stored in the `daphne-venv` directory on the host and, thus, keep existing after you shut down the container.
+
+## Don't Forget
+
+Every time you enter the `daphne-dev` container, make sure to activate the Python virtual environment again:
+
+```bash
+source daphne-venv/bin/activate
+```

mkdocs.yml

@@ -65,6 +65,7 @@ nav:
     - FPGAconfiguration.md
     - MPI-Usage.md
     - Profiling.md
+    - 'Custom Extensions': Extensions.md
   - 'Developers':
     - development/Contributing.md
     - development/BuildingDaphne.md
@@ -79,3 +80,4 @@ nav:
     - development/Profiling.md
     - development/WriteDocs.md
     - development/Testing.md
+    - development/InstallPythonLibsInContainer.md

scripts/examples/dnn-conv2d.daph

@@ -59,7 +59,7 @@ ema_var = rand(F, 1, 1.0, 1.0, 1.0, seed);
 eps = 0.00001;
 
 t_start_conv = now();
-X_conv, Hout, Wout = conv2d(X, W, N, F, Isize, Isize, Fsize, Fsize, 1, 1, 1, 1, b);
+X_conv, Hout, Wout = conv2d(X, W, N, C, Isize, Isize, Fsize, Fsize, 1, 1, 1, 1, b);
 t_end_conv = now();
 X_bn = batch_norm2d(X_conv, gamma, beta, ema_mean, ema_var, eps);
 t_end_bn = now();

src/api/cli/DaphneUserConfig.h

@@ -77,7 +77,7 @@ struct DaphneUserConfig {
     bool explain_mlir_codegen_sparsity_exploiting_op_fusion = false;
     bool explain_mlir_codegen_daphneir_to_mlir = false;
     bool explain_mlir_codegen_mlir_specific = false;
-    bool statistics = false;
+    bool enable_statistics = false;
 
     bool force_cuda = false;
 

src/api/internal/daphne_internal.cpp

@@ -499,7 +499,7 @@ int startDAPHNE(int argc, const char **argv, DaphneLibResult *daphneLibRes, int
         }
     }
 
-    user_config.statistics = enableStatistics;
+    user_config.enable_statistics = enableStatistics;
 
     if (user_config.use_distributed && distributedBackEndSetup == ALLOCATION_TYPE::DIST_MPI) {
 #ifndef USE_MPI
@@ -571,14 +571,48 @@ int startDAPHNE(int argc, const char **argv, DaphneLibResult *daphneLibRes, int
     // ************************************************************************
 
     KernelCatalog &kc = executor.getUserConfig().kernelCatalog;
-    // kc.dump();
-    KernelCatalogParser kcp(mctx);
-    kcp.parseKernelCatalog(user_config.libdir + "/catalog.json", kc);
-    if (user_config.use_cuda)
-        kcp.parseKernelCatalog(user_config.libdir + "/CUDAcatalog.json", kc);
-    // kc.dump();
-    if (!kernelExt.empty())
-        kcp.parseKernelCatalog(kernelExt, kc);
+    try {
+        // kc.dump();
+        KernelCatalogParser kcp(mctx);
+        kcp.parseKernelCatalog(user_config.libdir + "/catalog.json", kc, 0);
+        if (user_config.use_cuda)
+            kcp.parseKernelCatalog(user_config.libdir + "/CUDAcatalog.json", kc, 0);
+        // kc.dump();
+        if (!kernelExt.empty()) {
+            std::string extCatalogFile;
+            int64_t extPriority;
+
+            const std::string prioritySep = ":";
+            const size_t pos = kernelExt.rfind(prioritySep);
+            if (pos != std::string::npos) { // a priority was specified for the extension
+                extCatalogFile = kernelExt.substr(0, pos);
+                const std::string extPriorityStr(kernelExt.substr(pos + prioritySep.size()));
+                try {
+                    size_t idx;
+                    extPriority = std::stoll(extPriorityStr, &idx);
+                    if (idx != extPriorityStr.size())
+                        // stoll() did not consume all characters in extPriorityStr, there is some non-integer part at
+                        // the end of the string.
+                        throw std::runtime_error(""); // the error message is generated in the catch-block below
+                } catch (std::exception &e) {
+                    throw std::runtime_error("invalid priority for kernel extension, expected an integer after the '" +
+                                             prioritySep + "', but found '" + extPriorityStr + "': '" + kernelExt +
+                                             "'");
+                }
+            } else { // no priority was specified for the extension
+                extCatalogFile = kernelExt;
+                extPriority = 0;
+            }
+
+            kcp.parseKernelCatalog(extCatalogFile, kc, extPriority);
+        }
+    } catch (std::exception &e) {
+        logErrorDaphneLibAware(daphneLibRes, "Parser error: " + std::string(e.what()));
+        return StatusCode::PARSER_ERROR;
+    } catch (...) {
+        logErrorDaphneLibAware(daphneLibRes, "Parser error: Unknown exception");
+        return StatusCode::PARSER_ERROR;
+    }
 
     // ************************************************************************
     // Parse, compile and execute DaphneDSL script
@@ -671,7 +705,7 @@ int startDAPHNE(int argc, const char **argv, DaphneLibResult *daphneLibRes, int
         std::cerr << "}" << std::endl;
     }
 
-    if (user_config.statistics)
+    if (user_config.enable_statistics)
         Statistics::instance().dumpStatistics(KernelDispatchMapping::instance());
 
     // explicitly destroying the moduleOp here due to valgrind complaining about