|
| 1 | +# Copilot Instructions |
| 2 | + |
| 3 | +## Project Overview |
| 4 | + |
| 5 | +Linux kernel driver and userspace utilities for Intel RealSense D4XX series 3D depth cameras operating over GMSL (Gigabit Multimedia Serial Link) MIPI CSI-2 interface on NVIDIA Jetson platforms. Licensed under GPL-2.0. |
| 6 | + |
| 7 | +- **Supported platforms:** Jetson AGX Xavier (JetPack 4.6.1, 5.0.2, 5.1.2) and AGX Orin (JetPack 6.0, 6.1, 6.2, 6.2.1) |
| 8 | +- **Supported cameras:** D457 (primary), D401, D40x, D41x, D43x, D45x, D46x series |
| 9 | + |
| 10 | +## Coding Conventions |
| 11 | + |
| 12 | +### Kernel Driver (C — `kernel/realsense/d4xx.c`) |
| 13 | + |
| 14 | +- **Follow Linux kernel coding style**: tabs for indentation (8-space width), `/* */` block comments, max ~80–100 char lines. |
| 15 | +- **Function naming**: prefix all functions with `ds5_`. Mux-related functions use `ds5_mux_`. Examples: `ds5_read()`, `ds5_write()`, `ds5_probe()`, `ds5_mux_s_stream()`. |
| 16 | +- **Struct naming**: prefix with `ds5_`. Examples: `struct ds5`, `struct ds5_sensor`, `struct ds5_ctrls`, `struct ds5_format`. |
| 17 | +- **Macro naming**: prefix with `DS5_`. Register addresses: `DS5_FW_VERSION`, `DS5_START_STOP_STREAM`, `DS5_DEPTH_STREAM_DT`. |
| 18 | +- **Driver name**: `DS5_DRIVER_NAME` = `"d4xx"`, with variants `-awg`, `-asr`, `-class`, `-dfu`. |
| 19 | +- **I2C access**: use `ds5_read()` / `ds5_write()` wrappers around `regmap_raw_read()` / `regmap_raw_write()` with built-in retry logic (`DS5_I2C_RETRY_COUNT=5`, `DS5_I2C_RETRY_DELAY_US=5000`). |
| 20 | +- **Helper macros**: `ds5_read_with_check()`, `ds5_write_with_check()`, `ds5_raw_read_with_check()`, `ds5_raw_write_with_check()` — these return on error. |
| 21 | +- **Logging**: use `dev_err()`, `dev_warn()`, `dev_info()`, `dev_dbg()` with `&state->client->dev` as the device. Always include `__func__` in log messages. |
| 22 | +- **Locking**: `mutex_lock()` / `mutex_unlock()` for state synchronization. |
| 23 | +- **Module registration**: `module_i2c_driver()` pattern. |
| 24 | +- **Conditional compilation**: |
| 25 | + - `CONFIG_VIDEO_D4XX_SERDES` — SerDes (GMSL) support vs non-SerDes path. |
| 26 | + - `CONFIG_TEGRA_CAMERA_PLATFORM` — Tegra-specific camera platform integration. |
| 27 | + - `LINUX_VERSION_CODE` checks for API differences between kernel versions (4.9, 5.10, 5.15+). |
| 28 | + |
| 29 | +### V4L2 Subdev Architecture |
| 30 | + |
| 31 | +Each camera registers four sensor subdevices: Depth, RGB, IR (Y8/Y8I/Y12I), and IMU. Each has separate V4L2 subdev ops structs: |
| 32 | +- `ds5_depth_subdev_ops`, `ds5_ir_subdev_ops`, `ds5_rgb_subdev_ops`, `ds5_imu_subdev_ops` |
| 33 | +- Mux ops: `ds5_mux_subdev_ops`, `ds5_mux_pad_ops`, `ds5_mux_core_ops`, `ds5_mux_video_ops` |
| 34 | + |
| 35 | +A deserializer abstraction layer (`struct dser_interface`) provides function pointer tables for MAX9296 vs MAX96712 variants. |
| 36 | + |
| 37 | +### Video Device Layout (per camera) |
| 38 | + |
| 39 | +| Device | Stream | Format | |
| 40 | +|---------|-----------------|------------------------| |
| 41 | +| video0 | Depth | Z16 | |
| 42 | +| video1 | Depth metadata | D4XX custom format | |
| 43 | +| video2 | Color RGB | RGB888/YUV422 | |
| 44 | +| video3 | Color metadata | D4XX custom format | |
| 45 | +| video4 | IR | GREY, Y8I, Y12I | |
| 46 | +| video5 | IMU | Custom | |
| 47 | + |
| 48 | +### Tests (Python — `test/v4l2_test/`) |
| 49 | + |
| 50 | +- **Framework**: pytest with marker `@pytest.mark.d457` on all test classes. |
| 51 | +- **Class-based tests**: `class TestCameraDiscovery`, `class TestLaserControl`, `class TestFirmwareVersion`, etc. |
| 52 | +- **Test naming**: `test_at_least_one_camera`, `test_driver_name`, `test_six_devices_exist`, `test_fw_version_format`. |
| 53 | +- **Fixtures**: session-scoped `all_cameras`, `camera`; per-test `depth_device`, `fw_version`. |
| 54 | +- **Constants**: `test/v4l2_test/d4xx/constants.py` mirrors driver CIDs from `d4xx.c`. |
| 55 | +- **Categories**: Discovery, Streaming, Controls, Metadata, Error Handling. |
| 56 | +- **Streaming validation**: FPS tolerance 5%, frame count 60, min 90% frame arrival, consecutive drop limit 2. |
| 57 | +- **Test timeout**: 200 seconds (configured in `test/pytest.ini`). |
| 58 | + |
| 59 | +### Shell Scripts |
| 60 | + |
| 61 | +- Use `#!/bin/bash` with `set -e` (fail-on-error). |
| 62 | +- Source `scripts/setup-common` for JetPack version normalization. |
| 63 | +- Use `DEVDIR=$(cd \`dirname $0\` && pwd)` to resolve repo root. |
| 64 | +- Branch on JetPack major version (4.x / 5.x / 6.x) for platform-specific logic. |
| 65 | + |
| 66 | +### Device Tree |
| 67 | + |
| 68 | +- **Xavier (Tegra194)**: `.dtsi` includes, pattern `tegra194-camera-d4xx-{variant}.dtsi`. |
| 69 | +- **Orin (Tegra234)**: DT overlays (`.dts`), pattern `tegra234-camera-d4xx-overlay-{variant}.dts`, uses `/dts-v1/; /plugin/;`. |
| 70 | +- Variants: `single`, `dual`, `single.calib`, `dual.calib`, `fg12-16ch`, `max96712-EVB`. |
| 71 | + |
| 72 | +## Build System |
| 73 | + |
| 74 | +### Prerequisites |
| 75 | + |
| 76 | +```bash |
| 77 | +sudo apt install -y build-essential bc wget flex bison curl libssl-dev xxd |
| 78 | +``` |
| 79 | + |
| 80 | +### Full Build Flow |
| 81 | + |
| 82 | +```bash |
| 83 | +./setup_workspace.sh <version> # Clone NVIDIA sources, install toolchain |
| 84 | +./apply_patches.sh <version> # Apply D4XX patches to kernel + NVIDIA OOT modules |
| 85 | +./build_all.sh <version> # Build kernel, DTBs, and driver modules |
| 86 | +``` |
| 87 | + |
| 88 | +Build outputs go to `images/<version>/`. |
| 89 | + |
| 90 | +### Version Mapping |
| 91 | + |
| 92 | +| JetPack | L4T Revision | Kernel Dir | Normalized | |
| 93 | +|---------|-------------|-------------------------------|------------| |
| 94 | +| 4.6.1 | 32.7.1 | `kernel/kernel-4.9` | 4.6.1 | |
| 95 | +| 5.0.2 | 35.1 | `kernel/kernel-5.10` | 5.x | |
| 96 | +| 5.1.2 | 35.4.1 | `kernel/kernel-5.10` | 5.x | |
| 97 | +| 6.0 | 36.3 | `kernel/kernel-jammy-src` | 6.x | |
| 98 | +| 6.1 | 36.4 | `kernel/kernel-jammy-src` | 6.x | |
| 99 | +| 6.2 | 36.4.3 | `kernel/kernel-jammy-src` | 6.x | |
| 100 | +| 6.2.1 | 36.4.4 | `kernel/kernel-jammy-src` | 6.x | |
| 101 | + |
| 102 | +### Cross-Compilation Toolchains |
| 103 | + |
| 104 | +- JP 4.6.1: Linaro GCC 7.3 |
| 105 | +- JP 5.x: Bootlin GCC 9.3 |
| 106 | +- JP 6.x: Bootlin GCC 11.3 (`aarch64-buildroot-linux-gnu`) |
| 107 | + |
| 108 | +### Patch Application |
| 109 | + |
| 110 | +`apply_patches.sh` copies `kernel/realsense/d4xx.c`, device tree files, and `nvidia-oot/max96712.h` into the NVIDIA source tree, applies git patches, and commits with `"RS patched"`. The `reset` action uses `git reset --hard` to a stored base commit. |
| 111 | + |
| 112 | +Camera variant flags: `--one-cam`, `--dual-cam`, `--max96712-EVB`, `--fg12-16ch`, `--fg12-16ch-dual` (only some apply to specific JetPack versions). |
| 113 | + |
| 114 | +## Key Directories |
| 115 | + |
| 116 | +| Path | Description | |
| 117 | +|------|-------------| |
| 118 | +| `kernel/realsense/d4xx.c` | Main V4L2 I2C subdevice driver (~6900 lines) | |
| 119 | +| `kernel/kernel-4.9/` | Kernel patches for JetPack 4.6.1 | |
| 120 | +| `kernel/kernel-5.10/` | Kernel patches for JetPack 5.x | |
| 121 | +| `kernel/kernel-jammy-src/` | Kernel patches for JetPack 6.x | |
| 122 | +| `kernel/nvidia/` | NVIDIA driver patches (MAX9295/9296 SerDes, VI capture) | |
| 123 | +| `nvidia-oot/` | Out-of-tree NVIDIA module patches for JetPack 6.x | |
| 124 | +| `hardware/realsense/` | Device tree source files | |
| 125 | +| `hardware/nvidia/` | Platform-level DT patches | |
| 126 | +| `scripts/` | Build orchestration and SerDes configuration scripts | |
| 127 | +| `test/v4l2_test/` | V4L2 pytest test suite | |
| 128 | +| `utilities/streamApp/` | C++ V4L2 streaming application | |
| 129 | +| `utilities/JsonToBin/` | Python JSON-to-binary preset converter | |
| 130 | + |
| 131 | +## CI |
| 132 | + |
| 133 | +CI workflows (`.github/workflows/build-jp*.yml`) build for each JetPack version on pushes to `master`/`dev` and all PRs. The V4L2 test workflow runs on a self-hosted Jetson runner when `kernel/realsense/**` or `test/v4l2_test/**` paths change. |
| 134 | + |
| 135 | +CI requires `git config user.email/name` to be set before `apply_patches.sh`. |
| 136 | + |
| 137 | +## Branching |
| 138 | + |
| 139 | +- `master` — primary/release branch |
| 140 | +- `dev` — active development branch |
| 141 | +- |
| 142 | +## Refactoring notes (recent commit) |
| 143 | + |
| 144 | +Short summary of approaches used in the refactor of `kernel/realsense/d4xx.c`: |
| 145 | + |
| 146 | +- **Encapsulated shared state**: introduced `struct ds5_dev` (per-camera) and |
| 147 | + `struct dser_control` (per-deserializer) to centralize reset generation, |
| 148 | + cached device type and last-reset timestamps instead of scattered globals. |
| 149 | +- **Separated reset generation**: split camera-level and deserializer-level |
| 150 | + reset counters so camera resets and deserializer resets are independent; |
| 151 | + callers check both generation counters and invalidate cached state on mismatch. |
| 152 | +- **Atomic, lazy invalidation**: replaced many explicit peer-invalidation loops |
| 153 | + with `atomic_inc()` of the appropriate reset-gen; peers detect the bump in |
| 154 | + `ds5_configure()` and invalidate themselves lazily, reducing O(N) churn. |
| 155 | +- **Single registration/link path**: factored duplicate SERDES/board registration |
| 156 | + logic into `ds5_setup_and_link()` to avoid repeated code paths and make |
| 157 | + primary/peer linking explicit. |
| 158 | +- **Keep streaming visibility consistent**: driver now maintains per-camera |
| 159 | + streaming flags (kept in sync from `ds5_mux_s_stream()`) so recovery and |
| 160 | + sibling checks can reliably detect active streams. |
| 161 | +- **Non-SERDES compatibility**: added small fallbacks so callers can use the |
| 162 | + same reset-gen accessors in non-SERDES builds (where applicable). |
| 163 | + |
| 164 | +Advantages: clearer ownership of shared state, fewer global arrays and linear |
| 165 | +searches, more deterministic recovery behavior, and reduced duplicate code. |
0 commit comments