Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions .github/scripts/patch-openexr.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
#!/bin/bash
# Apply patches to OpenEXR submodule for modern C++ compiler compatibility

set -e

OPENEXR_DIR="src/deps/openexr/src/lib/OpenEXR"

echo "Applying OpenEXR patches..."

# Patch 1: ImfTiledMisc.h
FILE1="${OPENEXR_DIR}/ImfTiledMisc.h"
if ! grep -q "include <cstdint>" "$FILE1"; then
echo "Patching $FILE1"
sed -i '/^#include <ImathBox.h>/a #include <cstdint>' "$FILE1"
else
echo "$FILE1 already patched"
fi

# Patch 2: ImfDeepTiledInputPart.h
FILE2="${OPENEXR_DIR}/ImfDeepTiledInputPart.h"
if ! grep -q "include <cstdint>" "$FILE2"; then
echo "Patching $FILE2"
sed -i '/^#include <ImathBox.h>/a #include <cstdint>' "$FILE2"
else
echo "$FILE2 already patched"
fi

# Patch 3: ImfDeepTiledInputFile.h
FILE3="${OPENEXR_DIR}/ImfDeepTiledInputFile.h"
if ! grep -q "include <cstdint>" "$FILE3"; then
echo "Patching $FILE3"
sed -i '/^#include <ImathBox.h>/a #include <cstdint>' "$FILE3"
else
echo "$FILE3 already patched"
fi

echo "OpenEXR patches applied successfully"
93 changes: 93 additions & 0 deletions .github/workflows/publish-pypi.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
name: Publish to PyPI

on:
workflow_dispatch: # Allow manual trigger
inputs:
build_type:
description: 'Build type: full (compile from source) or quick (requires pre-built bindings)'
required: true
default: 'full'
type: choice
options:
- full
- quick
push:
tags:
- 'v*' # Trigger on version tags like v0.2.2a1

jobs:
publish:
runs-on: ubuntu-latest
permissions:
id-token: write # Required for trusted publishing
contents: read

steps:
- uses: actions/checkout@v4
with:
submodules: recursive

- name: Apply OpenEXR patches
run: |
chmod +x .github/scripts/patch-openexr.sh
./.github/scripts/patch-openexr.sh

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.8'

- name: Install dependencies
run: |
sudo apt-get update
sudo apt-get install -y \
cmake \
ninja-build \
libegl1-mesa-dev \
libgles2-mesa-dev \
libbullet-dev \
libxrandr-dev \
libxinerama-dev \
libxcursor-dev \
libxi-dev

- name: Install uv
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
echo "$HOME/.local/bin" >> $GITHUB_PATH

- name: Cache CMake build
uses: actions/cache@v4
with:
path: |
build
!build/bdist.*
!build/lib.*
key: cmake-build-${{ runner.os }}-py38-${{ hashFiles('src/**/*.cpp', 'src/**/*.h', 'src/CMakeLists.txt', 'cmake/**/*', 'src/deps/*/CMakeLists.txt') }}
restore-keys: |
cmake-build-${{ runner.os }}-py38-

- name: Full CMake build
if: github.event.inputs.build_type == 'full' || github.event_name == 'push'
run: |
python setup.py build_ext --inplace

- name: Copy compiled bindings to source tree
run: |
# Copy .so files from build dir to source tree for packaging
mkdir -p src_python/habitat_sim/_ext
cp -v build/RelWithDebInfo/lib/habitat_sim_bindings*.so src_python/habitat_sim/_ext/ || echo "Bindings already in place"
ls -lh src_python/habitat_sim/_ext/

- name: Build wheel
run: |
# Install wheel package for bdist_wheel command
pip install wheel
# Build wheel with bindings already in source tree
python setup.py bdist_wheel

- name: Publish to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
packages-dir: dist/
password: ${{ secrets.PYPI_API_TOKEN }}
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ __pycache__/
# Distribution / packaging
.Python
build/
build-headless/
develop-eggs/
dist/
downloads/
Expand Down
99 changes: 99 additions & 0 deletions CLAUDE.local.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# Local Fork Context

## Purpose

This fork exists to build and publish **habitat-sim wheels** for the **Thousand Brains Project (TBP) Monty** (`/home/vaskin/projects/tbp/`).

## Problem

**Habitat-sim** is officially conda-only and not available as wheels on PyPI. TBP Monty is a sensorimotor learning system that uses habitat-sim for 3D environment simulation. This project demonstrates that TBP can run entirely with UV/pip (no conda) by building custom habitat-sim wheels.

## Solution

This fork (`feature/uv-wsl2-support` branch) provides:

1. **UV/pip-compatible wheels** - Build habitat-sim as a wheel for UV/pip workflows
2. **WSL2 GPU support** - Environment variables to bypass CUDA device enumeration, enabling D3D12 backend
3. **Build flexibility** - Environment variables to control submodule updates and CMake builds
4. **Documentation** - Complete guides for building and integrating with TBP

## About TBP Monty

**Thousand Brains Project Monty** is an open-source sensorimotor learning system following neocortex principles. Named after Vernon Mountcastle, it implements learning through active sensing and motor interaction with 3D environments.

- **Project**: https://thousandbrainsproject.org
- **Docs**: https://thousandbrainsproject.readme.io/
- **Repo**: https://github.com/thousandbrainsproject/tbp.monty

Monty uses habitat-sim for realistic 3D object recognition tasks with vision and touch sensors.

## Built Wheels

Pre-built wheels are stored in `dist/` directory:

```bash
# Habitat-sim wheel for Python 3.8 (WSL2/Linux)
dist/habitat_sim-0.2.2-cp38-cp38-linux_x86_64.whl

# Magnum bindings (editable install from build directory)
build/deps/magnum-bindings/src/python/
```

TBP uses these via `pyproject.toml` configuration:
```toml
[tool.uv.sources]
habitat-sim = { path = "../habitat-sim/dist/habitat_sim-0.2.2-cp38-cp38-linux_x86_64.whl" }
magnum = { path = "../habitat-sim/build/deps/magnum-bindings/src/python", editable = true }
```

## Key Features (feature/uv-wsl2-support branch)

### Build Environment Variables
- `HSIM_NO_SUBMODULE_UPDATE=1` - Skip git submodule updates (preserve local patches)
- `HSIM_SKIP_CMAKE_BUILD=1` - Reuse existing CMake build (faster wheel packaging)

### WSL2 GPU Compatibility
- `HSIM_DISABLE_CUDA_DEVICE=1` - Bypass CUDA device enumeration
- Enables D3D12 backend rendering on WSL2 with NVIDIA RTX 4090

### Documentation
- `UV_WSL2_BUILD.md` - Complete build guide
- `OPENEXR_PATCHES.md` - Required OpenEXR submodule patches
- Cross-references TBP UV setup documentation

## Repository Structure

**Upstream**: `facebookresearch/habitat-sim` (v0.2.2)
**Fork**: `killerapp/habitat-sim`
**Branch**: `feature/uv-wsl2-support` (5 commits ahead)

### Commits on feature branch
1. `b0519a23` - feat: Add build environment variables for UV/pip workflow
2. `d3f83f76` - feat: Add HSIM_DISABLE_CUDA_DEVICE env var for WSL2 compatibility
3. `b22caa0e` - docs: Document required OpenEXR submodule patches
4. `47d87f24` - docs: Add comprehensive UV/WSL2 build guide
5. `d9ad079b` - chore: Add UV workflow files

## Related Projects

### TBP Monty (Main Project)
- **Path**: `/home/vaskin/projects/tbp/`
- **Fork**: https://github.com/killerapp/tbp
- **Purpose**: Sensorimotor learning system using this habitat-sim fork
- **Documentation**: See `../tbp/UV_PROJECT.md` for complete UV setup guide

### Habitat-Sim (This Fork)
- **Path**: `/home/vaskin/projects/habitat-sim/`
- **Fork**: https://github.com/killerapp/habitat-sim
- **Purpose**: Build UV/pip-compatible wheels for TBP

## Quick Start (For TBP Users)

If TBP is already set up with UV sources configured:

```bash
cd /home/vaskin/projects/tbp
uv sync --extra dev # Automatically installs habitat-sim wheel from this fork
```

See TBP's `UV_PROJECT.md` for full installation instructions.
3 changes: 3 additions & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
@@ -1 +1,4 @@
graft src_python/habitat_sim/sensors/noise_models/data
graft src_python/habitat_sim/_ext
include README_PYPI.md
include README_TBP_USERS.md
70 changes: 70 additions & 0 deletions OPENEXR_PATCHES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# OpenEXR Submodule Patches

## Overview

The OpenEXR submodule (`src/deps/openexr`) requires patches to compile with modern C++ compilers due to missing `<cstdint>` includes.

## Required Changes

Add `#include <cstdint>` to the following files in the OpenEXR submodule:

### File 1: `src/deps/openexr/openexr-2.3.0/IlmImf/ImfTiledMisc.h`

```cpp
// After existing includes (around line 19):
#include <ImathBox.h>

#include <cstdint> // ADD THIS LINE
#include <stdio.h>
#include <vector>
```

### File 2: `src/deps/openexr/openexr-2.3.0/IlmImf/ImfDeepTiledInputPart.h`

```cpp
// After existing includes (around line 13):
#include <ImathBox.h>
#include <cstdint> // ADD THIS LINE

OPENEXR_IMF_INTERNAL_NAMESPACE_HEADER_ENTER
```

### File 3: `src/deps/openexr/openexr-2.3.0/IlmImf/ImfDeepTiledInputFile.h`

```cpp
// After existing includes (around line 22):
#include <ImathBox.h>
#include <cstdint> // ADD THIS LINE

OPENEXR_IMF_INTERNAL_NAMESPACE_HEADER_ENTER
```

## Why This Is Needed

Modern C++ compilers (GCC 11+, Clang 13+) require explicit inclusion of `<cstdint>` for types like `int64_t` and `uint64_t`. The OpenEXR 2.3.0 headers used in this submodule predate this requirement.

## Application

These patches must be applied manually after initializing submodules:

```bash
git submodule update --init --recursive
cd src/deps/openexr
# Apply patches to the three files listed above
cd ../../..
```

Alternatively, set `HSIM_NO_SUBMODULE_UPDATE=1` when building if you've already applied the patches:

```bash
HSIM_NO_SUBMODULE_UPDATE=1 python -m build --wheel
```

## Upstream Status

These changes are local patches. The OpenEXR submodule points to an older version (2.3.0) that is no longer maintained. Newer versions of OpenEXR (3.x+) have fixed these issues but would require significant integration work.

## Related

- Part of the UV/WSL2 support effort
- See `UV_PROJECT.md` in the tbp.monty repository for full build instructions
Loading