Qwen 2.5 VL Dependency Decoupling and EO-1 Codebase Simplification (#16)

* This pull request sets up the initial GitHub repository configuration and automation for the EO-1 project (#5) (#6)

* Add initial project structure with configuration files, datasets, and example scripts

* Update .gitignore to include new demo data paths, modify pre-commit configuration to exclude additional directories, and enhance README with more examples and installation instructions. Adjust dataset handling in pipeline configuration and dataset classes for improved training flexibility. Remove deprecated demo scripts and refine evaluation scripts for clarity.

* Update .gitignore to include demo data paths, enhance README with additional examples, and modify Libero benchmark configuration files for improved clarity and structure. Adjust training scripts and evaluation settings across various experiments for consistency.

* Remove fast testing workflow configuration from GitHub Actions

* Update pre-commit configuration to refine exclusions, enhance README with structured examples, and remove unused imports in the EO model script.

Co-authored-by: dlqu_0010 <dlqu22@m.fudan.edu.cn>

* Refactor model input handling for multimodal data, including image and video features

* Refactor model input handling for multimodal data, including image and video features

* Refactor training scripts and configuration files for improved clarity and performance.

* Update .gitignore to include new output paths, modify dataset configurations for improved clarity, and adjust training scripts for consistency across experiments. Enhance README documentation for better guidance on dataset preparation and training processes.

* Refactor import order in test_vlm.py for improved readability and consistency.

* Update pre-commit configuration to exclude processing_eo1.py from bandit checks for improved security analysis.

* Refactor EO1 configuration and processing classes for improved structure and functionality. Updated EO1VisionFlowMatchingConfig to inherit from PretrainedConfig, streamlined initialization, and added keys_to_ignore_at_inference. Enhanced EO1VisionProcessor to support new text processing capabilities and improved handling of robot inputs and outputs. Adjusted class names for consistency and clarity.

* Update .gitignore to exclude hf_save_pretrained.py and enhance README with integration details for EO-1 with LERobot. Refactor dataset handling in MultimodaLeRobotDataset and adjust model architecture in EO1VisionFlowMatchingModel for improved functionality. Update training utilities for better configuration management and streamline processor methods for action selection.

---------

Co-authored-by: EO-Robotics-Team <22110240029@m.fudan.edu.cn>

Author: DelinQu

.gitignore

@@ -237,4 +237,5 @@ demo_data/demos25
 
 demo_data/libero_spatial_no_noops_1.0.0_lerobot
 experiments/test
+tools/hf_save_pretrained.py
 dev/

README.md

@@ -82,7 +82,7 @@ pip install --upgrade setuptools
 # install flash-attn 2
 MAX_JOBS=4 pip install flash-attn==2.8.3 --no-build-isolation
 
-# [recommended] install from source with H100 / H800 GPU, CUDA 12.8 for best performance
+# [recommended] ⭐️ install flash-attn 3 from source with H100 / H800 GPU, CUDA 12.8 for best performance
 # git clone https://github.com/Dao-AILab/flash-attn.git -b v2.8.3 --recursive --depth 1
 # cd hopper && python setup.py install
 
@@ -308,6 +308,7 @@ Robot Control Benchmark Results
 ## 📅 Roadmap
 
 - [x] 🤖 Release [EO-1](https://huggingface.co/IPEC-COMMUNITY/EO-1-3B) pretraining, finetune scripts, and documentations.
+- [x] Integrate into [LERobot](https://github.com/huggingface/lerobot). We have merged the [PR](https://github.com/huggingface/lerobot/pull/1971) into the main branch. You can now use EO-1 with LERobot without any modifications.
 - [ ] 🤗 Release [pre-training models](https://huggingface.co/collections/IPEC-COMMUNITY/eo-robotics-68ac4ff30e1f746cac28ca14), Interleaved Dataset `EO-Data1.5M` and benchmark `EO-Bench`.
 - [ ] ⚡️ Efficient LLM Inference over Long Sequences, Efficient KV-cache, etc.
 - [ ] 🤖 Integrate with human feedback fine-tuning.

eo/data/dataset.py

@@ -88,7 +88,7 @@ def __init__(
         if len(data_configs.mm_datasets) > 0:
             mm_dataset = MultimodaDataset(
                 data_configs=data_configs.mm_datasets,
-                max_packed_length=args.max_packed_length,
+                # max_packed_length=args.max_packed_length,
                 max_action_dim=args.max_action_dim,
                 meta_dataset=lerobot_dataset,
                 chunk_size=args.chunk_size,
@@ -327,6 +327,9 @@ def __getitem__(self, i) -> dict[str, torch.Tensor]:
     def info_qwen_vision_fetch(self):
         from qwen_vl_utils import smart_resize
 
+        if not self.lerobot_dataset:
+            return
+
         print(f"qwen2.5 vl min pixel {self.args.image_min_pixels}, max pixel {self.args.image_max_pixels}")
         for dataset in self.lerobot_dataset._datasets:
             meta_features, video_key = dataset.meta.features, dataset.select_video_keys

eo/data/lerobot_dataset.py

@@ -77,7 +77,6 @@ def __init__(
         delta_action: bool = False,
         effector_indices: list[int] | None = None,
         weight: float | None = None,
-        chunk_size: int = 32,
     ):
         super().__init__(
             repo_id=repo_id,

eo/model/configuration_eo1.py

@@ -12,66 +12,61 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from transformers.configuration_utils import PretrainedConfig
 from transformers.models.qwen2_5_vl.configuration_qwen2_5_vl import (
-    Qwen2_5_VLConfig,
     Qwen2_5_VLTextConfig,
     Qwen2_5_VLVisionConfig,
 )
 
 
-class EO1VisionVLTextConfig(Qwen2_5_VLTextConfig):
-    def __init__(
-        self,
-        state_token_id=None,
-        action_token_start_id=None,
-        action_token_id=None,
-        action_pass_id=None,
-        vision_token_start_id=None,
-        image_token_id=None,
-        video_token_id=None,
-        **kwargs,
-    ):
-        super().__init__(**kwargs)
-        self.state_token_id = state_token_id
-        self.action_token_start_id = action_token_start_id
-        self.action_token_id = action_token_id
-        self.action_pass_id = action_pass_id
-
-        self.vision_token_start_id = vision_token_start_id
-        self.image_token_id = image_token_id
-        self.video_token_id = video_token_id
-
-
-class EO1VisionFlowMatchingConfig(Qwen2_5_VLConfig):
-    model_type = "onevision_fm"
-    sub_configs = {"vision_config": Qwen2_5_VLVisionConfig, "text_config": EO1VisionVLTextConfig}
+class EO1VisionFlowMatchingConfig(PretrainedConfig):
+    model_type = "eo1"
+    sub_configs = {"vision_config": Qwen2_5_VLVisionConfig, "text_config": Qwen2_5_VLTextConfig}
+    keys_to_ignore_at_inference = ["past_key_values"]
 
     def __init__(
         self,
         text_config=None,
         vision_config=None,
         image_token_id=151655,
         video_token_id=151656,
-        # flow matching specific
         action_chunk_size=50,
         max_action_dim=32,
         num_denoise_steps=10,
         action_act="linear",
         num_action_layers=2,
+        state_token_id=151670,
+        action_token_id=151666,
+        action_pass_id=151667,
         **kwargs,
     ):
-        super().__init__(
-            text_config=text_config,
-            vision_config=vision_config,
-            image_token_id=image_token_id,
-            video_token_id=video_token_id,
-            **kwargs,
-        )
+        if isinstance(vision_config, dict):
+            self.vision_config = self.sub_configs["vision_config"](**vision_config)
+        elif vision_config is None:
+            self.vision_config = self.sub_configs["vision_config"](
+                hidden_size=1280,
+                out_hidden_size=2048,
+                tokens_per_second=2,
+            )
+
+        if isinstance(text_config, dict):
+            self.text_config = self.sub_configs["text_config"](**text_config)
+        elif text_config is None:
+            self.text_config = self.sub_configs["text_config"](**kwargs)
+
+        self.image_token_id = image_token_id
+        self.video_token_id = video_token_id
+        self.state_token_id = state_token_id
+        self.action_token_id = action_token_id
+        self.action_pass_id = action_pass_id
+
         self.action_chunk_size = action_chunk_size
         self.max_action_dim = max_action_dim
         self.num_denoise_steps = num_denoise_steps
         self.action_act = action_act
         self.num_action_layers = num_action_layers
 
+        super().__init__(**kwargs)
+
 
 EO1VisionFlowMatchingConfig.register_for_auto_class()