Release v1.5.3: VLA Ops Enhancements; Ray Repartition Pipeline; Scalability & Robustness

Major Updates

📊 Stats: 14 PRs merged, from 8 contributors

📈 Code diff: 172 files changed, with 19,085 insertions and 2,144 deletions

🤖 VLA ops enhancements: Expanded embodied-AI / Vision-Language-Action processing capabilities with 10+ new and renamed operators — including new camera calibration methods (DeepCalib, DroidCalib, MoGe), atomic action segmentation, hand action computation & motion smoothing, clip reassembly, trajectory overlay, and LeRobot export — plus a complete VLA pipeline demo for ego-hand action annotation. #931

🔄 Ray repartition pipeline: A new ray_repartition_pipeline enables dataset-level block repartitioning in Ray mode, giving users fine-grained control over data distribution across workers. #985

⚡ Scalable Ray Data reads: Wired override_num_blocks through the full call chain, allowing users to control Ray Data's block parallelism via CLI — essential for processing PB-scale datasets without overwhelming the scheduler. #984

🧪 Test coverage expansion: Added 409 new test cases across 18 test files covering utils, ops, format, config, download, and pipeline DAG modules. #990

New OPs

• export_to_lerobot_mapper: Exports processed data into the LeRobot dataset format for downstream robot learning. #931
• video_atomic_action_segment_mapper: Segments videos into atomic actions for fine-grained action annotation. #931
• video_camera_calibration_deepcalib_mapper (renamed from video_camera_calibration_static_deepcalib_mapper): Computes camera intrinsics and FOV using DeepCalib. #931
• video_camera_calibration_droidcalib_mapper: Computes camera intrinsics and FOV using DroidCalib. #931
• video_camera_calibration_moge_mapper (renamed from video_camera_calibration_static_moge_mapper): Computes camera intrinsics and FOV using MoGe-2. #931
• video_camera_pose_megasam_mapper (renamed from video_camera_pose_mapper): Extracts camera poses using MegaSaM and MoGe-2. #931
• video_clip_reassembly_mapper: Reassembles video clips for flexible clip-level data organization. #931
• video_hand_action_compute_mapper: Computes hand action data from video for manipulation tasks. #931
• video_hand_motion_smooth_mapper: Smooths hand motion trajectories for cleaner action signals. #931
• video_trajectory_overlay_mapper: Overlays trajectory visualizations onto video frames for debugging and presentation. #931
• ray_repartition_pipeline: A Ray-only pipeline for dataset-level block repartitioning, registered in config_all.yaml and operator docs. #985

Enhancements

• override_num_blocks CLI argument for Ray Data: Previously implemented only at the lowest layer (read_json_stream()), this parameter is now wired through the full call chain, making it accessible via CLI for controlling block parallelism on very large datasets. #984
• num_proc handling for vllm and Ray mode: TextTaggingByPromptMapper was unconditionally setting num_proc = 1, which broke parallelism in Ray mode. Now properly respects the configured value. #973

Fixed Bugs

• JSONStreamDatasource schema mismatch across batches: The first batch's inferred schema was locked and reused for all subsequent batches. When an early batch inferred a field as null and a later batch introduced a concrete type (e.g., string), the forced cast failed with ArrowInvalid. Schema is now unified across batches. #972
• OP env LATEST strategy returning unpinned version: The conflict resolution strategy incorrectly fell back to an unpinned version when the union of two conflicting specifiers contained ranges without an upper bound (e.g., numpy>=2.0 vs numpy<1.5). Now correctly resolves to a pinned version. #992
• FUSE-safe rmtree fallback missing in PartitionedRayExecutor: PR #943 fixed shutil.rmtree() failures on FUSE-mounted OSS buckets in RayExecutor, but the same pattern was missing in ray_executor_partitioned.py. All three rmtree sites now have the fallback. #988
• Deprecated model names in tests, demos, and docs: Replaced deprecated model names (e.g., qwen2.5-72b-instruct, qwen2.5-vl-3b-instruct) with available alternatives across test files, demo configs, and docstrings. #994

Acknowledgements

• @macroguo-ghy contributed the new ray_repartition_pipeline for Ray mode. #985
• @ArdalanM fixed num_proc handling for vllm and Ray mode. #973
• @CodyQin fixed the OP env LATEST strategy version resolution bug. #992
• @wenhaozhao011-cmd added FUSE-safe rmtree fallback to PartitionedRayExecutor. #988

New Contributors

• @ArdalanM made their first contribution in #973
• @CodyQin made their first contribution in #992
• @wenhaozhao011-cmd made their first contribution in #988

Full Changelog: v1.5.2...v1.5.3

Release v1.5.2: Semantic OPs; Agent Interaction Quality; Cross-Document Dedup; Robustness & Performance

Major Updates

📊 Stats: 15 PRs merged, from 11 contributors

📈 Code diff: 217 files changed, with 18,972 insertions and 1,685 deletions

🧠 Semantic OPs MVP shipped: Introduced a new semantic operations framework with extract and condition filter capabilities, laying the groundwork for join/aggregation/top-k in future releases. #948

🤖 Agent interaction quality evaluation: New OPs and recipe for assessing interaction quality, with bad-case HTML report generation, and robust JSONL / HuggingFace meta loading support. #957

📄 Cross-document line-level deduplication: A new DocumentLineDeduplicator enables deduplication at the line level across multiple documents — useful for removing boilerplate or repeated content shared between files. #961

⚡ Lighter default installation: Reduced the size of default dependencies to significantly speed up first-time installation. #959

New OPs

• DocumentLineDeduplicator: Performs cross-document line-level deduplication — ideal for removing shared boilerplate text, repeated headers/footers, or duplicated content across large corpora. #961
• semantic_extract_mapper & semantic_condition_filter (MVP): Extract structured fields from unstructured text and filter based on semantic conditions, powered by the new semantic ops framework. #948
• Interaction quality OPs (via dj-agents): Evaluate the quality of agent interactions with dedicated scoring operators and a recipe that generates HTML bad-case reports for human review. #957

Enhancements

• O(n²) → O(n) in FrequencySpecifiedFieldSelector: Replaced repeated sum() accumulation with itertools.chain, eliminating quadratic behavior on large batches. #975
• Exclude work_dir from fingerprint hashing: Cache fingerprints no longer change when work_dir differs, preventing unnecessary recomputation while preserving correct pickling behavior. #967
• Reduce default dependency size: Trimmed the default install footprint so new users can get started faster without pulling in heavyweight optional packages. #959
• Replace bs4 stub with beautifulsoup4: Switched to the canonical package name to prevent installation conflicts and deprecation warnings. #977

Fixed Bugs

• Invalid max_new_tokens injected into API chat completions: The parameter was being incorrectly passed to chat completion API calls that don't support it, causing request failures. Now properly excluded. #983
• initialize_ray crash without config: Ray initialization failed when no config was provided. Added proper fallback handling. #981
• exit(1) replaced with raise in dataset processing: Using exit() in a library context kills the entire process. Errors now raise proper exceptions for library-safe error handling. #974
• PyArrow 20.0.0+ batch JSON reading compatibility: Fixed an issue where open_json batch reading broke under PyArrow 20+. #942
• text_keys not propagated to ops in service mode: The DJ service was not passing text_keys through to operators via get_init_configs, causing ops to use wrong keys. #960
• Model utils sampling params & API client initialization: Fixed incorrect sampling parameter handling and optimized API client setup in model_utils.py. #962
• Aesthetics-predictor normalization check too strict: The substring match for model normalization detection now supports local model paths, not just HuggingFace hub identifiers. #946
• Temp dir removal failure in RayExecutor: Added fallback handling when temporary directory cleanup fails (e.g., due to permission issues or concurrent access). #943

Acknowledgements

• @dubin555 delivered key performance optimizations and library-safety improvements. #974 #975
• @amanyara contributed the new DocumentLineDeduplicator. #961
• @ideal added robustness to RayExecutor temp dir cleanup. #943
• @HunterLine fixed PyArrow 20+ batch JSON reading compatibility. #942
• @justinwolfington corrected the beautifulsoup4 package dependency. #977
• @macroguo-ghy fixed Ray initialization without config. #981

New Contributors

• @justinwolfington made their first contribution in #977
• @macroguo-ghy made their first contribution in #981
• @amanyara made their first contribution in #961
• @ideal made their first contribution in #943

Full Changelog: v1.5.1...v1.5.2

Release v1.5.1: LaTeX OPs; Compressed Format Support; Operator Robustness Fixes

Major Updates

• 📊 Stats: 13 PRs merged, from 7 contributors
• 📄 Two new LaTeX-focused mapper OPs shipped, extending data-juicer's document processing capabilities to handle .tex archives and figure contexts.
• 🗜️ Compressed dataset format support: json[l].gz files can now be loaded directly, and Ray datasets gain proper support for reading compressed JSON files.
• 📚 New documentation added covering cache, export, and tracing workflows to help users better understand and debug data processing pipelines.
• 🤖 Major refactor and upgrade of data-juicer-agents completed: The project architecture and CLI/session capabilities were comprehensively redesigned for better maintainability and extensibility. See date-juicer-agents for more details.

New OPs

• latex_merge_tex_mapper: Got a bunch of .tex files packed in an archive? This OP automatically extracts and merges them into a single unified LaTeX document, making it much easier to process multi-file LaTeX projects. #932
• latex_figure_context_extractor_mapper: Extracts figure-related context (e.g., captions, surrounding paragraphs) from LaTeX source files, so you can build richer multimodal datasets from academic papers. #923

Enhancements

• Load dataset with extra kwargs: You can now pass arbitrary extra arguments to datasets.load_dataset() via the new load_dataset_kwargs config field — handy for datasets that need non-standard loading options. #922
• Custom tokenizer in RemoveRepeatSentencesMapper: The mapper now accepts a custom tokenizer, so you're no longer stuck with the default sentence splitter — great for non-English text or domain-specific tokenization needs. #925
• Compressed JSON support: Added support for reading json[l].gz files directly, and fixed Ray datasets to properly handle compressed JSON — no more manual decompression before feeding data in. #919
• Faster TokenNumFilter with batch tokenization: Instead of tokenizing one sample at a time, TokenNumFilter now processes the whole batch in one shot, significantly speeding up token-count-based filtering. #929
• Cache redundant sum() calls in repetition filters: Repetition filters were calling sum() multiple times on the same data. These results are now cached, saving unnecessary computation on large batches. #924
• New docs: cache, export, and tracing: Added dedicated documentation pages explaining how data-juicer handles caching, result exporting, and execution tracing — a much-needed addition for debugging complex pipelines. #935
• Enhanced op_search with BM25/Regex & MCP Server upgrade: Added BM25 and regex search modes to op_search (no longer requiring dj-agents), and expanded the MCP server with four new tools covering op search, dataset analysis, config schema retrieval, and dataset loading strategy discovery. #937

Fixed Bugs

• Wrong cache key in ImageFaceCountFilter: The filter was using an incorrect key when reading from cache, causing it to miss cached results and redo redundant work. Now fixed. #921
• GeneralFusedOP silently dropping Mapper results: When running a fused pipeline, Mapper outputs were being discarded instead of passed downstream. This was a silent data loss bug — now properly fixed. #928
• Shared _default_kwargs mutation polluting other OP instances: Operator instances were accidentally sharing a mutable default kwargs dict, meaning modifying one OP's config could inadvertently affect other instances. Each instance now gets its own copy. #926
• NlpaugEnMapper only augmenting the first sample in a batch: Due to a bug in the batching logic, text augmentation was only being applied to the very first sample, leaving the rest untouched. All samples in a batch are now correctly augmented. #927

---

Acknowledgements

• @JohnGiorgi contributed three impactful improvements: load_dataset_kwargs support, custom tokenizer in RemoveRepeatSentencesMapper, and batch tokenization optimization in TokenNumFilter. #922 #925 #929
• @dubin555 squashed multiple operator bugs and added performance optimizations across filters and the fused pipeline. #921 #924 #926 #927 #928
• @leeyyi and @liyuyi-2001 made their first contributions with two brand-new LaTeX OPs. #923 #932
• @HunterLine added compressed JSON dataset support. #919

Full Changelog: v1.5.0...v1.5.1

Release v1.5.0: Partitioned Ray Executor; Embodied-AI OPs; OP-level Env Management

Major Updates

• 📊 Stats: 244 files changed with 22,394 additions and 2,053 deletions, from 12 contributors
• 🗂️ New partitioned ray executor: #748
  • Support data partitioning, checkpointing, event logging in ray mode.
  • Improved fault tolerence, extensibility, observability, flexibility, and processing performance.
• 🤖 New OPs for embodied AI: improved processing capability to handle camera-view videos.
• 🧩 Support OP-level isolated environment maintaining in ray mode to help resolve the dependency confliction issue from different OPs. #892
  • Allow to merge possible environments from different OPs that share common dependencies in different strategies and reuse the created environments.
  • Based on ray runtime environment.

New OPs

• video_camera_calibration_static_deepcalib_mapper: Compute the camera intrinsics and field of view (FOV) for a static camera using DeepCalib. #871
• video_camera_calibration_static_moge_mapper: Compute the camera intrinsics and field of view (FOV) for a static camera using Moge-2. #871
• video_undistort_mapper: Undistort raw videos with corresponding camera intrinsics and distortion coefficients. #871
• video_hand_reconstruction_hawor_mapper: Use HaWoR and MoGe-2 for hand reconstruction. #893
• video_camera_pose_mapper: Extract camera poses with MegaSaM and MoGe-2. #894

Enhancements

• Allow batch inference for image_captioning_mapper to improve processing performance. #901
• Optimize the logics of a branch by avoiding unnecessary function calls. #903 '
• Refactor Operator Search and Metadata Extraction for Enhanced Accuracy. #889
• Allow to return meta infos only for extract_keyframes func and remove the sample info in error logs to reduce the size of logs. #904
• Reduce the memory usage in convert_to_absolute_paths func by iterating only over the specified columns. #907
• Reorganize the main readme and update the tutorials in the playground to the latest version. #908
• Optimize issue templates: emphasize English usage and add Q&A Copilot check. #912
• Convert abs path for dataset in object store. #913

Fixed Bugs

• Fix the bug to make minhash deduplicator be able to trace all duplicate items. #906
• Fix the "multiple values for num_proc" bug in TextFormmater. #905
• Fix the homepage rendering issue and remove outdated OP docs. #910
• Fix several bugs in test stability and robustness. #918

Acknowledgement

• @dubin555 helps to improve the processing performance of some OPs and funcs. #901 #903
• @HunterLine helps to fix a bug in minhash deduplicator to trace all duplicate items. #906

New Contributors

• @HunterLine made their first contribution in #906
• @Dludora made their first contribution in #907

All Contributors

@HYLcool @dubin555 @claude @Qirui-jiao @cmgzn @Cathy0908 @Dludora @yxdyc @gemini-code-assist @HunterLine @ext.wanghao204 @cyruszhang

Full Changelog: v1.4.6...v1.5.0

Release v1.4.6: introduce Q&A Copilot; Video bytes I/O; Tracer for Ray mode

Major Updates

• 🤖 Our Q&A copilot is introduced to resolve questions from users. Now the robot is available in the docs, DingTalk group, Discord, etc. #891
• 🎬 I/O for video bytes: support bytes reading/storing for videos. #882
• 🫆 Tracer for ray mode: now the tracer supports to trace changed samples in ray mode. #885

Enhancements

• Prepare a new dockerfile for use case of embodied AI, and update the cuda/system/... versions of the basic docker image. #887
• Add Copilot News & Refined DingTalk link/QR code & Discord link/QR code in the docs. #891
• Convert the word retrieval from lists to sets to speed up two OPs. #890
• Add a new workflow to automatically fetch the traffic report from github insigts. #899 #900

Fixed Bugs

• Fix TypeError when using field_types in YAML config for RequiredFieldsValidator. #886
• Replace the deprecated concurrency parameter with compute parameter in the ray.data.Dataset.map_batches() call. #888
• Prevent divide-by-zero in calculate_ray_np when Ray cluster not ready. #864
• Add thread limiting for multi-process workloads to prevent over-subscription. #877
• Fix the bug where the unittest of standalone mode could be stuck. #896
• Update several out-of-date links in the docs. #898

Acknowledgement

• @dubin555 helps to fix several bugs and enhance the processing performance for some OPs. #886 #890
• @xyuzh helps to update the ray usage to the latest version in some OPs, fix some bugs and optimize the parallel strategies. #888 #864
• @XinyuLiu1999 helps to fix a bug of over-subscription on multi-process workloads. #877

Full Changelog: v1.4.5...v1.4.6

Release v1.4.5: Embodied-AI OPs; Doc System Upgrading

Major Updates

• Add several new OPs for embodied-AI.
• Upgrade to the documentation system: #842
  • transition the documentation generation and deployment to a unified Sphinx-based framework.
  • architectures, styles are maintained as an isolated repo. It will be pulled before building the docs of each sub-repo.

New OPs

Mapper

• video_captioning_from_vlm_mapper: generate video captions from latest VLMs (e.g., Qwen3-VL). #820
• video_object_segmenting_mapper: perform text-guided semantic segmentation of valid objects throughout the video (using YOLOE and SAM2), with support for saving segmentation visualization results. #801
• video_depth_estimation_mapper: perform depth estimation on the video, with support for saving both visualization results and point cloud data. #801
• image_mmpose_mapper: perform human keypoint detection inference using MMPose models. #800
• image_tagging_vlm_mapper: generates image tags with VLMs. #800
• image_sam_3d_body_mapper: perform single-image full-body 3D human mesh recovery (HMR) with the promptable model SAM 3D Body (3DB). #843
• s3_download_file_mapper: download files from S3 to local files or load them into memory. #839
• s3_upload_file_mapper: upload local files to S3 and update paths to S3 URLs. #839

Filter

• text_tagging_by_prompt_mapper: generate text tags using prompt with LLM. #408
• image_subplot_filter: detect and remove samples with images containing subplots. #840 #822
• video_motion_score_ptlflow_filter: a new motion score filter where the optical flows are computed by the ptlflow library. #824

Deduplicator

• document_minhash_deduplicator_with_uid: a more robust version of document_minhash_deduplicator for datasets with unique ID for each sample. #832 #677
• ray_bts_minhash_deduplicator_with_uid: a more robust version of ray_bts_minhash_deduplicator for datasets with unique ID for each sample. #832 #677
• ray_bts_minhash_cpp_deduplicator: enhance the performance of the basic BTS MinHash deduplicator by migrating its computationally intensive parts to C++ and Cython. #851

Pipeline

A new type of OP, which allows combine multiple OPs into one pipeline, or integrating a whole pipeline that is not easy to split into multiple atomic OPs. #835

• ray_vllm_engine_pipeline: basic OP for making use of the vLLM engine of Ray.
• llm_ray_vllm_engine_pipeline: generate response with LLMs using vLLM engine on Ray.
• vlm_ray_vllm_engine_pipeline: generate response with VLMs using vLLM engine on Ray.

Enhancements

• Several major dependencies of data-juicer are updated to the (nearly) latest version. #820
• Rename and align some core arguments of base OP about resource allocation to the ones in Ray. #837
• Refine the badges on the homepage to enhance user experience. #841
• Support to specify extra arguments for ffmpeg for some video OPs. #847
• Update Used by & Valuable Feedback from list to add new customers/users of Data-Juicer. #852
• Improve Ray-based deduplicators by lazily initializing actors, allowing the cluster to autoscale before actors consume resources. #855
• Improve the RayS3DataLoadStrategy class to provide better format detection, more informative logging, and support for loading multiple files from S3 directories in Ray mode. #860
• Support OpenAI Reponses API. #856
• Use consistent key naming and allow to include extra fields in the tracer output with trace_keys arg. #873 #874
• Support bytes data as input and add the auto_op_parallelism parameter to control whether to enable automatic calculation of OP parallelism. #867
• Support to save optical flows computed in the OP. #824
• Update the base image of official data-juicer docker image to cuda12.6.3 and ubuntu 24.04; update the python version to py311; install several embodied-ai-related packages. #881
• Add memory reservation parameters for Ray minhash deduplication to allow users to reserve memory for actors and tasks. #863

Fixed Bugs

• Fix the issue where export_aws_credentials cannot be read properly in RayExecutor when using S3 export paths. #834
• Change the logger level of import timing from info to debug. #859
• Cache the dataset columns once at the start of process() and pass the cached set through the operator pipeline to fix the issue where Ray's Dataset.columns() breaks streaming pipelines when called repeatedly during operator processing. #854
• Fix out-of-date URLs of PAI demos. #865
• Limit versions of several dependencies to avoid new issues from their latest versions. #876
• Use mount disk to avoid "No space" error on the / path when building docs. #857
• Use persist to avoid OOM in distributed deduplication of pyspark version. #836 #586
• Fix the issue where the number of OPs change can not trigger the OP doc building hook. #824

Acknowledgement

• @kyo-tom helps fix several bugs and enhance the functions of I/O, and implement 2 new OPs for s3. #834 #860 #839
• @xyuzh helps fix and enhance the core parts of ray distributed mode according to the professional techs from Ray repo. #859 #855 #854 #863
• @JohnGiorgi helps to support new type of OpenAI API and enhance the tracer. #856 #873 #874
• @coolderli helps to optimize the spark distributed deduplication to avoid OOM. #586
• @ZiyiTsang helps to implement a new OP image_subplot_filter. #822

Full Changelog: v1.4.4...v1.4.5

Release v1.4.4: NeurIPS 2025 Spotlight; New Video & Multimodal Ops; Repo Reorganization; S3 I/O Support

Major Updates

• 🎉 Update NeurIPS 2025 News: our Data-Juicer 2.0 paper is accepted as a NeurIPS'25 Spotlight (top 3.1% of all submissions)! And our two other works are also accepted by NeurIPS'25. #788
• 🧩 The sandbox component, data-juicer recipes, and data-juicer agents have been officially split from the main repository as data-juicer-sandbox/hub/agents respectively, to enable independent development and faster iteration. #817 #827 #830
• 🤝 S3 I/O support: Added S3 support in data loader and exporter for seamless cloud storage integration. #806

New OPs

• detect_main_character_mapper: Extract all main character names based on the given image and its caption. #795
• detect_character_locations_mapper: Given an image and a list of main character names, extract the bounding boxes for each present character. (YOLOE + MLLM) #795
• detect_character_attributes_mapper: Takes an image, a caption, and main character names as input to extract the characters' attributes. #795
• vggt_mapper: Input a video of a single scene, and use VGGT to extract information including Camera Pose, Depth Maps, Point Maps, and 3D Point Tracks. #804
• video_whole_body_pose_estimation_mapper : Input a video containing people, and use the DWPose model to extract the body, hand, feet, and face keypoints of the human subjects in the video, i.e., 2D Whole-body Pose Estimation. #812
• video_hand_reconstruction_mapper : Use the WiLoR model for hand localization and reconstruction. #818

Enhancements

• Enhanced documentation for operator details, significantly expanding coverage of effect demonstrations and usage examples, and improved homepage styling for better readability. #778 #819
• Added notebook detection and auto-redirect in logger setup for better user experience in Jupyter environments. #790
• Optimized the build_op_doc hook for more reliable documentation generation. #794
• Improved auto num_proc calculation in Ray mode for better resource utilization across operators. #789 #825
• Enabled support for videos and audios in WebDataset I/O, expanding multimodal data handling capabilities. #803
• Updated repository URLs and links across the project for consistency and correctness. #805
• Added support for FFmpeg and Decord backends in video data processing, improving flexibility and performance. #826 #829
• Added an MCP server CLI entry point to facilitate modular service deployment and upodate MCP documentation. #798

Fixed Bugs

• Fixed the Auto Prompt pipeline in sandbox to restore correct prompt generation behavior. #791
• Fixed a Ray connection error by properly passing the config parameter through resource utility functions. #808
• Fixed several CUDA-based operators to use internal resource monitor. #809
• Fixed custom op module loading issues and optimized video_extract_frames_mapper for saving extracted frames. #803
• Reset num_proc for vLLM and set default batch_size to 10 for CUDA operators to improve stability. #814
• Fixed Sphinx autodoc compatibility issue in the SpecialTokens metaclass to restore documentation build. #816
• Resolved a bug in trace_filter by excluding the __dj_stats__ column during dataset comparison. #828
• Fix several typos in video_split_by_scene_mapper. #744

Acknowledgement

• @kyo-tom helps to fix the ray connection error in #808
• @liuyuhanalex helps to fix several small typos in #744

Full Changelog: v1.4.3...v1.4.4

Release v1.4.3: OP Doc Enhancement; Optimized Auto Parallelism; Optimized Sandbox

Major Updates

• 🤝 OP Document Updates: Optimized multi-version docs; Doc strings are rewritten and enhanced by qwen-max. #755 #765 #768 #769 #787
• 💪🏻 Auto Parallelism Optimization: support cpu/gpu/mem requirement specification for each OP; optimize calculate_np for ray mode. #679 #774 #782 #786
• 🛠️ Sandbox Optimization: support iterative pipelines and early-stop targets; refactor the context infos; a new example on auto prompt optimization and several related hooks are added. #757
• 📈 Upgrade spacy from 3.8.0 to 3.8.7 due to the previous one is yanked. #763

New OPs

• image_detection_yolo_mapper: perform object detection (with YOLO) on images and return the bounding box values and class labels. #764
• optimize_prompt_mapper: optimize prompts based on the existing ones. #757

Enhancements

• Support shard_size and extra args for write methods in export_extra_args for RayExporter. #739
• Support min/max_closed_interval args to control filtering with open/closed intervals and reversed_range arg to allow keeping samples outside a specified range for Filters. #741
• Support API models for existing optimize_qa_mapper. #771

Fixed Bugs

• Fix and re-enable the disabled op_list_to_trace argument. #766
• Add missing skip tag to several API-based test cases for forked repos. #767
• Limit the version of transformers to "<4.55.0" to avoid computing on None value. #781
• Fix out-of-date invoking methods in several tools. #785 (from issue #750)
• Fix 500 error in API service. #785 (from issue #777)
• Remove specified_xxx_filter from NON_STATS_FILTER. #785 (from issue #783)

Full Changelog: v1.4.2...v1.4.3

Release v1.4.2: Python > 3.10 are supported; Data Attribution OPs; External OPs are supported; Install with "uv"

Major Updates

• 💪🏻 Data-Juicer now is compatible with Python 3.11 & 3.12. #749
• 🧩 5 OPs for data attribution are added. #735
• 🤝 Now Data-Juicer support register and apply custom OPs in external paths using the argument custom_operator_paths. #758
• 🔧 "uv" is the first choice to installing Data-Juicer now due to its capability to solve the dependency conflicts. #760

New Operators

Filter

• Validation-free
  • llm_perplexity_filter: Filter to keep samples with perplexity score, computed using a specified llm, within a specific range. #735
  • instruction_following_difficulty_filter: Filter to keep texts whose instruction follows difficulty (IFD, https://arxiv.org/abs/2308.12032) falls within a specific range. #735
• Validation-based
  • in_context_influence_filter: Filter to keep texts whose in-context influence upon validation set within a specific range. #735
  • llm_task_relevance_filter: Filter to keep sample with high relevance score to validation tasks estimated by LLM. #735
  • text_embd_similarity_filter: Filter to keep texts whose average embedding similarity to a set of given validation texts falls within a specific range. #735

Enhancements

• A new environment variable DATA_JUICER_EXTERNAL_MODELS_HOME is added to allow to specify some private or read-only paths to store external and extra models. #740
• Optimize the video link transformation and multi-version maintainence in the docs. Update demo videos with higher-resolution versions. #746
• Support custom save_dir for OPs that produce extra multimodal data. #751
• Add official and detailed docs about Data-Juicer Agent. #759
• Enhance unit tests: show the name of the current test cases; recycle resources after each test cases for ray mode. #749
• Refining developer guide for better practice on building new OPs. #760

Bugs Fixed

• Move the updating of special tokens of multimodal data in initialization of base_op, which fix the bug that special tokens might not be synced with the main process when processing data in parallel. #752
• Fix some test cases. #754

Acknowledgement

• @ShenQianli made their first contribution to 5 new OPs. #735

Full Changelog: v1.4.1...v1.4.2

Release v1.4.1: MCP server; GPU-based Minhash deduplicator; Improved unit test coverage.

Major Updates

• 🔧 Introduce Data-Juicer MCP server. Users can make use of the data processing capabilities in the MCP way conveniently. #690 #737
• 💪🏻 Unit test coverage rate is improved to 85%+ and several bugs in test cases are resolved (OOM, encoding error, and so on), which makes Data-Juicer more reliable. #698 #717 #720 #727
• 🤝 Minhash deduplication based on GPU is supported, collaborated with developers from Nvidia. #694 #644
• 🧩 RayExporter supports more formats to export a ray dataset in addition to json/jsonl. #687
• 🎥 Two demo videos are added to introduce the Data-Juicer core functions, agentic usages, and sandbox. #738

New Operators

• download_file_mapper downloads data from URLs to local files or specified fields. #709

Enhancements

• New analysis method: correlation analysis among stats is added. #663
• Several core dependencies are updated and fixed to a newer version, and dependency conflicts are resolved. #715 #717 #723
• The EasyAnimate pipelines in the sandbox are updated to follow the refactoring of sandbox. #710
• Apply more reliable pre-commit tools to improve the code style of Data-Juicer. #714
• Support store and process bytes data of images in the dataset. #725

Bugs Fixed

• The wheel & docker image building bug is fixed. #706
• Fix bugs in log_summarization. #710
• Fix "no module named data_juicer" error after installing from the wheel file. #727

Acknowledgement

• @fanronghai helps to fix the param error in dataset_splitting_by_language tool. #713
• @ayushdg helps to support a GPU-version Minhash deduplicator. #644
• @ricksun2023 helps to fix the bugs when there are more than one same-name OPs in the configs. #730

Full Changelog: v1.4.0...v1.4.1