MOMIJI

English | 日本語

MOMIJI (Modern Open Multimodal Japanese filtered Dataset) is a large-scale Japanese interleaved image-text dataset built from Common Crawl.

Interleaved image-text data preserves the order in which images and text appear in a web document and represents them together as a single record. MOMIJI uses placeholders to mark the original positions of images.

The source data comes from Common Crawl, a nonprofit project that continuously collects and publishes web pages from around the world. The collected HTML and crawl metadata are stored in WARC (Web ARChive) files.

This repository contains both the AWS implementation used to build MOMIJI and a local implementation that reproduces the core data-processing pipeline. See build_momiji/README.md for setup and execution instructions for the local pipeline.

Important

This repository publishes the code used to build MOMIJI. It does not include the original web-page text, image binaries, or momiji_generator, which generates text from the public MOMIJI records.

Published resources

MOMIJI provides three resources for different use cases.

Resource	Location	Contents
Public MOMIJI records	Hugging Face	Records identifying documents and images that passed the final text and image filters
MOMIJI data-generation utility	Separate repository: turingmotors/momiji_generator	Generates text and text_list from source web pages that remain available
MOMIJI build code	This repository: turingmotors/MOMIJI	The AWS implementation used for the original build and a local implementation of the core pipeline

Public MOMIJI records

Each record published on Hugging Face primarily contains:

• a document ID derived from a Common Crawl record (docId);
• the source web-page URL (url);
• placeholders marking image positions in the document;
• image URLs;
• image width and height; and
• image metadata such as alt text.

For copyright reasons, we do not redistribute the original web-page text or image binaries. The public records are therefore not a frozen corpus containing complete text and images. They are references that identify documents and images that passed the final filters and can be used to generate data from web resources that are still available.

MOMIJI data-generation utility

momiji_generator accesses the source web-page URLs in the public records and primarily generates these fields:

• text: the full document text, including image placeholders;
• text_list: an ordered sequence of image and text segments.

momiji_generator does not reproduce the exact page text captured when MOMIJI was built. It generates MOMIJI-formatted data from the page available at execution time. If a page has been updated or removed, the generated data may differ from the original build.

Source code, installation instructions, and usage documentation are maintained in the separate official repository turingmotors/momiji_generator.

MOMIJI build code

This repository provides two implementations:

• build_momiji_on_aws: the implementation used to build MOMIJI with AWS services such as Amazon S3 and AWS Lambda;
• build_momiji: a local implementation of the core processing pipeline that does not depend on AWS-specific services.

The local implementation replaces AWS storage, event-driven processing, and distributed execution with local paths and execution scripts. Its core document and image acceptance rules, filter order, principal thresholds, image-placeholder handling, and output fields correspond to the AWS implementation.

See build_momiji/README.md for detailed differences involving AWS-specific orchestration.

Local pipeline

build_momiji reproduces the main AWS data-processing steps locally without Amazon S3 or AWS Lambda. The numeric directory prefixes indicate execution order.

Stage 1 → Stage 2 → Stage 3 → Stage 4 → Stage 5 → Stage 6 → Stage 7

1. Read WARC files and apply coarse filtering

1_download_warc_files reads local WARC files. It can also download WARC files from Common Crawl, but only when network access is explicitly enabled.

It keeps HTML records that satisfy all of the following conditions:

• the record is an HTTP response;
• Content-Type contains text/html;
• the body contains at least one hiragana, katakana, or CJK ideograph character.

The <html lang="ja"> attribute is not used as a language criterion at this stage.

2. Detect Japanese with fastText

2_precise_extract_text_ja extracts the <title> element from raw HTML, removes image placeholders and line breaks, and then detects the language with fastText.

fastText is a machine-learning library that estimates the language of input text. Only HTML whose predicted label is ja with a score of at least 0.90 is retained.

The check is not skipped when <html lang="ja"> is present, and fastText is not run a second time after body-text extraction.

3. Extract body text and preserve image positions

3_extract_text_ja_from_warc_file extracts body text with BeautifulSoup and Trafilatura.

Images are not downloaded at this stage. Instead, each image is replaced with a [[IMAGE: ... ALT: ...]] placeholder containing its absolute URL and alt text, which preserves the image's position in the document.

4. Filter text for quality and safety

4_text_filtering applies 17 filters in the same order as the AWS implementation using Hojichar.

Category	Main criteria
Repetition	Repeated-line and repeated-paragraph ratios; character share occupied by repeated spans
Word repetition	Ratios of the most frequent word 2-gram, 3-gram, and 4-gram
Text quality	At least 400 characters, hiragana ratio, mean sentence length, and ratio of sentences ending in ellipses
Harmful content	Japanese and English adult terms, discriminatory expressions, and violent expressions
Other	Special-token ratio and consecutive identical characters

An n-gram is a sequence of n adjacent words. A 2-gram, for example, is a pair of adjacent words.

Exact thresholds and boundary conditions are documented in 4_text_filtering/README.md.

5. Select image URLs and download images

5_download_images filters image URLs extracted from the text and determines which images to download.

Repeated image URLs within a document are first deduplicated in the download-candidate list. The original image placeholders in the text are retained. The stage then checks URL suffixes, commas, blocked substrings, and inappropriate terms.

It also counts occurrences before within-document deduplication and removes URLs that occur at least 10 times within a processing unit. The remaining URLs are deduplicated in first-seen order.

Only when ALLOW_NETWORK=true is explicitly set does the stage use img2dataset to download images and produce a WebDataset containing JPEG images. WebDataset stores large collections of images and metadata in tar archives.

6. Filter images by dimensions and NSFW scores

6_image_filtering reads either the WebDataset from Stage 5 or a user-supplied local image manifest, then checks image dimensions and safety.

A manifest maps image URLs to local files. NSFW refers to content that may be inappropriate for viewing in workplaces or public settings.

Check	Acceptance rule
Width and height	150–20,000 px, inclusive
Aspect ratio	width ÷ height from 0.5 through 2.0, inclusive
NSFW	None of hentai > 0.9, porn > 0.85, or sexy > 1.0 is true

This stage does not perform near-duplicate image removal with pHash or a similar method. pHash compares images by perceptual hash to estimate visual similarity.

7. Build the final dataset

7_final_processing joins the URL-filtered documents from Stage 5 with the image metadata accepted by Stage 6.

It then segments Japanese text into sentences with Bunkai and writes gzip-compressed JSONL in MOMIJI or VILA format.

Filtering statistics and profiling

Use filtering_statistics to aggregate the numbers of documents and images accepted and rejected by each filter.

Setting PROFILE=true records elapsed time, CPU usage, memory usage, and throughput for each stage and its major phases. See profiling for configuration and aggregation instructions.

Profiling is disabled during normal execution.

Reproduction options and limitations

Depending on your goal and available compute resources, you can use or reproduce MOMIJI in one of the following ways:

1. Download the public MOMIJI records that passed the final filters from Hugging Face.
2. Use momiji_generator from its separate repository to generate text and text_list from source pages that remain available.
3. Run build_momiji or build_momiji_on_aws on the same Common Crawl WARC files, or on a subset of them.

Reprocessing the entire Common Crawl period requires substantial storage, compute, and network capacity. If a full rebuild is impractical, processing a small set of WARC files or records with build_momiji is sufficient to inspect filter rules, filter order, and intermediate outputs.

Text extraction and filtering can be rerun from the same WARC files. Images downloaded from external URLs, however, may not be bit-for-bit identical to the original build because of:

• expired image URLs;
• replaced images;
• access restrictions; or
• changes in delivery format.

Source pages accessed by momiji_generator can also be updated, deleted, moved, restricted, or structurally changed. Consequently, text, character counts, image counts, image positions, and document structure may differ from the state captured during the original build.

Character counts, image counts, and similar statistics reported in this README and the paper refer to the snapshot used to build MOMIJI.

License and terms of use

Materials in the Hugging Face dataset repository for which the publishers hold the necessary rights—such as the dataset card, statistics, and format documentation—are licensed under CC BY 4.0.

Rights in text, images, alt text, and other content originating from source web pages remain with their respective rights holders. CC BY 4.0 does not apply to third-party text or images. When using MOMIJI, review the terms in the Hugging Face dataset card and comply with applicable law.

MOMIJI is intended for uses such as training, evaluating, and auditing machine-learning models; text mining; and statistical analysis, including information analysis under Article 30-4 of the Copyright Act of Japan. The presence of a URL or metadata in MOMIJI does not imply permission to use the source content for every purpose. See the Hugging Face dataset card for full terms.

Source code in build_momiji_on_aws and build_momiji is provided under the MIT License. The momiji_generator source code is provided under the MIT License defined in its own repository.

These software licenses do not apply to third-party text or images retrieved or processed by the software.