Very poor compression performance

[andybbruno]

I tested Memvid with three text files totaling 94 KB:

After processing, the outputs were:

• docs.mp4 (video file)
• docs_index.json (JSON index)

Together, these files are roughly 2 MB, which is about 20× larger than the original text.

See also issue #49

---

Why the JSON Index Exists

The JSON file is required because video compression codecs are typically lossy, which risks corrupting the information embedded in QR codes. The JSON index serves as a fallback to preserve exact chunk data.

See issue #39 for a deeper analysis.

---

Problem with Current Approach

Encoding via a video compression pipeline introduces:

• Significant processing overhead
• Substantial storage inflation compared to traditional compression

Also discussed in issue #63

---

Strong Recommendation

Instead of relying on a video-based workflow, just use any general-purpose LZ compressor (e.g., gzip, zstd, LZMA).

This would:

• Eliminate the need for lossy video encoding and JSON fallback
• Achieve far better compression ratios
• Reduce both processing complexity and storage footprint

In short: an LZ compressor directly on the text or embeddings would be vastly more efficient than the current pipeline.

[GregBreak]

I think the meaning of the project is the speed of accessing data and in this scenario the video encode/decode is the best.
You have a tradeoff - the size of the video.
In my case I feed with a bunch of pdf and the ratio is not that bad

[andybbruno]

I think the meaning of the project is the speed of accessing data and in this scenario the video encode/decode is the best. You have a tradeoff - the size of the video.

That's not true at all. The claim of this project is not about encoding/decoding speed, but rather about compression ratio. Please read the README. It says: "we deliver 50-100x smaller storage than vector databases with zero infrastructure"

In my case I feed with a bunch of pdf and the ratio is not that bad

PDFs are not good at compressing text. Have you ever tried converting PDFs to text? If not, use this script:

import os

from pypdf import PdfReader

folder = "path/to/my_folder"

for file in os.listdir(folder):
    filename = os.path.join(folder, file)
    if filename.endswith(".pdf"):
        txt_file = filename.replace(".pdf", ".txt")
        reader = PdfReader(filename)
        with open(txt_file, "w") as f:
            for page in reader.pages:
                f.write(page.extract_text())

As an example, this is the result of the script over my files: