ramalama rag simple example does not seem to work

[mdengler]

Issue Description

I'm not sure if this is an issue of missing documentation (I can add), my misunderstanding the documentation, or something more technical, but I expect RAG to be able to be used with a single document and something is not right (see issue).

Steps to reproduce the issue

$ echo "Today is $(date), according to the ramalama oracle." > README.md && ramalama rag --format markdown README.md /tmp/output && ramalama run --rag /tmp/output deepseek-r1
Converting README.md .   2025-10-21 23:00:16,755 - INFO - detected formats: [<InputFormat.MD: 'md'>]
2025-10-21 23:00:16,755 - INFO - Going to convert document batch...
2025-10-21 23:00:16,756 - INFO - Initializing pipeline for SimplePipeline with options hash 995a146ad601044538e6a923bea22f4e
2025-10-21 23:00:16,766 - INFO - Loading plugin 'docling_defaults'
2025-10-21 23:00:16,767 - INFO - Registered picture descriptions: ['vlm', 'api']
2025-10-21 23:00:16,767 - INFO - Processing document README.md
2025-10-21 23:00:16,772 - INFO - Finished converting document README.md in 0.02 sec.
Finished converting README.md                     
🦭  > What is today, according to the ramalama oracle?
Error: Collection rag not found
$

Describe the results you received

"Error: Collection rag not found"

Describe the results you expected

"Today is [date]"

ramalama info output

{
    "Accelerator": "none",
    "Config": {
        "settings": {
            "config_files": [
                "/usr/share/ramalama/ramalama.conf"
            ]
        }
    },
    "Engine": {
        "Info": {
            "host": {
                "arch": "amd64",
                "buildahVersion": "1.41.5",
                "cgroupControllers": [
                    "cpu",
                    "memory",
                    "pids"
                ],
                "cgroupManager": "systemd",
                "cgroupVersion": "v2",
                "conmon": {
                    "package": "conmon-2.1.13-1.fc42.x86_64",
                    "path": "/usr/bin/conmon",
                    "version": "conmon version 2.1.13, commit: "
                },
                "cpuUtilization": {
                    "idlePercent": 99.89,
                    "systemPercent": 0.04,
                    "userPercent": 0.07
                },
                "cpus": 12,
                "databaseBackend": "boltdb",
                "distribution": {
                    "distribution": "fedora",
                    "version": "42"
                },
                "emulatedArchitectures": [
                    "linux/arm",
                    "linux/arm64",
                    "linux/arm64be",
                    "linux/loong64",
                    "linux/mips",
                    "linux/mips64",
                    "linux/ppc",
                    "linux/ppc64",
                    "linux/ppc64le",
                    "linux/riscv32",
                    "linux/riscv64",
                    "linux/s390x"
                ],
                "eventLogger": "journald",
                "freeLocks": 2039,
                "hostname": "hhh",
                "idMappings": {
                    "gidmap": [
                        {
                            "container_id": 0,
                            "host_id": 8001,
                            "size": 1
                        },
                        {
                            "container_id": 1,
                            "host_id": 524288,
                            "size": 65536
                        }
                    ],
                    "uidmap": [
                        {
                            "container_id": 0,
                            "host_id": 8001,
                            "size": 1
                        },
                        {
                            "container_id": 1,
                            "host_id": 524288,
                            "size": 65536
                        }
                    ]
                },
                "kernel": "6.14.5-300.fc42.x86_64",
                "linkmode": "dynamic",
                "logDriver": "journald",
                "memFree": 2714636288,
                "memTotal": 66807181312,
                "networkBackend": "netavark",
                "networkBackendInfo": {
                    "backend": "netavark",
                    "dns": {
                        "package": "aardvark-dns-1.16.0-1.fc42.x86_64",
                        "path": "/usr/libexec/podman/aardvark-dns",
                        "version": "aardvark-dns 1.16.0"
                    },
                    "package": "netavark-1.16.1-1.fc42.x86_64",
                    "path": "/usr/libexec/podman/netavark",
                    "version": "netavark 1.16.1"
                },
                "ociRuntime": {
                    "name": "crun",
                    "package": "crun-1.24-1.fc42.x86_64",
                    "path": "/usr/bin/crun",
                    "version": "crun version 1.24\ncommit: 54693209039e5e04cbe3c8b1cd5fe2301219f0a1\nrundir: /run/user/8001/crun\nspec: 1.0.0\n+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL"
                },
                "os": "linux",
                "pasta": {
                    "executable": "/usr/bin/pasta",
                    "package": "passt-0^20250919.g623dbf6-1.fc42.x86_64",
                    "version": "pasta 0^20250919.g623dbf6-1.fc42.x86_64-pasta\nCopyright Red Hat\nGNU General Public License, version 2 or later\n  <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>\nThis is free software: you are free to change and redistribute it.\nThere is NO WARRANTY, to the extent permitted by law.\n"
                },
                "remoteSocket": {
                    "exists": true,
                    "path": "/run/user/8001/podman/podman.sock"
                },
                "rootlessNetworkCmd": "pasta",
                "security": {
                    "apparmorEnabled": false,
                    "capabilities": "CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT",
                    "rootless": true,
                    "seccompEnabled": true,
                    "seccompProfilePath": "/usr/share/containers/seccomp.json",
                    "selinuxEnabled": true
                },
                "serviceIsRemote": false,
                "slirp4netns": {
                    "executable": "",
                    "package": "",
                    "version": ""
                },
                "swapFree": 42946842624,
                "swapTotal": 42949664768,
                "uptime": "3936h 41m 16.00s (Approximately 164.00 days)",
                "variant": ""
            },
            "plugins": {
                "authorization": null,
                "log": [
                    "k8s-file",
                    "none",
                    "passthrough",
                    "journald"
                ],
                "network": [
                    "bridge",
                    "macvlan",
                    "ipvlan"
                ],
                "volume": [
                    "local"
                ]
            },
            "registries": {
                "search": [
                    "registry.fedoraproject.org",
                    "registry.access.redhat.com",
                    "docker.io"
                ]
            },
            "store": {
                "configFile": "/home/xxx/.config/containers/storage.conf",
                "containerStore": {
                    "number": 9,
                    "paused": 0,
                    "running": 0,
                    "stopped": 9
                },
                "graphDriverName": "overlay",
                "graphOptions": {},
                "graphRoot": "/home/xxx/.local/share/containers/storage",
                "graphRootAllocated": 2951774167040,
                "graphRootUsed": 2529300402176,
                "graphStatus": {
                    "Backing Filesystem": "extfs",
                    "Native Overlay Diff": "true",
                    "Supports d_type": "true",
                    "Supports shifting": "false",
                    "Supports volatile": "true",
                    "Using metacopy": "false"
                },
                "imageCopyTmpDir": "/var/tmp",
                "imageStore": {
                    "number": 6
                },
                "runRoot": "/run/user/8001/containers",
                "transientStore": false,
                "volumePath": "/home/xxx/.local/share/containers/storage/volumes"
            },
            "version": {
                "APIVersion": "5.6.2",
                "BuildOrigin": "Fedora Project",
                "Built": 1759190400,
                "BuiltTime": "Tue Sep 30 02:00:00 2025",
                "GitCommit": "9dd5e1ed33830612bc200d7a13db00af6ab865a4",
                "GoVersion": "go1.24.7",
                "Os": "linux",
                "OsArch": "linux/amd64",
                "Version": "5.6.2"
            }
        },
        "Name": "podman"
    },
    "Image": "quay.io/ramalama/ramalama:latest",
    "Runtime": "llama.cpp",
    "Selinux": false,
    "Shortnames": {
        "Files": [
            "/usr/share/ramalama/shortnames.conf"
        ],
        "Names": {
            "cerebrum": "huggingface://froggeric/Cerebrum-1.0-7b-GGUF/Cerebrum-1.0-7b-Q4_KS.gguf",
            "deepseek": "ollama://deepseek-r1",
            "dragon": "huggingface://llmware/dragon-mistral-7b-v0/dragon-mistral-7b-q4_k_m.gguf",
            "gemma3": "hf://ggml-org/gemma-3-4b-it-GGUF",
            "gemma3:12b": "hf://ggml-org/gemma-3-12b-it-GGUF",
            "gemma3:1b": "hf://ggml-org/gemma-3-1b-it-GGUF/gemma-3-1b-it-Q4_K_M.gguf",
            "gemma3:27b": "hf://ggml-org/gemma-3-27b-it-GGUF",
            "gemma3:4b": "hf://ggml-org/gemma-3-4b-it-GGUF",
            "gemma3n": "hf://ggml-org/gemma-3n-E4B-it-GGUF/gemma-3n-E4B-it-Q8_0.gguf",
            "gemma3n:e2b": "hf://ggml-org/gemma-3n-E2B-it-GGUF/gemma-3n-E2B-it-Q8_0.gguf",
            "gemma3n:e2b-it-f16": "hf://ggml-org/gemma-3n-E2B-it-GGUF/gemma-3n-E2B-it-f16.gguf",
            "gemma3n:e2b-it-q8_0": "hf://ggml-org/gemma-3n-E2B-it-GGUF/gemma-3n-E2B-it-Q8_0.gguf",
            "gemma3n:e4b": "hf://ggml-org/gemma-3n-E4B-it-GGUF/gemma-3n-E4B-it-Q8_0.gguf",
            "gemma3n:e4b-it-f16": "hf://ggml-org/gemma-3n-E4B-it-GGUF/gemma-3n-E4B-it-f16.gguf",
            "gemma3n:e4b-it-q8_0": "hf://ggml-org/gemma-3n-E4B-it-GGUF/gemma-3n-E4B-it-Q8_0.gguf",
            "gpt-oss": "hf://ggml-org/gpt-oss-20b-GGUF",
            "gpt-oss:120b": "hf://ggml-org/gpt-oss-120b-GGUF",
            "gpt-oss:20b": "hf://ggml-org/gpt-oss-20b-GGUF",
            "granite": "ollama://granite3.1-dense",
            "granite-be-3.0:1b": "hf://taronaeo/Granite-3.0-1B-A400M-Instruct-BE-GGUF/granite-3.0-1b-a400m-instruct-be.Q2_K.gguf",
            "granite-be-3.3:2b": "hf://taronaeo/Granite-3.3-2B-Instruct-BE-GGUF/granite-3.3-2b-instruct-be.Q4_K_M.gguf",
            "granite-lab-7b": "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf",
            "granite-lab-8b": "huggingface://ibm-granite/granite-3.3-8b-instruct-GGUF/granite-3.3-8b-instruct-Q4_K_M.gguf",
            "granite-lab:7b": "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf",
            "granite:2b": "ollama://granite3.1-dense:2b",
            "granite:7b": "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf",
            "granite:8b": "ollama://granite3.1-dense:8b",
            "hermes": "huggingface://NousResearch/Hermes-2-Pro-Mistral-7B-GGUF/Hermes-2-Pro-Mistral-7B.Q4_K_M.gguf",
            "ibm/granite": "ollama://granite3.1-dense:8b",
            "ibm/granite:2b": "ollama://granite3.1-dense:2b",
            "ibm/granite:7b": "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf",
            "ibm/granite:8b": "ollama://granite3.1-dense:8b",
            "merlinite": "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf",
            "merlinite-lab-7b": "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf",
            "merlinite-lab:7b": "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf",
            "merlinite:7b": "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf",
            "mistral": "hf://lmstudio-community/Mistral-7B-Instruct-v0.3-GGUF/Mistral-7B-Instruct-v0.3-Q4_K_M.gguf",
            "mistral-small3.1": "hf://bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/mistralai_Mistral-Small-3.1-24B-Instruct-2503-IQ2_M.gguf",
            "mistral-small3.1:24b": "hf://bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/mistralai_Mistral-Small-3.1-24B-Instruct-2503-IQ2_M.gguf",
            "mistral:7b": "hf://lmstudio-community/Mistral-7B-Instruct-v0.3-GGUF/Mistral-7B-Instruct-v0.3-Q4_K_M.gguf",
            "mistral:7b-v1": "huggingface://TheBloke/Mistral-7B-Instruct-v0.1-GGUF/mistral-7b-instruct-v0.1.Q5_K_M.gguf",
            "mistral:7b-v2": "huggingface://TheBloke/Mistral-7B-Instruct-v0.2-GGUF/mistral-7b-instruct-v0.2.Q4_K_M.gguf",
            "mistral:7b-v3": "hf://lmstudio-community/Mistral-7B-Instruct-v0.3-GGUF/Mistral-7B-Instruct-v0.3-Q4_K_M.gguf",
            "mistral_code_16k": "huggingface://TheBloke/Mistral-7B-Code-16K-qlora-GGUF/mistral-7b-code-16k-qlora.Q4_K_M.gguf",
            "mistral_codealpaca": "huggingface://TheBloke/Mistral-7B-codealpaca-lora-GGUF/mistral-7b-codealpaca-lora.Q4_K_M.gguf",
            "mixtao": "huggingface://MaziyarPanahi/MixTAO-7Bx2-MoE-Instruct-v7.0-GGUF/MixTAO-7Bx2-MoE-Instruct-v7.0.Q4_K_M.gguf",
            "openchat": "huggingface://TheBloke/openchat-3.5-0106-GGUF/openchat-3.5-0106.Q4_K_M.gguf",
            "openorca": "huggingface://TheBloke/Mistral-7B-OpenOrca-GGUF/mistral-7b-openorca.Q4_K_M.gguf",
            "phi2": "huggingface://MaziyarPanahi/phi-2-GGUF/phi-2.Q4_K_M.gguf",
            "qwen2.5vl": "hf://ggml-org/Qwen2.5-VL-32B-Instruct-GGUF",
            "qwen2.5vl:2b": "hf://ggml-org/Qwen2.5-VL-2B-Instruct-GGUF",
            "qwen2.5vl:32b": "hf://ggml-org/Qwen2.5-VL-32B-Instruct-GGUF",
            "qwen2.5vl:3b": "hf://ggml-org/Qwen2.5-VL-3B-Instruct-GGUF",
            "qwen2.5vl:7b": "hf://ggml-org/Qwen2.5-VL-7B-Instruct-GGUF",
            "smollm:135m": "hf://HuggingFaceTB/smollm-135M-instruct-v0.2-Q8_0-GGUF",
            "smolvlm": "hf://ggml-org/SmolVLM-500M-Instruct-GGUF",
            "smolvlm:256m": "hf://ggml-org/SmolVLM-256M-Instruct-GGUF",
            "smolvlm:2b": "hf://ggml-org/SmolVLM-Instruct-GGUF",
            "smolvlm:500m": "hf://ggml-org/SmolVLM-500M-Instruct-GGUF",
            "stories-be:260k": "hf://taronaeo/tinyllamas-BE/stories260K-be.gguf",
            "tiny": "hf://TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF",
            "tinyllama": "hf://TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF"
        }
    },
    "Store": "/home/xxx/.local/share/ramalama",
    "UseContainer": true,
    "Version": "0.12.3"
}

Upstream Latest Release

No

Additional environment details

No response

Additional information

No response

[bmahabirbu]

Hi! The --format markdown is only used for outputting the preprocessed files from docling to a folder. As of this moment ramalama run --rag does not have a built in pipeline that injects the preprocessed markdown back into a vector database for use with rag. The --format markdown really is only for people who want to use docling to preprocess there files to put it into there own RAG pipeline i.e chunking then vectorizing etc.

Ramalama already does the chunking vectorizing by default! ramalama rag test.md image:latest will preprocess the files using docling chunk them and add it to a vector database neatly wrapped in a container. Then ramalama serve --rag image:latest will pull the vector database from that image to run against

You should remove this for using rag with ramalama and it will work! By default ramalama uses qdrant for its vector database.

[bmahabirbu]

If you already have preprocessed markdown from using --format markdown just take that existing folder as /folder/ and do ramalama rag folder image:latest! Now the process should be a lot faster!

1. pdf -> vector data base (slow)
2. pdf -> markdown (slow) (optional)
3. markdown -> vectordatabase (faster)

Internally ramalama rag does 1->2->3 while 2 in this case is converting to a docling json argument its basically the middle step before the chunking vectorizing etc.

Saving to markdown is kinda like a checkpoint that if you ever needed to re process those files if you have the markdown itll be faster