Allow a persistence volume for model weights

[dolfim-ibm]

Problem description

Docling requires AI models for converting the documents. Which models is only known at runtime and it might be different for each API request.
In Docling Serve we took the approach of storing the model common model weights in the container image, but this creates issues with users who request the features provided by the extra models.

Proposed solution

To allow users to download all the required models, and to keep them "cached" in the cluster, we suggest adding the optional possibility to:

1. add a persistent volume to the deployment
2. add an initContainers which downloads the requested models in the volume

For completeness, we could give the users 3 choices:

• Option 1: No volume. Just use what is there by default
• Option 2: Request the operator to provision a pvc and specify which models are requested
• Option 3: Provide pointers to an existing pvc and specify which models are requested

Proposed specs

kind: DoclingServe
spec:
  artifactsVolume:
    enable: false | true  # default false
    models: ""  # a space separated list like"layout tableformer code_formula picture_classifier smolvlm granite_vision easyocr"
    volumeClaimTemplates:  # optional
      - {}  # the specs for a volumeClaim (see below examples)

The default pvc settings should request for 16GB storage and accessMode: ReadWriteMany.

Inspired by other operators (e.g. elasticsearch) Users could provide the volumeClaim details for advanced changes, for example:

• A different size or storage class:

  volumeClaimTemplates:
    - metadata:
      name: docling-models
      spec:
        accessModes:
          - ReadWriteMany
        resources:
          requests:
            storage: 20Gi
        storageClassName: standard

• Point to an existing volume:

  volumeClaimTemplates:
    - metadata:
      name: docling-models
      spec:
        storageClassName: ""
        volumeName: foo-pv    

Changes in the Deployment

1. Mount the volume, e.g. in /opt/docling/models (just a proposal)

2. Set the ENV DOCLING_SERVER_ARTIFACTS_PATH=/opt/docling/models (the actual mount point)

3. Add a initContainers using the same image but a different command:

   initContainers:
     - name: docling-models-download
       image: <spec.apiServer.image>
       command: ['sh', '-c', "docling-tools models download -o <mountPoint> <spec.artifactsVolume.models>"]

[bcrochet]

I'd prefer models be an actual list. No sense in making it harder to parse.

[dolfim-ibm]

I'd prefer models be an actual list. No sense in making it harder to parse.

Either is ok with me. Eventually the CLI will need the space-separated list.

[acornett21]

We probably don't need enabled as well, since the presence of artifactsVolume will let us know to do custom things.

The default pvc settings should request for 16GB storage and accessMode: ReadWriteMany.

@dolfim-ibm Can you explain the above further? Are you saying that the docling-serve workload now needs a PVC by default?

[dolfim-ibm]

Not really by default. The key is the three options described in the description above.

[bcrochet]

I'd prefer models be an actual list. No sense in making it harder to parse.

Either is ok with me. Eventually the CLI will need the space-separated list.

That part is easy enough. I'll stick with a list.

[bcrochet]

Should models actually be a top-level attribute? That would allow us to get rid of the enabled attribute.

1. No artifactsVolume specified. This would be the default, and make that part completely optional.

kind: DoclingServe
spec:
    models:
        - layout
        - tableformer
        - code_formula
        - picture_classifier

2. artifactsVolume specified, but no template. This would give a default (16G as stated above) pvc, with accessmodes of ROX and RWO. (As far as I can tell this is allowed?)

kind: DoclingServe
spec:
    models:
        - layout
        - tableformer
        - code_formula
        - picture_classifier
    artifactsVolume: {}

3. artifactsVolume specified, and additional attributes, like size, name, and storage class are specified. These are liktely the only parts of the pvc template that we want to allow. The operator needs to handle the other parts.

kind: DoclingServe
spec:
    models:
        - layout
        - tableformer
        - code_formula
        - picture_classifier
    artifactsVolume:
        storage: 16G
        name: modelvol
        storageClassName: mysuperfaststorage

[dolfim-ibm]

1. Ok with me. Let's avoid making the list static typed (i.e. with only fixed values), because they will change "quickly".
2. Nice
3. What is name? The name given to the pvc or the name of the matched pv?
   • my proposal (very verbose) was based on what I've seen other operators using (e.g. elasticsearch, etc)
   • I think it is in general better to meet the conventions on what people know and expect to be able to configure.
   • I didn't look around in all operators, but if you know that it is also common to use the "simplified version", it is much better.

[acornett21]

@dolfim-ibm Can you point to where you see the elasticsearch operator exposes a CRD that references volumeClaimTemplates? I'm not finding any such CRD, but I might be missing it.

[dolfim-ibm]

@dolfim-ibm Can you point to where you see the elasticsearch operator exposes a CRD that references volumeClaimTemplates? I'm not finding any such CRD, but I might be missing it.

https://www.elastic.co/docs/deploy-manage/deploy/cloud-on-k8s/volume-claim-templates

[dolfim-ibm]

On the other hand, other operator like the datascience pipelines are a bit simpler, see https://github.com/opendatahub-io/data-science-pipelines-operator/blob/main/config/samples/dspa-all-fields/dspa_all_fields.yaml

[acornett21]

@dolfim-ibm Can you point to where you see the elasticsearch operator exposes a CRD that references volumeClaimTemplates? I'm not finding any such CRD, but I might be missing it.

https://www.elastic.co/docs/deploy-manage/deploy/cloud-on-k8s/volume-claim-templates

From what I can tell this info isn't in the operators CRDs, and the above is for a manual configuration. I might be incorrect though and missing something, though I see nothing in the below CRDs that lets a user define this.

https://github.com/redhat-openshift-ecosystem/certified-operators/tree/main/operators/elasticsearch-eck-operator-certified/2.16.1/manifests

Or when I check artifacthub
https://artifacthub.io/packages/olm/community-operators/elastic-cloud-eck

[dolfim-ibm]

From what I can tell this info isn't in the operators CRDs, and the above is for a manual configuration. I might be incorrect though and missing something, though I see nothing in the below CRDs that lets a user define this.

https://github.com/redhat-openshift-ecosystem/certified-operators/tree/main/operators/elasticsearch-eck-operator-certified/2.16.1/manifests

it is these lines, or?
https://github.com/redhat-openshift-ecosystem/certified-operators/blob/main/operators/elasticsearch-eck-operator-certified/3.0.0/manifests/elasticsearches.elasticsearch.k8s.elastic.co.crd.yaml#L2227

[bcrochet]

1. Ok with me. Let's avoid making the list static typed (i.e. with only fixed values), because they will change "quickly".

Most certainly. It will only be validated as a list. The contents are completely variable.

2. Nice

Excellent.

3. What is name? The name given to the pvc or the name of the matched pv?

The name of the pvc. We can make that more explicit.

• my proposal (very verbose) was based on what I've seen other operators using (e.g. elasticsearch, etc)
• I think it is in general better to meet the conventions on what people know and expect to be able to configure.
• I didn't look around in all operators, but if you know that it is also common to use the "simplified version", it is much better.

You mentioned the datascience pipelines, and I like that approach much better, with just pvcSize and storageclassname. To me, that would be a good first cut.

[acornett21]

@dolfim-ibm It looks like your search is better than mine, yes those would be the lines. TY.

I can see this both ways, do we want to be like other operators and expose everything for users that they see in other places ie volumeClaimTemplates. But I can also see having a smaller API surface in this operator, ie have less knobs for a user that have more clarity on the end use case(s).