Help with Debug multiple worker setup

[rwilkhu]

Hello,

We have marker-API running as a single worker but we've tried to make updates to make our application run with multiple workers against marker-API but unfortunately our process stalls or crashes when multiple pdf's and workers are enabled.

Could someone help us fix this issue for us urgently? May be some implementation related to this in our app is incorrect hence would be great to get someone to help us fix this problem. (As a thank you, although a small amount, we'd offer $100 :-) -- better I offer it here to an expert than finding someone on upwork who may take longer)

Regards

Randeep

[rwilkhu]

Here are more details:

In our implementation of the marker-api distributed approach we manage to scale up and run 3 workers by following this approach https://github.com/adithya-s-k/marker-api?tab=readme-ov-file#step-2-scale-celery-workers-with-docker-compose.
But we are facing issues where we try to process 3 pdf documents parallely using (/convert) endpoint. And we are receiving "POST /convert HTTP/1.1" 408 Request Timeout error on all 3 workers and they are not recovering from that state
Our expectation is to create the pdf to markdown pipeline with at least 20 maker workers to be executed parallely.

Machine type: n1-standard-8 (8 vCPUs, 30 GB Memory) on GCP
CPU platform: Intel Broadwell
GPUs: 1 x NVIDIA T4

[adithya-s-k]

Hey, I will look into the issue.

The thing is, with 1 T4 GPU, you can run a maximum of 3 worker nodes, as the T4 has 16GB VRAM and each worker takes up to 5GB VRAM.

So, in order to scale to 20 workers, you will need a Kubernetes (K8s) setup, which I am planning to add to the repository.

I will first investigate the crashing issue and then try to test it out with K8s. I’ll also add documentation to make it easier to deploy on the cloud.

Let me know if you have any other requirements , I can try to add them as well!

[rwilkhu]

Thanks @adithya-s-k , what kind of server do you recommend if we were to run 3-10 workers initially until you have the K8 setup? (We are using the servers on GCP)

We are looking to introduce our application next week and this is the only piece we are trying to resolve urgently :-( ... is the crash issue something you can look at this week? (we are also investigating in parallel)

[adithya-s-k]

so a rough calculation of how much u can scale

assuming 5 gb per worker node then
t4 - 16 gb - 3 workers
L4 - 24gb - 4 to 5 workers
V100 - 32gb - 6 - 7 workers
A10 - 40 gb - 8 - 10 workers
A100 - 80gb - 16 workers

so if you want maximum number of workers then A100 would be the best but its also 4 to 6x the code of A100
i would suggest going with A10 as it gives the best price per compute

[rwilkhu]

Thank you

We are looking to test it on A100 - 40Gb and will let you know if the crash issues goes away.

[adithya-s-k]

@rwilkhu let me know if you were able to succesfully run it ?

[rwilkhu]

Hi - still working on it - some issues with the celery workers.

[adithya-s-k]

@rwilkhu any updates ?