Issue with file node on kubernetes

[ibarrick]

I have all four components of anytype running in kubernetes, but file uploads do not make it into minio and instead I get the error "file upload limit has been reached"

I cannot figure out how to get rid of this error. I already have
fileLimit: limitDefault: 1099511627776 limitAlphaUsers: 1099511627776 limitNightlyUsers: 1099511627776
In the yaml file for the coordinator node.

[ibarrick]

After updating the anytype-ts client to the latest appimage 0.38.9-alpha, the error has changed to: "calculate file size: walk DAG: get root node: CID not found". This seems like the more accurate error as when I investigated the badger spacestore db the cached status showed limit -1 and bytesleft was correctly set to the value implied by treating -1 as unsigned when computing remaining bytes.

The error is preceded by: "logger":"anytype-files","msg":"FileByHash: failed to retrieve from IPFS: CID not found" and "logger":"fileobject","msg":"build details: file not found"

I've determined from the source code that the latter error is caused by the FileByHash error, so I guess the only piece I'm missing is why the CID is missing from IPFS. I will try digging through more source to find how it's supposed to get in IPFS to begin with but would appreciate some guidance.

[fb929]

the any-sync server consists of multiple services, most of which are stateful.
for example, any-sync-node essentially, it's a database.
Kubernetes, on the other hand, is designed for stateless services.
any-sync also does not fully support dynamic configuration via an orchestrator, which is another important aspect of application design for Kubernetes.

[tbrasser]

the any-sync server consists of multiple services, most of which are statefull. for example, any-sync-node essentially, it's a database. Kubernetes, on the other hand, is designed for stateless services. any-sync also does not fully support dynamic configuration via an orchestrator, which is another important aspect of application design for Kubernetes.

this makes no sense, k8s is just a container scheduler / extensible api to accomplish that. There is no real difference between running containers with docker-compose or with k8s. What do you even mean with 'dynamic configuration via an orchestrator'?

I can go on for hours, but to keep it short: statefullsets and operators are specifically targeted to running statefull workloads on k8s. Stateless is just easier (again, exactly the same with single-machine docker(compose).

[tbrasser]

the any-sync server consists of multiple services, most of which are statefull. for example, any-sync-node essentially, it's a database. Kubernetes, on the other hand, is designed for stateless services. any-sync also does not fully support dynamic configuration via an orchestrator, which is another important aspect of application design for Kubernetes.

this makes no sense, k8s is just a container scheduler / extensible api to accomplish that. There is no real difference between running containers with docker-compose or with k8s. What do you even mean with 'dynamic configuration via an orchestrator'?

I can go on for hours, but to keep it short: statefullsets and operators are specifically targeted to running statefull workloads on k8s. Stateless is just easier (again, exactly the same with single-machine docker(compose).

took a look at the docker compose yaml, seems the config gen and bootstrap can just be initcontainers or even k8s jobs. or you know, just generate config manually / using the compose stack and then hardcode the k8s manifests. @ibarrick maybe try a different S3 compatible storage backend? I hear good things about cubefs

[fb929]

by dynamic configuration, I mean systems like Consul, ZooKeeper, etcd.

if you believe it makes sense to run databases in k8s, that's your choice - I won't try to convince you otherwise.
I'm just informing you that the server-side part of any-sync is not adapted for running in k8s, and we have no plans to support that.
however, you're always free to do it yourself - it might even be useful for the community.

[tbrasser]

Don't want to nitpick, possibly also misunderstood your (initial) message. But just for clarification:

• consul / zookeeper have nothing to do with k8s (and as far as I can see also not with anytype) (maybe you're refering to non-vanilla / more opinionated stacks like openshift, not sure)
• etcd is just the default db for k8s apiserver, application containers on k8s don't need to be aware of it (with kine you can swap it for postgresql / sqlite / etc)
• I'm not sure how one would "adapt" a container for running on k8s (other than optional things that also benefit non-k8s like running nonroot)

Fully understand that providing readymade k8s deployment options next to the docker compose stack is not something you want to commit to or provide support for. If (big if) I can find the time I'd be happy to experiment writing some equivalent helm chart for anytype stack on k8s

cheers!