Currently aws_efs_file_system.langfuse resource is defined (link) with elastic throughput mode. In practice this configuration alone causes pretty big spending on EFS I/O.
Our internal research showed that our self-hosted Langfuse installation can work fine with 10-20 MiB/sec provisioned throughput mode, bringing 80% reduction of cost associated with EFS volumes (I assume for clickhouse tables).
So it will be great if decision about throughput mode could be delegated to the user, keeping elastic by default for backward compatibility.
Have you considered switching to FSx for Lustre instead of EFS by the way? We use it for I/O heavy workflows on k8s cluster, and observed much better performance and significant lag reduction, with I/O practically unbounded. However, it comes with a limitation of a single AZ for an FS instance.
Currently
aws_efs_file_system.langfuseresource is defined (link) withelasticthroughput mode. In practice this configuration alone causes pretty big spending on EFS I/O.Our internal research showed that our self-hosted Langfuse installation can work fine with 10-20 MiB/sec provisioned throughput mode, bringing 80% reduction of cost associated with EFS volumes (I assume for clickhouse tables).
So it will be great if decision about throughput mode could be delegated to the user, keeping
elasticby default for backward compatibility.Have you considered switching to
FSx for Lustreinstead of EFS by the way? We use it for I/O heavy workflows on k8s cluster, and observed much better performance and significant lag reduction, with I/O practically unbounded. However, it comes with a limitation of a single AZ for an FS instance.