Open
Description
Firestore has a limitation for the maximum field size of 1MB. Some manifests in the wild already exceed this size.
Options:
- Always use unpkg or similar CDN for manifests. This would add latency and network ingress for every page request, but unpkg is cached by Cloudflare and very fast.
- Store manifests in Cloud Storage. Cheap, fast, the preferred architecture for GCP. Requires making a blob storage interface for the Catalog class to use.
- Destructure the manifest into actual Firestore fields. JSON maps to Firestore's document model very well, and the manifest format is relatively shallow - each package contains a flat list of modules. Modules would exceed 1MB exceedingly rarely. This would allow for some more structured searching of manifest data, but we don't have use-cases for that yet.
- Compress the mainfest. Easy to do with current Firestore schema. We can serve the gzipped data directly, saving CPU.