Skip to content

[catalog-server] Manifests have a max size of 1MB #1358

Open
@justinfagnani

Description

@justinfagnani

Firestore has a limitation for the maximum field size of 1MB. Some manifests in the wild already exceed this size.

Options:

  1. Always use unpkg or similar CDN for manifests. This would add latency and network ingress for every page request, but unpkg is cached by Cloudflare and very fast.
  2. Store manifests in Cloud Storage. Cheap, fast, the preferred architecture for GCP. Requires making a blob storage interface for the Catalog class to use.
  3. Destructure the manifest into actual Firestore fields. JSON maps to Firestore's document model very well, and the manifest format is relatively shallow - each package contains a flat list of modules. Modules would exceed 1MB exceedingly rarely. This would allow for some more structured searching of manifest data, but we don't have use-cases for that yet.
  4. Compress the mainfest. Easy to do with current Firestore schema. We can serve the gzipped data directly, saving CPU.

Metadata

Metadata

Assignees

No one assigned

    Labels

    New SiteIssues for the rewrite of the site and catalog.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions