RFC: Portable EmDash site export and import

[ascorbic]

EmDash should provide a core-owned, versioned format for moving a site between EmDash installations without copying a database or depending on a particular storage provider.

The format would represent EmDash concepts rather than database tables. It would carry the content model, editorial content and history, relationships, site presentation data, and media bytes while deliberately excluding users, credentials, secrets, and deployment state. Import would be planned and validated before it writes, run as a resumable operation, and finish with a receipt that identifies the exact artifact and what was verified on the target.

The same core engine should support three entry points:

• an Export / Import site workflow in the admin;
• CLI, authenticated API, and MCP operations for operators and automation; and
• hosting-provider workflows such as moving an anonymously authored site into a stable production EmDash instance.

This is a portability and migration feature. It is not database disaster recovery, continuous synchronization, or a replacement for raw database and media backups.

Motivation

EmDash already has several related mechanisms, but none is a safe, complete site-transfer contract:

• Seed files describe initial schema and sample content. They are intentionally declarative bootstrap inputs, not lossless site snapshots.
• emdash export-seed can turn some schema and content into a seed, but it does not preserve the complete editorial state or media binaries.
• The preview snapshot and JSON backup export database-shaped tables for preview and inspection. EmDash explicitly cannot restore those archives.
• Raw SQLite, PostgreSQL, or D1 backups are correct for recovery into the same deployment shape, but they are backend-specific and include state that must not cross a trust boundary.
• The WordPress importers prove the value of analyze-before-import and bounded execution, but they are source-specific and do not define EmDash-to-EmDash portability.

This leaves several product needs without a supported path:

1. Move an EmDash site between SQLite, PostgreSQL, and D1 deployments.
2. Let a hosting provider give a customer a portable copy of their site.
3. Clone a site into a newly provisioned environment without cloning users, tokens, or secrets.
4. Move authoring state from a disposable development environment into stable production storage, then prove that the production copy is complete before making it live.
5. Build future environment-diff and reviewed-promotion tools on one logical representation rather than reverse-engineering storage tables.

Without a core contract, each host has to choose between incomplete API replay, unsafe raw database transfer, or its own private serialization format.

Goals

• Define a versioned, storage-independent representation of a portable EmDash site.
• Preserve stable logical identities and relationships without relying on mutable slugs.
• Include the media bytes required to render the transferred site.
• Exclude authentication, credentials, secrets, and environment-local runtime state by construction.
• Validate the whole artifact and produce a reviewable import plan before any target mutation.
• Make imports bounded, idempotent, resumable, and observable on Workers as well as Node deployments.
• Verify the imported result and return a machine-readable receipt tied to the artifact digest.
• Expose one core implementation through the admin, CLI, authenticated APIs, and MCP.
• Keep the artifact useful to third-party hosting providers without making Cloudflare concepts part of the format.

Non-goals

• A raw database backup or point-in-time disaster-recovery format.
• Bidirectional or continuous synchronization between two live sites.
• Automatically merging arbitrary changes into a non-empty production site.
• Silently applying destructive schema changes.
• Moving user accounts, sessions, passkeys, OAuth state, API tokens, encryption keys, or other credentials.
• Moving generated application source, a Worker bundle, or deployment configuration. Those are separate release artifacts.
• Guaranteeing portability for arbitrary plugin-owned state in the first version.
• Making a multi-record import globally atomic. The contract instead makes an incomplete import non-live, inspectable, and safely resumable.

Terminology

• Site package: the portable artifact produced by core.
• Origin: the EmDash site that produced a package.
• Target: the EmDash site receiving it.
• Analyze: a read-only validation and compatibility pass.
• Import plan: the exact proposed creates, mappings, warnings, and blockers for a package and target.
• Import operation: the durable, resumable execution of an accepted plan.
• Import receipt: the signed or digest-verifiable result of a completed import.
• Portable identity: an entity identity from the origin namespace, distinct from any target database row ID.

Proposal

1. Core owns the logical contract

The site-package format and its exporter, validator, importer, and verifier should live in EmDash core. Consumers may store or transport the bytes, but must not need to understand EmDash tables or reconstruct schema semantics.

The contract should expose opaque artifact metadata to hosts:

interface SitePackageReference {
	kind: "emdash-site-package";
	formatVersion: string;
	digest: `sha256:${string}`;
	byteLength: number;
}

A host should be able to ask core to export, analyze, import, resume, and verify an artifact while treating its internal records as opaque.

2. Use a logical, checksummed package

The outer package should be a streamable archive with a small manifest, logical record streams, and content-addressed media. The exact archive container can be settled during implementation; the normative contract is the manifest and logical record schemas, not ZIP paths or database tables.

An illustrative package is:

manifest.json
records/schema.ndjson
records/content.ndjson
records/editorial.ndjson
records/navigation.ndjson
records/settings.ndjson
media/<sha256>
checksums.json

The manifest should include at least:

{
	"format": "emdash-site-package",
	"formatVersion": "1",
	"packageId": "...",
	"originSiteId": "...",
	"createdAt": "...",
	"createdByEmDashVersion": "...",
	"profile": "full-transfer",
	"features": ["revisions", "scheduling", "media"],
	"recordCounts": {},
	"media": {
		"count": 0,
		"totalBytes": 0
	},
	"checksums": {}
}

This example is intentionally not a final wire schema. The implementation RFC must define canonical serialization so the package digest is reproducible and every accepted format version has an executable conformance suite.

3. Define the full-transfer profile

The initial profile should prioritize complete migration into a new target, not selective export. It should include portable core-owned state for:

• collection, field, locale, block-type, and taxonomy definitions;
• content entries, including live and draft state, scheduling, trash state, revision history, and translation groups;
• references and other content relationships;
• taxonomy terms and assignments;
• bylines and presentation credits;
• menus and menu items;
• sections, widget areas, and widgets;
• SEO records and redirects;
• portable site settings such as title, tagline, locale, display preferences, social profiles, and SEO defaults;
• media metadata, variants where required by the logical model, and the source media bytes; and
• comments when the target supports them.

The package must not contain:

• users, sessions, credentials, passkeys, invitations, or authorization state;
• API tokens, OAuth clients, grants, codes, or tokens;
• encryption keys or encrypted plugin credentials;
• setup-complete flags, preview signing secrets, rate limits, edit locks, jobs, leases, caches, or migration locks;
• deployment URLs, storage keys, bucket names, database names, binding names, or provider credentials;
• audit logs unless a later, separately permissioned profile defines their privacy and retention semantics; or
• arbitrary plugin tables or plugin settings.

Target-local values such as the canonical site URL must be preserved or recomputed according to an explicit target policy, never copied accidentally because they happened to share an option prefix.

4. Make media self-contained and content-addressed

Media metadata without media bytes is not a complete transfer. Local media in the package should refer to a blob digest rather than an origin storage key or URL. The importer should:

1. validate each declared size and digest;
2. deduplicate identical bytes;
3. write through the target's configured storage adapter;
4. create target media records and variants according to target capabilities;
5. rewrite logical content references through the identity map; and
6. report any missing, corrupt, or unsupported asset before verification.

Strict mode should be the default for a full transfer: missing referenced media blocks a verified receipt. A future selective profile may allow explicitly accepted omissions.

Large sites may need a manifest plus separately uploaded, content-addressed chunks. That transport optimization must not change the logical package or verification semantics.

5. Separate portable identity from target row identity

Slugs are mutable and cannot identify content across imports. Every exported entity should carry a portable identity scoped to the origin site. References inside the package use those identities.

The target should keep a durable mapping similar to:

(origin site, entity kind, portable ID) -> target ID

The importer may preserve an ID when safe, but consumers must not depend on that. The mapping provides deterministic relationship resolution and makes a retry of the same operation idempotent without matching by slug.

An origin site ID is a namespace, not proof of authority. Import authorization comes from the target operation, not from a value inside an untrusted package.

6. Analyze before mutation

Uploading or selecting a package first creates an analysis operation. Analysis must not mutate site data. It should verify:

• archive structure, canonical manifest, supported format and declared features;
• digests, sizes, record counts, limits, and media references;
• logical record schemas and cross-record references;
• required target EmDash version and capabilities;
• field and block-type compatibility;
• target emptiness and conflicting portable identities;
• settings that require target-local decisions; and
• plugin extension requirements, if extensions are present.

The resulting plan should have a stable digest and report:

• records and bytes to create;
• identity mappings to establish;
• compatible transformations that will be applied;
• warnings that require acknowledgement;
• blockers that prevent import; and
• an estimate of the bounded work batches.

Execution accepts the package digest and plan digest. If either changes, the caller must analyze again.

7. Scope v1 import to a new or empty target

The first version should import only into a freshly initialized target with no user-created schema or content. Target-local admin users may exist because auth state is outside the transfer, but the portable CMS domain must be empty.

This gives useful site migration and hosting-provider publication without pretending that arbitrary merge semantics are safe. It also lets incomplete imports remain unreachable until they verify.

A later proposal can add import into a non-empty target. That work needs a real diff model, _rev or equivalent conflict checks, backups, additive-by-default schema policy, explicit delete handling, and durable source-to-target baselines. It should be described as reviewed promotion or merge, not generic sync.

8. Execute as a durable, resumable operation

Import must be decomposed into bounded stages that fit the target runtime. A representative order is:

1. reserve the package and operation idempotency key;
2. establish portable-identity mappings;
3. create compatible schema and block definitions;
4. ingest and verify media;
5. create content and revisions;
6. resolve relationships, translations, taxonomies, and media references;
7. import navigation, presentation records, comments, and settings;
8. rebuild derived indexes and usage data; and
9. verify the complete target state.

Progress must be stored server-side. Retrying the same operation or batch must not duplicate entities or lose mappings. A browser or CLI disconnect must not lose the resume position.

Only one transfer operation may mutate a target at a time. Normal CMS writes should remain disabled until a new-site import completes or is explicitly abandoned.

9. Verify and return an import receipt

Success means more than every write request returning 2xx. Core should read the target's logical state back and verify counts, identities, relationships, publication pointers, and media digests against the accepted package.

The completed operation should return a receipt such as:

interface SiteImportReceipt {
	operationId: string;
	packageDigest: `sha256:${string}`;
	planDigest: `sha256:${string}`;
	targetSiteId: string;
	formatVersion: string;
	importerEmDashVersion: string;
	completedAt: string;
	logicalDigest: `sha256:${string}`;
	counts: Record<string, number>;
	warnings: Array<{ code: string; message: string }>;
	verification: "verified";
}

The exact receipt schema is part of the core contract. A hosting control plane can require a receipt whose package digest and target site match the pending publication before it promotes a release.

An operation with unresolved omissions or verification failures is not successful and must not produce a verified receipt.

10. Expose one engine through admin, CLI, API, and MCP

The authenticated API is the canonical operation surface. The admin, CLI, and MCP tools should be policy-aware clients of it, not separate implementations with different transfer semantics.

Admin

Add a Transfer area rather than presenting import as restore under Backups. The distinction matters: backups are operational recovery points; site packages are logical and portable.

The admin workflow should be:

Export site
-> choose full-transfer profile
-> review exclusions and estimated media size
-> create and download package

Import site
-> upload or select package
-> analyze without writing
-> review source, counts, target decisions, warnings, and blockers
-> explicitly confirm
-> follow durable progress
-> receive verification report

The import action should appear only where an empty-target import is valid. A future setup-wizard entry point could offer Create from EmDash package.

CLI

The CLI should use the same operations, for example:

emdash site export --output site.emdash
emdash site import site.emdash --analyze
emdash site import site.emdash --plan <digest> --confirm
emdash site import status <operation-id>
emdash site import resume <operation-id>

Names are illustrative. Machine-readable JSON output is required for provider automation. A local command must not bypass the core validator or importer by writing tables directly.

API

Authenticated APIs should support export creation, artifact upload or selection, analysis, confirmed execution, status, resume, and receipt lookup. Long-running calls should return operation IDs rather than holding one request open.

Permissions should distinguish export, import analysis, and import execution. Bearer-token scopes must not make a browser admin's broad privileges the only automation option.

MCP

MCP should expose transfer as asynchronous, coarse-grained tools rather than streaming archive bytes through model context. A possible surface is:

site_transfer_capabilities
site_export_start
site_export_status
site_import_analyze
site_import_start
site_import_status
site_import_resume
site_import_receipt

Artifacts are passed as opaque, server-side references. Status and receipt tools return bounded summaries; package contents, media bytes, credentials, and signed download URLs should not be placed in model context.

The tools need separate scopes for export, analysis, and execution. Export can exfiltrate an entire site's content and media, so it must not be implied by an ordinary content-read grant. Import execution mutates the target, so it must not be implied by schema or content write access.

For interactive agent use, site_export_start should also require an explicit one-time approval or OAuth consent that names the site and export profile. An MCP client must not be able to mint a reusable download capability merely because it can read individual content entries.

For interactive agent use, site_import_start should require a short-lived approval grant bound to all of:

• the target site;
• the package digest;
• the analyzed plan digest; and
• the authenticated principal.

The approval is created through an explicit admin or host-control-plane action, not by another MCP tool available under the same grant. A server-to-server MCP client may execute without a human gesture only when it has been deliberately issued the import-execution scope.

MCP import is still not publication. In a hosting product, a verified import receipt may satisfy one precondition for a later publish action, but the model must not be able to promote a release merely because it can call transfer tools. EmDash Build should invoke production transfer from its trusted control plane, not add these production tools to the site-building agent.

Plugin-owned data

Arbitrary plugin tables and settings should be excluded in v1. Copying them would leak secrets and bind the package to private database layouts.

A future extension mechanism can let a plugin register versioned logical export/import handlers. Each extension would declare:

• plugin identity and extension format version;
• whether it is required or optional for a valid site;
• required target plugin and compatibility range;
• whether its records contain personal or sensitive data; and
• its own validation, import, and verification behavior.

Unknown required extensions should block analysis. Unknown optional extensions may be skipped only with an explicit warning recorded in the plan and receipt. Plugin secrets remain excluded even when an extension exists.

Security and privacy requirements

Treat every package as untrusted input, including one exported by another EmDash installation.

• Enforce compressed and expanded byte limits, record-count limits, nesting limits, and per-record limits before expensive work.
• Reject path traversal, duplicate paths, unsafe archive links, digest mismatches, undeclared files, and decompression bombs.
• Parse logical records through strict schemas; never execute SQL or code from the package.
• Reject dangling or type-invalid relationships before execution.
• Keep staged packages private and subject to retention and deletion policy.
• Do not fetch arbitrary origin URLs during import. Media must be in the accepted package or an explicitly authorized companion transport.
• Redact sensitive record values from operation logs and error telemetry.
• Require authorization again when accepting a plan for execution; possession of an artifact or operation ID is not authorization.
• Record who initiated export and import in the target audit system without placing target user identity in the portable artifact.
• Never regard a package digest as a signature. If provenance is required, add a separate signature envelope and trust policy.

Failure, retry, and cancellation semantics

• Analysis failures do not mutate the target.
• Execution records the last completed durable stage and batch.
• A retry with the same idempotency key resumes the same operation.
• A retry with different bytes under the same package identity is rejected.
• Cancellation stops after the current bounded batch and leaves the target marked incomplete. It does not claim to roll back already committed work.
• The target cannot serve the imported site or accept ordinary writes until verification succeeds.
• Abandoning an incomplete new-site import may delete that target through a separately authorized lifecycle operation; import itself should not perform an unbounded destructive rollback.
• Errors use stable machine-readable codes and distinguish retryable transport failures from compatibility or validation blockers.

Versioning and compatibility

The site-package format version is independent from the EmDash package version. Core should publish:

• the logical schemas for each supported format version;
• a feature/capability registry;
• an exporter and importer support matrix;
• canonical digest rules; and
• cross-database conformance fixtures that export on one adapter and import on another.

Readers must reject unknown required features. Additive optional fields may be ignored only when their enclosing schema explicitly permits it. A migration between format versions should be a pure, testable transformation before target mutation.

Relationship to backups, seeds, and previews

Mechanism	Purpose	Importable	Media bytes	Auth/secrets
Seed	Bootstrap a model and sample data	Yes, with seed semantics	No	No
Preview snapshot	Populate isolated rendering state	Preview-only	No	No
JSON backup	Inspect selected database-shaped state	No	No	No
Raw DB + media backup	Disaster recovery for one deployment	Operational restore	Separate	May contain sensitive state
Site package	Logical EmDash-to-EmDash transfer	Yes, through this proposal	Yes	No

The existing JSON backup should not silently change meaning. It can share low-level readers with the exporter where appropriate, but a site package needs its own format name, validation rules, compatibility guarantees, and UI.

Alternatives considered

Restore the current JSON backup

The backup is database-shaped, deliberately incomplete, lacks media bytes, and is documented as non-restorable. Turning it into a public import contract would couple portability to internal tables and retroactively change its safety promise.

Use seed files

Seeds are excellent for templates and reproducible initial data. They omit important site state and do not define resumable migration, conflict, verification, or complete media semantics.

Copy raw SQLite or D1 state

This is fast for a controlled recovery but backend-specific, tightly coupled to migrations, and liable to transfer users, credentials, locks, and runtime state. It also cannot support SQLite-to-PostgreSQL portability.

Replay existing content REST or MCP operations from a host

Public APIs are useful building blocks but do not provide a consistent export, stable portable identity, complete relationship ordering, media packaging, durable resume state, or a verified receipt. Every host would implement a different partial importer.

Let each hosting provider define a format

This creates lock-in and makes core schema evolution unsafe for hosts. Hosts should own transport and deployment orchestration; core should own CMS serialization and compatibility.

Proposed delivery sequence

1. Contract and fixtures: define the logical entities, exclusions, canonical digests, capability registry, conformance packages, and security limits.
2. Exporter and validator: stream a full-transfer package with media and validate it independently of import.
3. Empty-target importer: analyze, plan, execute in durable batches, keep identity mappings, rebuild derived data, and issue a verified receipt.
4. CLI, API, and MCP: expose the same engine with operation IDs, bounded tool responses, explicit scopes, and JSON output.
5. Admin Transfer UI: export, analyze, review, confirm, monitor, and show the receipt without conflating transfer with backup restore.
6. Cross-adapter qualification: test SQLite, PostgreSQL, and D1 in every supported source/target direction at meaningful scale and under retries.
7. Later RFCs: plugin extensions, selective packages, signatures, and reviewed import into non-empty targets.

Acceptance criteria for v1

• A representative site can export from each supported database adapter and import into an empty target on every other supported adapter.
• The transferred public pages and admin-visible content retain schema, publication state, revisions, translations, relationships, navigation, settings, and media.
• No user, token, credential, secret, environment binding, or origin storage key appears in the package or target import.
• A corrupt record, dangling relationship, unsupported required feature, or missing media blob is detected before a verified receipt is issued.
• Killing the client or runtime during every import stage can be retried without duplicates or lost mappings.
• Replaying a completed operation returns the same result rather than importing a second copy.
• The admin, CLI, API, and MCP produce equivalent plans and receipts because they use the same core engine.
• Import verification catches a case where metadata exists but a referenced body image is absent.
• The existing backup and seed behavior remains backwards compatible and is still documented for its original purpose.

Open questions

1. Should comments be required in the v1 full-transfer profile or advertised as an optional feature when the target supports them?
2. Should trash history be included by default, or should v1 preserve only active, draft, scheduled, and revision state?
3. What is the smallest consistent-read primitive the exporter needs across SQLite, PostgreSQL, and D1 while writes continue?
4. Should the first transport be one archive or a manifest plus independently uploaded media chunks from the start?
5. Which target-local settings need an explicit decision in the import plan rather than a fixed preserve/reset rule?
6. Should import receipts be core-signed, host-signed, or digest-verifiable only in v1?
7. Where should import-from-package appear during first-run setup, given that a target still needs a new local administrator?

Related discussions

• #190: Migrations/Exporting structure as config proposes schema/config portability but does not cover a complete site package.
• #412: Native Ghost CMS import concerns a source-specific migration path.
• #285: WXR admin import date preservation captures correctness requirements from the existing WordPress importer.
• #2483: Draft revisions for imported/synchronized content is relevant to preserving editorial and promotion semantics.
• #308: Managed hosting for EmDash provides hosting context but does not define a transfer format.