On-demand CKAsset downloads for asset-heavy apps

[yunhao]

SQLiteData is great, especially its CloudKit sync and sharing support. I’m wondering whether it could offer more flexibility around CKAsset for apps with large amounts of binary data.

Use case

Consider a Notes-like app:

• Users can create folders and notes, and attach many images, PDFs, audio files, or videos.
• Data syncs through CloudKit.
• A folder can be shared, with participants collaborating on all notes and attachments under it.

The problem

This works well at small scale, but becomes problematic when attachments grow from hundreds of MB to many GB. Every device may end up storing a full local copy, including attachments from old notes that are rarely opened.

Discussion #130 already touches on external storage, but I think there are two separate concerns:

• External storage keeps large binaries out of SQLite.
• On-demand CKAsset fetching avoids keeping every binary downloaded on every device.

Desired behavior

Ideally, an asset-heavy app could:

• Store attachment metadata and local file references in SQLite.
• Store the binary in CloudKit as a CKAsset.
• Sync metadata normally.
• Download the asset only when needed, and evict/re-download it later if necessary.
• Treat downloaded assets as a local cache, so the app can offer a storage cleanup option that deletes re-downloadable files without losing data.
• Keep the attachment record inside the normal sharing hierarchy.

The main limitation seems to be CKSyncEngine: its fetch API does not expose field-level filtering like CKFetchRecordZoneChangesOperation.ZoneConfiguration.desiredKeys, so it does not seem possible to sync the record metadata while intentionally leaving the asset field unfetched.

Question

Would SQLiteData be interested in supporting this kind of asset model as a first-class feature?

Alternatively, could SyncEngine expose hooks that let applications manage CKAsset fetching themselves, while SQLiteData continues to handle metadata sync, hierarchy, conflict resolution, and CloudKit Sharing?

[mbrandonw]

Hi @yunhao, thanks for starting up this discussion, and we definitely agree that certain apps out there would benefit from this kind of system. But, as you mention, it does not exactly play nicely with CKSyncEngine, which wants to sync everything.

And given that this library is so focused on interfacing with CKSyncEngine, we would first like to explore if there is some way for people to support this outside of the library. After all, you have full access to SQLite, which means you are free to create triggers to listen to things happening inside the database.

So is there a chance you could build some kind of AssetSyncEngine that takes inspiration from our SyncEngine, and whose whole responsibility is to synchronize assets on demand? The StructuredQueries library that powers SQLiteData exposes all of the static information for tables and so I'm willing to bet some really nice tools could be built.

If you ever come across something that is internal/private to SQLiteData/StructuredQueries that you need, just let us know and we can determine if there is a workaround, or if we should make that API public.

And for reference, there is already someone who has built an undo system on top of SQLiteData, and it follows the pattern I described above. A new object for managing undo along with triggers for listening to changes in order to push data to an undo stack.