Store some fields in CloudKit unencrypted

[finnvoor]

Hi, sqlite-data seems to encrypt every value stored on CloudKit-synced CKRecords at the moment. CloudKit doesn’t support indexes on encrypted fields, so I can't query the CloudKit database using any predicates or sort descriptors (i.e. sort by addedAt date). It would be nice if some fields could be marked as unencrypted for use when querying the CloudKit database outside of sqlite-data.

[stephencelis]

@finnvoor We could definitely support something like:

try SyncEngine(
  for: $0.defaultDatabase,
  tables: RemindersList.self, Reminder.self,
  unencryptedFields: RemindersList.title, Reminder.title
)

And maybe even a flipped initializer for encryptedField. Folks would just need to take care when adding this configuration to an existing deployed app.

Can you explain a bit more about your use case, though? What kind of CloudKit querying do you need to do that can't be done using SQLiteData's tools?

[finnvoor]

In this case I'm interested in creating a web based version to view data using CloudKit js alongside an app, similar to Apple Notes/Reminders.

[mahee96]

@stephencelis may I know if this is already available in any released version or is planned, Thanks.

[mbrandonw]

@mahee96 This is not released or currently planned. There is more discussion below that you can follow.

[pfandrade]

I believe CKQuery based subscriptions also do not work with encrypted fields. That is, you can't have a predicate on your query referencing an encrypted field.

[lukaskubanek]

While I don’t have a concrete use case for configuring fields as encrypted or unencrypted either (aside from the CloudKit bug described in #315, which might be related to encrypted fields used by default in SQLiteData), the idea of dedicated per-field macros for configuring sync-engine behavior sounds very interesting.

I recently sketched out an API for conflict handling, where I introduced macros to assemble a merge function from the merge policies specified for individual fields (approach D). I’ve also struggled with @Table being too generic and not extendable, which is why I ended up with a secondary type-level macro, @MergeableTable.

Seeing this discussion now, I think it’s worth exploring whether the field annotations for name, encryption, and merge policy could be unified into a single field-level macro. That would still require a type-level macro to generate those field-level macros when not specified explicitly.

In addition to @Table and @Column, there could be a secondary macro pair for configuring the sync behavior of a table, such as @SyncConfigured and @SyncFieldConfiguration.

It might also make sense to make the notion of a synced table explicit in SQLiteData. One approach would be switching the type-level macro from @Table to something like @SyncedTable, which would keep the existing @Table logic and additionally generate field-level macros such as @SyncedField. On the other hand, this might be too big of a change for the public API.

In the API ideation for conflict handling I also considered specifying merge policies in SyncEngine.init (approach B), but that felt too detached, especially if the goal is to support more per-field configuration options like those outlined in this thread.

Edit: The type-level macro @SyncedTable could theoretically be extended with the flag for indicating shareable vs. private tables in order to treat all CloudKit configuration in the same way.

[pfandrade]

Honestly, I haven't created macros yet. I just assumed @SyncEngine could add some static methods on that type that could be consulted by the SyncEngine at runtime. I may be completely off here.

Either way, the @SyncedTable suggestion from @lukaskubanek does sound good. If we're considering configuring field encryption via a macro on the type itself, might as well consider configuring private/shared tables as well.

[lukaskubanek]

Honestly, I haven't created macros yet. I just assumed @SyncEngine could add some static methods on that type that could be consulted by the SyncEngine at runtime.

Neither do I have hands-on experience with implementing macros, but I think I get the general principle. And generating some sort of a type-level registry was also the idea I had in mind.

However, thinking about this a bit more, this likely requires additional guardrails, as the configuration cannot be changed freely once the CloudKit schema is deployed (or data has been written in the development schema).

Changing the custom field name must be disallowed, similar to how regular column renames are disallowed in CloudKit context.

The same applies to the encryption flag for fields, which must also be treated as immutable for the same reason. Moreover, CloudKit would even crash when encountering records that attempt to store a value as both encrypted and unencrypted. See this simple local example:

let record = CKRecord(recordType: "MyRecord")
record.encryptedValues["field"] = 42
record["field"] = 1
// Terminating app due to uncaught exception 'NSInvalidArgumentException', 
// reason: 'You cannot set the same key field on both CKRecord and -[CKRecord encryptedValues]'

This means that both custom field names and encryption settings need to be treated as immutable once locked in. We would likely need to cache them to detect changes, rather than relying directly on the configuration produced by the macro.

This doesn’t apply to the merge policy, though, since conflict resolution is entirely a matter of the client and it doesn’t affect the server-side schema.

[lukaskubanek]

See also the follow-up in #336.

[lukaskubanek]

See follow-up discussion on the macro-based pattern in #352.

[jaanussiim]

Just wanted to add my use-case for making some fields queriable - initial sync on clean install.

Use cases:

• 60k poker hands over 1.5 years - make date queriable.
• 10k meals in calories consumed app - make date queriable.
• 15k day transactions summary in budget app - make date queriable

In all these case, on clean install, first pull latest data for x days. Then let the sync engine do it's thing.

I'm assuming, that when filling sqlite with custom pulled data, as they have id's from iCloud, sync engine will later figure it out, that these are the same record.

[mbrandonw]

Hi @jaanussiim, sorry but I don't understand the situation you are describing. Are you saying you want to perform queries directly on the CKDatabase from the client? If so, then you will only be getting back CKRecords, which is basically a [String: Any] dictionary. You won't be able to turn that into your models that are backed by SQLite.

[jaanussiim]

Yes, for targeted sync I would query iCloud directly for raw CKRecord objects. From these I would pick out values for Model.Draft initialization and save them to SQLite database.

This is done before SyncEngine has started and has nothing to do with SQLiteData library.

[mbrandonw]

That is a pretty dangerous thing to do. Manually synchronizing records outside the purview of the sync engine is bound to cause some problems. For one thing, you have a higher risk of not properly having all associations loaded and so will get foreign key constraints. And there's a lot of manual work required to turn a CKRecord into a statically typed model, and it will be easy to get those steps wrong.

But, even if you really do want to do this, it seems possible without us moving fields to unencrypted. Wouldn't you want to just query for all records that have been edited in the past X days? I believe that is possible regardless of how other fields are stored in the CKRecord because CloudKit tracks a creationDate field for you.

[ArchimedesTrajano]

My case is that I'm sharing a record with someone. When I delete or create a record, I want the other user to be notified as:

"\(title) is deleted"
"\(title) is created"

I can't recover the record from the database because, when a delete notification is received, the record has already been deleted. Similarly, for a create notification, the record may not have been fetched yet when the notification is processed.

The only way to include the title in the notification itself is to add it to desiredKeys. However, since title is an Encrypted String, I cannot add it to desiredKeys in the subscription.

It would be nice to at least have an unencrypted version available when needed for cases like this, for example:

Field	Type
title	Encrypted String
__title	String

[mbrandonw]

The only way to include the title in the notification itself is to add it to desiredKeys. However, since title is an Encrypted String, I cannot add it to desiredKeys in the subscription.

Is this documented somewhere? That you can't use encrypted keys in desiredKeys?

[ArchimedesTrajano]

It says here:

CloudKit doesn’t support indexes on encrypted fields. Don’t include encrypted fields in your predicate or sort descriptors when fetching records with CKQuery and CKQueryOperation.

and later on:

CloudKit encrypts the fields’ values on-device before saving them to iCloud, and decrypts the values only after fetching them from the server.

It throws an error when you try to subscribe to such a field: unsupported field type in notification.

This also seems relevant:

The encrypted fields can’t have indexes because the server can’t read the fields.

[mbrandonw]

I'm not familiar with how CloudKit notifications work, but are you sure you can get desiredKeys for a record that has been deleted? How does moving fields to non-encrypted help with this?

[ArchimedesTrajano]

Yes, there is a dedicated option for the .firesOnRecordDeletion subscription, which triggers the notification when the record is deleted. At that exact moment, CloudKit server still has access to the record, but since the encryption seems to happen on-device, the ck server cannot include encrypted fields in the notification.

So, if you subscribe to title, which is an encrypted field, you immediately encounter this error and the subscription fails:

unsupported field type in notification

However, if you subscribe to __title, which is unencrypted and uses one of the supported types, such as String, the subscription succeeds, and the value is available in the notification’s userInfo payload under the recordFields key.