Skip to content

Commit eb21a7b

Browse files
ericmigiclaude
andcommitted
Add full formatting display + WYSIWYG editing
- Decode all paragraph + inline attribute_run fields (corrects README's earlier mapping: 0=Title not Body, 103=Checkbox not 101). Adds DASHED_LIST style, font_weight/underline/strike/link/attachment fields. - MergeableDataDecoder for table CKAttachments — walks the CRDT graph (KeyItems, TypeItems, GraphObjects: List/Dict/String/CustomMap/ OrderedSet) and extracts a real 2D grid via cellColumns iteration. - Image attachments fetched via Media CKReference → Asset downloadURL. - FormattedNoteBody renders styled paragraphs + AnnotatedString inline, table grids, inline images, attachment placeholder cards, clickable links via LinkAnnotation, and tap-to-toggle checkboxes (auto-saves). - Edit mode is WYSIWYG via VisualTransformation: heading sizes, inline bold/italic/underline/strike/link styling, list markers (• – 1. ☐ ☑) visible while editing, with OffsetMapping keeping cursor/selection aligned to the underlying chars. - Formatting toolbar (Material 3, animated) with paragraph style picker, list style toggles, B/I/U/S, and link dialog. Save path threads new AttrSpan list through NoteAppender; format-only changes use a rewriteAttributeRunsOnly fast path that preserves the substring tree. - 7 JVM unit tests round-trip a real iCloud-pulled fmt-baseline-1.proto through encoder/decoder, asserting bold/heading/checkbox edits survive. - GitHub Action builds debug APK on push/PR/dispatch and uploads it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 712e294 commit eb21a7b

12 files changed

Lines changed: 2290 additions & 124 deletions

File tree

.github/workflows/build-apk.yml

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
name: Build APK
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
branches: [main]
8+
workflow_dispatch:
9+
10+
jobs:
11+
build:
12+
runs-on: ubuntu-latest
13+
steps:
14+
- uses: actions/checkout@v4
15+
16+
- uses: actions/setup-java@v4
17+
with:
18+
distribution: temurin
19+
java-version: '17'
20+
21+
- uses: gradle/actions/setup-gradle@v4
22+
23+
- name: Build debug APK
24+
run: ./gradlew :app:assembleDebug --no-daemon
25+
26+
- name: Upload debug APK
27+
uses: actions/upload-artifact@v4
28+
with:
29+
name: app-debug
30+
path: app/build/outputs/apk/debug/app-debug.apk
31+
if-no-files-found: error

README.md

Lines changed: 66 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -21,9 +21,12 @@ report for the next person (or agent) who picks it up.
2121
| Auto-save on lifecycle pause | Works (lifecycle `ON_PAUSE` only — see [AppleNotesApp.kt:1052](app/src/main/java/com/example/applenotes/ui/AppleNotesApp.kt)) |
2222
| Share OUT (Android intent chooser) | Works (plaintext only) |
2323
| Delete | Uses CloudKit `forceDelete`; Mac surfaces as "Recently Deleted" (server-managed, not a hard wipe) |
24-
| Format display (headings, lists, checkboxes) | Decoded but not rendered yet (Phase D1 done; D2 pending) |
25-
| Format input toolbar | Not built |
26-
| Image attachments | Renders `` stub; no image display |
24+
| Format display (paragraph styles + inline) | Works — Title / Heading / Subheading / Body / Monospaced; Bulleted / Dashed / Numbered / Checkbox; bold / italic / underline / strikethrough; clickable links |
25+
| Table attachments | Decoded from `MergeableDataEncrypted` (CRDT graph), rendered as a real grid |
26+
| Image attachments | Fetched as CKAsset bytes from the Media reference, rendered inline |
27+
| Other attachments (sketches, maps, etc.) | Rendered as a labeled placeholder card |
28+
| Format input toolbar | Works — paragraph style picker, list toggles (bullet/dash/numbered/checkbox), inline B/I/U/S, link |
29+
| Tap-to-toggle checkbox | Works in formatted view (auto-saves) |
2730
| Conflict UX | Auto-retry on CONFLICT/oplock; no concurrent-edit merge yet |
2831

2932
## The most important thing we think we learned (not yet end-to-end confirmed)
@@ -163,22 +166,73 @@ of Mac trashing — we ruled this out by sampling.
163166
- `` (U+FFFC OBJECT REPLACEMENT) is Apple's placeholder for inline attachments.
164167
We pass through but don't render the attachment.
165168

166-
### Paragraph styles (in `AttributeRun.f2.f3`)
169+
### Paragraph styles (`AttributeRun.f2.f1`)
167170

168171
| Code | Style |
169172
|---|---|
170-
| 0 | Body |
171-
| 1 | Title |
172-
| 2 | Heading |
173-
| 3 | Subheading |
173+
| 0 | Title (also: implicit for the first line) |
174+
| 1 | Heading |
175+
| 2 | Subheading |
176+
| 3 | Body (also: f1 absent → Body) |
174177
| 4 | Monospaced |
175178
| 100 | Bulleted list |
176-
| 101 | Checkbox (with `f4=1` if checked) |
179+
| 101 | Dashed list |
177180
| 102 | Numbered list |
181+
| 103 | Checkbox list |
178182

179-
`f9` inside `f2` is a 16-byte paragraph UUID (Mac uses these; iCloud.com doesn't,
180-
yet both round-trip). Decode via
181-
[`NoteBodyEditor.parseAttributeRuns`](app/src/main/java/com/example/applenotes/proto/NoteBodyEditor.kt).
183+
(The earlier draft of this README listed 0=Body / 1=Title / 101=Checkbox. Wrong:
184+
verified by sampling a baseline note that contained one of every style.)
185+
186+
Checkbox checked-state lives at `AttributeRun.f2.f5` — a 20-byte sub-message
187+
containing `{f1: 16-byte UUID, f2: 0|1 (done flag)}`, NOT in `AttributeRun.f2.f4`.
188+
189+
### Inline formatting (siblings of `AttributeRun.f2`)
190+
191+
| Field | Meaning |
192+
|---|---|
193+
| `f5` (varint) | font weight enum: 1=Bold, 2=Italic, 3=BoldItalic |
194+
| `f6` (varint) | underlined (bool) |
195+
| `f7` (varint) | strikethrough (bool) |
196+
| `f9` (bytes) | link URL (UTF-8 string) |
197+
| `f12` (msg) | attachment_info `{f1: UUID-string, f2: UTI-string}` (table/image/etc.) |
198+
199+
Decode via [`NoteBodyEditor.parseAttributeRuns`](app/src/main/java/com/example/applenotes/proto/NoteBodyEditor.kt).
200+
Re-encode via [`NoteBodyEditor.encodeAttributeRunField`](app/src/main/java/com/example/applenotes/proto/NoteBodyEditor.kt).
201+
202+
### Table attachments
203+
204+
Tables are stored as separate `Attachment` CKRecords (UTI
205+
`com.apple.notes.table`). Their content lives in `MergeableDataEncrypted` — a
206+
zlib-deflate'd protobuf encoding Apple's MergeableData CRDT graph. The graph
207+
nodes form a typed object soup:
208+
209+
- **KeyItems** (field 4): string field names — `self`, `crRows`, `crColumns`,
210+
`cellColumns`, `crTableColumnDirection`, `identity`, `UUIDIndex`.
211+
- **TypeItems** (field 5): string type names — `com.apple.CRDT.NSString`,
212+
`com.apple.CRDT.NSUUID`, `com.apple.notes.ICTable`, etc. Indexed.
213+
- **UUIDItems** (field 6): 16-byte UUIDs. Indexed.
214+
- **GraphObjects** (field 3): repeated, indexed. Each is one of:
215+
- `f1` List, `f6` Dictionary, `f10` String (with f2 = current text),
216+
`f13` CustomMap (typed map, like ICTable), `f16` OrderedSet.
217+
218+
The root ICTable's CustomMap has entries `crRows` (OrderedSet of row UUIDs),
219+
`crColumns` (OrderedSet of column UUIDs), and `cellColumns` (Dict of
220+
col_uuid → Dict of row_uuid → string_obj_id). Walk those to extract a 2D
221+
grid; resolve each string_obj_id to its NSString.currentText.
222+
223+
See [`MergeableDataDecoder.kt`](app/src/main/java/com/example/applenotes/proto/MergeableDataDecoder.kt).
224+
The decoder uses cellColumns iteration order as the column order and
225+
first-appearance order across columns as the row order — crRows/crColumns
226+
OrderedSets carry the canonical visual order via a separate UUIDIndex layer
227+
that we don't fully reconstruct yet, but the iteration order matches the
228+
visual order for tables created left-to-right top-to-bottom.
229+
230+
### Image attachments
231+
232+
Image Attachment records carry a `Media` field that's a CKReference to a
233+
separate "Media" record. Look up the media record; its `Asset` (or
234+
`MediaEncrypted`) field is a CKAsset whose `value.downloadURL` is the binary's
235+
signed download URL. GET it with the session's cookies.
182236

183237
## What we tried that failed
184238

app/build.gradle.kts

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,10 @@ android {
3737
excludes += "/META-INF/{AL2.0,LGPL2.1}"
3838
}
3939
}
40+
41+
testOptions {
42+
unitTests.isReturnDefaultValues = true
43+
}
4044
}
4145

4246
kotlin {

app/src/main/AndroidManifest.xml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,8 @@
3535
<action android:name="com.example.applenotes.APPEND_BY_TITLE" />
3636
<action android:name="com.example.applenotes.SET_BODY_BY_TITLE" />
3737
<action android:name="com.example.applenotes.DELETE_BY_TITLE" />
38+
<action android:name="com.example.applenotes.DUMP_BY_TITLE" />
39+
<action android:name="com.example.applenotes.DUMP_RECORD" />
3840
<action android:name="com.example.applenotes.CREATE" />
3941
</intent-filter>
4042
</receiver>

app/src/main/java/com/example/applenotes/client/AppleNotesClient.kt

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,11 @@ import com.example.applenotes.auth.USER_AGENT
77
import com.example.applenotes.proto.NoteBodyEditor
88
import io.ktor.client.HttpClient
99
import io.ktor.client.call.body
10+
import io.ktor.client.request.get
1011
import io.ktor.client.request.headers
1112
import io.ktor.client.request.post
1213
import io.ktor.client.request.setBody
14+
import io.ktor.client.statement.HttpResponse
1315
import io.ktor.http.ContentType
1416
import io.ktor.http.HttpHeaders
1517
import io.ktor.http.contentType
@@ -310,6 +312,24 @@ class AppleNotesClient(
310312
* Delete a note via a hard CloudKit delete operation. Mac/iOS Notes
311313
* surface this as moving the note to "Recently Deleted" (server-managed).
312314
*/
315+
/**
316+
* Download a CKAsset's binary body via its `downloadURL`. iCloud signs the
317+
* URL with the session's auth so we just GET it with our cookie jar.
318+
*/
319+
suspend fun fetchAssetBytes(downloadUrl: String): ByteArray? = runCatching {
320+
val response: HttpResponse = httpClient.get(downloadUrl) {
321+
headers {
322+
append(HttpHeaders.UserAgent, USER_AGENT)
323+
append(HttpHeaders.Cookie, session.cookieHeader)
324+
}
325+
}
326+
if (response.status.value !in 200..299) {
327+
Log.w(TAG, "fetchAssetBytes HTTP ${response.status.value} for $downloadUrl")
328+
return@runCatching null
329+
}
330+
response.body<ByteArray>()
331+
}.getOrNull()
332+
313333
suspend fun deleteNote(recordName: String, recordChangeTag: String): String {
314334
val payload: JsonObject = buildJsonObject {
315335
put("zoneID", buildJsonObject { put("zoneName", "Notes") })
@@ -657,6 +677,18 @@ internal fun extractStringValue(field: JsonElement): String? {
657677
return runCatching { value.jsonPrimitive.content }.getOrNull()
658678
}
659679

680+
/**
681+
* CKAsset fields look like
682+
* { "type": "ASSETID", "value": { "downloadURL": "...", "size": N, "fileChecksum": "..." } }
683+
* Pull `value.downloadURL`. Some download URLs come back with `${f}` placeholders
684+
* that need substitution; we surface the raw URL and let the caller request it.
685+
*/
686+
internal fun extractAssetDownloadUrl(field: JsonElement): String? {
687+
val obj = field as? JsonObject ?: return null
688+
val value = obj["value"] as? JsonObject ?: return null
689+
return runCatching { value["downloadURL"]?.jsonPrimitive?.content }.getOrNull()
690+
}
691+
660692
/**
661693
* CKReference fields look like
662694
* { "type": "REFERENCE", "value": { "recordName": "...", "action": "NONE", "zoneID": {...} } }

app/src/main/java/com/example/applenotes/debug/DebugReceiver.kt

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -153,6 +153,75 @@ class DebugReceiver : BroadcastReceiver() {
153153
}
154154
}
155155
}
156+
ACTION_DUMP_RECORD -> {
157+
val recordName = intent.getStringExtra("recordName")
158+
if (recordName == null) {
159+
Log.e(TAG, "DUMP_RECORD missing recordName")
160+
return
161+
}
162+
launchOp("DUMP_RECORD") { client, _ ->
163+
val record = client.lookupNote(recordName)
164+
Log.i(TAG, "DUMP_RECORD type=${record.recordType} tag=${record.recordChangeTag}")
165+
for ((name, _) in record.rawFields) {
166+
val s = record.stringField(name)
167+
if (s == null) continue
168+
Log.i(TAG, "FIELD $name (b64.len=${s.length})")
169+
val chunkSize = 800
170+
val chunks = (s.length + chunkSize - 1) / chunkSize
171+
for (i in 0 until chunks) {
172+
val start = i * chunkSize
173+
val end = minOf(start + chunkSize, s.length)
174+
Log.i(TAG, "FIELD_B64 $name [$i/$chunks] ${s.substring(start, end)}")
175+
}
176+
}
177+
}
178+
}
179+
ACTION_DUMP_BY_TITLE -> {
180+
val title = intent.getStringExtra("title")
181+
if (title == null) {
182+
Log.e(TAG, "DUMP_BY_TITLE missing title")
183+
return
184+
}
185+
launchOp("DUMP_BY_TITLE") { client, _ ->
186+
val match = findByTitle(client, title)
187+
if (match == null) {
188+
Log.w(TAG, "DUMP_BY_TITLE_NOT_FOUND title='$title'")
189+
} else {
190+
Log.i(TAG, "DUMP_BY_TITLE_FOUND '$title' -> ${match.recordName}")
191+
val record = client.lookupNote(match.recordName)
192+
val b64 = record.stringField("TextDataEncrypted")
193+
if (b64 == null) {
194+
Log.w(TAG, "DUMP_BY_TITLE_NO_BODY")
195+
} else {
196+
Log.i(TAG, "DUMP_KIND ${NoteBodyEditor.probeBase64(b64)}")
197+
// Log b64 in 1KB chunks so logcat doesn't truncate.
198+
val chunkSize = 800
199+
val chunks = (b64.length + chunkSize - 1) / chunkSize
200+
Log.i(TAG, "DUMP_B64_BEGIN total=${b64.length} chunks=$chunks")
201+
for (i in 0 until chunks) {
202+
val start = i * chunkSize
203+
val end = minOf(start + chunkSize, b64.length)
204+
Log.i(TAG, "DUMP_B64[$i/$chunks] ${b64.substring(start, end)}")
205+
}
206+
Log.i(TAG, "DUMP_B64_END")
207+
Log.i(TAG, "DUMP_SUMMARY ${NoteBodyEditor.summarizeBase64(b64)}")
208+
// describeBase64 is multi-line; split for logcat.
209+
val desc = NoteBodyEditor.describeBase64(b64)
210+
Log.i(TAG, "DUMP_DESCRIBE_BEGIN")
211+
for (line in desc.lines()) Log.i(TAG, "DESC $line")
212+
Log.i(TAG, "DUMP_DESCRIBE_END")
213+
val ar = NoteBodyEditor.dumpAttributeRunsBase64(b64)
214+
Log.i(TAG, "DUMP_ATTRRUNS_BEGIN")
215+
for (line in ar.lines()) Log.i(TAG, "AR $line")
216+
Log.i(TAG, "DUMP_ATTRRUNS_END")
217+
val tree = NoteBodyEditor.dumpSubstringTreeBase64(b64)
218+
Log.i(TAG, "DUMP_TREE_BEGIN")
219+
for (line in tree.lines()) Log.i(TAG, "TREE $line")
220+
Log.i(TAG, "DUMP_TREE_END")
221+
}
222+
}
223+
}
224+
}
156225
ACTION_CREATE -> {
157226
val title = intent.getStringExtra("title") ?: ""
158227
val body = intent.getStringExtra("body") ?: ""
@@ -275,6 +344,8 @@ class DebugReceiver : BroadcastReceiver() {
275344
private const val ACTION_APPEND_BY_TITLE = "com.example.applenotes.APPEND_BY_TITLE"
276345
private const val ACTION_SET_BODY_BY_TITLE = "com.example.applenotes.SET_BODY_BY_TITLE"
277346
private const val ACTION_DELETE_BY_TITLE = "com.example.applenotes.DELETE_BY_TITLE"
347+
private const val ACTION_DUMP_BY_TITLE = "com.example.applenotes.DUMP_BY_TITLE"
348+
private const val ACTION_DUMP_RECORD = "com.example.applenotes.DUMP_RECORD"
278349
private const val ACTION_CREATE = "com.example.applenotes.CREATE"
279350

280351
private val httpClient by lazy {

0 commit comments

Comments
 (0)