[Local Catalog] Persist catalog downloads in the background #16342

joshheald · 2025-11-11T15:01:37Z

Merge after: #16337

Description

This PR moves the handling (parsing and persistence) of background full catalog sync downloads, to be completed in the background. This happens in the 30s that Apple give us after the background completes.

Note than none of this will be live for a while, because we need the Catalog endpoints in Core first.

Test Steps

Backgrounded testing is tricky, and you can't really do it with the debugger attached. I've found network breakpoints are useful.

Use a large catalog store with the catalog generation/file endpoints, e.g. largefuntesting.mystagingwebsite.com
Turn on the pointOfSaleCatalogAPI feature flag
Set POSLocalCatalogEligibilityService.Constants.defaultCatalogSizeLimit to 100000
Set network conditions to something slow, like 3g or 2g. In Proxyman, it's Tools > Network conditions > + then set the host to largefuntesting.mystagingwebsite.com and use the preset profile. Note that this is only wanted for the file download, so no need to do it on public-api.wordpress.com. For these settings, 3g takes about 30s to download, and 2g takes about 2 minutes.

Run the app, then stop it.
Launch the app from the device, not Xcode.
Go to POS > Settings and tap refresh catalog
Watch the requests to /wc/v3/products/catalog, wait for the completed status response with a download URL (this takes a few minutes)
As soon as you see that, background the app
Wait for the download to complete (see Proxyman to check that it has)
Wait another 3-5 minutes to give time for the persistence and parsing to complete
Open the app again and go to Menu > Settings > Help and Support > Application logs > Current
Observe that the download completed and was persisted when we opened the app again. Note that it's fine to only see Clearing catalog data for site 247136746 in the background. It's possible to run out of time with this approach to handling downloads, but they get completed when you open the app again.

Screenshots

2025/11/11 12:55:15:570  🔵 Tracked pos_settings_open, properties: [site_url: https://largefuntesting.mystagingwebsite.com, blog_id: 247136746, store_id: 5127f96a-6b26-4aca-bdf3-7f92defeb6ce, is_wpcom_store: false, plan: jetpack_security_t1_yearly, was_ecommerce_trial: false]
2025/11/11 12:55:21:478  📋 POSCatalogSyncCoordinator: Site 247136746 eligible (within grace period: 0 days since first sync)
2025/11/11 12:55:21:481  📋 POSCatalogSyncCoordinator: Last sync for site 247136746 was 2408s ago (max: 0s), sync needed
2025/11/11 12:55:21:491  🔄 POSCatalogSyncCoordinator starting full sync for site 247136746
2025/11/11 12:55:21:492  🔄 Starting full catalog sync for site ID: 247136746 with regenerateCatalog: true and allowCellular: true
2025/11/11 12:55:21:492  🟣 Starting catalog request...
2025/11/11 12:55:23:918  🟣 Catalog request pending... (attempt 1/1000)
2025/11/11 13:00:22:596  🟣 Catalog request processing... (attempt 141/1000)
2025/11/11 13:00:24:749  🟣 Catalog ready for download: https://largefuntesting.mystagingwebsite.com/wp-content/uploads/wc-product-catalog/pos-catalog-feed.json
2025/11/11 13:01:07:064  Scheduling background refresh task ordersAndDashboardSync in 1799s
2025/11/11 13:01:07:073  🔵 Tracked application_closed, properties: [plan: jetpack_security_t1_yearly, blog_id: 247136746, is_wpcom_store: false, store_id: 5127f96a-6b26-4aca-bdf3-7f92defeb6ce, site_url: https://largefuntesting.mystagingwebsite.com, time_in_app: 363.0, was_ecommerce_trial: false]
2025/11/11 13:01:07:074  ⏱️ ForegroundPOSCatalogSyncDispatcher: Stopping timer
2025/11/11 13:02:21:601  🟣 Handling background URLSession events for identifier: com.woocommerce.pos.catalog.download.247136746.8BB90F4D-65B6-4E07-A82F-1EB22314BA04
2025/11/11 13:02:21:603  🟣 Handling background session event for: com.woocommerce.pos.catalog.download.247136746.8BB90F4D-65B6-4E07-A82F-1EB22314BA04
2025/11/11 13:02:21:605  🟣 Reconnecting to background session: com.woocommerce.pos.catalog.download.247136746.8BB90F4D-65B6-4E07-A82F-1EB22314BA04
2025/11/11 13:02:21:620  🟣 Background download completed, file moved to: /private/var/mobile/Containers/Data/Application/12167735-BF6A-4EBA-A4C5-0619BE4432E6/tmp/pos-catalog-feed.json
2025/11/11 13:02:36:575  🟣 Catalog download completed - Time: 435.08s
2025/11/11 13:02:36:575  ✅ Loaded 4977 products and 12458 variations for siteID 247136746
2025/11/11 13:02:36:575  💾 Persisting catalog with 4977 products and 12458 variations
2025/11/11 13:02:53:010  🗑️ Clearing catalog data for site 247136746
2025/11/11 13:07:01:056  🔵 Tracked application_opened, properties: [widgets: , is_wpcom_store: false, store_id: 5127f96a-6b26-4aca-bdf3-7f92defeb6ce, was_ecommerce_trial: false, site_url: https://largefuntesting.mystagingwebsite.com, blog_id: 247136746, plan: jetpack_security_t1_yearly]
2025/11/11 13:07:15:013  ✅ Database write complete for site 247136746
2025/11/11 13:07:15:048  ✅ Catalog persistence complete
2025/11/11 13:07:15:050  📋 POSCatalogSyncCoordinator: Site 247136746 eligible (within grace period: 0 days since first sync)
2025/11/11 13:07:15:051  Persisted 4960 products, 5694 product images, 1992 product attributes, 12288 variations, 12288 variation images, 32023 variation attributes
2025/11/11 13:07:15:052  ✅ Persisted 4977 products and 12458 variations to database for siteID 247136746
2025/11/11 13:07:15:079  ✅ POSCatalogSyncCoordinator completed full sync for site 247136746

I have considered if this change warrants user-facing release notes and have added them to RELEASE-NOTES.txt if necessary.

wpmobilebot · 2025-11-11T15:11:29Z

📲 You can test the changes from this Pull Request in WooCommerce iOS Prototype by scanning the QR code below to install the corresponding build.

	App Name	WooCommerce iOS Prototype
	Build Number	`pr16342-d0558a2`
	Version	`23.6`
	Bundle ID	`com.automattic.alpha.woocommerce`
	Commit	`d0558a2`
	Installation URL	2lha4pb2uuf58

Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

…nto feat/WOOMOB-1173-parse-catalog-downloads-in-the-background

iamgabrielma

Works well! Added some non-blocking comments/questions below

iamgabrielma · 2025-11-14T02:34:25Z

Modules/Sources/Yosemite/Tools/POS/POSCatalogFullSyncService.swift

        throw POSCatalogSyncError.timeout
    }
+
+    public func parseAndPersistBackgroundDownload(fileURL: URL, siteID: Int64) async throws -> POSCatalog {


Nit: We can move this signature to a different extension or to the class. I get a warning here: 'public' modifier conflicts with extension's default access of 'private'

iamgabrielma · 2025-11-14T02:46:49Z

Modules/Tests/NetworkingTests/Network/BackgroundCatalogDownloadCoordinatorTests.swift

+        #expect(parsedFileURL?.path == "/tmp/test.json")
+
+        // Cleanup
+        BackgroundDownloadState.clear()


AFAU this clean up is not needed, as we already perform it at the end of handleBackgroundSessionEvent right?

Not necessarily a suggestion, just something to think about: I wonder if there could be test isolation issues when these run in parallel, or a test crashes/fails before cleanup which could affect subsequent tests since these share UserDefaults.standard

iamgabrielma · 2025-11-14T03:05:47Z

Modules/Tests/NetworkingTests/Remote/POSCatalogSyncRemoteTests.swift

Not from this PR, but consider making a helper function in this test suite for clean up the temp files and background state, I think it's easy to miss the difference between .removeItem(at: documentsURL), .removeItem(at: tempFile) or when these should be called per test basis.

iamgabrielma · 2025-11-14T03:07:59Z

Modules/Tests/NetworkingTests/Mocks/MockBackgroundDownloader.swift

-        // Simulates async behavior
-        try await Task.sleep(nanoseconds: 1_000_000) // 1ms
+        // Notify test that download has started
+        onDownloadStarted?()


iamgabrielma · 2025-11-14T03:15:55Z

Modules/Sources/PointOfSale/Presentation/Settings/POSSettingsLocalCatalogViewModel.swift

+                    switch newState {
+                    case .syncCompleted, .syncFailed, .initialSyncFailed:
+                        // Sync finished - clear the refreshing state if it was set
+                        if isRefreshingCatalog {


We only toggle the value of isRefreshingCatalog when is user-initiated through refreshCatalog(). If a non-user-initiated background sync completes this always be false, so UI would only update when user-initiated. Is intended for now?

Yep, that's totally intended. It'd be strange to show a button spinner for something the user didn't do, IMO

iamgabrielma · 2025-11-14T03:31:12Z

Modules/Sources/PointOfSale/Presentation/Settings/POSSettingsLocalCatalogViewModel.swift

+                await withCheckedContinuation { (continuation: CheckedContinuation<Void, Never>) in
+                    withObservationTracking {
+                        // Access the observed property to register observation
+                        _ = currentSyncState
+                    } onChange: {
+                        // When state changes, resume the continuation
+                        continuation.resume()
+                    }
+                }


This is a bit tricky to wrap my head around, I have a question: When we setup the observation through withCheckedContinuation, and then becomes inactive before reading the new state (let newState = currentSyncState)... could there be state changes that are missed before we loop back to the beginning of the loop?

It was my understanding that when .onChange fires we must callwithObservationTracking again to re-register, so the gap between these calls is not observed. But this could be just a miss-understanding on my part, since the signature does not return any cancellable/subscription object but is just a closure.

Yeah, it's a little odd. The while loop takes care of re-upping the observation tracking, but I've changed it to make that a little clearer.

dangermattic · 2025-11-14T14:00:32Z

	1 Warning
⚠️	This PR is larger than 300 lines of changes. Please consider splitting it into smaller PRs for easier and faster reviews.

Generated by 🚫 Danger

joshheald added 3 commits November 11, 2025 07:40

Persist background downloads in the background

fabb1de

Tests for background catalog parsing

a93899e

Stop spinner when foregrounding POS

be0cf24

joshheald added type: task An internally driven task. feature: POS labels Nov 11, 2025

joshheald changed the base branch from trunk to feat/WOOMOB-1173-background-catalog-download-updated November 11, 2025 15:02

joshheald changed the title ~~Feat/woomob 1173 parse catalog downloads in the background~~ [Local Catalog] Persist catalog downloads in the background Nov 11, 2025

joshheald added this to the 23.7 milestone Nov 11, 2025

joshheald requested a review from iamgabrielma November 11, 2025 15:55

joshheald added 3 commits November 11, 2025 16:10

Remove unused code

5046bfd

Merge branch 'feat/WOOMOB-1173-background-catalog-download-updated' i…

f350297

…nto feat/WOOMOB-1173-parse-catalog-downloads-in-the-background

Improve test timing/blocking approach

db3f9d4

joshheald marked this pull request as ready for review November 12, 2025 09:56

joshheald mentioned this pull request Nov 12, 2025

[Local catalog] Analytics for sync, launch, and loading #16345

Merged

1 task

Base automatically changed from feat/WOOMOB-1173-background-catalog-download-updated to trunk November 13, 2025 11:41

joshheald modified the milestones: 23.7, 23.8 Nov 13, 2025

iamgabrielma self-assigned this Nov 14, 2025

iamgabrielma approved these changes Nov 14, 2025

View reviewed changes

joshheald added 4 commits November 14, 2025 10:03

Fix placement of public function

5d29062

Remove unnecessary cleanup

06ae494

Inject a new UserDefault instance in the unit tests

bc5c1bf

Improve state observation for refresh control

fa75864

Periphery

d0558a2

joshheald enabled auto-merge November 14, 2025 14:44

joshheald merged commit 5a7449f into trunk Nov 14, 2025
14 checks passed

joshheald deleted the feat/WOOMOB-1173-parse-catalog-downloads-in-the-background branch November 14, 2025 15:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Local Catalog] Persist catalog downloads in the background #16342

[Local Catalog] Persist catalog downloads in the background #16342

Uh oh!

joshheald commented Nov 11, 2025 •

edited

Loading

Uh oh!

wpmobilebot commented Nov 11, 2025 •

edited

Loading

Uh oh!

iamgabrielma left a comment

Uh oh!

iamgabrielma Nov 14, 2025

Uh oh!

iamgabrielma Nov 14, 2025

Uh oh!

iamgabrielma Nov 14, 2025

Uh oh!

iamgabrielma Nov 14, 2025

Uh oh!

iamgabrielma Nov 14, 2025

Uh oh!

joshheald Nov 14, 2025

Uh oh!

iamgabrielma Nov 14, 2025

Uh oh!

joshheald Nov 14, 2025

Uh oh!

dangermattic commented Nov 14, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[Local Catalog] Persist catalog downloads in the background #16342

[Local Catalog] Persist catalog downloads in the background #16342

Uh oh!

Conversation

joshheald commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Steps

Screenshots

Uh oh!

wpmobilebot commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iamgabrielma left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dangermattic commented Nov 14, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

joshheald commented Nov 11, 2025 •

edited

Loading

wpmobilebot commented Nov 11, 2025 •

edited

Loading