Skip to content

Conversation

@joshheald
Copy link
Contributor

@joshheald joshheald commented Nov 11, 2025

Merge after: #16337

Description

This PR moves the handling (parsing and persistence) of background full catalog sync downloads, to be completed in the background. This happens in the 30s that Apple give us after the background completes.

Note than none of this will be live for a while, because we need the Catalog endpoints in Core first.

Test Steps

Backgrounded testing is tricky, and you can't really do it with the debugger attached. I've found network breakpoints are useful.

  • Use a large catalog store with the catalog generation/file endpoints, e.g. largefuntesting.mystagingwebsite.com
  • Turn on the pointOfSaleCatalogAPI feature flag
  • Set POSLocalCatalogEligibilityService.Constants.defaultCatalogSizeLimit to 100000
  • Set network conditions to something slow, like 3g or 2g. In Proxyman, it's Tools > Network conditions > + then set the host to largefuntesting.mystagingwebsite.com and use the preset profile. Note that this is only wanted for the file download, so no need to do it on public-api.wordpress.com. For these settings, 3g takes about 30s to download, and 2g takes about 2 minutes.
  1. Run the app, then stop it.
  2. Launch the app from the device, not Xcode.
  3. Go to POS > Settings and tap refresh catalog
  4. Watch the requests to /wc/v3/products/catalog, wait for the completed status response with a download URL (this takes a few minutes)
  5. As soon as you see that, background the app
  6. Wait for the download to complete (see Proxyman to check that it has)
  7. Wait another 3-5 minutes to give time for the persistence and parsing to complete
  8. Open the app again and go to Menu > Settings > Help and Support > Application logs > Current
  9. Observe that the download completed and was persisted when we opened the app again. Note that it's fine to only see Clearing catalog data for site 247136746 in the background. It's possible to run out of time with this approach to handling downloads, but they get completed when you open the app again.

Screenshots

2025/11/11 12:55:15:570  🔵 Tracked pos_settings_open, properties: [site_url: https://largefuntesting.mystagingwebsite.com, blog_id: 247136746, store_id: 5127f96a-6b26-4aca-bdf3-7f92defeb6ce, is_wpcom_store: false, plan: jetpack_security_t1_yearly, was_ecommerce_trial: false]
2025/11/11 12:55:21:478  📋 POSCatalogSyncCoordinator: Site 247136746 eligible (within grace period: 0 days since first sync)
2025/11/11 12:55:21:481  📋 POSCatalogSyncCoordinator: Last sync for site 247136746 was 2408s ago (max: 0s), sync needed
2025/11/11 12:55:21:491  🔄 POSCatalogSyncCoordinator starting full sync for site 247136746
2025/11/11 12:55:21:492  🔄 Starting full catalog sync for site ID: 247136746 with regenerateCatalog: true and allowCellular: true
2025/11/11 12:55:21:492  🟣 Starting catalog request...
2025/11/11 12:55:23:918  🟣 Catalog request pending... (attempt 1/1000)
2025/11/11 13:00:22:596  🟣 Catalog request processing... (attempt 141/1000)
2025/11/11 13:00:24:749  🟣 Catalog ready for download: https://largefuntesting.mystagingwebsite.com/wp-content/uploads/wc-product-catalog/pos-catalog-feed.json
2025/11/11 13:01:07:064  Scheduling background refresh task ordersAndDashboardSync in 1799s
2025/11/11 13:01:07:073  🔵 Tracked application_closed, properties: [plan: jetpack_security_t1_yearly, blog_id: 247136746, is_wpcom_store: false, store_id: 5127f96a-6b26-4aca-bdf3-7f92defeb6ce, site_url: https://largefuntesting.mystagingwebsite.com, time_in_app: 363.0, was_ecommerce_trial: false]
2025/11/11 13:01:07:074  ⏱️ ForegroundPOSCatalogSyncDispatcher: Stopping timer
2025/11/11 13:02:21:601  🟣 Handling background URLSession events for identifier: com.woocommerce.pos.catalog.download.247136746.8BB90F4D-65B6-4E07-A82F-1EB22314BA04
2025/11/11 13:02:21:603  🟣 Handling background session event for: com.woocommerce.pos.catalog.download.247136746.8BB90F4D-65B6-4E07-A82F-1EB22314BA04
2025/11/11 13:02:21:605  🟣 Reconnecting to background session: com.woocommerce.pos.catalog.download.247136746.8BB90F4D-65B6-4E07-A82F-1EB22314BA04
2025/11/11 13:02:21:620  🟣 Background download completed, file moved to: /private/var/mobile/Containers/Data/Application/12167735-BF6A-4EBA-A4C5-0619BE4432E6/tmp/pos-catalog-feed.json
2025/11/11 13:02:36:575  🟣 Catalog download completed - Time: 435.08s
2025/11/11 13:02:36:575  ✅ Loaded 4977 products and 12458 variations for siteID 247136746
2025/11/11 13:02:36:575  💾 Persisting catalog with 4977 products and 12458 variations
2025/11/11 13:02:53:010  🗑️ Clearing catalog data for site 247136746
2025/11/11 13:07:01:056  🔵 Tracked application_opened, properties: [widgets: , is_wpcom_store: false, store_id: 5127f96a-6b26-4aca-bdf3-7f92defeb6ce, was_ecommerce_trial: false, site_url: https://largefuntesting.mystagingwebsite.com, blog_id: 247136746, plan: jetpack_security_t1_yearly]
2025/11/11 13:07:15:013  ✅ Database write complete for site 247136746
2025/11/11 13:07:15:048  ✅ Catalog persistence complete
2025/11/11 13:07:15:050  📋 POSCatalogSyncCoordinator: Site 247136746 eligible (within grace period: 0 days since first sync)
2025/11/11 13:07:15:051  Persisted 4960 products, 5694 product images, 1992 product attributes, 12288 variations, 12288 variation images, 32023 variation attributes
2025/11/11 13:07:15:052  ✅ Persisted 4977 products and 12458 variations to database for siteID 247136746
2025/11/11 13:07:15:079  ✅ POSCatalogSyncCoordinator completed full sync for site 247136746

  • I have considered if this change warrants user-facing release notes and have added them to RELEASE-NOTES.txt if necessary.

@joshheald joshheald added type: task An internally driven task. feature: POS labels Nov 11, 2025
@joshheald joshheald changed the base branch from trunk to feat/WOOMOB-1173-background-catalog-download-updated November 11, 2025 15:02
@joshheald joshheald changed the title Feat/woomob 1173 parse catalog downloads in the background [Local Catalog] Persist catalog downloads in the background Nov 11, 2025
@joshheald joshheald added this to the 23.7 milestone Nov 11, 2025
@wpmobilebot
Copy link
Collaborator

wpmobilebot commented Nov 11, 2025

App Icon📲 You can test the changes from this Pull Request in WooCommerce iOS Prototype by scanning the QR code below to install the corresponding build.

App NameWooCommerce iOS Prototype
Build Numberpr16342-d0558a2
Version23.6
Bundle IDcom.automattic.alpha.woocommerce
Commitd0558a2
Installation URL2lha4pb2uuf58
Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

@joshheald joshheald marked this pull request as ready for review November 12, 2025 09:56
Base automatically changed from feat/WOOMOB-1173-background-catalog-download-updated to trunk November 13, 2025 11:41
@joshheald joshheald modified the milestones: 23.7, 23.8 Nov 13, 2025
@iamgabrielma iamgabrielma self-assigned this Nov 14, 2025
Copy link
Contributor

@iamgabrielma iamgabrielma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works well! Added some non-blocking comments/questions below

throw POSCatalogSyncError.timeout
}

public func parseAndPersistBackgroundDownload(fileURL: URL, siteID: Int64) async throws -> POSCatalog {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: We can move this signature to a different extension or to the class. I get a warning here: 'public' modifier conflicts with extension's default access of 'private'

#expect(parsedFileURL?.path == "/tmp/test.json")

// Cleanup
BackgroundDownloadState.clear()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAU this clean up is not needed, as we already perform it at the end of handleBackgroundSessionEvent right?

Not necessarily a suggestion, just something to think about: I wonder if there could be test isolation issues when these run in parallel, or a test crashes/fails before cleanup which could affect subsequent tests since these share UserDefaults.standard

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not from this PR, but consider making a helper function in this test suite for clean up the temp files and background state, I think it's easy to miss the difference between .removeItem(at: documentsURL), .removeItem(at: tempFile) or when these should be called per test basis.

// Simulates async behavior
try await Task.sleep(nanoseconds: 1_000_000) // 1ms
// Notify test that download has started
onDownloadStarted?()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💯

switch newState {
case .syncCompleted, .syncFailed, .initialSyncFailed:
// Sync finished - clear the refreshing state if it was set
if isRefreshingCatalog {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We only toggle the value of isRefreshingCatalog when is user-initiated through refreshCatalog(). If a non-user-initiated background sync completes this always be false, so UI would only update when user-initiated. Is intended for now?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, that's totally intended. It'd be strange to show a button spinner for something the user didn't do, IMO

Comment on lines 100 to 108
await withCheckedContinuation { (continuation: CheckedContinuation<Void, Never>) in
withObservationTracking {
// Access the observed property to register observation
_ = currentSyncState
} onChange: {
// When state changes, resume the continuation
continuation.resume()
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit tricky to wrap my head around, I have a question: When we setup the observation through withCheckedContinuation, and then becomes inactive before reading the new state (let newState = currentSyncState)... could there be state changes that are missed before we loop back to the beginning of the loop?

It was my understanding that when .onChange fires we must callwithObservationTracking again to re-register, so the gap between these calls is not observed. But this could be just a miss-understanding on my part, since the signature does not return any cancellable/subscription object but is just a closure.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it's a little odd. The while loop takes care of re-upping the observation tracking, but I've changed it to make that a little clearer.

@dangermattic
Copy link
Collaborator

1 Warning
⚠️ This PR is larger than 300 lines of changes. Please consider splitting it into smaller PRs for easier and faster reviews.

Generated by 🚫 Danger

@joshheald joshheald enabled auto-merge November 14, 2025 14:44
@joshheald joshheald merged commit 5a7449f into trunk Nov 14, 2025
14 checks passed
@joshheald joshheald deleted the feat/WOOMOB-1173-parse-catalog-downloads-in-the-background branch November 14, 2025 15:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature: POS type: task An internally driven task.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants