[Local Catalog] Add GRDB data source for item list #16231

joshheald · 2025-10-09T14:28:59Z

Description

This PR adds a new datasource, analagous to PointOfSaleItemService, but with observation of the database so that any changes result in updates to the observable properties that GRDBObservableDataSource provides.

Steps to reproduce

This isn't used in the app yet. Unit tests are all you have! This PR is the base of a stack, and the top PR should allow you to use it in the item list, so functional testing can wait until then.

I have considered if this change warrants user-facing release notes and have added them to RELEASE-NOTES.txt if necessary.

wpmobilebot · 2025-10-09T14:41:38Z

📲 You can test the changes from this Pull Request in WooCommerce iOS Prototype by scanning the QR code below to install the corresponding build.

	App Name	WooCommerce iOS Prototype
	Build Number	`pr16231-728ac9a`
	Version	`23.5`
	Bundle ID	`com.automattic.alpha.woocommerce`
	Commit	`728ac9a`
	Installation URL	4pb6gqusqa2d8

Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

Moves mock from YosemiteTests to PointOfSaleTests/Mocks where it can be used by PointOfSale module tests. Changes: - Moved file to Modules/Tests/PointOfSaleTests/Mocks/ - Made properties mutable for test control - Removed public access (internal to test module) - Simplified methods (tests set properties directly) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Follow the pattern from PointOfSaleItemService by accepting CurrencySettings and creating the PointOfSaleItemMapper internally. This avoids exposing PointOfSaleItemMapper publicly from Yosemite. Changes: - GRDBObservableDataSource now takes currencySettings parameter - Creates PointOfSaleItemMapper internally with optional override - Updated tests to use new initializer

joshheald · 2025-10-10T08:12:22Z

@staskus I'll have the other PR in the stack up this morning, which will let you see it working in the views. You may want to delay testing until then.

I know you don't have all the context of this yet, so happy to have a call or async chat to go over anything

Replace sleep-based polling with proper Swift Observation tracking. The waitForCondition helper now: - Uses withObservationTracking to detect changes to observable properties - Recursively re-observes until the condition is met - Has a 2-second timeout as a backstop to prevent hanging - Checks both loading completion AND expected item count - No primary polling - purely event-driven with timeout safety This fixes intermittent failures in: - test_load_products_maps_to_pos_items_correctly - test_load_products_sets_loading_state_and_fetches_items

Created reusable request methods on PersistedProduct and PersistedProductVariation: - posProductsRequest: Filters to simple/variable, non-downloadable products, sorts by name - posVariationsRequest: Filters to non-downloadable variations, sorts by ID This fixes the pagination issue where unsupported/downloadable product types caused fewer items than pageSize to be returned, stopping pagination early. Before: 20 fetched → 18 mapped → pagination stopped (18 < 20) After: 20 fetched → 20 mapped → pagination continues correctly Changes: - Add ProductType enum to PersistedProduct for type-safe filtering - Filter out unsupported product types at DB level (not in mapper) - Filter out downloadable products/variations - Products sorted alphabetically by name (case-insensitive, localized) - Variations sorted consistently by ID - Use weak self capture in Task blocks to avoid retain cycles - Centralized query logic for maintainability Benefits: - All supported products now load correctly with proper pagination - Cleaner separation of concerns (filtering in DB, not mapper) - More efficient (don't fetch items we'll filter out anyway)

Ensure the observe() function and condition check always run on MainActor to prevent data races when accessing observable properties. This fixes intermittent 'Index out of range' crashes on line 72 where the continuation would resume before the productItems array was actually populated on the main actor.

@mainactor

Use .receive(on: DispatchQueue.main) to ensure ValueObservation results are delivered on the main queue, and update properties synchronously rather than dispatching via Task { @mainactor }. This simplifies the threading model and ensures property updates happen synchronously when observations fire, which should improve reliability and reduce potential race conditions. Applied to all three observations: products, variations, and statistics.

dangermattic · 2025-10-10T13:33:30Z

	1 Warning
⚠️	This PR is larger than 300 lines of changes. Please consider splitting it into smaller PRs for easier and faster reviews.

Generated by 🚫 Danger

staskus · 2025-10-15T11:34:38Z

Modules/Sources/Yosemite/PointOfSale/Items/GRDBObservableDataSource.swift

+                    .fetchAll(database)
+
+                return try persistedProducts.map { persistedProduct in
+                    let images = try persistedProduct.request(for: PersistedProduct.images).fetchAll(database)


For each product, we request images and attributes. Is this an effective way?

Could we flatten it all by:

Getting all products

Getting all attributes (for product ids) and make dictionary

Getting all images (for product ids) and make dictionary

Map into POSProduct with attributes and images.

Or I see there also .including method which maybe could be used.

Same for variations

Improved in b832cd6, 2x faster comparing test speed with 1000s of products.

staskus · 2025-10-15T11:40:22Z

Modules/Sources/Yosemite/PointOfSale/Items/GRDBObservableDataSource.swift

+                    .fetchCount(database)
+
+                let variationCount = try PersistedProductVariation
+                    .filter(PersistedProductVariation.Columns.siteID == siteID)


Should it be using posVariationsRequest or least .filter(Columns.downloadable == false) so we would get the same count in case there are downloadable variations?

I think it also affects hasMoreVariations since it checks totalVariationCount which is not filtered per-product and compares it to variationItems is is loaded per-product.

Improved 5354d0a

staskus · 2025-10-15T12:03:23Z

Modules/Sources/Yosemite/PointOfSale/Items/GRDBObservableDataSource.swift

+                },
+                receiveValue: { [weak self] observedProducts in
+                    guard let self else { return }
+                    let posItems = itemMapper.mapProductsToPOSItems(products: observedProducts)


You mentioned the performance with a lot of items which may be connected to mapping, could we do the mapping outside the main thread and only receive on main thread?

.publisher(in: grdbManager.databaseConnection) .map { [itemMapper] observedProducts in // Mapping happens on background Combine scheduler itemMapper.mapProductsToPOSItems(products: observedProducts) } .receive(on: DispatchQueue.main)

We could also leave such performance considerations for the future. I'm sure there are even more things that we could do if we reach use cases with extremely large catalogs.

Yeah, I thought about this approach after I finished up last week, and it should work. But I previously had the mapping happen off the main thread (with different syntax) and it didn't make a noticable performance difference, so I think it's fine to leave for now.

wpmobilebot · 2025-10-17T04:52:09Z

Version 23.5 has now entered code-freeze, so the milestone of this PR has been updated to 23.6.

Products and variations with attributes are 2x faster loaded when using included

staskus

✅ Approving after changes for merge.

Extensively tested internally with large datasets and behaves as expected.
Tested with the integration #16235 as well.

We can keep improving as we go on

hasMoreVariations may not yet be loaded when products load since statistics are loaded separately

joshheald · 2025-10-20T10:20:23Z

✅ Approving after changes for merge.
* Extensively tested internally with large datasets and behaves as expected.

* Tested with the integration #16235 as well.
We can keep improving as we go on

Thanks so much for the review and improvements!

joshheald added type: task An internally driven task. status: feature-flagged Behind a feature flag. Milestone is not strongly held. feature: POS labels Oct 9, 2025

joshheald added this to the 23.5 milestone Oct 9, 2025

joshheald and others added 3 commits October 9, 2025 17:19

Add GRDB data source for item list

fd80f54

Periphery ignore horizontal slices

a84aebc

joshheald force-pushed the issue/woomob-1088-use-GRDB-in-itemlist branch from 4e79a59 to a84aebc Compare October 9, 2025 16:23

joshheald marked this pull request as ready for review October 9, 2025 16:52

joshheald requested a review from staskus October 10, 2025 08:11

joshheald added 4 commits October 10, 2025 11:12

joshheald mentioned this pull request Oct 10, 2025

[Local Catalog] Show items from GRDB in POS #16235

Merged

1 task

staskus reviewed Oct 15, 2025

View reviewed changes

wpmobilebot modified the milestones: 23.5, 23.6 Oct 17, 2025

staskus added 5 commits October 17, 2025 11:35

Merge branch 'trunk' into issue/woomob-1088-use-GRDB-in-itemlist

8c63b39

Limit variation statistics to a parent variation

5354d0a

Add new tests to verify products with attributes and images are loading

794dca7

Add attributes key to PersistedProductAttribute to use in associations

96c7d30

Use including to more efficiently load products with associated values

b832cd6

Products and variations with attributes are 2x faster loaded when using included

staskus approved these changes Oct 17, 2025

View reviewed changes

Wait for hasMoreProducts/hasMoreVariations value

728ac9a

hasMoreVariations may not yet be loaded when products load since statistics are loaded separately

staskus merged commit 56aa52a into trunk Oct 17, 2025
13 checks passed

staskus deleted the issue/woomob-1088-use-GRDB-in-itemlist branch October 17, 2025 14:16

[Local Catalog] Add GRDB data source for item list #16231

[Local Catalog] Add GRDB data source for item list #16231

Uh oh!

Conversation

joshheald commented Oct 9, 2025

Description

Steps to reproduce

Uh oh!

wpmobilebot commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joshheald commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dangermattic commented Oct 10, 2025

Uh oh!

staskus Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

staskus Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

staskus Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

staskus Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

staskus Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

joshheald Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

wpmobilebot commented Oct 17, 2025

Uh oh!

staskus left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joshheald commented Oct 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

wpmobilebot commented Oct 9, 2025 •

edited

Loading

joshheald commented Oct 10, 2025 •

edited

Loading

staskus Oct 17, 2025 •

edited

Loading