Support circular references in Sync Streams

[simolus3]

For Sync Streams, building a graph of parameter lookups is implemented by going through expressions and recursively handling dependencies. E.g. if a stream has outputs of a table a and we encounter a filter expression a.foo = b.bar, we'd try to resolve b as a parameter lookup recursively before looking at additional expressions. This generally works well, because it both resolves partition keys and parameter outputs while sorting the flat list of expressions topologically, giving us a description for a querier.

Because of this recursion, we can't support circular references between result sets. Initially, I believed that circular references are inherently unsupported (we need to look up parameters in sequence, after all). Then I looked at an actual instance of this in the wild, and realized that the circular reference isn't really all that circular:

-- This currently crashes the compiler
SELECT invoices.* FROM invoices, stores WHERE invoices.store_name = stores.name AND invoices.region = stores.region AND stores.owner = auth.user_id()

Currently, we start by analyzing invoices, track stores as a dependency due to invoices.store_name = stores.name. When parsing stores, we discover invoices.region = stores.region and crash due to a circular reference. The fix is very simple: Ignore expressions referencing a result set that is currently being analyzed, so we'd then:

1. Start by analyzing invoices, track stores as a dependency due to invoices.store_name = stores.name.
2. Resolve stores, find stores.owner = auth.user_id() as the only expression and create a pending parameter lookup partitioned by owner without any outputs.
3. Resolve the invoices.store_name = stores.name expression by adding stores.name as an output to the lookup and invoices.store_name as a bucket parameter.
4. Encounter the invoices.region = stores.region expression in the context of invoices, realize we've resolved stores already and just do the same add output + bucket parameter step with the resolved parameter lookup.

The same strategy works for larger reference chains as well.

[changeset-bot]

🦋 Changeset detected

Latest commit: 8650664

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 18 packages

Name	Type
@powersync/service-sync-rules	Patch
@powersync/service-jpgwire	Patch
@powersync/service-core-tests	Patch
@powersync/service-core	Patch
@powersync/lib-services-framework	Patch
@powersync/service-module-mongodb-storage	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-module-mssql	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-postgres-storage	Patch
@powersync/service-module-postgres	Patch
@powersync/lib-service-postgres	Patch
@powersync/service-module-core	Patch
@powersync/service-image	Patch
test-client	Patch
@powersync/service-rsocket-router	Patch
@powersync/lib-service-mongodb	Patch
@powersync/service-schema	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[rkistner]

Another case still failing (picked up by Codex):

SELECT a.*
    FROM a, b, json_each(a.tags) AS j
    WHERE a.k = b.k AND j.value = b.v

I'm not sure if it is actually feasible or useful to support this case, but maybe we can have a more descriptive error message?

[simolus3]

I'm not sure if it is actually feasible or useful to support this case, but maybe we can have a more descriptive error message?

We should be able to support that as well, I've fixed that and added a test 👍