Firestore sub collections: collection group queries

[joppehoekstra]

I was wondering if it is possible to use this integration with Firestore collection groups (so create an index of documents across multiple sub collections)? In the configuration, I only see there is the option to provide a path for a single collection to index.

I can imagine having such an option might cause problems when having documents in the different sub collections with the same id.

[bidoubiwa]

Hey @joppehoekstra sorry for the delay in my answer. The issue flew under my radar.

You are right about the fact that for this moment the plugin only work with simple collections. How would you imagine it to work? What would the setting look like?

[joppehoekstra]

I'm not quite sure to be honest. Perhaps add a field that allows the user to define a certain path? E.g. "mycollection/*/subcollection".

[bidoubiwa]

Hey @joppehoekstra I will look into this and if it is possible after the pre-release period of Meilisearch :) Sorry for the delay in answer

[bidoubiwa]

Hello @joppehoekstra
Sorry for the delay. Unfortunately the API only gives me the possibility to watch one collection at the time (unless I misread the documentation).

Do you have another extension example that successfully implemented this?

[mogmet]

Hi.
In the import documentation, there is an option for collection group query.
Is it possible to read the data to be imported into meilisearch in the same way, using collection group?

https://github.com/meilisearch/firestore-meilisearch/blob/main/guides/IMPORT_EXISTING_DOCUMENTS.md

[bidoubiwa]

That's interesting, though I'm not sure it works even in the import script. We'll have to investigate to see how we can implement it. It might not be hard to do, but it requires to look into it.

[jontybrook]

I can confirm it does work in the import script. I'm using Meilisearch v1.0.0 and it's working beautifully.

Hoping that now that v1 is released you might have the time to investigate this feature @bidoubiwa - Would be highly appreciated.

[jontybrook]

I note that the BigQuery extension appears to support what I need (wildcard path matching). This isn't quite collectionGroup matching, but seems like it could be easier to implement?

[jontybrook]

TLDR; Collection path wildcards just work!

I was just looked into how BigQuery supports wildcard collection paths and it seems they do nothing special. Turns out, you can use wildcards in the collection path with this extension and it just works.

Problem

Our use-case is that we have a multi-tenant app. We're storing data belonging to tenants in common sub-collections nested under a tenant collection, eg tenants/{tenantId}/customers and tenants/{tenantId}/orders etc.

Without wildcard or collectionGroup support, we'd need to set up an instance of this extension for every one of our tenants (or so I thought!)

Solution

I've just installed this extension with the following config:

• Collection path: tenants/{tenantId}/orders (note the wildcard {tenantId})
• Melisearch Index Name: all_orders
• (etc)

.. and it works! Any order written under any of our tenants sub-collections is successfully indexing. No changes needed.

Supporting Multi-tenancy tokens

Now that we can index all our tenant's data into one Meilisearch index, our next challenge is preventing tenants from accessing other tenant's results! Our solution to this is that we always add the owning tenant's ID to documents (under a metadata object) within any document anywhere in our tenant sub-collections. Therefore we can use Meilisearch tenant tokens to generate tokens which limit the scope of results to the tenant.

[bidoubiwa]

Hey @jontybrook, thanks a lot for the review. It helps a lot!
After some digging at Algolia's side as well, it seems they do not manage the collection group in their extension.

I was just looked into how BigQuery supports wildcard collection paths and it seems they do nothing special. Turns out, you can use wildcards in the collection path with this extension and it just works.

Using * as my collection path, I tried it both locally and on the firebase platform but the hook did not trigger on both.

Did you manage to make it work? What was the value of your collection path?

Also looking at the documentation of functions.firestore.document(config.collectionPath) it states:

* @param path Full database path to listen to. This includes the name of
 * the collection that the document is a part of. For example, if the
 * collection is named "users" and the document is named "Ada", then the
 * path is "/users/Ada".

It does not mention wildcards.

Meanwhile, the way we suggest acting on collection group is to add multiple times the Meilisearch extension with each one of their own collection pointing to the same index

[jontybrook]

I have it working with the my collection path set as:

tenants/{tenantId}/orders

Note that {tenantId} is the 'wildcard'. For example, the following paths all match:

/tenants/tenant_a/orders/order_1234 /tenant/tenant_b/orders/order_2345 /tenant/tenant_c/orders/order_48292

So for my use case, this is already working by simply using {arbitary_reference} in the collection path (this is what I mean by wildcard. I don't think * would work). So, instead of having several instances with collection paths set like:

1. /tenants/tenant_a/orders
2. /tenants/tenant_b/orders
3. /tenants/tenant_c/orders

I can just have one instance with the collection path:

/tenants/{anything_here}/orders

.. and it works :).

The confusing thing about this is that firestore doesn't support using wildcards in queries; but in the context of this extension we're not actually doing a query, we're using a cloud function trigger. Under the hood, Google uses EventArc (or at least something very similar) to trigger the cloud functions in response to events, and this does support wildcard matching. See here in the documentation.

So.. what the OP @joppehoekstra describes above actually already works - just the syntax of his wildcard is wrong (use {something} instead of * in your path .

It's important to note that the terminology matters here. "Collection group queries" are a different thing to "matching sub collections by path". Collection group queries are used in the context of making a query to the database, whereas in this case we're not making a query to the database at all... we're triggering a function in response to an event.

The point in the OP about potential problems with documents with the same ID is valid. A potential solution to this would be to concatenate the ID onto any wildcards used in the collection path (which can be found in context.params in the function trigger) when writing to meilisearch. eg wildcard_a-wildcard_b-original_id.

Hope this helps

[bidoubiwa]

Hey @jontybrook

It helps a lot 🔥 Thanks 🙏

A potential solution to this would be to concatenate the ID onto any wildcards used in the collection path (which can be found in context.params in the function trigger) when writing to meilisearch. eg wildcard_a-wildcard_b-original_id.

It's already something I do in other plugins, and I should have considered it for this use case as well.