Skip to content

Commit

Permalink
Add option to map fieldnames to different typesearch fieldnames
Browse files Browse the repository at this point in the history
  • Loading branch information
@guenth39
guenth39 committed Apr 24, 2023
1 parent 44e01a7 commit e218e11
Show file tree
Hide file tree
Showing 5 changed files with 38 additions and 20 deletions.
18 changes: 9 additions & 9 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -48,15 +48,15 @@ firebase ext:install typesense/firestore-typesense-search --project=[your-projec

When you install this extension, you'll be able to configure the following parameters:

| Parameter | Description |
|-----------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Firestore Collection Path | The Firestore collection that needs to be indexed into Typesense. |
| Firestore Collection Fields | A comma separated list of fields that need to be indexed from each Firestore document. Leave blank to index all fields. |
| Flatten Nested Documents | Should nested documents in Firestore be flattened before they are indexed in Typesense? Set to "Yes" for Typesense Server versions v0.23.1 and below, since indexing Nested objects is natively supported only in Typesense Server v0.24 and above. |
| Typesense Hosts | A comma-separated list of Typesense Hosts (only domain without https or port number). For single node clusters, a single hostname is sufficient. For multi-node Highly Available or (Search Delivery Network) SDN Clusters, please be sure to mention all hostnames in a comma-separated list. |
| Typesense API Key | A Typesense API key with admin permissions. Click on "Generate API Key" in cluster dashboard in Typesense Cloud. |
| Typesense Collection Name | Typesense collection name to index data into (you need to create this collection in Typesense yourself. This extension does not create the Typesense Collection for you). |
| Cloud Functions location | Where do you want to deploy the functions created for this extension? You usually want a location close to your database. For help selecting a location, refer to the [location selection guide](https://firebase.google.com/docs/functions/locations). |
| Parameter | Description |
|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Firestore Collection Path | The Firestore collection that needs to be indexed into Typesense. |
| Firestore Collection Fields | A comma separated list of fields that need to be indexed from each Firestore document. Leave blank to index all fields. With a "=" separated another field name can be specified for a field, e.g. "field1=otherFieldName1". Thus the firestore field "field1" in typesense is called "otherFieldName1". |
| Flatten Nested Documents | Should nested documents in Firestore be flattened before they are indexed in Typesense? Set to "Yes" for Typesense Server versions v0.23.1 and below, since indexing Nested objects is natively supported only in Typesense Server v0.24 and above. |
| Typesense Hosts | A comma-separated list of Typesense Hosts (only domain without https or port number). For single node clusters, a single hostname is sufficient. For multi-node Highly Available or (Search Delivery Network) SDN Clusters, please be sure to mention all hostnames in a comma-separated list. |
| Typesense API Key | A Typesense API key with admin permissions. Click on "Generate API Key" in cluster dashboard in Typesense Cloud. |
| Typesense Collection Name | Typesense collection name to index data into (you need to create this collection in Typesense yourself. This extension does not create the Typesense Collection for you). |
| Cloud Functions location | Where do you want to deploy the functions created for this extension? You usually want a location close to your database. For help selecting a location, refer to the [location selection guide](https://firebase.google.com/docs/functions/locations). |

> ⚠️ You'll notice that there is no way to configure the port number or protocol.
This is because this extension only supports connecting to Typesense running HTTPS on Port 443, since your data goes from Firebase to Typesense over the public internet and we want your data to be encrypted in transit.
Expand Down
4 changes: 2 additions & 2 deletions extension.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -69,8 +69,8 @@ params:
- param: FIRESTORE_COLLECTION_FIELDS
label: Firestore Collection Fields
description: >-
A comma separated list of fields that need to be indexed from each Firestore document. Leave blank to index all fields.
example: field1,field2,field3
A comma separated list of fields that need to be indexed from each Firestore document. Leave blank to index all fields. With a "=" separated another field name can be specified for a field, e.g. "field1=otherFieldName1". Thus the firestore field "field1" in typesense is called "otherFieldName1".
example: field1,field2=otherFieldName2,field3
default: ""
required: false
- param: FLATTEN_NESTED_DOCUMENTS
Expand Down
2 changes: 1 addition & 1 deletion functions/src/backfillToTypesenseFromFirestore.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ const validateBackfillRun = (snapshot) => {
module.exports = functions.handler.firestore.document
.onWrite(async (snapshot, context) => {
functions.logger.info("Backfilling " +
`${config.firestoreCollectionFields.join(",")} fields in Firestore documents ` +
`${Array.from(config.firestoreCollectionFields.keys()).join(",")} fields in Firestore documents ` +
`from ${config.firestoreCollectionPath} ` +
`into Typesense Collection ${config.typesenseCollectionName} ` +
`on ${config.typesenseHosts.join(",")}`);
Expand Down
26 changes: 22 additions & 4 deletions functions/src/config.js
View file
Original file line number Diff line number Diff line change
@@ -1,10 +1,27 @@
/**
* @param {string} fieldNames
* @return {Map} map of field names where key is the field name in Firestore and value is the field name in Typesense
* @example
* parseFieldNames("foo=bar, baz") => Map { "foo" => "bar", "baz" => "baz" }
* parseFieldNames("foo=bar, baz=qux") => Map { "foo" => "bar", "baz" => "qux" }
* parseFieldNames("foo=bar, baz=qux,") => Map { "foo" => "bar", "baz" => "qux" }
* parseFieldNames("foo, baz = qux, bar , ") => Map { "foo" => "foo", "baz" => "qux", "bar" => "bar" }
*/
const parseFieldNames = (fieldNames) => new Map(
fieldNames.split(",")
.filter((v) => v)
.map(
(f) => {
const [key, value = key] = f.split("=").map((p) => p.trim());
return [key, value];
},
),
);

module.exports = {
firestoreCollectionPath: process.env.FIRESTORE_COLLECTION_PATH,
firestoreCollectionFields:
(process.env.FIRESTORE_COLLECTION_FIELDS || "")
.split(",")
.map((f) => f.trim())
.filter((f) => f),
parseFieldNames(process.env.FIRESTORE_COLLECTION_FIELDS || ""),
shouldFlattenNestedDocuments: process.env.FLATTEN_NESTED_DOCUMENTS === "true",
typesenseHosts:
process.env.TYPESENSE_HOSTS.split(",").map((e) => e.trim()),
Expand All @@ -15,3 +32,4 @@ module.exports = {
typesenseBackfillTriggerDocumentInFirestore: "typesense_sync/backfill",
typesenseBackfillBatchSize: 1000,
};

8 changes: 4 additions & 4 deletions functions/src/utils.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ const mapValue = (value) => {

/**
* @param {DocumentSnapshot} firestoreDocumentSnapshot
* @param {Array} fieldsToExtract
* @param {Map<string,string>} fieldsToExtract
* @return {Object} typesenseDocument
*/
exports.typesenseDocumentFromSnapshot = (
Expand All @@ -33,12 +33,12 @@ exports.typesenseDocumentFromSnapshot = (

let entries = Object.entries(data);

if (fieldsToExtract.length) {
entries = entries.filter(([key]) => fieldsToExtract.includes(key));
if (fieldsToExtract.size) {
entries = entries.filter(([key]) => fieldsToExtract.has(key));
}

// Build a document with just the fields requested by the user, and mapped from Firestore types to Typesense types
const mappedDocument = Object.fromEntries(entries.map(([key, value]) => [key, mapValue(value)]));
const mappedDocument = Object.fromEntries(entries.map(([key, value]) => [fieldsToExtract.get(key), mapValue(value)]));

// using flat to flatten nested objects for older versions of Typesense that did not support nested fields
// https://typesense.org/docs/0.22.2/api/collections.html#indexing-nested-fields
Expand Down

0 comments on commit e218e11

Please sign in to comment.