Algolia Integration

[SachaG]

I've been working on a new Algolia integration with @pixelastic. Couple things that need to be changed in core:

• Create new all.new.async, all.edit.async, etc. callback hooks that affect all collections.
• Add support for adding custom properties to createCollection:

const Rooms = createCollection({
  collectionName: 'Rooms',
  typeName: 'Room',
  schema,
  resolvers: getDefaultResolvers('Rooms'),
  mutations: getDefaultMutations('Rooms'),
  description: 'An individual property listing',
  algolia: {
    foo: 123,
    bar: {
      baz: 'lorem ipsum'
    }
  }
});

[pixelastic]

As a reference, here is the working code that can add, edit and delete records on Algolia when they are updated in Vulcan:

import { addCallback } from 'meteor/vulcan:core';
import algoliasearch from 'algoliasearch';
const client = algoliasearch('APP_ID', 'API_KEY');
const index = client.initIndex('vulcan');


function recordFromItem(input, collection) {
  const record = {
    ...input,
    objectID: input._id
  }
  delete record._id;
  return record;
}


async function addOrEditRecord(newItem, user, collection) {
  const record = recordFromItem(newItem, collection);
  index.saveObject(record);
}

async function deleteRecord(input) {
  index.deleteObject(input._id);
}

addCallback('movies.new.async', addOrEditRecord);
addCallback('movies.edit.async', addOrEditRecord);
addCallback('movies.remove.async', deleteRecord);

[pixelastic]

The API we had in mind would be something similar to what we have for the Rails integration:

class Product < ActiveRecord::Base
  include AlgoliaSearch

  algoliasearch do
    # list of attribute used to build an Algolia record
    attributes :title, :subtitle, :description, :likes_count, :seller_name

    # the `searchableAttributes` (formerly known as attributesToIndex) setting defines the attributes
    # you want to search in: here `title`, `subtitle` & `description`.
    # You need to list them by order of importance. `description` is tagged as
    # `unordered` to avoid taking the position of a match into account in that attribute.
    searchableAttributes ['title', 'subtitle', 'unordered(description)']

    # the `customRanking` setting defines the ranking criteria use to compare two matching
    # records in case their text-relevance is equal. It should reflect your record popularity.
    customRanking ['desc(likes_count)']
  end

end

Basically it's a way, at the collection level, to define which attributes should be saved in Algolia (you might not want to send emails and/or password hashes). It also allow to define some Algolia-specific config, like which fields should be made searchable, or what should be used for the popularity ranking.

We would listen to a callback when a collection is created. We would then check if a matching Algolia index exists. If it does not exist, we create it with the needed config. If it exists, we don't do anything. If we need to update the config, we should allow for some kind of cli manual command like vulcan algolia reindex movies.

Also, if people don't add anything to their collection config, we should be able to create a default Algolia configuration from the schema that should be "good enough". It will search into all fields equally (not the best relevance), but it would work.

Happy to help provide more guidance on feedback on this from the Algolia side if needed.

[Discordius]

Just as a reference:

I needed to do a bunch of more complicated sharding with Algolia, breaking posts and comments down into paragraphs, since you can't send objects that are larger than a few kb to Algolia.

But doing that automatically seems really hard, but it does seem like a pretty common use-case, if you want any form of longform text. Here is the code that I wrote for this:

Posts.toAlgolia = (post) => {
  const algoliaMetaInfo = {
    _id: post._id,
    userId: post.userId,
    url: post.url,
    title: post.title,
    slug: post.slug,
    baseScore: post.baseScore,
    score: post.score,
    status: post.status,
    legacy: post.legacy,
    commentCount: post.commentCount,
    userIP: post.userIP,
    createdAt: post.createdAt,
    postedAt: post.postedAt,
    isFuture: post.isFuture,
    viewCount: post.viewCount,
    lastCommentedAt: post.lastCommentedAt,
    draft: post.draft,
  };
  const postAuthor = Users.findOne({_id: post.userId});
  if (postAuthor) {
    algoliaMetaInfo.authorSlug = postAuthor.slug;
    algoliaMetaInfo.authorDisplayName = postAuthor.displayName;
    algoliaMetaInfo.authorUserName = postAuthor.username;
  }
  const postFeed = RSSFeeds.findOne({_id: post.feedId});
  if (postFeed) {
    algoliaMetaInfo.feedName = postFeed.nickname;
    algoliaMetaInfo.feedLink = post.feedLink;
  }
  let postBatch = [];
  let paragraphCounter =  0;
  let algoliaPost = {};
  const body = (post.content ? htmlToText.fromString(contentToHtml(post.content)) : (post.htmlBody ? htmlToText.fromString(post.htmlBody) : post.body))
  if (body) {
    body.split("\n\n").forEach((paragraph) => {
      algoliaPost = {
        ...algoliaMetaInfo,
        objectID: post._id + "_" + paragraphCounter,
        body: paragraph,
      }
      paragraphCounter++;
      postBatch.push(_.clone(algoliaPost));
    })
  } else {
    postBatch.push(_.clone(algoliaMetaInfo));
  }
  return postBatch;
}

[pixelastic]

For large texts, what we suggest is to split into smaller chunks as you mentioned. What we do in our WordPress, Jekyll and DocSearch projects is having one record for each <p> of text, and then group them using the distinct setting of Algolia (usually on the url). This keeps you out of the 10kb limit on records and also greatly improve the relevance of results (as you can now display exactly the part of the text that is matching).

We have plans on making this whole process externalized into its own npm package at some point, but I have no ETA for that at the moment.

[SachaG]

@pixelastic I created a new repo and invited you: https://github.com/VulcanJS/vulcan-algolia

Can you push what you have so far? (just the vulcan-algolia package though, not the whole app)

[pixelastic]

✔ Done. There isn't much more than what I actually pasted above, but it might serve as a base. Happy to help on Algolia-related questions if you need a hand on best practices/architecture of code.