Best Practices for getting/creating an index

[tacman]

This code feels like it could be done better. I simply want to get an index if it exists, or optionally create it if it doesn't.

There is no hasIndex method, so is the best way to try to get it and look for a 404?

Is creating an index an async call? It seems like it shouldn't be, because after I create an index I want to use it, and not have to check if it exists. So in this case I'm waiting for the index creating task to complete, but it feels like creating an index should be really high priority. Everything else can be queued, but you can't queue to an index that doesn't exist.

    public function getOrCreateIndex(string $indexName, string $key = 'id', bool $autoCreate = true): ?Indexes
    {
        $client = $this->getMeiliClient();
        try {
            $index = $client->getIndex($indexName);
        } catch (ApiException $exception) {
            if ($exception->httpStatus === 404) {
                if ($autoCreate) {
                    $task = $this->waitForTask($this->getMeiliClient()->createIndex($indexName, ['primaryKey' => $key]));
                    $index = $client->getIndex($indexName);
                } else {
                    $index = null;
                }
            } else {
                dd($exception, $exception::class);
            }
        }
        return $index;
    }

The documentation example has a 0 returned for creating the index. Is there any way to make this sync, or is how I'm doing it above the correct way?

[tacman]

I guess I could alter my wait method to accept a callback or dispatch an event when the task is complete.

    public function waitForTask(array|string|int $taskId, ?Indexes $index = null, bool $stopOnError = true, mixed $dataToDump = null): array
    {

        if (is_array($taskId)) {
            $taskId = $taskId['taskUid'];
        }
        if ($index) {
            $task = $index->waitForTask($taskId);
        } else {
            // e.g index creation, when we don't have an index.  there's probably a better way.
            while (
                ($task = $this->getMeiliClient()->getTask($taskId))
                && (($status = $task['status']) && !in_array($status, ['failed', 'succeeded']))
            ) {
                if (isset($this->logger)) {
//                    $this->logger->info(sprintf("Task %s is at %s", $taskId, $status));
                }
                usleep(50_000);
            };
            if ($status == 'failed') {
                if ($stopOnError) {
                    $this->logger?->warning(json_encode($dataToDump ?? [], JSON_PRETTY_PRINT+JSON_UNESCAPED_SLASHES));
                    throw new \Exception("Task has failed " . $task['error']['message']);
                }
            }
        }

        return $task;
    }

[tacman]

On a related note, I often purge an index when I need to configure new settings, usually related to tweaking the data itself. So I have this cumbersome method, which could be combined with the create (above), to do something with waiting for the task to complete.

    public function reset(string $indexName)
    {
        $client = $this->getMeiliClient();
        try {
            $index = $client->getIndex($indexName);
            $response = $client->deleteIndex($indexName);
            $task = $this->waitForTask($response['taskUid'], $index);
//            $this->io()->info("Deletion Task is at " . $task['status']);
            $this->logger->info("Index " . $indexName . " has been deleted.");
        } catch (ApiException $exception) {
            if ($exception->errorCode == 'index_not_found') {
                try {
//                    $this->io()->info("Index $indexName does not exist.");
                } catch (\Exception) {
                    //
                }
//                    continue;
            } else {
                dd($exception);
            }
        }
    }

Feedback welcome about how to improve.

[irevoire]

Hey @tacman,

I’m not 100% sure why you want to do that, but I’ll answer your questions and propose another solution, maybe depending on your workflow.

Is creating an index an async call? It seems like it shouldn't be, because after I create an index I want to use it, and not have to check if it exists. So in this case I'm waiting for the index creating task to complete, but it feels like creating an index should be really high priority. Everything else can be queued, but you can't queue to an index that doesn't exist.

Yes, creating an index is async. This has not always been the case, and it caused a lot of issues to have it sync while the document addition was async.
But since all operations are async, you can immediately start queueing your other tasks (settings update, document addition, etc...) without waiting for the index to be created.

---

I guess your issue is that when doing searches, you receive a 404, and you would prefer the index to be created instantaneously and empty, right?
An alternative solution might be to keep your outdated index, create a new temporary index, add your settings, document additions, etc., and finally swap both indexes so the search moves automatically from the outdated index to the new index.

[tacman]

Thanks! I didn't realize you could enqueue documents before an index was completely created. Knowing that means I can get rid of the waitForTask for the most part, though if creating the index fails I guess I will have a lot of other tasks that will fail as well.

On a related best practices note, is there any advantage to updating the settings before adding documents? I'm thinking there probably is.

I also realize that I've been wasting space by not deleting tasks when they're complete.

[Kerollmops]

though if creating the index fails I guess I will have a lot of other tasks that will fail as well.

DocumentAddition, SettingsUpdates, and all the other tasks create the index if it doesn't exist. An index creation failure is really rare and unlikely to happen. Document updates have more chances to fail.

[irevoire]

On a related best practices note, is there any advantage to updating the settings before adding documents? I'm thinking there probably is.

Yes definitely! If you update your settings later on, the engine will have to re-index everything on the settings update.
Also, it may be doing too much work during the document addition, so both the initial document addition and settings update may be slower when you send the settings at the end.

I also realize that I've been wasting space by not deleting tasks when they're complete.

About that, if you know for sure that you can remove the tasks it’s a good practice to do it, but the engine should do it automatically anyway at some point.

[tacman]

Hmm. You can enqueue a delete index request on an index that doesn't exist. It will fail, but I won't know that until I check the task. I guess I was using an earlier version of the API that returned an error code if the index didn't exist. Or something.

Regardless, I now have a much better handle on what's going on, thanks for your clarification.

[tacman]

OK, here's where I'm stuck. I dispatch a message to create an index. Then I want to update the settings. Do do that, in the PHP API, I call

$index = $client->getIndex($uId);

but that will fail if the index isn't created yet, as it is not an async call. Then I can call $index->updateSettings(), which is an async call, and then I can add documents (also async).

Is there a way to call updateSettings as an async call that doesn't require getIndex()? I think that is my chicken-and-egg problem.

[dureuill]

sounds like an issue in the PHP SDK 🤔

The corresponding JS SDK function doesn't make the same check and the Rust SDK goes through the trouble of specifying "without any check or doing an HTTP call."

Is there maybe an alternative PHP function that gets a handle to an index without checking for its existence?

[tacman]

in the php client, $client->index($uId) gets the index endpoint, $client->getIndex($uId) does a fetch and requires that the index exist. The example use the latter. The documentation for the client could be improved to clarify that, I created an issue there.