How to properly check document for field value exist?

[d47081]

I'm using this library for simple web search project where crawler check URL for duplicates before insert new one

$url      = trim($url);
$crc32url = crc32($url);

if (!$index->search('@url "' . $url . '"')
            ->filter('crc32url', $crc32url)
            ->limit(1)
            ->get()
            ->getTotal())
{
    $index->addDocument(
        [
            'url' => $url,
            'crc32url' => $crc32url
        ]
    );
}

I'm not sure this way is correct, and it's also write requests to the query.log

Thanks for any advice

[sanikolaev]

where crawler check URL for duplicates before insert new one

Doesn't https://github.com/manticoresoftware/manticoresearch-php/blob/master/docs/usage.md#replace work? Can you use your crc32url as the ID for your table?

[d47081]

Thanks for reply, I've just replaced tables to use crc32url as ID.

Currently, request looks like this:
$index->search('')->filter('id', $crc32url)->limit(1)->get()->getTotal();

But the problem I can't use the replace construction, because my crawler scan pages for new documents (links there) and I can't use replace because link in document could be already indexed, but in current document know nothing about it content until crawler parse it in queue.

In other words,
https://github.com/YGGverse/Yo/blob/main/src/cli/document/crawl.php#L439

This solution works, but make query.log full of empty requests.