Returned documents as objects, for metadata

[Pierstoval]

Hello, I've only seen your slides (wasn't there to see the show unfortunately), and this looks extremely promising!

In your setup, you show that returned documents are usually arrays.

I wondered if you had thought about using a few layers here, to have an internal class, like SearchResult, and each item in it would be an instance of a SearchDocument or whatever naming suits better, which would contain an internal private array $data property that would contain the actual data.

The role of this layer would allow 3 things IMO:

• Internal data is still an array, because that's the best normalized method and that's what's returned by most APIs anyway, and it would be the "default way" to fetch the documents, for DX and ease of "getting started" with the project
• Storing metadata, like "formatted" fields, and other query results (like total queries, pagination information, etc.) would be extremely convenient. These metadata would have to be optional, because it must be taken in account that not all providers might provide them, but nullable fields are definitely the way to go here: null means the provider didn't give the information (so either change method or change provider), and non-null means you can interact with your index more fluently.
• The SearchDocument class might contain an object version of the data: it would drastically ease the integration of a data mapper, deserializer, denormalizer, making both accesses possible, and data mappers wouldn't even need to be strict about fields, so you could always have access to both the array version and the object one, mostly to allow the provider to return "old data" that isn't in sync with "new mapping": this way, userland can detect that and act accordingly: update the object with new kind of data, trigger deprecations, drop indices, etc.

WDYT?

[alexander-schranz]

@Pierstoval Really thank you, I appreciate your feedback and time to go through my slides 🙏 .

We had in the past discussion before SEAL about such a general SearchDocument for the different CMS and how such document should look like and we didn't find a common structure which we all were happy with it. So when created SEAL it was really about creating the common communication part to the different search engines like doctrine DBAL does and at current state not use or define any kindof SearchDocument object, as ontop of SEAL that can still be achieved via a Wrapper which returns the Document in there favorite format.

That is also kind of planned that on top of SEAL additonal packages should provide that Typesafety Objects. So the target is that in future you can create your own Search Document class specially optimized for your usecase and and your own properties. Similar how a ODM or ORM works: #81

so you could always have access to both the array version and the object one, mostly to allow the provider to return "old data" that isn't in sync with "new mapping"

I kind of like your input and thinking here. But would definitly see this as part of #81 to have that logic not inside the core of SEAL instead of an own package. But I think we are also kind of limited in SEAL because of the wide range of supported Engines to a very strict schema which need to match, not all engine support nested objects and so SEAL in some cases need to flatten it and the strict schema is required to unflatten it again and unknown fields so thrown away. So like in for database migration for real blue green deployment also for SEAL you should add the new field, deploy it and later if you migrated all data to the new field remove it.

I will link your points inside the #81 as I think it some great feedback and things which we could keep in mind for that part.

[Pierstoval]

Maybe a more thorough audit should be done in order to determine which metadata are commonly returned alongside documents by most search engines. I mean, pagination is often taken in account, but I remember 10 years ago an ElasticSearch setup using its metadata capabilities to return things like "similar results", or "most appreciated similar products" or things like that, which were purely metadata and not documents themselves.
As this might be engine-specific, but some engines might share common concepts, conducting an audit would definitely help knowing whether an "upper abstraction level" could help.

For example, if we consider a SearchResult object, containing an array<Document> $documents and array<array> $rawData properties:

• $rawData could be used for both migrations (aka "fields not in the mapping but returned by the provider", whatever the reason), so it would be quite niche, but still useful, and will be used for the mapping/deserialisation/denormalisation
• $documents would be a list of Document objects, as said above.

Each Document object contains a ->getData() and a ->getRawData() methods. The ->getData() would be the only data that can be altered or mutated by SEAL internals, with a "hydration" process that will be taken care of by a list of "hydrators": a hydrator is just a class/callback receiving 2 things: the currently being altered data (so order matters), and the never-altered raw data (to make checks and hydration in a middleware-like process).

This way, you have an internal method to automatically map your $rawData <=> Document::$data to whatever mapping is already set by userland (which is currently denormalised as an array by SEAL), but you can also add another layer of hydration that would actually use a real data mapper (jane's auto mapper, symfony's new data mapper, symfony serializer, etc.) to denormalise the array into an Object.
Something like a "generic-like" phpdoc could be used to get something like this with the example from #81 :

use Schranz\Search\SEAL\Schema\Field;
use Schranz\Search\SEAL\Schema\Index;

#[Index(name: 'news')]
class News {
    #[Field\IdentifierField('id')]
    private string $id;
    
    #[Field\TextField('title')]
    private string $title;
    
    #[Field\TypedField('header', 'type', [
        'image' => [
            'media' => new Field\IntegerField('media'),
        ],
        'video' => [
            'media' => new Field\TextField('media', searchable: false),
        ],
    ])]
    private array $header;
    
    // ...
}



/** @var CmsIg\Seal\SearchResult<News> $results */
$results = $sealEngine->createSearchBuilder('news')
    ->addFilter(/* ... */)
    ->getResult();

This is not a perfect solution, mostly because PHP doesn't have generics (well, maybe soon? 😁), but many IDEs and static analysers are actually capable of understanding this kind of syntax (or by using @template and other kind of phpdoc tags).

The default behavior would still be the same: just a normal PHP array, similar to the $rawData but stripped from the unmapped fields. Maybe you could even have zero fields mapped and just go wild with nothing but raw calls (nobody should ever recommend that, but for small projects, I don't know, it helps bootstrapping stuff quickly, and some people like that).
And with a small setup, i.e a configured data mapper, and a mapped object (like in #81) referenced to with phpdoc, and you're set.

The role of the container classes (SearchResult and Document) are only to keep a bit more control over it, having metadata both internal to SEAL and related to the actual provider, while still allowing to do the most unimaginable things in the internal $data property with custom hydrators.