[FEATURE] Make complex Python objects more easily working with CocoIndex engine

[georgeh0]

This originates from the comment from #1083 by @nightscape:

would this also be a step toward allowing "registering" custom datatypes?
I have classes that are not fully under my control and not compatible with CocoIndex (e.g. Azure DocAI AnalyzeResult which is neither a DataClass nor a Pydantic model, or LlamaIndex ImageNode which is a Pydantic model but with a complex field that CocoIndex does not understand). Currently I'm converting them back and forth between a CocoIndex-compatible representation and the format that the tools need, but this is rather cumbersome and does not scale well.
It would be nice if I could provide an instance of some class which gives CocoIndex the required field information so that it can use the custom types like the predefined ones.

---

❤️ Contributors, please refer to 📙Contributing Guide.
Unless the PR can be sent immediately (e.g. just a few lines of code), we recommend you to leave a comment on the issue like I'm working on it or Can I work on this issue? to avoid duplicating work. Our Discord server is always open and friendly.

[georgeh0]

CocoIndex has limitations of type bindings mostly because we need our data to be serializable, so currently we use disciplined type systems and not all Python object can be directly used (we have similar constraints as a database).

To address the problem raised by @nightscape, there're two directions that can help:

• Leverage Json type already supported by CocoIndex. Allow users t o register Python types together with serialize/deserialize methods. Users can directly use these types more conveniently.

• Support native data types residing in host language (Python) - i.e. we don't even need to serialization/deserialization between two languages, which will be more efficient. We'll define a few interfaces like "get a field", "iterate over elements" and CocoIndex engine can leverage them for certain purposes. Serialize/deserialize capability is optional - if missing, it means we'll lose some power like caching the value for reuse in next time reprocessing, for this single step.

For me to have a better idea about the real use case and make the better design decision, a question for @nightscape: in you use case, are you more likely to treat the object (e.g. Azure DocAI AnalyzeResult, LlamaIndex ImageNode, as you mentioned) as a whole object that you directly pass the entire thing to the next transformation step, or you need to be able to operate on finer granularity by CocoIndex (e.g. do transformation for each element of a specific field)? Thanks!

[nightscape]

Hi @georgeh0 , thanks for getting back so quickly!
Both are actually useful:
The Azure DocAI AnalyzeResult would be passed and used as a full structure, for the LlamaIndex ImageNode we update individual fields and want to pipe the rest of the fields through unmodified.
Serialization/on-disk caching would also be great in order to e.g. retry runs failed due to exceeded rate limits or after fixing a bug in a later stage of the pipeline (which would require a restart and therefore lose in-memory objects).
Right now we are using Dagster and it's working very well, but its mechanisms to update downstream assets when something changes upstream is much less elegant than CocoIndex.