Features to Enhance Search Capabilities Using OpenSearch

[dpengpeng]

What is the transformation progress of using OpenSearch to enhance the search capability? The latest version of OpenSearch is the code from a year ago. Currently, the interface is the v2 interface of the beta version.

[boring-cyborg]

Thanks for opening your first issue in the Marquez project! Please be sure to follow the issue template!

[thijs-s]

Hi @dpengpeng

Just to let you know, we are working on updating the OpenSearch to the newest version, it's still not released. But you can already use the simple search functionality, which does not need opensearch to work. You can check the api(which will be updated during the next few days after the release of 0.53.1)

Just in case it's here - https://github.com/ilum-cloud/marquez

[dpengpeng]

Hi @dpengpeng

Just to let you know, we are working on updating the OpenSearch to the newest version, it's still not released. But you can already use the simple search functionality, which does not need opensearch to work. You can check the api(which will be updated during the next few days after the release of 0.53.1)

Just in case it's here - https://github.com/ilum-cloud/marquez

@thijs-s Thank you. I will try out your enhanced version. Has the Marquez team stopped maintaining this project? There hasn't been any code update in the last six months.

[thijs-s]

unfortunately yes :(

this is why we created a fork while upstream activity was paused so we could ship performance and security fixes quickly without breaking compatibility. All changes are additive (API-compatible). From 0.53.x we’re aligning with upstream and contributing improvements back. We also don’t use marquez-web internally, we run our own UI optimized for large lineages and plan to open-source it.

[dpengpeng]

unfortunately yes :(

this is why we created a fork while upstream activity was paused so we could ship performance and security fixes quickly without breaking compatibility. All changes are additive (API-compatible). From 0.53.x we’re aligning with upstream and contributing improvements back. We also don’t use marquez-web internally, we run our own UI optimized for large lineages and plan to open-source it.

@thijs-s
Why did the Marquez project choose the PG database to store lineage relationships when theoretically, graph databases should perform better than PG databases? Do you have plans to switch your private project to a graph database?

[thijs-s]

@dpengpeng simplicity with starting a fresh project without too much of external dependencies, but you are totally right. There is no better way of moving forward for enterprise level tools than applying a real graph db over it.

At ilum we are planning to move the current structure, but still hold it in pg with the use of extensions like Apache AGE for full graph support, but if you have any recommendations for an alternative approach, I'm all ears.

[dpengpeng]

@dpengpeng simplicity with starting a fresh project without too much of external dependencies, but you are totally right. There is no better way of moving forward for enterprise level tools than applying a real graph db over it.

At ilum we are planning to move the current structure, but still hold it in pg with the use of extensions like Apache AGE for full graph support, but if you have any recommendations for an alternative approach, I'm all ears.

Apache AGE is indeed a good idea, I will also look into this plugin. My original idea is to completely replace the PG database with the graph database and insert the Openlineage data into the graph database. However, the reconstruction workload will be large.

[thijs-s]

indeed, that might affect just too many users, but I will ask my team to reconsider both options. Maybe we will come up with some strategies for supporting both options during the transition period.

[swar00pduthks]

@thijs-s please keep us updated as well. As I have implemented run lineage and denormalized tables to improve read and write in my fork . so may be it would be good if the original creaters no longer want to maintain this we could help them to keep it updating?

[thijs-s]

@swar00pduthks will do! I'm almost ready with the performance fixes for lineage with more than 800nodes and then will continue with OpenSearch. I would also love the check your improvements! Would you mind if I somehow put it here https://github.com/ilum-cloud/marquez ?