Secondary index and full tet query in Hugegraph

[mkj-git]

In Janusgraph secondary index used to be e.g elastic search, same way in hugegraph also has that concept?

 graph.schema().indexLabel("personByAge")
               .onV("person") // Apply to vertices of type "person"
               .by("age") // Index the "age" property
               .secondary() // Use a secondary index
               .ifNotExist()
               .create();

What is the role of secondary() here and how it works?

Also, full text search isnt supported yet, so what is the future plan for this?

    // For full-text query (not supported now)
    SEARCH(3, "search"),

[dosubot[bot]]

The secondary() method in HugeGraph is used to create a secondary index on a vertex or edge label. This type of index improves the performance of queries that filter vertices or edges based on property values. In your example, the secondary() method is used to create an index on the "age" property of the "person" vertex label, allowing for efficient querying by this property ^[1].

Regarding full-text search, it is currently not supported in HugeGraph, and I couldn't find specific information about future plans or a roadmap for supporting full-text search in the repository ^[2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[mkj-git]

[dosubot[bot]]

I couldn't find specific information on whether HugeGraph's secondary index is dependent on external storage or search servers. However, HugeGraph supports various storage backends for storing secondary indexes, including Cassandra, ScyllaDB, HBase, RocksDB, MySQL, Palo, PostgreSQL, and Hstore ^[1]. Unfortunately, details on how secondary indexes are stored internally across these different storage backends are not available.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[imbajin]

In Janusgraph secondary index used to be e.g elastic search, same way in hugegraph also has that concept?

 graph.schema().indexLabel("personByAge")
               .onV("person") // Apply to vertices of type "person"
               .by("age") // Index the "age" property
               .secondary() // Use a secondary index
               .ifNotExist()
               .create();

What is the role of secondary() here and how it works?

Also, full text search isnt supported yet, so what is the future plan for this?

    // For full-text query (not supported now)
    SEARCH(3, "search"),

@dosu, you should refer #884 (comment) to answer the question in English

The hugegraph database index support is already relatively comprehensive. Graph database indexes generally include the following aspects:

• Graph Index / Edge Index: Mainly used to accelerate the retrieval of a vertex's associated edges, generally using adjacency lists or cross-linked lists, and can also use hash indexes. hugegraph uses adjacency lists.

• Supernode Index: Mainly used to accelerate the retrieval of some associated edges of supernodes, such as querying the call edges of the 12315 number in the most recent week. hugegraph is implemented through sortkey, which is equivalent to an enhanced version of the adjacency list.

• Secondary Index (ordinary attribute index): Mainly used to accelerate the retrieval of vertices based on attributes, or the retrieval of edges based on attributes. Databases generally use btree or hash implementations.

• Range Index (comparable attribute index): Mainly used to accelerate the retrieval of vertices or edges based on attribute ranges, such as people with age>18. Databases generally use btree implementations.

• Shard Index (attribute index that can be retrieved by prefix + comparison): Mainly used to accelerate the retrieval of vertices or edges based on attribute ranges, such as people with city=Beijing && age>18.

• Search Index (full-text searchable attribute index): Mainly used to accelerate the retrieval of vertices or edges based on any text segment or word in the attribute value, such as people with city contains Bay.

The implementation of attribute indexes in hugegraph depends on the storage and mainly relies on lsm-tree; the implementation principle of full-text indexes is to perform word segmentation on the attribute text and then store the index.

Reference document: 2.5 IndexLabel

Perhaps we can use some automatic translation plugins or LLM to automatically summarize and translate some common historical questions

[dosubot[bot]]

The secondary() method in HugeGraph is used to create a secondary index on a vertex or edge label, which helps improve query performance by allowing efficient retrieval based on specific attributes. This type of index is typically implemented using B-tree or hash structures ^[1].

Regarding full-text search, HugeGraph already supports it through the search() method, which allows for full-text search capabilities ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[mkj-git]

So what I think is that Hugegraph right now has very limited capability in terms of search. Since external search engine support isn't supported right now therefore that pain will be there till the time its not implemented.

I feel supporting external search engine like Elasticsearch, and vector DB will make this even more powerful.

[imbajin]

1. The reason why we changed from JG's cheat mode to a built-in tokenizer to achieve simple full-text indexing is because in production environment, ES will encounter many problems in a large number of point and edge situations(And it has increased a significant amount of write/query process burden and maintenance costs, also have the issue of data consistency)
   • In addition, based on actual user feedback, the demand for complete full-text indexing is not very strong, and graph databases are not attempting to create a universal DB collection similar to PostgreSQL
   • This requires a significant amount of manpower to maintain different backend/storage systems. Initially, this may seem like a good idea, but later iterations may encounter a lot of trouble. This is why JG's data structure has not changed much over the years, as it provides 2 heavy backend abstractions, making it difficult to optimize and adjust performance separately (otherwise the cost would be too high)
2. We do consider implementing a built-in vector index engine(like faiss-java or lucene) , but please note that it is not an external implementation of a separate vector database (Too heavy)

[mkj-git]

I agree that having a inbuilt full-text search is better than external one, but is it scalable and does it support same level of query support as ES or any other search engine server?

Does hugegraph support any vector indexing right now?

[tomasciudad-indra]

Thanks for this discussion, we are in the same position now, we are looking to integrate hugegraph with elastic/opensearch for full text search, do you have in mind or in the roadmap?

[imbajin]

@tomasciudad-indra Thanks for the follow-up. Short answer: we do not have a short-term plan to add a separate full-text indexing engine in HugeGraph; vector indexing is the current priority, but we are open to well-argued proposals from the community.

A practical direction would be an external module that writes full-text indexes asynchronously. That keeps the core HugeGraph codebase minimally invasive, allows the module to be maintained independently, and keeps the impact on OLTP latency low while still preserving eventual consistency.

Also, PR #2922 already contains the initial vector index integration design, so comments and ideas there are very welcome.

[tomasciudad-indra]

Thanks @imbajin for your quick answer, I will review the PR of course. Thanks