[GraphRAG] Develop Data Ingestion Module with Schema for Flare Mainnet Transactions into Neo4j

[dineshpinto]

• Description:
  Create a robust data ingestion module for the GraphRAG engine. This module builds on the ecosystem Goldsky integration
  and normalizes the raw transaction data, and transforming it into a graph-friendly format suitable for storage in the Neo4j database.
• Acceptance Criteria:
  • The module can successfully connect to the Flare Mainnet to receive a stream of transaction data or batch updates.
  • Incoming data is correctly parsed, normalized to a standard internal representation, and transformed into graph structures (nodes and edges).
  • The transformed graph data is ingested into the Neo4j database, adhering to the pre-defined graph schema.
  • The ingestion process is designed to be robust, capable of handling potential data inconsistencies, retries, and network issues.
  • A detailed graph schema document is produced, clearly defining node labels (e.g., Transaction, Account, Contract, Token), edge types (e.g., SENT_TO, INTERACTED_WITH, DEPLOYED_CONTRACT, TRANSFERRED_TOKEN), and the properties associated with each.
  • The schema is designed to effectively capture the relationships and data points necessary for answering complex contextual queries about blockchain activity.
  • The designed schema is implemented in the Neo4j setup.
• Key Files/Modules Involved (Tentative):
  • flare_ai_kit/rag/graph/schema.py (New file to be created)
  • flare_ai_kit/rag/graph/transformer.py (New file to be created)
• Tasks / Implementation Steps:
  • Create a schema for Flare transactions in flare_ai_kit/rag/graph/schema.py
  • Define how raw transaction data from Goldsky (see [Ecosystem Engine] Integrate Goldsky for Blockchain Data Querying with GraphRAG Engine #30 ) can be transformed using flare_ai_kit/rag/graph/transformer.py into this schema

[Chukwuemeka-James]

Hi dineshpinto, I’d love to take this on. I have solid experience building robust data ingestion pipelines, transforming raw blockchain data into graph-friendly formats, and working with Neo4j. I'm confident in handling schema design, normalization, and ingestion logic. I haven’t used Goldsky directly, but I’ve reviewed its ecosystem and can ramp up on the integration quickly.

[Chukwuemeka-James]

Title: Goldsky Integration: Add v2 Subgraph Support + Suggestion to Load Chain Slug from .env

Hello dineshpinto — the above task which I am currently rapping up builds on [#30], and so I cloned [PR #50] to build on the Goldsky integration work by [@ritik4ever].

While testing the ingestion pipeline, I ran into a couple of things that might be worth improving:

---

v2 Subgraphs Not Currently Supported

It looks like the current Goldsky implementation supports v1-style subgraphs, but doesn’t yet handle v2 subgraphs (with versioned endpoints and slightly different GraphQL structures).

Since my use case depends on a v2-style subgraph, I hit a few blockers here. It might be helpful to:

• Allow custom GraphQL endpoints to be passed in
• Or add logic to detect/handle v2 subgraph structures

Will be wiling to collaborate and implement or test any changes here!

---

Let me know what you think — and thanks again [@ritik4ever] for the good work so far!

[dineshpinto]

Hi @Chukwuemeka-James what do you mean by v2 subgraphs? Can you link the Goldsky documentation on this

[Chukwuemeka-James]

Hi @dineshpinto, the V1 and V2 I mentioned earlier refer to the GOLDSKY_BASE_URL patterns used in the Goldsky class definition (not an official name). I've been working on the Goldsky integration but ran into some issues with the API connection.

I initially used the below url which is what we have in the Goldsky client config:

GOLDSKY_BASE_URL=https://api.goldsky.com

But received 404 errors, even though all .env variables were correctly set:

GOLDSKY_API_KEY  
GOLDSKY_CHAIN_SLUG=flare  
GOLDSKY_PROJECT_NAME=Flare_Goldsky

To debug, I deployed a subgraph using the contract at:
https://flare-explorer.flare.network/address/0x0F081cad5BCed7B2acA1c1D22CdafcB21322B280

I then updated the base URL to:

GOLDSKY_BASE_URL=https://api.goldsky.com/api/public/my_project_id/subgraphs/my_subgraph_name/1.0.0/gn

I will be happy to hop on a short call (google meet) to walk through my progress, or I can open a PR if preferred. Kindly let me know what works best.