modify fabric to calculate emissions from REST api

[sichen1234]

Modify fabric to get emissions factors from postgres database in data/postgres/ instead of directly from Fabric's couchdb, possibly by specifying ip address/port connection to postgres or using a docker container if really necessary.

[sichen1234]

After #500, once the data is loaded into postgres, modify the chaincode that is accessing this seed data, for example in https://github.com/hyperledger-labs/blockchain-carbon-accounting/blob/main/emissions-data/chaincode/emissionscontract/typescript/src/lib/emissionsFactor.ts, to access it directly from postgres database.

As a test try accessing any postgres database from chaincode and see if it works.

[brioux]

@sichen1234 getting caught up on this issue as it relates to the data integration mentorship assigned to @Ackintya.
This blog discusses support for other state DBs, and highlights that Fabric only support couchDb or levelDB natively. He mentions a working document on using GO plugins for pluggable ledger state databases.
Is this what you have in mind? If so we need to put together a plan as the solution requires forking Fabric. (they recommend just using couch or levelDb)

[sichen1234]

I don't want to use a different database with Fabric, only that the chain code be able to access an external data source. All the Fabric transactions should stay on whatever database it uses. In the very beginning we got the chain code to connect to Amazon's Dynamodb using external API calls. Can we set it up so the chain code can access postgres as well to get the emissions factors?

…

----- Si Chen Open Source Strategies, Inc. *Why aren't we decarbonizing when it's profitable? How can we fix it? See our Blog <https://www.opensourcestrategies.com/2022/04/13/using-the-blockchain-for-supply-chain-decarbonization-with-emissions-transfers/> and White Paper <https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4082449> *

On Wed, Jun 15, 2022 at 9:30 AM Bertrand Rioux ***@***.***> wrote: @sichen1234 <https://github.com/sichen1234> getting caught up on this issue as it relates to the data integration mentorship <https://wiki.hyperledger.org/display/INTERN/Multiple+Data+Integration+to+Fabric+Climate+Accounting+Network> assigned to @Ackintya <https://github.com/Ackintya>. This blog <https://wiki.hyperledger.org/display/INTERN/Multiple+Data+Integration+to+Fabric+Climate+Accounting+Network> discusses support for other state DBs, and highlights that Fabric only support couchDb or levelDB natively. He mentions a working document on using GO plugins for pluggable ledger state databases <https://docs.google.com/document/d/1ZdxPWdxUwEDwRAKY8tewgjqh2qTZCm6RZ6tU9ccm4dM/edit#heading=h.supvt45riocz> . Is this what you have in mind? If so we need to put together a plan as the solution requires forking Fabric <https://lists.hyperledger.org/g/fabric/topic/replacing_couchdb_with_a/74953512>. (they recommend just using couch or levelDb) — Reply to this email directly, view it on GitHub <#501 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANAS4LKWMDW5A5DENWIGOLVPIAKJANCNFSM5RANDD2A> . You are receiving this because you were mentioned.Message ID: <hyperledger-labs/blockchain-carbon-accounting/issues/501/1156685900@ github.com>

[brioux]

Ok, I understand better the issue now.
The emissions will not be stored on the Fabric state DB (i.e., couch), instead the chaincode calls an external sevice (i.e., postgreSQL DB). This makes sense as it will avoid loading entire datasets into Fabric stateDb.

[brioux]

In this case I think a good first task for @Ackintya is to work on revising the chaincode to pull data from an external resource.

He can start by working on modifying the emissions chaincode to connect to the external postgreSQL database where all the emissions data is now being loaded.

This directory will replace the data loaded into fabric couchDB using egrid-data-loader.

[brioux]

@Ackintya here is an example of where the emissions chaincode will need to be modified.
The function getEmissionsRecord of the EmissionsRecordState should query external Postgres DB for the uuid of the emission record, instead of the internal stateDB (couch).

[brioux]

I have been thinking about this issue over the weeknd. There are the two approaches to using an external database to access emission records.

1. This new approach: the chaincode is designed to query records using an API call. each fabric peer has to submit a query and expects the API to return same result.
2. Old approach: An organization queries the eternal database (e.g., postgres) before calling the chaincode. The organization requests peers to store the record on the Fabric internal state DB (e.g., importUtilityFactor chaincode function). Peers handle audit requests by querying the shared state DB, no API calls.

[brioux]

@Ackintya, @sichen1234 is asking to implement 1. The organization tells the network where to get the data from, or the API address/functions are hard coded into the chaincode.

I read this threads that warns setting up external API calls inside Fabric could result in consensus issues, i.e., if peers receive different results.

There is still Fabric documentation on how to do this here, but may not be updated for recent versions. If we take this route, we need to research this further.

This can be an issue if running a network with a large number of peers and access to the external service is unstable -> peers don't receive the same result. With only a few organizations/peers on the audit channel, and stable connection to the external DB, this should not be a major issue.

If we stick to the old approach, my understanding is that the only difference from how the Fabric network is currently setup is that all the emission records are not written directly to the stateDB (e.g., using the egrid-data.load.sh script). Each organization can setup is own connection to an external emission database (e.g., the postgres data-loader). Only records submitted for audit are written to the state DB (so peers do not have to query the external service).

[brioux]

Benefits of the new approach (assuming consensus is not an issue).

1. The chaincode can be configured to whitelist recognized/trusted emission record APIs
2. No need for internal state DB to replicate existing emission record database