Market Data Ingestion and BigQuery Integration

Introduction

Building upon the foundation of the original Market Data Ingestion Project, this enhanced version takes a leap forward by adopting the Infrastructure as Code paradigm.

By utilizing Terraform, the project ensures consistent, repeatable and version-controlled infrastructure setups.

Getting Started

Prerequisites

Before you begin, make sure you have the following:

• Python 3.x installed
• Google Cloud Platform account with BigQuery access
• API access credentials for the market data source

Installation

1. Clone this repository to your local machine:

   git clone https://github.com/your-username/market-data-ingesting-terraform.git

2. Navigate to the project directory:

   cd market-data-ingesting-terraform

3. Initialize all the environment variables

   source gcp_project/init.sh

4. Init project :

   make init

5. Build project:

   make build

This command init init terraform and install requirements. This command build create seamlessly GCP ressources through the execution of the terraform apply command. For a detailed view of the Terraform configuration refer to the build folder.

For uninstallation of the project, execute the following command :

make cleanup

This command remove all the GCP resources through the execution of the terraform destroy command.

Usage

See Market Data Ingestion Project

Configuration

See Market Data Ingestion Project

Testing

See Market Data Ingestion Project

Contributing

Contributions are welcome! If you have any ideas, improvements, or bug fixes, please feel free to open an issue or submit a pull request.

License

This project is licensed under the MIT License.