Skip to content

Commit 70dc8b6

Browse files
davidgasquezdavidgasquez
authored
📦 Restore legacy portal snapshot for archival (#92)
1 parent ac66ad6 commit 70dc8b6

46 files changed

Lines changed: 6365 additions & 83 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.devcontainer/devcontainer.json‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
{
2+
"name": "Gitcoin Grants Data Portal",
3+
"build": {
4+
"dockerfile": "../Dockerfile",
5+
"context": ".."
6+
},
7+
"customizations": {
8+
"vscode": {
9+
"settings": {
10+
"quarto.render.previewType": "external",
11+
"files.associations": {
12+
"*.sql": "jinja-sql"
13+
},
14+
"[jinja-sql]": {
15+
"editor.formatOnSave": false
16+
},
17+
"python.analysis.autoImportCompletions": true,
18+
"python.analysis.typeCheckingMode": "basic",
19+
"[python]": {
20+
"editor.formatOnSave": true,
21+
"editor.codeActionsOnSave": {
22+
"source.fixAll": "explicit",
23+
"source.organizeImports": "explicit"
24+
},
25+
"editor.defaultFormatter": "charliermarsh.ruff"
26+
},
27+
"notebook.formatOnSave.enabled": true,
28+
"notebook.codeActionsOnSave": {
29+
"source.fixAll": true,
30+
"source.organizeImports": true
31+
}
32+
},
33+
"extensions": [
34+
"charliermarsh.ruff",
35+
"EditorConfig.EditorConfig",
36+
"innoverio.vscode-dbt-power-user",
37+
"ms-python.python",
38+
"ms-python.vscode-pylance",
39+
"ms-toolsai.jupyter",
40+
"quarto.quarto",
41+
"redhat.vscode-yaml",
42+
"tamasfe.even-better-toml",
43+
"VisualStudioExptTeam.vscodeintellicode"
44+
]
45+
}
46+
},
47+
"postCreateCommand": "make setup"
48+
}

‎.editorconfig‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
root = true
2+
3+
[*]
4+
indent_style = space
5+
indent_size = 4
6+
end_of_line = lf
7+
charset = utf-8
8+
trim_trailing_whitespace = true
9+
insert_final_newline = true
10+
11+
[*.yml]
12+
indent_size = 2
13+
14+
[*.json]
15+
indent_size = 2

‎.github/workflows/link_check.yml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
name: Check Portal for broken links
2+
on:
3+
schedule:
4+
- cron: "10 10 * * 1"
5+
workflow_dispatch: # allow manual trigger
6+
jobs:
7+
crawl_for_broken_links:
8+
runs-on: ubuntu-latest
9+
name: Broken-Links-Crawler
10+
steps:
11+
- name: Checking links
12+
uses: ScholliYT/Broken-Links-Crawler-Action@v3
13+
with:
14+
website_url: 'https://grantsdataportal.xyz/'
15+
verbose: 'true'
16+
exclude_url_contained: 'twitter'
17+
max_retry_time: 30
18+
max_retries: 5
19+
max_depth: 1

‎.github/workflows/run.yml‎

Lines changed: 93 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -16,16 +16,99 @@ jobs:
1616
run:
1717
runs-on: ubuntu-latest
1818
steps:
19-
- name: Checkout
20-
uses: actions/checkout@v4
19+
- name: Checkout
20+
uses: actions/checkout@v4
2121

22-
- name: Install uv
23-
uses: astral-sh/setup-uv@v5
24-
with:
25-
enable-cache: true
22+
- name: Set up Python
23+
uses: actions/setup-python@v5
24+
with:
25+
python-version: 3.11
26+
cache: pip
2627

27-
- name: Install dependencies
28-
run: uv sync --no-dev --frozen
28+
- name: Install dependencies
29+
run: |
30+
pip install -e .
2931
30-
- name: Run
31-
run: make run
32+
- name: Run
33+
run: |
34+
make run
35+
env:
36+
COVALENT_API_KEY: ${{ secrets.COVALENT_API_KEY }}
37+
DUNE_API_KEY: ${{ secrets.DUNE_API_KEY }}
38+
39+
- name: Export Database and Log File Sizes
40+
run: |
41+
make tables
42+
echo "Files exported:" >> $GITHUB_STEP_SUMMARY
43+
{
44+
echo "| Filename | File Size |"
45+
echo "| --- | --- |"
46+
ls -lh data/tables | tail -n +2 | awk '{printf "| %s | %s |\n", $9, $5}'
47+
} >> $GITHUB_STEP_SUMMARY
48+
ls -lh data/tables
49+
50+
- name: Install Quarto
51+
uses: quarto-dev/quarto-actions/setup@v2
52+
53+
- name: Render
54+
run: |
55+
make render
56+
57+
- name: Upload Pages Artifact
58+
if: github.event_name == 'push' || github.event_name == 'schedule'
59+
uses: actions/upload-pages-artifact@v3
60+
with:
61+
path: portal/_site
62+
63+
- name: Copy Data Tables and Database
64+
run: |
65+
cp -r data/tables/ portal/_site/data/
66+
cp -r data/local.duckdb portal/_site/data/database.duckdb
67+
68+
- name: Push to IPFS
69+
if: github.event_name == 'push' || github.event_name == 'schedule'
70+
uses: filebase/ipfs-action@master
71+
id: ipfs
72+
with:
73+
path: portal/_site/
74+
service: filebase
75+
pinName: gitcoin-grants-data-portal-tables
76+
filebaseBucket: gitcoin-grants-data-portal
77+
filebaseKey: ${{ secrets.FILEBASE_KEY }}
78+
filebaseSecret: ${{ secrets.FILEBASE_SECRET }}
79+
key: gitcoin-grants-data-portal-db
80+
81+
- name: Update CID file
82+
if: github.event_name == 'push' || github.event_name == 'schedule'
83+
run: echo ${{ steps.ipfs.outputs.cid }} > data/IPFS_CID
84+
85+
- name: Commit and Push
86+
if: github.event_name == 'push' || github.event_name == 'schedule'
87+
run: |
88+
git config --global user.name "GitHub Action"
89+
git config --global user.email "actions@users.noreply.github.com"
90+
git add data/IPFS_CID
91+
if git diff-index --quiet HEAD --; then
92+
echo "No changes to commit"
93+
else
94+
git commit -m "chore: :id: Update CID"
95+
git push
96+
fi
97+
98+
deploy:
99+
if: github.event_name == 'push' || github.event_name == 'schedule'
100+
needs: run
101+
102+
permissions:
103+
pages: write
104+
id-token: write
105+
106+
environment:
107+
name: github-pages
108+
url: ${{ steps.deployment.outputs.page_url }}
109+
110+
runs-on: ubuntu-latest
111+
steps:
112+
- name: Deploy to GitHub Pages
113+
id: deployment
114+
uses: actions/deploy-pages@v4

‎.gitignore‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ dist/
1414
downloads/
1515
eggs/
1616
.eggs/
17-
!lib/
17+
lib/
1818
lib64/
1919
parts/
2020
sdist/
@@ -158,3 +158,9 @@ cython_debug/
158158
# and can be added to the global gitignore or merged into this file. For a more nuclear
159159
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
160160
#.idea/
161+
162+
# All under data except gitkeep
163+
data/*
164+
!data/.gitkeep
165+
!data/IPFS_CID
166+
!data/index.html

‎Dockerfile‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
FROM mcr.microsoft.com/devcontainers/python:3.11
2+
3+
# Install Quarto
4+
RUN curl -sL $(curl https://quarto.org/docs/download/_prerelease.json | grep -oP "(?<=\"download_url\":\s\")https.*${ARCH}\.deb") --output /tmp/quarto.deb \
5+
&& dpkg -i /tmp/quarto.deb \
6+
&& rm /tmp/quarto.deb
7+
8+
# Environment Variables
9+
ENV DAGSTER_HOME "/home/vscode"
10+
ENV WORKSPACE "/workspaces/gitcoin-grants-data-portal"
11+
12+
# Working Directory
13+
WORKDIR ${WORKSPACE}

‎Makefile‎

Lines changed: 32 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,38 @@
1-
.DEFAULT_GOAL := all
1+
.DEFAULT_GOAL := run
22

3-
.PHONY: .uv
4-
.uv:
5-
@uv --version || echo 'Please install uv: https://docs.astral.sh/uv/getting-started/installation/'
3+
run:
4+
@dagster asset materialize --select \* -m ggdp
65

7-
.PHONY: setup
8-
setup: .uv
9-
uv sync --frozen
6+
tables:
7+
@python -c 'from ggdp import db; db.export_database_to_parquet("data/local.duckdb", "data/tables");'
108

11-
.PHONY: dev
129
dev:
13-
uv run dg dev
10+
@dagster dev
1411

15-
.PHONY: run
16-
run:
17-
uv run dg launch --assets '*'
12+
setup:
13+
@command -v uv >/dev/null 2>&1 || pip install -U uv
14+
@uv venv
15+
@uv pip install -U -e ".[dev]"
16+
17+
sql:
18+
@harlequin "./data/local.duckdb"
19+
20+
test:
21+
@cd dbt && dbt test
22+
23+
preview:
24+
@quarto preview portal
25+
26+
render:
27+
@cd dbt && dbt docs generate
28+
@mkdir -p dbt/target/docs
29+
@cp dbt/target/*.json dbt/target/index.html dbt/target/graph.gpickle dbt/target/docs/
30+
@quarto render portal
31+
@cp -r dbt/target/docs/ portal/_site/dbt
32+
33+
publish:
34+
@quarto publish gh-pages portal --no-prompt
1835

19-
.PHONY: web
20-
web:
21-
uv run python -m http.server 8000 -d web
36+
clean:
37+
@rm -rf portal/.quarto data/*.parquet data/*.duckdb portal/_site data/tables
38+
@rm -rf dbt/target dbt/logs dbt/dbt_packages

‎README.md‎

Lines changed: 47 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -1,59 +1,68 @@
1-
<h1 align="center">࿂ Grants Data Portal ࿂</h1>
1+
<p align="center">
2+
<img src="https://user-images.githubusercontent.com/1682202/271937380-10d6e036-5fe4-4ea6-b3b4-8e3001c21289.png" data-canonical-src="https://user-images.githubusercontent.com/1682202/271937380-10d6e036-5fe4-4ea6-b3b4-8e3001c21289.png" width="100" />
3+
</p>
24

3-
<div align="center">
4-
<img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/davidgasquez/grants-data-portal?style=flat-square">
5-
<img alt="GitHub" src="https://img.shields.io/github/license/davidgasquez/grants-data-portal?style=flat-square">
6-
<img alt="GitHub Workflow Status" src="https://img.shields.io/github/actions/workflow/status/davidgasquez/grants-data-portal/run.yml?style=flat-square">
7-
<a href="https://www.drips.network/app/projects/github/davidgasquez/gitcoin-grants-data-portal" target="_blank"><img src="https://www.drips.network/api/embed/project/https%3A%2F%2Fgithub.com%2Fdavidgasquez%2Fgitcoin-grants-data-portal/support.png?background=light&style=github&text=project&stat=none" alt="Support gitcoin-grants-data-portal on drips.network" height="20"></a>
8-
</div>
5+
# 🌲 Gitcoin Grants Data Portal
96

10-
11-
Open source, serverless, and local-first Data Platform for Public Goods Grants Data. This data hub improves data access and empower data scientists to conduct research that guides community driven analysis.
7+
Open source, serverless, and local-first Data Platform for Gitcoin Grants Data. This data hub improves data access and empower data scientists to conduct research that guides community driven analysis.
128

139
> [!IMPORTANT]
1410
>
15-
> The previous version of the portal (browse it [here](https://github.com/davidgasquez/grants-data-portal/tree/ea74827476e25fbc3f94aa052c15c7b681a2d183)) is **deprecated**. You can find better and more up to date data on [Open Source Observer](https://www.opensource.observer/). This new approach/version **focuses on curating Open Source Observer datasets with some smaller datasets** and distributing them.
16-
17-
## 📖 Overview
11+
> This project is now **deprecated** and **archived**. It is no longer maintained.
12+
> The underlying Grants Stack infrastructure was sunset by Gitcoin. `grants-stack-indexer-v2` is no longer running.
13+
> Gitcoin confirmed that Grants Stack and Grants Lab were shut down, and indexer-backed apps are no longer accessible ([governance post](https://gov.gitcoin.co/t/focusing-gitcoins-future-sunsetting-grants-stack-eol-may-2025/20333/7)).
14+
> You can find better and more up to date datasets on [Open Source Observer](https://www.opensource.observer/).
1815
19-
The repository contains code and artifacts to help process grants around the Public Goods Ecosystem. The portal is based on the principles of [Datadex](https://datadex.datonic.io/).
16+
![DAG](https://github.com/davidgasquez/gitcoin-grants-data-portal/assets/1682202/2095974c-f8c4-430b-9c93-dd2a0598127e)
2017

21-
### 📦 Key Features
18+
## 📂 Gitcoin Grants Data
2219

23-
- **Open**: Code and data are open source and relies on open standards and formats.
24-
- **Permissionless Collaboration**: Collaborate on data, models, and pipelines. Fork the repo and run the platform locally in minutes. No constraints or platform lock-ins.
25-
- **Decentralization Options**: Runs on a laptop, server, CI runner, or even on decentralized compute networks like Bacalhau. No local setup required.
26-
- **Data as Code**: Each commit generates and pushes all datasets as files to an Object Storage.
27-
- **Modular Flexibility**: Replace, extend, or remove individual components. Compatible with tons of tools. At the end of the day, datasets are Parquet files.
28-
- **Low Friction Data Usage**: Raw and processed data is available to anyone openly. Use whatever tool you want!
29-
- **Modern Data Engineering**: Supports data engineering essentials such as typing, testing, materialized views, and development branches. Utilizes best practices, including declarative transformations, and utilizes state-of-the-art tools like DuckDB.
30-
31-
## ⚙️ Quick Start
20+
Datasets are living as Parquet files on IPFS! You can get them all at the [IPFS CID](https://raw.githubusercontent.com/davidgasquez/gitcoin-grants-data-portal/main/data/IPFS_CID) pointer available in this repository, surfaced also in the [Data Portal Data section](https://grantsdataportal.xyz/data/).
3221

33-
Make sure you have [uv](https://docs.astral.sh/uv/getting-started/installation/) installed. Clone the repository and install dependencies:
22+
The following command will give you a working URL to explore the available tables.
3423

3524
```bash
36-
git clone https://github.com/davidgasquez/grants-data-portal.git
37-
cd grants-data-portal
38-
make setup
25+
# Get the latest IPFS CID
26+
LATEST_IPFS_CID=$(curl https://raw.githubusercontent.com/davidgasquez/gitcoin-grants-data-portal/main/data/IPFS_CID)
27+
28+
# Print the Gateway URL with all the tables
29+
echo https://ipfs.filebase.io/ipfs/$LATEST_IPFS_CID/data/
3930
```
4031

41-
Run the Dagster UI:
32+
### 📌 IPNS
4233

43-
```bash
44-
make dev
45-
```
34+
You can also go to [`ipns://k51qzi5uqu5dhn3p5xdkp8n6azd4l1mma5zujinkeewhvuh5oq4qvt7etk9tvc`](https://k51qzi5uqu5dhn3p5xdkp8n6azd4l1mma5zujinkeewhvuh5oq4qvt7etk9tvc.ipns.cf-ipfs.com/data/), which points to the latest available data via IPNS.
35+
36+
You can now read the files from your favorite tools. E.g: `pd.read_parquet('https://grantsdataportal.xyz/data/allo_rounds.parquet')`
37+
38+
## 📖 Overview
39+
40+
The repository contains code and artifacts to help process Gitcoin Grants data from the [Grants Stack Indexer API](https://github.com/gitcoinco/grants-stack-indexer). It is an instance of [Datadex](https://github.com/davidgasquez/datadex) allowing you and everyone else to:
41+
42+
- Add new data sources to the portal, collaborate on better models (ala Dune) or submit an interesting analysis.
43+
- All in a permissionless way. Don't ask, fork it and improve the models, add a new source or update any script.
44+
- Declarative stateless transformations tracked in git, executed in GitHub Actions and published to IPFS. Data, artifacts (like the entire DuckDB database), and models all version controlled.
45+
- Share and explore dashboards and report with the world!
46+
47+
> [!TIP]
48+
> You can read more on the [motivation and the approach on my blog](https://davidgasquez.github.io/gitcoin-data/)!
49+
50+
## ⚙️ Quick Start
51+
52+
The fastest way to start working on the Data Portal is via [VSCode Remote Containers](https://code.visualstudio.com/docs/remote/containers). Once inside the develpment environment, you can run `make dev` to spin up the Dagster UI.
53+
54+
The development environment can also run in your browser thanks to GitHub Codespaces! Just click on the badge below to get started.
4655

47-
You can now access the Dagster UI at [http://localhost:3000](http://localhost:3000)!
56+
[![badge](https://github.com/codespaces/badge.svg)](https://codespaces.new/davidgasquez/gitcoin-grants-data-portal)
4857

49-
## 🛠️ Contributing
58+
### 🛠️ Contributing
5059

51-
This project is in active development. You can help by giving ideas, answering questions, reporting bugs, proposing enhancements, improving the documentation, and fixing bugs.
52-
Some ways you can contribute to this project:
60+
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change. There are multiple interesting ways to contribute to this project:
5361

54-
- Adding new data sources.
55-
- Improving the data quality of existing datasets.
56-
- Adding tests to the data pipelines.
62+
- Propose a new [dataset](https://github.com/davidgasquez/gitcoin-grants-data-portal/issues/new)
63+
- Add [new data sources](ggdp/assets/allo.py)
64+
- Improve [dbt project](dbt/) models
65+
- Write a one off [report](reports/)
5766

5867
### 📄 License
5968

‎data/.gitkeep‎

Whitespace-only changes.

‎data/IPFS_CID‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
bafybeiatd2kw56ulooqd2n6sbfbvadzqp54alyfyvypvapnsj3pznwbm6i

0 commit comments

Comments
 (0)