Skip to content

Commit fee6ad0

Browse files
patconpatcon
authored
Initial commit
0 parents  commit fee6ad0

46 files changed

Lines changed: 7553 additions & 0 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/deploy.yml‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# An example workflow configuration assuming your kedro project is
2+
# at the root directory
3+
4+
# The name of your workflow
5+
name: Publish and share Kedro Viz
6+
7+
on:
8+
push:
9+
branches:
10+
- main
11+
workflow_dispatch:
12+
13+
jobs:
14+
build:
15+
# The action is currently tested on ubuntu-latest (Recommended)
16+
runs-on: ubuntu-latest
17+
env:
18+
DO_NOT_TRACK: true
19+
KEDRO_POLIS_URL: https://pol.is/report/r29kkytnipymd3exbynkd
20+
steps:
21+
- name: Fetch the repository
22+
uses: actions/checkout@v4
23+
24+
- name: Install lxml build dependencies
25+
run: |
26+
sudo apt-get update
27+
sudo apt-get install -y libxml2-dev libxslt-dev
28+
29+
- name: "Set up Python"
30+
uses: actions/setup-python@v5
31+
with:
32+
python-version: "3.10"
33+
34+
- name: Install uv
35+
uses: astral-sh/setup-uv@v6
36+
37+
- name: Run all pipelines
38+
# Run pipelines for: #ChileDespertó polis conversation (Chilean protest)
39+
# Polis report: https://pol.is/report/r29kkytnipymd3exbynkd
40+
run: uv run make run-pipelines PARAMS="polis_url=$KEDRO_POLIS_URL"
41+
42+
- name: Build static site
43+
run: |
44+
uv run make build
45+
env:
46+
POLIS_URL: ${{ env.KEDRO_POLIS_URL }}
47+
48+
- name: Upload static files as artifact
49+
uses: actions/upload-pages-artifact@v3
50+
with:
51+
path: build/
52+
53+
deploy:
54+
needs: build
55+
permissions:
56+
# The pages and id-token write permissions are required to use
57+
# the action if your GitHub publishing source is a custom
58+
# GitHub Actions workflow
59+
pages: write
60+
id-token: write
61+
62+
# Deploy to the github-pages environment
63+
environment:
64+
name: github-pages
65+
url: ${{ steps.deployment.outputs.page_url }}
66+
67+
runs-on: ubuntu-latest
68+
env:
69+
DO_NOT_TRACK: true
70+
steps:
71+
- name: Deploy to GitHub Pages
72+
id: deployment
73+
uses: actions/deploy-pages@v4

‎.gitignore‎

Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
##########################
2+
# KEDRO PROJECT
3+
4+
__pycache__
5+
6+
# ignore all local configuration
7+
conf/local/**
8+
!conf/local/.gitkeep
9+
10+
# ignore potentially sensitive credentials files
11+
conf/**/*credentials*
12+
13+
# ignore everything in the following folders
14+
data/**
15+
16+
# except their sub-folders
17+
!data/**/
18+
19+
# also keep all .gitkeep files
20+
!.gitkeep
21+
22+
# also keep the example dataset
23+
!data/01_raw/foo.csv
24+
25+
# ignore kedro-viz metadata
26+
.viz
27+
28+
# ignore file based logs
29+
*.log
30+
31+
##########################
32+
# Common files
33+
34+
# IntelliJ
35+
.idea/
36+
*.iml
37+
out/
38+
.idea_modules/
39+
40+
### macOS
41+
*.DS_Store
42+
.AppleDouble
43+
.LSOverride
44+
.Trashes
45+
46+
# Vim
47+
*~
48+
.*.swo
49+
.*.swp
50+
51+
# emacs
52+
*~
53+
\#*\#
54+
/.emacs.desktop
55+
/.emacs.desktop.lock
56+
*.elc
57+
58+
# JIRA plugin
59+
atlassian-ide-plugin.xml
60+
61+
# C extensions
62+
*.so
63+
64+
### Python template
65+
# Byte-compiled / optimized / DLL files
66+
__pycache__/
67+
*.py[cod]
68+
*$py.class
69+
70+
# Distribution / packaging
71+
.Python
72+
build/
73+
develop-eggs/
74+
dist/
75+
downloads/
76+
eggs/
77+
.eggs/
78+
lib/
79+
lib64/
80+
parts/
81+
sdist/
82+
var/
83+
wheels/
84+
*.egg-info/
85+
.installed.cfg
86+
*.egg
87+
MANIFEST
88+
89+
# PyInstaller
90+
# Usually these files are written by a python script from a template
91+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
92+
*.manifest
93+
*.spec
94+
95+
# Installer logs
96+
pip-log.txt
97+
pip-delete-this-directory.txt
98+
99+
# Unit test / coverage reports
100+
htmlcov/
101+
.tox/
102+
.coverage
103+
.coverage.*
104+
.cache
105+
nosetests.xml
106+
coverage.xml
107+
*.cover
108+
.hypothesis/
109+
110+
# Translations
111+
*.mo
112+
*.pot
113+
114+
# Django stuff:
115+
*.log
116+
.static_storage/
117+
.media/
118+
local_settings.py
119+
120+
# Flask stuff:
121+
instance/
122+
.webassets-cache
123+
124+
# Scrapy stuff:
125+
.scrapy
126+
127+
# Sphinx documentation
128+
docs/_build/
129+
130+
# PyBuilder
131+
target/
132+
133+
# Jupyter Notebook
134+
.ipynb_checkpoints
135+
136+
# pyenv
137+
.python-version
138+
139+
# celery beat schedule file
140+
celerybeat-schedule
141+
142+
# SageMath parsed files
143+
*.sage.py
144+
145+
# Environments
146+
.env
147+
.venv
148+
env/
149+
venv/
150+
ENV/
151+
env.bak/
152+
venv.bak/
153+
154+
# mkdocs documentation
155+
/site
156+
157+
# mypy
158+
.mypy_cache/
159+
160+
# mlflow local runs
161+
mlruns/*
162+
163+
.telemetry
164+
test_cache.sqlite

‎Makefile‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# Kedro Polis Pipelines Makefile
2+
# Based on the original Makefile structure with targets for build, run-pipelines, and dev
3+
4+
run-pipelines: ## Run the pipeline script (use PIPELINES= and PARAMS= env vars)
5+
@echo "🚀 Running Kedro pipelines..."
6+
@if [ -n "$(PIPELINES)" ]; then \
7+
if [ -n "$(PARAMS)" ]; then \
8+
echo "Running pipelines: $(PIPELINES) with params: $(PARAMS)"; \
9+
python scripts/run_pipelines.py $(PIPELINES) --params "$(PARAMS)"; \
10+
else \
11+
echo "Running pipelines: $(PIPELINES)"; \
12+
python scripts/run_pipelines.py $(PIPELINES); \
13+
fi; \
14+
else \
15+
if [ -n "$(PARAMS)" ]; then \
16+
echo "Running all pipelines with params: $(PARAMS)"; \
17+
python scripts/run_pipelines.py --all --params "$(PARAMS)"; \
18+
else \
19+
echo "Running all pipelines"; \
20+
python scripts/run_pipelines.py --all; \
21+
fi; \
22+
fi
23+
24+
dev: ## Run kedro viz for development
25+
@echo "🔧 Starting Kedro Viz development server..."
26+
kedro viz
27+
28+
set-tmp-build-config: ## Set temporary build config in base globals (requires POLIS_ID or POLIS_URL env var)
29+
@echo "🔧 Setting temporary build configuration in base globals..."
30+
@if [ -n "$(POLIS_URL)" ]; then \
31+
echo "Using POLIS_URL: $(POLIS_URL)"; \
32+
sed -i.bak 's|build_polis_url: null|build_polis_url: "$(POLIS_URL)"|' conf/base/globals.yml; \
33+
echo "✅ build_polis_url set to $(POLIS_URL) in conf/base/globals.yml"; \
34+
elif [ -n "$(POLIS_ID)" ]; then \
35+
echo "Using POLIS_ID: $(POLIS_ID)"; \
36+
sed -i.bak 's/build_polis_id: null/build_polis_id: $(POLIS_ID)/' conf/base/globals.yml; \
37+
echo "✅ build_polis_id set to $(POLIS_ID) in conf/base/globals.yml"; \
38+
else \
39+
echo "❌ Error: Either POLIS_ID or POLIS_URL environment variable is required"; \
40+
echo "Usage: POLIS_ID=6carwc4nzj make set-tmp-build-config"; \
41+
echo " or: POLIS_URL=https://polis.example.com/6carwc4nzj make set-tmp-build-config"; \
42+
exit 1; \
43+
fi
44+
@echo "⚠️ Remember to run 'make reset-tmp-build-config' after building to reset to null"
45+
46+
reset-tmp-build-config: ## Reset temporary build config to null in base globals
47+
@echo "🔄 Resetting temporary build configuration to null in base globals..."
48+
@sed -i.bak 's|build_polis_id: [^[:space:]]*|build_polis_id: null|' conf/base/globals.yml
49+
@sed -i.bak 's|build_polis_url: "[^"]*"|build_polis_url: null|' conf/base/globals.yml
50+
@rm -f conf/base/globals.yml.bak
51+
@echo "✅ Temporary build configuration reset to null in conf/base/globals.yml"
52+
53+
build: ## Build static site in build directory (requires POLIS_ID or POLIS_URL env var)
54+
@echo "🏗️ Building static site..."
55+
@mkdir -p build
56+
@$(MAKE) set-tmp-build-config
57+
@echo "🔧 Building Kedro Viz..."
58+
@kedro viz build --include-previews || (echo "Build failed, restoring globals..."; $(MAKE) reset-tmp-build-config; exit 1)
59+
@echo "🔄 Restoring globals to null..."
60+
@$(MAKE) reset-tmp-build-config
61+
python scripts/copy_data.py
62+
@echo "🔧 Fixing API file paths..."
63+
python scripts/fix_api_paths.py
64+
@echo "✅ Build completed! Static site ready in build/ directory"
65+
66+
serve: ## Serve the build directory with Python HTTP server (with CORS headers)
67+
@echo "🌐 Starting HTTP server for build directory with CORS support..."
68+
@if [ ! -d "build" ]; then \
69+
echo "❌ Build directory not found. Run 'make build' first."; \
70+
exit 1; \
71+
fi
72+
python scripts/serve_with_cors.py
73+
74+
# Legacy targets for backward compatibility
75+
set-build-polis-id: set-tmp-build-config # Legacy alias for set-tmp-build-config
76+
reset-build-polis-id: reset-tmp-build-config # Legacy alias for reset-tmp-build-config
77+
set-build-polis-url: set-tmp-build-config # Legacy alias for set-tmp-build-config
78+
reset-build-polis-url: reset-tmp-build-config # Legacy alias for reset-tmp-build-config
79+
80+
.PHONY: help build run-pipelines dev serve
81+
82+
help:
83+
@echo 'Usage: make <command>'
84+
@echo
85+
@echo 'where <command> is one of the following:'
86+
@echo
87+
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-15s\033[0m %s\n", $$1, $$2}'
88+
89+
.DEFAULT_GOAL := help

‎README.md‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# Kedro Polis-like Pipelines: Template
2+
3+
This project is an attempt to use Kedro to model [Polis](https://pol.is/home)-like data pipelines.
4+
5+
This repo builds and runs a set of polis-like pipelines via continuous integration, and publishes a small micro-site that exposes all the pipelines and data.
6+
7+
## Usage
8+
9+
You can fork this repo directly, or use it as a template repo for your own Polis conversations.
10+
11+
### Existing Pipeline Repos
12+
13+
- Blacksky: Community Guidelines. [about](https://bsky.app/profile/rude1.blacksky.team/post/3lxx52acerc2s)
14+
- [pipeline repo](https://github.com/patcon/kedro-polislike-pipelines-blacksky) | [micro-site](https://patcon.github.io/kedro-polislike-pipelines-blacksky/?types=parameters,datasets&pid=mean_localmap_bestkmeans&expandAllPipelines=false&sid=b9680c9f) | [data explorer 🔭](https://main--68c53b7909ee2fb48f1979dd.chromatic.com/iframe.html?args=kedroBaseUrl%3Ahttps__COLON____SLASH____SLASH__patcon__DOT__github__DOT__io__SLASH__kedro-polislike-pipelines-blacksky&globals=&id=components-app-kedro-mode--kedro-mode-with-animation&viewMode=story)
15+
- original polis: [report](https://assembly.blacksky.community/report/r9pnvme4e39uy5a3uptmr)
16+
- Austrian Klimarat. [about](https://klimarat.org/)
17+
- Energy
18+
- [pipeline repo](https://github.com/patcon/kedro-polislike-pipeline-klimarat-energy) | [micro-site](https://patcon.github.io/kedro-polislike-pipeline-klimarat-energy/?types=parameters,datasets&pid=mean_localmap_bestkmeans&expandAllPipelines=false&sid=b9680c9f) | [data explorer 🔭](https://main--68c53b7909ee2fb48f1979dd.chromatic.com/iframe.html?args=kedroBaseUrl%3Ahttps__COLON____SLASH____SLASH__patcon__DOT__github__DOT__io__SLASH__kedro-polislike-pipeline-klimarat-energy&globals=&id=components-app-kedro-mode--kedro-mode-with-animation&viewMode=story)
19+
- original polis: [report](https://pol.is/report/r8nssrnnnf2bewvtd5f5h)
20+
- Mobility
21+
- [pipeline repo](https://github.com/patcon/kedro-polislike-pipelines-Klimarat-mobility) | [micro-site](https://patcon.github.io/kedro-polislike-pipelines-Klimarat-mobility/?types=parameters,datasets&pid=mean_localmap_bestkmeans&expandAllPipelines=false&sid=b9680c9f) | [data explorer 🔭](https://main--68c53b7909ee2fb48f1979dd.chromatic.com/iframe.html?args=kedroBaseUrl%3Ahttps__COLON____SLASH____SLASH__patcon__DOT__github__DOT__io__SLASH__kedro-polislike-pipelines-Klimarat-mobility&globals=&id=components-app-kedro-mode--kedro-mode-with-animation&viewMode=story)
22+
- original polis: [report](https://pol.is/report/r5bbmenm6nt3nnmf9dpvk)
23+
- San Juan Islands Land Trust (testing geographic projections)
24+
- [pipeline repo](https://github.com/patcon/kedro-polislike-pipelines-san-juan-islands) | [micro-site]() | [data explorer 🔭](https://main--68c53b7909ee2fb48f1979dd.chromatic.com/iframe.html?args=kedroBaseUrl%3Ahttps__COLON____SLASH____SLASH__patcon__DOT__github__DOT__io__SLASH__kedro-polislike-pipelines-san-juan-islands&globals=&id=components-app-kedro-mode--kedro-mode-with-animation&viewMode=story)
25+
- original polis: [report](https://pol.is/report/r7bhuide6netnbr8fxbyh)
26+
27+
## Background
28+
29+
Polis is a collective intelligence tool for collecting simple agree/disagree data and from that
30+
building maps of the opinion space in which participants reside. This allows sensemaking by
31+
surfacing complexity in the groups that agree/disagree together.
32+
33+
## Goals
34+
35+
- allow for more visibility into existing Polis pipeline
36+
- support exploration of new parameters and algorithms
37+
- support collaboration on these new pipeline variants
38+
- support generation of standardized data types that new UI can be built around
39+
- modularization of pipeline steps
40+
- help determine best architecture for the standalone [`red-dwarf` algorithm library](https://github.com/polis-community/red-dwarf/)
41+
42+
## Usage
43+
44+
```bash
45+
# Build static site
46+
uv run make build
47+
# or: make build
48+
49+
# Run all pipelines
50+
uv run make run-pipelines
51+
# or: make run-pipelines
52+
53+
# Run specific pipelines with parameters
54+
uv run make run-pipelines PIPELINES=bestkmeans PARAMS="polis_id=r29kkytnipymd3exbynkd"
55+
# or: make run-pipelines PIPELINES=bestkmeans PARAMS="polis_id=r29kkytnipymd3exbynkd"
56+
57+
# Start development server
58+
uv run make dev
59+
# or: make dev
60+
61+
# Serve build directory
62+
uv run make serve
63+
# or: make serve
64+
65+
# Show help
66+
make
67+
# or: make help
68+
```

0 commit comments

Comments
 (0)