Skip to content

Commit 54816db

Browse files
NewJerseyStyleNewJerseyStyle
authored
Alpha release to help contributors get started building LitePolis packages (#23)
* Create requirements.txt for front end * Create main.py * Create dashboard.py * Create portal.py * Update dashboard.py * Create main.py * Create requirements.txt * Create auth.py * Create db.py * Create public.py * Create secure.py * Update requirements.txt * get-set functions of tables * Create db.sql * Update comments * Create dockerfile * define APIs * update env var list * Update auth.py with docstrings * Update db.py with new format The function `get_user_from_api_key` will cause error at the moment, waiting to be resolved Added docstring waiting for edit, templates for now Added classes for each table waiting for data dictionary before detailed implementation, and docstring * Create db.dbml * Update db.sql * Update README.md * Update README.md * test and debug with StarRocks this file works * Update db.dbml Update with tested SQL * Add files via upload * Update requirements.txt * Update README.md * Update dockerfile * Update dockerfile * Create database.md * Update requirements.txt * Comment py module and Impl Users class * Implemented CRUD on tables * Implement CRUD for apikeys table * Added DB connection health chheck * Implement API endpoints except for few Comments methods * dependency for Unit test * Create test_main.py * Update requirements.txt * Create README.md * Update README.md * Define test_main.py * Correct DIR name * correct DIR name * bugfix non-exist import item * Update public.py formatting returns * implement test cases implemented: - test_read_main() - test_read_secure_main() - test_read_main_invalid_parameter() - test_read_main_not_found() - test_read_user_role() * Update db.py bugfix query setting database of connection changing query string from mysql style to starrocks compatiable by removing qoute `"` in query string `query.get_sql()` format * Update db.py bugfix queryreturn val return is a tuple, get the first item * Update db.py remove hardcoded config & bugfix db fetch * Update secure.py implemented get role api * Update db.py adding docstring * Update db.py bugfix conversation insert keys * Create test_db.py unitests * Update test_db.py bugfix kwarg name errors * Update db.py bugfix class attr name mistake * Update test_db.py bugfix import * Update db.py bugfix based on test cases * Update test_db.py bugfix kwargs * Update db.py bugfix post processing of db return val format * Update test_db.py bugfix kwarg name and args * Update db.py bugfix wrong operator in comparison * Update db.py bugfix API_KEY is Primary Key cannot update so `delete` and then `insert` as update * Update db.py bugfix wrong operator in comparison * Update test_db.py bugfix incomplete test data * Update db.py bugfix inits optional arg must check is not None * Update db.py Added return id for inserts * Update test_db.py bugfix verify DB ops by ID of row * Update test_db.py bugfix sequences and missing id * Update test_db.py * Update db.py starrocks does not support cursor.lastrowid * Update test_db.py * Update db.py bugfix align column names * Update db.py bugfix comment insert data format * Update test_db.py bugfix autoincremental key start from 1 * Update test_db.py bugfix interpret db return val format * Update test_db.py bugfix wrong param * Update db.py bugfix variable scope * Update test_db.py bugfix interpret db return val format * Update db.py impl comments update/approve * Update public.py with docstring * Update public.py * Update test_main.py * Update secure.py docs will be added by tags_metadata respond format to be defined in response_models * Update db.py adding reject of comments * Update secure.py * Update db.py resolving issue #3 * Update test_db.py align with #6 * Update auth.py align with #6 * Update db.py align with #3 and #6 * Update test_db.py align with db.py * Update test_db.pfy fix missing alignment * Update test_db.py bugfix wrong test case * Update secure.py align with db.py update * Update secure.py adding metadata tags #3 * Update secure.py all endpoint uses Request Body and add docstr #3 Request Body based on doc https://fastapi.tiangolo.com/tutorial/body/ Add docstring to all functional endpoints * Update requirements.txt add email-validator * Update secure.py impl Response Model #3 #2 * Create README.md * Update secure.py bugfix `renew` endpoint code deprecation * Update test_main.py update align api doc * align the return related to api keys * fix endpoint url * fix urls and http error codes, param name * bugfix request json verification * beautify * bugfix request to create user * bugfix parameters and asyncs * fix syntax missing comma * bugfix parameter name * handle empty password * temporary bugfix for moderation before impl * Update and rename dockerfile to Dockerfile * Create quickstart.sh * Update and rename Dockerfile to Dockerfile-api * Rename src/Dockerfile-api to src/api/Dockerfile * Update Dockerfile * Update Dockerfile * Update Dockerfile * Update Dockerfile * Create Dockerfile * Update Dockerfile add pytest case * Update secure.py * Update quickstart.sh * Update quickstart.sh import sql file * Update quickstart.sh add docker built * Update quickstart.sh * Update quickstart.sh * Update quickstart.sh * Update secure.py streamlit login with template * Update main.py added a test and split login and allow join conversation without login * Update requirements.txt * Update requirements.txt add streamlit-cookie-controller * Update main.py * Update db.py with validation of user and API key param name beautify * Update secure.py add login endpoint * Update main.py * Update main.py * Update secure.py * Update main.py * Update main.py * Update main.py fix syntax error * Update requirements.txt fix typo * Update main.py import missing dep * Update main.py * Create config.toml * Update main.py * Update portal.py * Added role guest for users not login but trying to view and comment A feature from polis and #13 * Add Python SDK generated by OpenAPI * Add Python SDK generated by OpenAPI * restructured the dirs to reduce scope for core module of LitePolis * changed package name, add setuppy, litepolis as core engine for modularization * Refactor to extreme minimal package manager * Update core.py passes unittest * refactored to use click for CLI interface * resolve #16 test and app break problem * syntax correct for get_apps() * update URI and passing integration test with routers * added cli command to download template for develop new module * add config interface for modules * adding ABC class for integration * hide WIP `RouterIntegrationInterface` * merge README from release branch * Create python-publish.yml (#22) Enable auto publish to PyPI after creating tag and release on GitHub * add config server service and clean up unused file * allow retrival of whole sector * improved automated create script to help developer get started * improved robustness * Update setup.py version format * add database template and finalize the create feature
1 parent cf29d7e commit 54816db

File tree

8 files changed

+766
-9
lines changed

8 files changed

+766
-9
lines changed

‎.github/workflows/python-publish.yml‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
# This workflow will upload a Python Package to PyPI when a release is created
2+
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python#publishing-to-package-registries
3+
4+
# This workflow uses actions that are not certified by GitHub.
5+
# They are provided by a third-party and are governed by
6+
# separate terms of service, privacy policy, and support
7+
# documentation.
8+
9+
name: Upload Python Package
10+
11+
on:
12+
release:
13+
types: [published]
14+
15+
permissions:
16+
contents: read
17+
18+
jobs:
19+
release-build:
20+
runs-on: ubuntu-latest
21+
22+
steps:
23+
- uses: actions/checkout@v4
24+
25+
- uses: actions/setup-python@v5
26+
with:
27+
python-version: "3.x"
28+
29+
- name: Build release distributions
30+
run: |
31+
# NOTE: put your own distribution build steps here.
32+
python -m pip install build
33+
python -m build
34+
35+
- name: Upload distributions
36+
uses: actions/upload-artifact@v4
37+
with:
38+
name: release-dists
39+
path: dist/
40+
41+
pypi-publish:
42+
runs-on: ubuntu-latest
43+
needs:
44+
- release-build
45+
permissions:
46+
# IMPORTANT: this permission is mandatory for trusted publishing
47+
id-token: write
48+
49+
# Dedicated environments with protections for publishing are strongly recommended.
50+
# For more information, see: https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment#deployment-protection-rules
51+
environment:
52+
name: pypi
53+
# OPTIONAL: uncomment and update to include your PyPI project URL in the deployment status:
54+
# url: https://pypi.org/p/YOURPROJECT
55+
#
56+
# ALTERNATIVE: if your GitHub Release name is the PyPI project version string
57+
# ALTERNATIVE: exactly, uncomment the following line instead:
58+
# url: https://pypi.org/project/YOURPROJECT/${{ github.event.release.name }}
59+
60+
steps:
61+
- name: Retrieve release distributions
62+
uses: actions/download-artifact@v4
63+
with:
64+
name: release-dists
65+
path: dist/
66+
67+
- name: Publish release distributions to PyPI
68+
uses: pypa/gh-action-pypi-publish@release/v1
69+
with:
70+
packages-dir: dist/

‎README.md‎

Lines changed: 219 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,23 +1,233 @@
11
# LitePolis
2+
Infrastructure for E-democracy
23
![](https://img.shields.io/badge/status-under_development-red) ![](https://img.shields.io/badge/release-no_release-red)
34

4-
Python port of [polis](https://github.com/compdemocracy/polis)
5-
try to be data scientist friendly and easy to deploy
6-
support horizontal scaling on cloud
5+
Polis is a real-time system for gathering, analyzing and understanding
6+
what large groups of people think in their own words,
7+
enabled by advanced statistics and machine learning.
8+
9+
LitePolis is a Python-based, developer-friendly iteration of [Polis](https://github.com/compdemocracy/polis),
10+
designed to provide a scalable and flexible platform for data scientists and developers.
11+
Our goal is to make it easy to build and deploy data-driven applications giving more flexibility to the community.
12+
13+
## Overview
14+
15+
LitePolis is a modular Python-based system, refactored from [Polis](https://github.com/compdemocracy/polis), designed for scalability and performance. It uses a microservice-like architecture for flexibility and integration. This repository acts as the central package manager, orchestrating the integration of various components for deployment. The package manager automatically discovers and manages routers (APIs), middleware, and UI packages, including dependency resolution. Database interactions are handled as a router (API). Future development will include Docker SDK support for managing dependencies like database servers.
16+
17+
Routers and middleware can be developed independently in separate repositories and integrated during deployment based on the dependencies declared by UI packages. This allows for a highly modular and extensible system.
18+
19+
```mermaid
20+
graph TD
21+
subgraph "Local Development Environment"
22+
CLI[LitePolis CLI]
23+
PM[Package Manager]
24+
CF[Static Config File]
25+
end
26+
27+
subgraph "Package Sources"
28+
PYPI[PyPI Repository]
29+
GH[GitHub Templates]
30+
end
31+
32+
subgraph "Ray Cluster Deployment"
33+
RAY[Ray Cluster]
34+
SERVE[Ray Serve]
35+
MON[Ray Dashboard/Monitoring]
36+
37+
subgraph "Application Components"
38+
API[FastAPI Application]
39+
R[Router Services]
40+
M[Middleware Services]
41+
UI[UI Static Files]
42+
end
43+
end
44+
45+
subgraph "Database Layer"
46+
SR[(StarRocks Data Lakehouse)]
47+
end
48+
49+
CLI -->|User Commands| PM
50+
PM -->|Read/Write| CF
51+
PM -->|Install Dependencies| PYPI
52+
53+
PM -->|Deploy Application| SERVE
54+
SERVE -->|Run| API
55+
API -->|Include| R
56+
API -->|Apply| M
57+
API -->|Serve| UI
58+
59+
R <-->|Query/Store Data| SR
60+
61+
RAY -->|Monitor| MON
62+
SERVE -.->|Part of| RAY
63+
```
64+
65+
**Routers, UI Packages, and Middlewares**
66+
67+
In LitePolis, these components play distinct roles, though their relationship to traditional MVC might not be a direct mapping due to the distributed nature of the system:
68+
69+
* **Routers (APIs):** These are analogous to *Controllers* in a traditional MVC framework. They define the endpoints and logic for handling incoming requests and returning responses. They act as the entry point for all interactions with the system. A database interaction is handled *through* a router, meaning a dedicated router is responsible for communicating with the database. The database itself, along with the data processing logic, represents the *Model* in this context.
70+
71+
* **UI Packages:** These are closer to the *View* component of MVC. They consume the APIs exposed by the routers to present data and interact with the user. While they might contain some control logic, their primary function is to render the user interface and handle user interactions. They then communicate these interactions back to the routers.
72+
73+
* **Middlewares:** These components sit between the routers and the UI, acting as intermediaries. They handle cross-cutting concerns like authentication, authorization, logging, and rate limiting. They are not directly tied to the MVC paradigm but are essential for managing access control, security, and other system-wide functionalities. Thinking of them as handling access control is a reasonable simplification.
74+
75+
```mermaid
76+
graph TD
77+
subgraph "LitePolis Core"
78+
PM[Package Manager] --> DP[Dependency Resolution]
79+
PM --> CP[Component Discovery]
80+
PM --> ORC[Orchestration]
81+
end
82+
83+
subgraph "Component Types"
84+
R[Routers/APIs] --> RA[API Endpoints]
85+
R --> RC[Request Handling]
86+
R --> RDB[Database Access Router]
87+
88+
M[Middlewares] --> MA[Authentication]
89+
M --> MB[Authorization]
90+
M --> MC[Logging]
91+
M --> MD[Rate Limiting]
92+
93+
UI[UI Packages] --> UIA[Static Files]
94+
UI --> UIB[User Interface]
95+
end
96+
97+
PM --> R
98+
PM --> M
99+
PM --> UI
100+
101+
R --> RDB
102+
RDB -.-> DK[Docker Containers]
103+
104+
UI --> DPC[Dependencies Declaration]
105+
DPC --> PM
106+
```
107+
108+
**Scalability and Infrastructure**
109+
110+
LitePolis is designed for scalability and can handle high-volume usage through horizontal scaling:
111+
112+
* **Routers/APIs (Controllers):** Multiple instances of the API servers can be deployed and managed by a load balancer to distribute traffic and ensure high availability. LitePolis support autoscaling on cloud platforms like Google Cloud out-of-box.
113+
* **UI Packages (Views):** Static files for the UI can be served from a content delivery network (CDN) to minimize latency and improve performance. The UI itself can be designed to be stateless, allowing for easy horizontal scaling of the application servers.
114+
* **Model (Database and Data Processing):** LitePolis leverages distributed databases and data processing systems (like [StarRocks on Kubernetes](https://github.com/StarRocks/starrocks-kubernetes-operator/tree/main/examples/starrocks)) that can scale horizontally to handle increasing data volumes and query loads.
115+
116+
This distributed architecture, combined with cloud infrastructure and autoscaling, allows LitePolis to adapt to varying levels of demand and maintain performance even under heavy load, enabling nation-wide high-volume usage.
117+
118+
```mermaid
119+
graph TD
120+
subgraph "Scalability"
121+
SCALE[Scalable Architecture] --> LB[Load Balancing]
122+
SCALE --> CDN[Content Delivery Network]
123+
SCALE --> DB[Distributed Database]
124+
SCALE --> AS[Auto Scaling]
125+
end
126+
```
127+
128+
## Conclusion
129+
130+
This flexible architecture, coupled with a central package manager, simplifies the development process. Developers can focus on building their applications (routers, middleware, UI) while LitePolis handles the underlying infrastructure for scalability and performance. This separation of concerns allows for rapid development and deployment of new features and functionalities.
131+
132+
133+
134+
135+
# TODOs
136+
## Features
137+
138+
* [ ] **Real-time Sentiment Gathering:** Polis gathers and analyzes opinions from large groups of people in real-time.
139+
* [ ] **Open-Ended Feedback:** Participants can express their views in their own words, going beyond simple surveys or polls.
140+
* [ ] **Anonymous Participation:** Participants can contribute anonymously, fostering open and honest dialogue.
141+
* [ ] **Voting Mechanism:** Participants can vote on statements submitted by others, indicating agreement, disagreement, or neutrality.
142+
* [ ] **Advanced Statistical Analysis:** Polis uses machine learning algorithms to identify consensus statements, divisive statements, and patterns in opinions.
143+
* [ ] **Data Visualization and Reporting:** The platform provides real-time data visualization and reporting tools to understand the results of the conversation.
144+
* [ ] **Moderation Tools:** Moderators can manage the conversation, address spam, and ensure a productive environment.
145+
* [ ] **Scalability:** The platform can handle large-scale conversations with many participants.
146+
147+
**Core Functionality:**
148+
149+
* **User Interface:**
150+
* [ ] User registration and authentication.
151+
* [ ] Interface for creating and joining conversations.
152+
* [ ] Interface for submitting statements and voting on them.
153+
* [ ] Interface for viewing conversation data and reports.
154+
* [ ] User-friendly and intuitive design.
155+
* **Backend System:**
156+
* [ ] Database to store user data, conversation data, and voting data.
157+
* [x] API for interacting with the frontend and other services.
158+
* [ ] Secure and reliable infrastructure.
159+
* [x] Scalable architecture to handle large volumes of data and users.
160+
* **Machine Learning Algorithms:**
161+
* [ ] Algorithms to analyze sentiment and identify consensus and divisive statements.
162+
* [ ] Algorithms to personalize the user experience based on their voting history and interests.
163+
* [ ] Algorithms to detect spam and inappropriate content.
164+
* **Data Visualization and Reporting:**
165+
* [ ] Real-time visualizations of conversation data, including sentiment trends, agreement maps, and key statements.
166+
* [ ] Customizable reports and dashboards for analyzing data.
167+
* [ ] Ability to export data in various formats.
168+
* **Moderation Tools:**
169+
* [ ] Tools for managing user accounts and roles.
170+
* [ ] Tools for flagging and removing inappropriate content.
171+
* [ ] Tools for moderating discussions and resolving conflicts.
172+
* **Integration with Third-Party Services:**
173+
* [ ] Integration with translation services for multilingual conversations.
174+
* [ ] Integration with spam filtering services to prevent abuse.
175+
* [ ] Integration with other platforms and tools for data analysis and visualization.
176+
* **Security:**
177+
* [ ] Compliance with relevant privacy regulations (e.g., GDPR).
178+
* **Scalability:**
179+
* [x] Ability to handle large numbers of users and conversations.
180+
* [x] Scalable infrastructure to accommodate increasing demand.
181+
* [ ] Performance optimization for efficient data processing.
182+
* **Accessibility:**
183+
* [ ] Accessible design for users with disabilities.
184+
* [ ] Support for multiple languages and cultural contexts.
185+
* **Documentation:**
186+
* [ ] Comprehensive documentation for developers and users.
187+
* [ ] Tutorials and guides to help users understand the platform.
188+
* **Testing:**
189+
* [ ] Thorough testing of all features and functionality.
190+
* [x] Automated testing to ensure code quality and stability.
7191

8192
## Getting started
9193
Under development...
10194
<!-- something about deployment and configuration -->
195+
### Tryout
196+
use all in one dockerfile
197+
## Advanced usage
198+
### Separate storage
199+
https://www.starrocks.io/blog/four-simple-ways-to-deploy-starrocks
200+
### Separate front end
201+
disable `streamlit`
202+
develop your own front end web/mobile/desktop application with RESTful API docs
203+
### Product deployment
204+
- MVC architecture
205+
- Data lakehouse
206+
- https://www.starrocks.io/blog/four-simple-ways-to-deploy-starrocks
207+
- scaling
208+
- scaling of UI
209+
- scaling of API server
210+
- scaling of database
11211

12212
## Developer manual
13213
### Tech stack
214+
Relationship of containers
14215
```mermaid
15-
title Containers
16216
graph TD;
17-
Streamlit-->FastAPI;
18-
FastAPI-->StarRock;
19-
StarRock-->*storage;
217+
Streamlit-->|View|FastAPI;
218+
FastAPI-->|Control|StarRocks;
219+
StarRocks-->*storage;
20220
```
21-
*storage: Such as Amazon S3, Google Cloud Storage, Azure Blob Storage, and other S3-compatible storage
22221

23-
Access control is provided by `Streamlit-Authenticator`
222+
Data flows from UI to API and store in database
223+
then been accessed by other UI widget through other API endpoints
224+
```mermaid
225+
graph TD;
226+
Streamlit-input-->|input|FastAPI-C_UD;
227+
FastAPI-C_UD-->|write-in|StarRocks;
228+
StarRocks-->*storage;
229+
*storage-->StarRocks;
230+
StarRocks-->|read-out|FastAPI-_R__;
231+
FastAPI-_R__-->|output|Streamlit-display;
232+
```
233+
*storage: Such as Amazon S3, Google Cloud Storage, Azure Blob Storage, and other S3-compatible storage

‎litepolis/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
from .utils import get_config

0 commit comments

Comments
 (0)