Merge pull request #30 from NewJerseyStyle/develop

release candidate

Author: NewJerseyStyle

README.md

@@ -2,239 +2,102 @@
 Infrastructure for E-democracy  
 ![](https://img.shields.io/badge/status-alpha_test-orange) ![](https://img.shields.io/badge/release-developer_facing-yellow) ![PyPI - Version](https://img.shields.io/pypi/v/litepolis)
 
-Polis is a real-time system for gathering, analyzing and understanding
-what large groups of people think in their own words,
-enabled by advanced statistics and machine learning.
+## Overview
 
-LitePolis is a Python-based, developer-friendly iteration of [Polis](https://github.com/compdemocracy/polis),
-designed to provide a scalable and flexible platform for data scientists and developers.
-Our goal is to make it easy to build and deploy data-driven applications giving more flexibility to the community.
+LitePolis is an advanced, Python-based infrastructure designed for building **customizable opinion collection systems**, extending beyond the capabilities of the original [Polis](https://github.com/compdemocracy/polis). It offers a **developer-friendly** environment focused on flexibility and power.
 
-## TL;DR
-* [Build Your First LitePolis API Router in 5 Minutes! 🚀](https://github.com/NewJerseyStyle/LitePolis/wiki/Tutorial:-Creating-New-LitePolis-Packages-with-litepolis%E2%80%90cli)
-* [Example of database package](https://github.com/NewJerseyStyle/LitePolis-database-example)
-* [Example of router package](https://github.com/NewJerseyStyle/LitePolis-router-example)
+Built with a modular, microservice-like architecture, LitePolis is **distributed by default**, leveraging the [Ray framework](https://www.ray.io/) for inherent **auto-scaling** capabilities right out of the box. This ensures your applications can handle large groups and high traffic volumes efficiently.
 
-> ⚠️ The LitePolis is a complex system with this package manager to enable serve in cluster. Current stage is beta testing with developer while building more documents and testing middleware integration.
+The core of LitePolis is a central package manager that discovers and orchestrates various components – API Routers, Middleware, and UI Packages. This modularity allows developers and data scientists to easily build, deploy, and iterate on sophisticated data-driven e-democracy tools and other opinion-gathering applications.
 
-## Overview
+## Getting Started: Develop to Deploy LitePolis App in 7 Steps\!
+
+This tutorial guides you through building and deploying a minimal LitePolis application, consisting of a simple API backend (a Router package).
+
+**Prerequisites:**
+
+  * Python (3.8+ recommended) and pip installed.
+  * Git installed.
+  * Access to a running Ray Cluster. For local development, you can start one easily:
+    ```bash
+    pip install ray[serve] # Install Ray with the Serve component
+    ray start --head --dashboard-host 0.0.0.0 # Start a local single-node cluster
+    # Access the Ray Dashboard at http://127.0.0.1:8265
+    ```
+    *(Refer to [Ray Cluster Setup Docs](https://docs.ray.io/en/latest/cluster/getting-started.html) for more advanced setups like multi-node or Kubernetes)*
+
+### 1. Install the LitePolis CLI
 
-LitePolis is a modular Python-based system, refactored from [Polis](https://github.com/compdemocracy/polis), designed for scalability and performance.  It uses a microservice-like architecture for flexibility and integration. This repository acts as the central package manager, orchestrating the integration of various components for deployment.  The package manager automatically discovers and manages routers (APIs), middleware, and UI packages, including dependency resolution. Database interactions are handled as a router (API). Future development will include Docker SDK support for managing dependencies like database servers.
-
-Routers and middleware can be developed independently in separate repositories and integrated during deployment based on the dependencies declared by UI packages.  This allows for a highly modular and extensible system.
-
-```mermaid
-graph TD
-    subgraph "Local Development Environment"
-        CLI[LitePolis CLI]
-        PM[Package Manager]
-        CF[Static Config File]
-    end
-
-    subgraph "Package Sources"
-        PYPI[PyPI Repository]
-        GH[GitHub Templates]
-    end
-
-    subgraph "Ray Cluster Deployment"
-        RAY[Ray Cluster]
-        SERVE[Ray Serve]
-        MON[Ray Dashboard/Monitoring]
-        
-        subgraph "Application Components"
-            API[FastAPI Application]
-            R[Router Services]
-            M[Middleware Services]
-            UI[UI Static Files]
-        end
-    end
-
-    subgraph "Database Layer"
-        SR[(StarRocks Data Lakehouse)]
-    end
-
-    CLI -->|User Commands| PM
-    PM -->|Read/Write| CF
-    PM -->|Install Dependencies| PYPI
-    
-    PM -->|Deploy Application| SERVE
-    SERVE -->|Run| API
-    API -->|Include| R
-    API -->|Apply| M
-    API -->|Serve| UI
-    
-    R <-->|Query/Store Data| SR
-    
-    RAY -->|Monitor| MON
-    SERVE -.->|Part of| RAY
+The `litepolis-cli` is your main tool for creating, managing, and deploying LitePolis packages.
+
+```bash
+pip install litepolis
+# Verify installation
+litepolis --version
 ```
 
-**Routers, UI Packages, and Middlewares**
-
-In LitePolis, these components play distinct roles, though their relationship to traditional MVC might not be a direct mapping due to the distributed nature of the system:
-
-* **Routers (APIs):** These are analogous to *Controllers* in a traditional MVC framework. They define the endpoints and logic for handling incoming requests and returning responses.  They act as the entry point for all interactions with the system.  A database interaction is handled *through* a router, meaning a dedicated router is responsible for communicating with the database. The database itself, along with the data processing logic, represents the *Model* in this context.
-
-* **UI Packages:** These are closer to the *View* component of MVC. They consume the APIs exposed by the routers to present data and interact with the user.  While they might contain some control logic, their primary function is to render the user interface and handle user interactions.  They then communicate these interactions back to the routers.
-
-* **Middlewares:** These components sit between the routers and the UI, acting as intermediaries. They handle cross-cutting concerns like authentication, authorization, logging, and rate limiting.  They are not directly tied to the MVC paradigm but are essential for managing access control, security, and other system-wide functionalities.  Thinking of them as handling access control is a reasonable simplification.
-
-```mermaid
-graph TD
-    subgraph "LitePolis Core"
-        PM[Package Manager] --> DP[Dependency Resolution]
-        PM --> CP[Component Discovery]
-        PM --> ORC[Orchestration]
-    end
-
-    subgraph "Component Types"
-        R[Routers/APIs] --> RA[API Endpoints]
-        R --> RC[Request Handling]
-        R --> RDB[Database Access Router]
-        
-        M[Middlewares] --> MA[Authentication]
-        M --> MB[Authorization]
-        M --> MC[Logging]
-        M --> MD[Rate Limiting]
-        
-        UI[UI Packages] --> UIA[Static Files]
-        UI --> UIB[User Interface]
-    end
-
-    PM --> R
-    PM --> M
-    PM --> UI
-    
-    R --> RDB
-    RDB -.-> DK[Docker Containers]
-    
-    UI --> DPC[Dependencies Declaration]
-    DPC --> PM    
+### 2. Create Your First API Router Package
+
+API Routers handle backend logic and data requests. Let's create a simple "hello world" router.
+
+```bash
+# Use the CLI to bootstrap a new router package
+litepolis create router litepolis-router-simple-api
+
+# Navigate into the new directory
+cd litepolis-router-simple-api
 ```
 
-**Scalability and Infrastructure**
+This command creates a standard Python package structure. Look inside `litepolis_router_simple_api/main.py` (or similar). It will contain boilerplate code using FastAPI. Let's make sure it has a basic endpoint:
 
-LitePolis is designed for scalability and can handle high-volume usage through horizontal scaling:
+```python
+# litepolis_router_simple_api/main.py (ensure/modify it looks like this)
+from fastapi import APIRouter
 
-* **Routers/APIs (Controllers):**  Multiple instances of the API servers can be deployed and managed by a load balancer to distribute traffic and ensure high availability. LitePolis support autoscaling on cloud platforms like Google Cloud out-of-box.
-* **UI Packages (Views):**  Static files for the UI can be served from a content delivery network (CDN) to minimize latency and improve performance. The UI itself can be designed to be stateless, allowing for easy horizontal scaling of the application servers.
-* **Model (Database and Data Processing):**  LitePolis leverages distributed databases and data processing systems (like [StarRocks on Kubernetes](https://github.com/StarRocks/starrocks-kubernetes-operator/tree/main/examples/starrocks)) that can scale horizontally to handle increasing data volumes and query loads.
+router = APIRouter()
 
-This distributed architecture, combined with cloud infrastructure and autoscaling, allows LitePolis to adapt to varying levels of demand and maintain performance even under heavy load, enabling nation-wide high-volume usage.
+@router.get("/hello")
+async def read_root():
+    return {"message": "Hello from my-simple-api!"}
 
-```mermaid
-graph TD
-    subgraph "Scalability"
-        SCALE[Scalable Architecture] --> LB[Load Balancing]
-        SCALE --> CDN[Content Delivery Network]
-        SCALE --> DB[Distributed Database]
-        SCALE --> AS[Auto Scaling]
-    end
+# LitePolis package manager will discover this 'router' object
 ```
 
-## Conclusion
-
-This flexible architecture, coupled with a central package manager, simplifies the development process.  Developers can focus on building their applications (routers, middleware, UI) while LitePolis handles the underlying infrastructure for scalability and performance.  This separation of concerns allows for rapid development and deployment of new features and functionalities.
-
-
-
-
-# TODOs
-## Features
-
-* [ ] **Real-time Sentiment Gathering:** Polis gathers and analyzes opinions from large groups of people in real-time.
-* [ ] **Open-Ended Feedback:** Participants can express their views in their own words, going beyond simple surveys or polls.
-* [ ] **Anonymous Participation:** Participants can contribute anonymously, fostering open and honest dialogue.
-* [ ] **Voting Mechanism:** Participants can vote on statements submitted by others, indicating agreement, disagreement, or neutrality.
-* [ ] **Advanced Statistical Analysis:** Polis uses machine learning algorithms to identify consensus statements, divisive statements, and patterns in opinions.
-* [ ] **Data Visualization and Reporting:** The platform provides real-time data visualization and reporting tools to understand the results of the conversation.
-* [ ] **Moderation Tools:** Moderators can manage the conversation, address spam, and ensure a productive environment.
-* [ ] **Scalability:** The platform can handle large-scale conversations with many participants.
-
-**Core Functionality:**
-
-* **User Interface:**
-    * [ ] User registration and authentication.
-    * [ ] Interface for creating and joining conversations.
-    * [ ] Interface for submitting statements and voting on them.
-    * [ ] Interface for viewing conversation data and reports.
-    * [ ] User-friendly and intuitive design.
-* **Backend System:**
-    * [ ] Database to store user data, conversation data, and voting data.
-    * [x] API for interacting with the frontend and other services.
-    * [ ] Secure and reliable infrastructure.
-    * [x] Scalable architecture to handle large volumes of data and users.
-* **Machine Learning Algorithms:**
-    * [ ] Algorithms to analyze sentiment and identify consensus and divisive statements.
-    * [ ] Algorithms to personalize the user experience based on their voting history and interests.
-    * [ ] Algorithms to detect spam and inappropriate content.
-* **Data Visualization and Reporting:**
-    * [ ] Real-time visualizations of conversation data, including sentiment trends, agreement maps, and key statements.
-    * [ ] Customizable reports and dashboards for analyzing data.
-    * [ ] Ability to export data in various formats.
-* **Moderation Tools:**
-    * [ ] Tools for managing user accounts and roles.
-    * [ ] Tools for flagging and removing inappropriate content.
-    * [ ] Tools for moderating discussions and resolving conflicts.
-* **Integration with Third-Party Services:**
-    * [ ] Integration with translation services for multilingual conversations.
-    * [ ] Integration with spam filtering services to prevent abuse.
-    * [ ] Integration with other platforms and tools for data analysis and visualization.
-* **Security:**
-    * [ ] Compliance with relevant privacy regulations (e.g., GDPR).
-* **Scalability:**
-    * [x] Ability to handle large numbers of users and conversations.
-    * [x] Scalable infrastructure to accommodate increasing demand.
-    * [ ] Performance optimization for efficient data processing.
-* **Accessibility:**
-    * [ ] Accessible design for users with disabilities.
-    * [ ] Support for multiple languages and cultural contexts.
-* **Documentation:**
-    * [ ] Comprehensive documentation for developers and users.
-    * [ ] Tutorials and guides to help users understand the platform.
-* **Testing:**
-    * [ ] Thorough testing of all features and functionality.
-    * [x] Automated testing to ensure code quality and stability.
-
-## Getting started
-Under development...
-<!-- something about deployment and configuration -->
-### Tryout
-use all in one dockerfile
-## Advanced usage
-### Separate storage
-https://www.starrocks.io/blog/four-simple-ways-to-deploy-starrocks
-### Separate front end
-disable `streamlit`
-develop your own front end web/mobile/desktop application with RESTful API docs
-### Product deployment
-- MVC architecture
-- Data lakehouse
-  - https://www.starrocks.io/blog/four-simple-ways-to-deploy-starrocks
-- scaling
-  - scaling of UI
-  - scaling of API server
-  - scaling of database
-
-## Developer manual
-### Tech stack
-Relationship of containers
-```mermaid
-  graph TD;
-      Streamlit-->|View|FastAPI;
-      FastAPI-->|Control|StarRocks;
-      StarRocks-->*storage;
+  * **(Optional) Explore:** Check out the [Example Router Package](https://github.com/NewJerseyStyle/LitePolis-router-example) for more complex examples, including database interactions ([Example DB Package](https://github.com/NewJerseyStyle/LitePolis-database-example)).
+
+### 3. Prepare Your Packages for Deployment
+
+For the LitePolis CLI to find your local packages during deployment, you often need to install them in "editable" mode.
+
+```bash
+# In the litepolis-router-simple-api directory:
+pip install -e .
 ```
 
-Data flows from UI to API and store in database
-then been accessed by other UI widget through other API endpoints
-```mermaid
-  graph TD;
-      Streamlit-input-->|input|FastAPI-C_UD;
-      FastAPI-C_UD-->|write-in|StarRocks;
-      StarRocks-->*storage;
-      *storage-->StarRocks;
-      StarRocks-->|read-out|FastAPI-_R__;
-      FastAPI-_R__-->|output|Streamlit-display;
+This makes your local package code importable by the LitePolis deployment process.
+
+### 4. Deploy to Ray\!
+
+Make sure your local Ray cluster is running (see Prerequisites). Now, use the LitePolis CLI to deploy your application based on the configuration file.
+
+```bash
+litepolis-cli deploy add-deps litepolis-router-simple-api
+litepolis-cli deploy serve
 ```
-*storage: Such as Amazon S3, Google Cloud Storage, Azure Blob Storage, and other S3-compatible storage
+
+## Next Steps:
+
+  * Explore the [Example UI](https://github.com/NewJerseyStyle/LitePolis-ui-example) and [Example Router](https://github.com/NewJerseyStyle/LitePolis-router-example) repositories in more detail.
+  * Try adding a **Middleware** package (`litepolis create middleware ...`) for things like logging or authentication.
+  * Learn more about configuring **auto-scaling**, replicas, and other deployment options in your `deploy.yaml` or via CLI arguments (refer to LitePolis documentation).
+  * Integrate a database using a dedicated router package (see the [Database Example](https://github.com/NewJerseyStyle/LitePolis-database-example)).
+
+## Deployment & Maintenance Strengths
+
+While the tutorial covers the *how*, remember the key *why*s for using LitePolis, especially for administrators:
+
+  * **Default Auto-Scaling:** Ray Serve automatically scales API replicas based on traffic.
+  * **Modular Feature Addition:** Add functionality by adding package dependencies and redeploying.
+  * **Simplified Orchestration:** The `litepolis` tool manages component discovery and dependency resolution.
+  * **Resilience:** Ray provides fault tolerance for running components.
+  * **Cloud Native:** Runs effectively on Kubernetes and cloud platforms.

litepolis/core.py

@@ -90,10 +90,10 @@ def add_deps(ctx, package):
             line = line.strip()
             if len(line) and not line.startswith('#'):
                 line = line.replace('-', '_')
-                if 'litepolis_' in line.lower():
+                if line.lower().startswith('litepolis_'):
                     packages.append(line)
     check_import(package)
-    if package not in packages:
+    if package.replace('-', '_').lower() not in packages:
         with open(ctx.obj['packages_file'], 'a') as f:
             f.write(f"{package}\n")
 
@@ -108,9 +108,9 @@ def remove_deps(ctx, package):
         line = line.strip()
         if len(line) and not line.startswith('#'):
             line = line.replace('-', '_')
-            if 'litepolis_' in line.lower():
+            if line.lower().startswith('litepolis_'):
                 packages.append(line)
-    if package not in packages:
+    if package.replace('-', '_').lower() not in packages:
         raise ValueError(f"Package '{package}' not found in dependencies file.")
     else:
         packages.remove(package)
@@ -193,7 +193,14 @@ def get_apps(ctx, monolithic=False):
         # ).remote()
         pass
 
-    for line in routers + user_interfaces:
+    for line in user_interfaces:
+        m = importlib.import_module(line)
+        try:
+            app.include_router(m.prefix, m.files, name=m.name)
+        except Exception as e:
+            print(f"Error importing router {line}: {e}")
+
+    for line in routers:
         m = importlib.import_module(line)
         try:
             app.include_router(
@@ -211,11 +218,6 @@ def get_apps(ctx, monolithic=False):
         except Exception as e:
             print(f"Error importing middleware {line}: {e}")
 
-    app.include_router(
-        public.router,
-        prefix="/api"
-    )
-
     return [app]
 
 
@@ -368,7 +370,7 @@ def ui(project_name):
     import git
 
     # Clone the repository
-    repo_url = "https://github.com/NewJerseyStyle/LitePolis-router-template.git"
+    repo_url = "https://github.com/NewJerseyStyle/LitePolis-ui-template.git"
     repo = git.Repo.clone_from(repo_url, project_name)
     git_reinit(project_name, repo_url)