docs: refactor documentation into separate file and apply formatting

sumant1122 · sumant1122 · commit 08a430eefaef · 2026-04-20T19:06:34.000+05:30
diff --git a/README.md b/README.md
@@ -2,146 +2,45 @@
 
 A lightweight, local-first CI/CD tool written in Rust with a modern, real-time Terminal User Interface (TUI).
 
-## Features
-- **Artifact Management**: Define `artifacts` in your jobs. Conveyor will automatically collect and preserve these files (binaries, reports, etc.) in the build history.
-- **Interactive History**: Browse previous builds, view their statuses, and inspect full terminal logs directly from the history tab.
-- **Log Highlighting**: Real-time log search (`/`) not only filters lines but also highlights matching substrings for better visibility.
-- **Cron Scheduling**: Automate your pipelines with standard cron expressions (e.g., `schedule: "0 */6 * * *"`).
-- **Stages & Jobs**: Group related jobs into stages for better organization and a clearer overview of your pipeline's progress.
-- **Build History & Persistence**: Automatically saves build results and logs to a local history, allowing you to review past performance and failures.
-- **Sequential by Default**: Jobs run one-by-one in the order defined in your pipeline, ensuring a predictable flow.
-- **Explicit Parallelism**: Use the `parallel: true` flag to run independent jobs concurrently.
-- **Dependency Tracking (DAG)**: Fine-tune execution order with the `needs` keyword for complex dependency graphs.
-- **Log Search & Filtering**: Quickly find errors or specific output with real-time log filtering (`/`).
-- **Pipeline Hooks**: Define `on_success` and `on_failure` commands to run after the pipeline completes.
-- **Responsive TUI**: A spacious, modern interface with OneDark colors that adapts to your terminal size.
-- **Headless Mode**: Run pipelines in a non-interactive mode with real-time log streaming to stdout. Ideal for AI agents, CI automation, and scripts.
-- **Git Integration**: Live display of current branch and latest commit info in the header.
-- **Credential & Secret Management**: Declare required secrets in your pipeline. Conveyor will securely prompt for missing values at startup and automatically mask them (as `****`) in all TUI logs.
-- **Environment Variables**: Support for pipeline-level, job-specific, local `env.yaml` variables, and secure `secrets.yaml`.
-- **Cross-Platform**: Automatically selects the correct shell (`cmd` for Windows, `sh` for Linux/macOS).
-- **Graceful Process Termination**: Safely cancels running jobs and cleans up orphaned compilation processes instantly when quitting (`Q`) or retrying (`R`).
-- **Isolated Workspaces**: Remote pipelines clone into dynamically generated, globally unique directories to prevent file locking and stomping collisions.
+## Key Features
+- **Real-time TUI**: Modern, OneDark-inspired interface for live build monitoring.
+- **Artifacts & History**: Preserve build outputs and browse full logs of previous runs.
+- **Dependency Tracking**: Fine-grained execution control with DAG-based job dependencies.
+- **Flexible Scheduling**: Automate builds with standard Cron expressions.
+- **Secure by Default**: Automatic masking of secrets in logs and interactive credential prompting.
+- **Headless Mode**: Ideal for AI agents and CLI automation.
 
-## Installation
-Ensure you have the Rust toolchain installed.
+## Quick Start
 
+### 1. Installation
 ```bash
-git clone https://github.com/yourusername/conveyor.git
+git clone https://github.com/sumant1122/conveyor.git
 cd conveyor
 cargo build --release
 ```
 
-## Usage
-1. Create a `pipeline.yaml` in your project root.
-2. (Optional) Create an `env.yaml` for local/secret environment variables.
-3. Run Conveyor:
-   ```bash
-   cargo run
-   ```
-
-### Remote Repositories
-Conveyor can also run pipelines directly from a remote Git repository. It will clone the repository into a temporary workspace and automatically load the `pipeline.yaml` from its root.
-
-```bash
-# Run the default branch (usually main)
-cargo run -- https://github.com/user/repo.git
-
-# Run a specific branch
-cargo run -- https://github.com/user/repo.git my-feature-branch
-```
-
-### Navigation & Controls
-- **'1' / '2' / '3' / '4'**: Switch between **Dashboard**, **History**, **Pipeline Config**, and **Env Variables**.
-- **Up/Down Arrows**: 
-  - **Dashboard**: Select a job to view its logs.
-  - **History**: Select a previous build record.
-- **'Enter'**:
-  - **History**: View the full details and logs of the selected historical build.
-- **'Esc'**: 
-  - **History Detail**: Return to the build list.
-  - **Search**: Clear search query or exit search mode.
-- **'R'**: **Retry** the current pipeline (resets states and starts fresh).
-- **'/'**: Enter **Search Mode** to filter and **highlight** logs in real-time.
-- **'PgUp' / 'PgDn'**: Scroll through logs.
-- **'q'**: Quit the application.
-
-## Pipeline Configuration (`pipeline.yaml`)
-Example `pipeline.yaml` using modern features like **Stages**, **Cron**, and **Artifacts**, with **Simplified Syntax**:
-
+### 2. Define your Pipeline
+Create a `pipeline.yaml` in your project root:
 ```yaml
-# name: Example Service (Optional: Defaults to 'Conveyor Build')
-schedule: "0 */1 * * * *" # Run every minute
-on_success: "echo 'Success!'"
-
-stages:
+jobs:
   - name: Build
-    jobs:
-      - name: Compile
-        command: cargo build --release
-        artifacts:
-          - "target/release/conveyor"
-
+    command: cargo build
   - name: Test
-    jobs:
-      - name: Unit Tests
-        steps:
-          - cargo test # Shorthand for simple commands
-        artifacts:
-          - "target/debug/deps/"
-
-  - name: Deploy
-    jobs:
-      - name: Push Image
-        needs: ["Unit Tests"]
-        command: echo "Pushing..."
-```
-
-*Note: The older flat `jobs:` format is still supported for backward compatibility.*
-
-## Local Environment Variables (`env.yaml` & `secrets.yaml`)
-Store sensitive or machine-specific variables in an `env.yaml` file. Use `secrets.yaml` for credentials; any value defined here will be **masked** in the TUI logs.
-
-Example `env.yaml`:
-```yaml
-DEBUG: "true"
-LOG_LEVEL: "info"
-```
-
-Example `secrets.yaml`:
-```yaml
-API_KEY: "my-very-secret-key"
-SSH_PRIVATE_KEY: |
-  -----BEGIN RSA PRIVATE KEY-----
-  ...
-```
-
-To enforce secret entry, add them to your `pipeline.yaml`:
-```yaml
-name: My Pipeline
-secrets:
-  - API_KEY
-  - DOCKER_PASSWORD
+    command: cargo test
 ```
-If these are not present in `secrets.yaml`, Conveyor will prompt for them securely at startup.
-
-### Headless Mode (for AI Agents & Automation)
-To run Conveyor without the TUI, use the `--headless` flag. This will stream all logs directly to stdout and exit with a proper status code (0 for success, 1 for failure).
 
+### 3. Run
 ```bash
-# Run local pipeline
-cargo run -- --headless
-
-# Run remote repository
-cargo run -- https://github.com/user/repo.git --headless
+cargo run
 ```
 
-## Roadmap / Upcoming Enhancements
-To closely mirror the capabilities of professional CI systems like Jenkins, the following features are planned:
+## Documentation
+For detailed information on configuration, navigation, and advanced features, see **[documentation.md](./documentation.md)**.
 
-- **🎛️ Input Parameters**: Support for "Build with Parameters," allowing users to select options (like environment or version) before a pipeline starts.
-- **🏗️ Distributed Agents**: The ability to delegate jobs to remote machines via SSH or a custom agent protocol.
-- **🐳 Container Isolation**: Run jobs inside Docker/Podman containers for clean, reproducible environments.
+- **[Pipeline Configuration](./documentation.md#pipeline-configuration-pipelineyaml)**: Stages, Jobs, DAG, and Simplified Syntax.
+- **[Artifacts & Cron](./documentation.md#artifact-management)**: How to preserve build outputs and automate triggers.
+- **[Secrets Management](./documentation.md#environment--secrets)**: Local variables and automatic log masking.
+- **[Navigation Reference](./documentation.md#navigation--controls)**: Full list of TUI keybindings.
 
 ## License
 MIT
diff --git a/documentation.md b/documentation.md
@@ -0,0 +1,137 @@
+# Conveyor Documentation 🏗️
+
+This document provides a comprehensive guide to configuring and using Conveyor, a lightweight, local-first CI/CD runner.
+
+---
+
+## Table of Contents
+1. [Navigation & Controls](#navigation--controls)
+2. [Pipeline Configuration (pipeline.yaml)](#pipeline-configuration-pipelineyaml)
+    - [Stages & Jobs](#stages--jobs)
+    - [Simplified Syntax](#simplified-syntax)
+    - [Dependency Tracking (DAG)](#dependency-tracking-dag)
+    - [Artifact Management](#artifact-management)
+    - [Cron Scheduling](#cron-scheduling)
+3. [Environment & Secrets](#environment--secrets)
+    - [env.yaml](#envyaml)
+    - [secrets.yaml](#secretsyaml)
+4. [Headless Mode](#headless-mode)
+5. [Remote Repositories](#remote-repositories)
+
+---
+
+## Navigation & Controls
+
+Conveyor provides a real-time TUI for managing your builds.
+
+- **'1' / '2' / '3' / '4'**: Switch between **Dashboard**, **History**, **Pipeline Config**, and **Env Variables**.
+- **Up/Down Arrows**: 
+  - **Dashboard**: Select a job to view its logs.
+  - **History**: Select a previous build record.
+- **'Enter'**:
+  - **History**: View the full details and logs of the selected historical build.
+- **'Esc'**: 
+  - **History Detail**: Return to the build list.
+  - **Search**: Clear search query or exit search mode.
+- **'R'**: **Retry** the current pipeline (resets states and starts fresh).
+- **'/'**: Enter **Search Mode** to filter and **highlight** logs in real-time.
+- **'PgUp' / 'PgDn'**: Scroll through logs.
+- **'q'**: Quit the application.
+
+---
+
+## Pipeline Configuration (pipeline.yaml)
+
+Conveyor searches for `pipeline.yaml` in your project root.
+
+### Stages & Jobs
+Stages allow you to group related jobs. Jobs within a stage can run in parallel, while stages themselves generally progress sequentially.
+
+### Simplified Syntax
+You can omit verbose fields for simple tasks:
+- **Shorthand Step**: A string instead of an object if only a command is needed.
+- **Shorthand Job**: Use `command` directly on the job if it only has one step.
+
+```yaml
+# Optional: Defaults to 'Conveyor Build'
+name: My Pipeline 
+schedule: "0 */1 * * * *" # Cron: sec min hour day month dow
+
+stages:
+  - name: Build
+    jobs:
+      - name: Compile
+        command: cargo build --release # Shorthand job
+        artifacts:
+          - "target/release/app"
+
+  - name: Test
+    jobs:
+      - name: Unit Tests
+        steps:
+          - cargo test # Shorthand step
+```
+
+### Dependency Tracking (DAG)
+Use `needs` to define a Directed Acyclic Graph of dependencies.
+```yaml
+jobs:
+  - name: Deploy
+    needs: ["Unit Tests", "Integration Tests"]
+    command: ./deploy.sh
+```
+
+### Artifact Management
+Specify files or directories to preserve after a successful job. They are stored in `history/build_{id}/artifacts/`.
+```yaml
+artifacts:
+  - "target/release/binary"
+  - "docs/html/"
+```
+
+### Cron Scheduling
+Standard 6-field cron expressions are supported for automated runs.
+`sec min hour day month dow`
+
+---
+
+## Environment & Secrets
+
+### env.yaml
+Store non-sensitive local variables here.
+```yaml
+DEBUG: "true"
+API_URL: "http://localhost:8080"
+```
+
+### secrets.yaml
+Store sensitive credentials here. Values are **automatically masked** in all TUI logs.
+```yaml
+DATABASE_PASSWORD: "super-secret-password"
+```
+
+To require a secret, declare it in `pipeline.yaml`. Conveyor will prompt for it at startup if missing.
+```yaml
+secrets:
+  - DATABASE_PASSWORD
+```
+
+---
+
+## Headless Mode
+
+For use in CI environments or by AI agents. Logs stream to `stdout` and the process exits with `0` (success) or `1` (failure).
+
+```bash
+cargo run -- --headless
+```
+
+---
+
+## Remote Repositories
+
+Run a pipeline directly from a Git URL. Conveyor clones it into a unique temporary workspace.
+
+```bash
+cargo run -- https://github.com/user/repo.git [optional-branch]
+```
diff --git a/src/main.rs b/src/main.rs
@@ -285,7 +285,7 @@ jobs:
                         r_clone.reset().await;
                         r_clone.run().await;
                     });
-                    
+
                     if let Some(schedule_str) = &pipeline.schedule {
                         if let Ok(schedule) = schedule_str.parse::<cron::Schedule>() {
                             next_run = schedule.upcoming(chrono::Local).next();
diff --git a/src/runner.rs b/src/runner.rs
@@ -629,7 +629,10 @@ impl Runner {
             None => return,
         };
 
-        let artifact_root = self.history.root.join(format!("build_{}/artifacts/{}", self.build_id, job_name));
+        let artifact_root = self
+            .history
+            .root
+            .join(format!("build_{}/artifacts/{}", self.build_id, job_name));
         let _ = tokio::fs::create_dir_all(&artifact_root).await;
 
         for art_path in artifacts {
@@ -643,36 +646,57 @@ impl Runner {
 
                 {
                     let mut states = self.states.lock().await;
-                    states[index].logs.push(format!("📦 Collecting artifact: {}", art_path).dim().to_string());
+                    states[index].logs.push(
+                        format!("📦 Collecting artifact: {}", art_path)
+                            .dim()
+                            .to_string(),
+                    );
                 }
 
                 if src.is_file() {
                     if let Err(e) = tokio::fs::copy(&src, &dest).await {
                         let mut states = self.states.lock().await;
-                        states[index].logs.push(format!("❌ Failed to copy artifact {}: {}", art_path, e).red().to_string());
+                        states[index].logs.push(
+                            format!("❌ Failed to copy artifact {}: {}", art_path, e)
+                                .red()
+                                .to_string(),
+                        );
                     }
                 } else if src.is_dir() {
                     // For directories, we'll do a simple recursive copy or just log it's not supported for now if we want to be strict "Exact Paths"
                     // But usually people mean the folder too. Let's do a simple recursive copy logic.
                     if let Err(e) = self.copy_dir_recursive(&src, &dest).await {
                         let mut states = self.states.lock().await;
-                        states[index].logs.push(format!("❌ Failed to copy artifact directory {}: {}", art_path, e).red().to_string());
+                        states[index].logs.push(
+                            format!("❌ Failed to copy artifact directory {}: {}", art_path, e)
+                                .red()
+                                .to_string(),
+                        );
                     }
                 }
             } else {
                 let mut states = self.states.lock().await;
-                states[index].logs.push(format!("⚠️  Artifact not found: {}", art_path).yellow().to_string());
+                states[index].logs.push(
+                    format!("⚠️  Artifact not found: {}", art_path)
+                        .yellow()
+                        .to_string(),
+                );
             }
         }
     }
 
-    async fn copy_dir_recursive(&self, src: &std::path::Path, dst: &std::path::Path) -> std::io::Result<()> {
+    async fn copy_dir_recursive(
+        &self,
+        src: &std::path::Path,
+        dst: &std::path::Path,
+    ) -> std::io::Result<()> {
         tokio::fs::create_dir_all(&dst).await?;
         let mut entries = tokio::fs::read_dir(src).await?;
         while let Some(entry) = entries.next_entry().await? {
             let file_type = entry.file_type().await?;
             if file_type.is_dir() {
-                Box::pin(self.copy_dir_recursive(&entry.path(), &dst.join(entry.file_name()))).await?;
+                Box::pin(self.copy_dir_recursive(&entry.path(), &dst.join(entry.file_name())))
+                    .await?;
             } else {
                 tokio::fs::copy(entry.path(), dst.join(entry.file_name())).await?;
             }
diff --git a/src/ui.rs b/src/ui.rs
