Merge pull request #67 from seanwevans/claude/project-cleanup-testing-7e97st

Refactor pg_os into single SQL file and improve documentation

Author: seanwevans

.github/workflows/test.yml

@@ -1,20 +1,43 @@
 name: CI
+
 on:
   push:
   pull_request:
+
 jobs:
   test:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
 
-      - name: Install PostgreSQL and headers
+      - name: Install PostgreSQL and development headers
         run: |
           sudo apt-get update
-          # Install the server, contrib modules and the development headers
-          sudo apt-get install -y postgresql postgresql-contrib libpq-dev postgresql-server-dev-16
+          # Server, contrib modules, client library and the PGXS build headers
+          sudo apt-get install -y postgresql postgresql-contrib libpq-dev postgresql-server-dev-all
 
       - name: Start PostgreSQL
         run: |
-          # Determine the installed major version and start the cluster
-          sudo pg_ctlcluster $(pg_config --version | awk '{print $2}' | cut -d. -f1) main start
+          # Determine the installed major version and start the default cluster
+          PG_MAJOR=$(pg_config --version | awk '{print $2}' | cut -d. -f1)
+          echo "PG_MAJOR=$PG_MAJOR" >> "$GITHUB_ENV"
+          sudo pg_ctlcluster "$PG_MAJOR" main start
+
+      - name: Create a superuser role for the runner
+        run: |
+          # pg_regress connects over the local socket as the current OS user via
+          # peer authentication, so that user needs a matching superuser role.
+          sudo -u postgres createuser --superuser "$(whoami)"
+          sudo -u postgres createdb "$(whoami)"
+
+      - name: Build and install the extension
+        run: |
+          make
+          sudo make install
+
+      - name: Run regression tests
+        run: make installcheck
+
+      - name: Show regression diffs on failure
+        if: failure()
+        run: cat tmp_pg_os_regress/regression.diffs || true

.gitignore

@@ -13,6 +13,9 @@
 # Regression test outputs
 /results/
 /tmp_check/
+/tmp_pg_os_regress/
+regression.diffs
+regression.out
 
 # Backup and temp files
 *~

Makefile

@@ -1,7 +1,19 @@
 EXTENSION = pg_os
 DATA = pg_os--1.0.sql
 PG_CONFIG ?= pg_config
-REGRESS = pg_os_basic create_user lock_file create_process allocate_memory load_unload_module modules check_permission free_all_memory_for_process file_versioning locks_security create_file create_process
+REGRESS = pg_os_basic \
+          create_user \
+          check_permission \
+          create_process \
+          allocate_memory \
+          allocate_page \
+          free_all_memory_for_process \
+          create_file \
+          file_versioning \
+          lock_file \
+          load_unload_module \
+          modules \
+          locks_security
 REGRESS_OPTS = --outputdir=$(CURDIR)/tmp_pg_os_regress
 PGXS := $(shell $(PG_CONFIG) --pgxs)
 include $(PGXS)

README.md

@@ -3,148 +3,148 @@
 
 
 ## Overview
-`pg_os` is a PostgreSQL extension that provides operating system-level functionality directly within the database environment. By leveraging SQL procedures and functions, `pg_os` enables users to interact with system processes, manage memory, handle inter-process communication (IPC), schedule tasks, and enforce security policies at the OS level.
+`pg_os` is a PostgreSQL extension that models operating-system concepts entirely
+inside the database. Tables stand in for kernel objects (processes, threads,
+memory segments, files, mutexes, semaphores, devices, …) and PL/pgSQL functions
+play the role of system calls that operate on them under a small permission
+model.
 
-The extension bridges the gap between the database and the underlying OS, allowing administrators to gain deep insights into system performance and control resource usage efficiently.
+It is a teaching/experimentation toy rather than a way to control the host OS:
+everything happens in SQL, so you can poke at schedulers, page allocation, file
+permissions, IPC and locking without leaving `psql`.
 
 ## Features
-- **Process Management** – Monitor and control OS-level processes directly from PostgreSQL.
-- **File System Operations** – Interact with the file system to read, write, and manipulate files.
-- **Memory Management** – Analyze and manage memory allocations.
-- **IPC (Inter-Process Communication)** – Facilitate communication between processes.
-- **Scheduler** – Schedule jobs and tasks within PostgreSQL.
-- **Locks** – Manage and analyze system and database locks.
-- **Security** – Handle user roles and permissions at the OS level.
-- **Network Monitoring** – Monitor and manage network interfaces and traffic.
-- **Power Management** – Access and control power states of the system.
-- **I/O Operations** – Track and manage input/output processes.
-- **Signal Management** – Send and receive system signals.
-- **Garbage Collection** – Monitor and optimize system garbage collection processes.
-- **Module Management** – Inspect and control system modules.
+- **Users, roles & permissions** – a minimal RBAC layer that gates the other subsystems.
+- **Process management** – create, start, execute, prioritise and terminate simulated processes.
+- **Scheduler** – priority / round-robin / SJF process scheduling and a simple thread scheduler.
+- **Memory management** – allocate and free memory segments, and allocate paged virtual addresses per thread.
+- **Filesystem** – a hierarchical file table with permissions, locking and content versioning.
+- **IPC** – named channels and a per-user mailbox.
+- **Locks** – mutexes and counting semaphores exposed through `SECURITY DEFINER` helpers.
+- **Signals** – send and handle `SIGTERM` / `SIGSTOP` / `SIGCONT`.
+- **Garbage collection** – free memory for terminated processes and reap stale ones.
+- **Devices & I/O, networking, modules, power** – small simulated subsystems for completeness.
 
 ## Requirements
-- PostgreSQL 12+
-- `plpgsql` extension installed (required)
-- Extension is not relocatable and will install into the database's default schema
-- Sufficient OS-level permissions for system interaction
+- PostgreSQL 12 or newer
+- `plpgsql` (bundled with PostgreSQL; declared as a dependency)
+- The PGXS build headers (`postgresql-server-dev-*`) to build and install
 
 ## Installation
 1. Clone the repository:
    ```bash
-   git clone https://github.com/your-repo/pg_os.git
+   git clone https://github.com/seanwevans/pg_os.git
+   cd pg_os
    ```
-2. Build and install the extension using PGXS:
+2. Build and install with PGXS:
    ```bash
-   cd pg_os
    make
    sudo make install
    ```
-3. Connect to your PostgreSQL instance:
+   To build against a specific PostgreSQL install, point `PG_CONFIG` at it:
    ```bash
-   psql -U postgres -d your_database
+   make PG_CONFIG=/usr/lib/postgresql/16/bin/pg_config
    ```
-4. Install the extension:
+3. Create the extension in your database. You may install it into a dedicated
+   schema if you prefer:
    ```sql
-   CREATE EXTENSION pg_os;
+   CREATE EXTENSION pg_os;                       -- into the current schema
+   -- or
+   CREATE EXTENSION pg_os WITH SCHEMA os;        -- into schema "os"
    ```
+   Creating the extension also ensures a `pgos_admin` role exists; it owns the
+   `SECURITY DEFINER` locking helpers (see [Security model](#security-model)).
 
 ## Usage
-### 1. Process Management
-```sql
-SELECT * FROM os_processes();
-```
-- Lists all current OS processes and their statuses.
+The functions are intentionally small. A typical session looks like this:
 
-### 2. File System Operations
 ```sql
-SELECT * FROM fs_list('/var/log');
+CREATE EXTENSION pg_os;
+
+-- 1. Set up a user, a role, and grant it some permissions
+SELECT create_user('alice')              AS alice_id;   -- => 1
+SELECT create_role('operator')           AS role_id;    -- => 1
+SELECT assign_role_to_user(1, 1);
+SELECT grant_permission_to_role(1, 'process', 'execute');
+SELECT grant_permission_to_role(1, 'memory',  'allocate');
+SELECT grant_permission_to_role(1, 'file',    'write');
+
+-- 2. Create and inspect a process (create_process is a PROCEDURE)
+CALL create_process('worker', 1, 5);        -- name, owner_id, priority
+SELECT name, state, priority FROM processes;
+
+-- 3. Allocate memory to the process
+INSERT INTO memory_segments (size) VALUES (4096);
+SELECT allocate_memory(1, 1, 1024);         -- user_id, process_id, size
+SELECT process_id, segment_id FROM process_memory;
+
+-- 4. Create, write and read a file (with automatic versioning)
+SELECT create_file(1, 'notes.txt', NULL, FALSE) AS file_id;   -- => 1
+SELECT write_file(1, 1, 'hello');
+SELECT write_file(1, 1, 'hello world');
+SELECT read_file(1, 1);                                       -- => 'hello world'
+SELECT version_number, contents FROM file_versions WHERE file_id = 1 ORDER BY version_number;
+
+-- 5. Concurrency primitives (callable by unprivileged roles, see below)
+SELECT create_semaphore('jobs', 1, 1);
+SELECT acquire_semaphore(1, 'jobs');
+SELECT release_semaphore(1, 'jobs');
 ```
-- Retrieves the contents of a directory.
 
-### 3. Memory Management
-```sql
-SELECT * FROM os_memory_usage();
+Permissions are enforced with `check_permission(user_id, resource_type, action)`,
+where `resource_type` is one of `process`, `file`, `resource`, `memory` and
+`action` is one of `read`, `write`, `execute`, `allocate`, `delete`. Calls that
+require a permission the user lacks raise an exception.
+
+### Selected functions by subsystem
+| Subsystem | Functions |
+|-----------|-----------|
+| Users/roles | `create_user`, `create_role`, `assign_role_to_user`, `grant_permission_to_role`, `check_permission` |
+| Processes | `create_process`, `start_process`, `execute_process`, `terminate_process`, `set_process_priority`, `pause_all_processes`, `resume_all_waiting_processes`, `list_processes_by_state`, `process_count_by_state` |
+| Scheduler | `schedule_processes`, `schedule_threads` |
+| Memory | `allocate_memory`, `free_memory`, `allocate_page`, `free_all_memory_for_process` |
+| Filesystem | `create_file`, `read_file`, `write_file`, `change_file_permissions`, `lock_file`, `unlock_file`, `version_file` |
+| IPC | `register_channel`, `write_channel`, `read_channel`, `send_mail`, `check_mail` |
+| Locks | `create_mutex`, `lock_mutex`, `unlock_mutex`, `create_semaphore`, `acquire_semaphore`, `release_semaphore` |
+| Signals | `send_signal`, `handle_signals` |
+| GC | `cleanup_terminated_processes`, `free_all_memory_for_process` |
+| I/O / Net / Modules / Power | `enqueue_io_request`, `process_device_queue`, `send_packet`, `load_module`, `unload_module`, `set_power_state` |
+
+## Security model
+Most functions run with the privileges of the caller. The mutex and semaphore
+helpers (`create_mutex`, `lock_mutex`, `unlock_mutex`, `create_semaphore`,
+`acquire_semaphore`, `release_semaphore`) are `SECURITY DEFINER` and owned by the
+`pgos_admin` role, with `EXECUTE` granted to `PUBLIC`. This lets an otherwise
+unprivileged role manipulate locks and semaphores through the controlled API
+without being granted direct access to the underlying tables. Their
+`search_path` is pinned to the extension's schema (`@extschema@`) plus `pg_temp`
+to avoid search-path injection.
+
+## Testing
+The extension ships a PGXS regression suite under `sql/` (inputs) and
+`expected/` (expected output). With a running PostgreSQL cluster and a superuser
+role for your OS user:
+
+```bash
+make
+sudo make install
+make installcheck
 ```
-- Displays memory consumption statistics.
 
-### 4. IPC (Inter-Process Communication)
-```sql
-SELECT ipc_signal(12345, 'SIGTERM');
-```
-- Sends a termination signal to the process with PID 12345.
+`make installcheck` creates a throwaway `contrib_regression` database, installs
+the extension and runs every test listed in `REGRESS` in the `Makefile`. The same
+flow runs on every push/PR via the GitHub Actions workflow in
+`.github/workflows/test.yml`.
 
-### 5. Locks
-```sql
-SELECT * FROM os_locks();
-```
-- Lists all current locks on system resources.
-
-### 6. Scheduler
-```sql
-SELECT * FROM os_schedule_job('backup', '0 3 * * *', 'pg_dump your_database > backup.sql');
-```
-- Schedules a nightly backup at 3 AM.
-
-### 7. Network Monitoring
-```sql
-SELECT * FROM os_network_interfaces();
-```
-- Lists all network interfaces and their statuses.
-
-### 8. Power Management
-```sql
-SELECT os_power_state('suspend');
-```
-- Suspends the system.
-
-### 9. I/O Operations
-```sql
-SELECT * FROM os_io_stats();
-```
-- Displays current I/O statistics.
-
-### 10. Signal Management
-```sql
-SELECT os_send_signal(12345, 'SIGHUP');
-```
-- Sends a hang-up signal to the process with PID 12345.
-
-### 11. Garbage Collection
-```sql
-SELECT * FROM os_gc_status();
-```
-- Monitors garbage collection processes.
-
-### 12. Module Management
-```sql
-SELECT * FROM os_modules();
-```
-- Lists all loaded system modules.
-
-### 13. Users, Roles, and Permissions
-```sql
-SELECT * FROM os_users_roles_permissions();
-```
-- Displays user roles and permissions.
-
-## Source Files
-- `audit.sql` – Handles audit logging and tracking.
-- `fs.sql` – Implements file system operations.
-- `ipc.sql` – Facilitates inter-process communication.
-- `locks.sql` – Manages lock operations.
-- `memory.sql` – Provides memory management utilities.
-- `os.sql` – Core OS interaction functions.
-- `processes.sql` – Monitors and manages system processes.
-- `scheduler.sql` – Implements job scheduling.
-- `usersrolespermissions.sql` – Manages users, roles, and permissions.
-- `network.sql` – Handles network monitoring and management.
-- `power.sql` – Implements power management functions.
-- `io.sql` – Tracks I/O operations.
-- `signals.sql` – Manages system signals.
-- `gc.sql` – Monitors and controls garbage collection.
-- `modules.sql` – Manages system modules.
-- `pg_os--1.0.sql` – Extension script combining all modules.
-- `pg_os.control` – Extension control file.
+## Project layout
+- `pg_os--1.0.sql` – the extension install script (tables, functions, indexes). Single source of truth.
+- `pg_os.control` – extension control file.
+- `Makefile` – PGXS build/test configuration.
+- `sql/` – regression test inputs (plus `fs_constraints_migration.sql`, a one-off migration that tightens the `files` constraints on older installs).
+- `expected/` – expected regression output.
 
 ## Contributing
-Contributions are welcome! Please open an issue or submit a pull request with any improvements or bug fixes.
+Contributions are welcome. Please keep `pg_os--1.0.sql` and the regression tests
+in sync: add or update a test under `sql/` and its `expected/` output for any
+behavioural change, and make sure `make installcheck` passes before opening a
+pull request.