WIP: some major improvements

.github/CONTRIBUTING.md

@@ -0,0 +1,14 @@
+# Contributing to VSR
+
+Thank you for your interest in contributing to VSR (vlt serverless registry)!
+
+Please see our [main CONTRIBUTING guide](../CONTRIBUTING.md) in the repository root for comprehensive information about:
+- Development setup
+- Project structure
+- Testing procedures
+- Deployment instructions
+- Contribution guidelines
+- Pull request process
+- Code style guidelines
+
+We look forward to your contributions!

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,36 @@
+---
+name: Bug report
+about: Create a report to help us improve VSR
+title: "[BUG] "
+labels: bug
+assignees: ''
+---
+
+## Bug Description
+A clear and concise description of what the bug is.
+
+## Reproduction Steps
+Steps to reproduce the behavior:
+1. Run '...'
+2. Call API endpoint '....'
+3. Configure '...'
+4. See error
+
+## Expected Behavior
+A clear and concise description of what you expected to happen.
+
+## Actual Behavior
+A clear description of what actually happened.
+
+## Environment
+- OS: [e.g. macOS, Windows, Linux]
+- Node.js Version: [e.g. 18.15.0]
+- NPM Version: [e.g. 9.5.0]
+- VSR Version: [e.g. 0.1.1]
+- Deployment: [e.g. local, Cloudflare Workers]
+
+## Additional Context
+Add any other context about the problem here. Include logs, screenshots, or related information.
+
+## Possible Solution
+If you have suggestions on how to fix the bug, please describe them here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,26 @@
+---
+name: Feature request
+about: Suggest an idea for VSR
+title: "[FEATURE] "
+labels: enhancement
+assignees: ''
+---
+
+## Feature Description
+A clear and concise description of the feature you're requesting.
+
+## Problem / Motivation
+Describe the problem or limitation that this feature would address.
+Example: "I'm always frustrated when..."
+
+## Proposed Solution
+Describe how you envision this feature working. Be as specific as possible.
+
+## Alternatives Considered
+Describe any alternative solutions or features you've considered.
+
+## Additional Context
+Add any other context, screenshots, mockups, or examples about the feature request here.
+
+## Impact
+How would this feature benefit users of VSR? Is it something that could be generally useful for many users or specific to a particular use case?

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,37 @@
+## Description
+<!-- Describe the changes you've made and the reason for the changes -->
+
+## Related Issue
+<!-- Link to the issue this PR addresses (if applicable) -->
+Fixes #(issue)
+
+## Type of Change
+<!-- Check the applicable boxes -->
+- [ ] Bug fix (non-breaking change that fixes an issue)
+- [ ] New feature (non-breaking change that adds functionality)
+- [ ] Breaking change (causes existing functionality to not work as expected)
+- [ ] Documentation update
+- [ ] Performance improvement
+- [ ] Code refactor (no functional changes)
+- [ ] Test changes
+
+## How Has This Been Tested?
+<!-- Describe the tests you ran to verify your changes -->
+- [ ] Unit tests
+- [ ] Integration tests
+- [ ] Manual testing
+- [ ] Other (please describe)
+
+## Checklist
+<!-- Check the applicable boxes -->
+- [ ] My code follows the style guidelines of this project
+- [ ] I have performed a self-review of my own code
+- [ ] I have commented my code, particularly in hard-to-understand areas
+- [ ] I have made corresponding changes to the documentation
+- [ ] My changes generate no new warnings
+- [ ] I have added tests that prove my fix is effective or that my feature works
+- [ ] New and existing unit tests pass locally with my changes
+- [ ] Any dependent changes have been merged and published
+
+## Additional Context
+<!-- Add any other context or screenshots about the PR here -->

.github/workflows/publish.yml

@@ -0,0 +1,35 @@
+name: Publish
+
+on:
+  workflow_dispatch:
+    inputs:
+      ref:
+        description: 'the GitHub ref to checkout and publish'
+        type: string
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  lint:
+    name: Publish vlt serverless registry
+
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout Repository
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref }}
+
+      - name: Install dependencies
+        run: npm install
+
+      - name: Build
+        run: npm run build
+
+      - name: Run Publish
+        run: cd dist && ls -la
+        # env:
+        #   VLT_CLI_PUBLISH_TOKEN: ${{ secrets.VLT_CLI_PUBLISH_TOKEN }}

.gitignore

@@ -172,3 +172,6 @@ out
 .dev.vars
 .wrangler/
 local-store
+
+# build output
+dist/

CONTRIBUTING.md

@@ -0,0 +1,239 @@
+# Contributing to VSR (vlt serverless registry)
+
+Thank you for your interest in contributing to VSR! This document provides guidelines and instructions for development, testing, and contributing to the project.
+
+## Table of Contents
+
+- [Development Setup](#development-setup)
+- [Project Structure](#project-structure)
+- [Testing](#testing)
+  - [Running Tests](#running-tests)
+  - [Testing Background Refresh Functionality](#testing-background-refresh-functionality)
+  - [Testing Patterns](#testing-patterns)
+- [Deployment](#deployment)
+  - [Local Deployment](#local-deployment)
+  - [Production Deployment](#production-deployment)
+- [Caching Implementation](#caching-implementation)
+- [Contribution Guidelines](#contribution-guidelines)
+- [Pull Request Process](#pull-request-process)
+- [Code Style](#code-style)
+
+## Development Setup
+
+1. **Clone the repository**:
+   ```bash
+   git clone https://github.com/vltpkg/vsr.git
+   cd vsr
+   ```
+
+2. **Install dependencies**:
+   ```bash
+   npm install
+   ```
+
+3. **Run the development server**:
+   ```bash
+   npm run dev
+   ```
+   This will start a local development server at http://localhost:1337 with the Cloudflare Workers environment simulated locally.
+
+4. **Database setup**:
+   ```bash
+   npm run db:setup
+   ```
+   This initializes the local D1 database for development.
+
+## Project Structure
+
+- `src/` - Core application source code
+  - `db/` - Database models and migrations
+  - `routes/` - API route handlers
+  - `middleware/` - Middleware functions
+  - `utils/` - Utility functions
+- `test/` - Test files and utilities
+- `scripts/` - Helper scripts
+- `bin/` - CLI entry point
+
+## Testing
+
+VSR has a comprehensive test suite that covers various aspects of the codebase, including the stale-while-revalidate caching pattern implemented with Cloudflare Workers' `waitUntil` API.
+
+### Running Tests
+
+```bash
+# Run all tests
+npm test
+
+# Run specific test groups
+npm run test:cache     # Run caching-related tests
+npm run test:improved  # Run optimized `waitUntil` tests
+
+# Clean up problematic test files
+npm run test:cache:clean
+```
+
+The test suite is configured with the following npm scripts:
+
+- `test`: Runs the full test suite with cleanup, setup, and execution
+- `test:run`: Executes the tests using Vitest
+- `test:cache`: Runs caching-related tests
+- `test:improved`: Runs optimized tests for the `waitUntil` pattern
+- `test:setup`: Sets up the test environment
+- `test:cleanup`: Cleans up processes from previous test runs
+
+### Testing Background Refresh Functionality
+
+The registry implements a "stale-while-revalidate" caching pattern for faster responses:
+
+1. When a request is received, we first check if the data is in our cache
+2. If the data is in the cache, we return it immediately, even if it's stale (old)
+3. If the data is stale, we queue a background task to refresh it from the upstream registry
+4. The background task updates the cache with fresh data for future requests
+5. This approach ensures users get a fast response (cached data) while keeping our cache up-to-date
+
+This pattern is implemented in the following key files:
+- `getPackagePackument`: For fetching package metadata
+- `getPackageManifest`: For fetching specific package versions
+
+#### How `waitUntil` Works
+
+Cloudflare Workers provide a special API called `waitUntil` that enables background tasks to continue running after a response has been sent. Here's a simplified example:
+
+```javascript
+export default {
+  async fetch(request, env, ctx) {
+    // Return a response immediately
+    const response = new Response("Hello World");
+
+    // Queue a background task that continues after response is sent
+    ctx.waitUntil(
+      (async () => {
+        // This runs in the background after response is sent
+        await doLongRunningTask();
+      })()
+    );
+
+    return response;
+  }
+};
+```
+
+### Testing Patterns
+
+Testing `waitUntil` behavior can be tricky because of how JavaScript executes Immediately Invoked Function Expressions (IIFEs). The common mistake is to use code like this:
+
+```javascript
+// ❌ PROBLEM: This executes the function immediately!
+c.waitUntil((async () => {
+  // Background work
+  await refreshCache();
+})());
+```
+
+Our testing approach:
+
+1. Correctly simulates the Cloudflare `waitUntil` API
+2. Captures background tasks without executing them immediately
+3. Allows precise control over when background tasks run
+
+For detailed documentation on testing this pattern, refer to `test/waituntil-README.md`.
+
+#### Test Files
+
+- `waituntil-correct.test.js`: Demonstrates the core pattern with simple examples
+- `hono-context.test.js`: Shows how to mock the Hono context with proper `waitUntil` support
+- `route-with-waituntil.test.js`: Tests a route handler implementing the stale-while-revalidate pattern
+
+## Deployment
+
+### Local Deployment
+
+For local usage, you can simply run:
+
+```bash
+npx -y vltpkg/vsr
+```
+
+This will start the registry locally at http://localhost:1337.
+
+### Production Deployment
+
+VSR is designed to be deployed on Cloudflare Workers. To deploy to production:
+
+1. **Set up Cloudflare account and create necessary resources**:
+   - Create a Cloudflare account if you don't have one
+   - Create a new D1 Database
+   - Create a new R2 Bucket
+
+2. **Configure your `wrangler.toml`**:
+   Make sure your configuration includes the correct bindings for D1 and R2.
+
+3. **Deploy using Wrangler**:
+   ```bash
+   npm run build   # Create deployment bundle
+   npm run deploy  # Deploy to Cloudflare Workers
+   ```
+
+4. **Post-deployment setup**:
+   - Initialize your database with the schema
+   - Create admin tokens as needed
+
+## Caching Implementation
+
+The stale-while-revalidate caching pattern is a core feature of VSR. It provides several benefits:
+
+1. **Improved Response Times**: By returning cached data immediately, users experience fast response times.
+2. **Reduced Load on Upstream Registry**: By caching package data, we reduce the number of requests to the upstream npm registry.
+3. **Background Updates**: By refreshing stale data in the background, cache stays current without impacting user response time.
+
+### Implementation Details
+
+The caching logic is implemented as follows:
+
+1. **Cache Check**: When a request comes in, we check if the data exists in our cache (D1 database).
+2. **Freshness Check**: If the data exists, we check if it's fresh (typically less than 5 minutes old).
+3. **Response Strategy**:
+   - If data is fresh: Return it immediately
+   - If data is stale: Return it immediately AND queue a background refresh
+   - If data doesn't exist: Fetch from upstream, cache it, then return
+
+4. **Background Refresh**:
+   ```javascript
+   // Example pattern (simplified)
+   if (isStale) {
+     c.executionCtx.waitUntil(new Promise(async (resolve) => {
+       const bgRefresh = async () => {
+         const freshData = await fetchFromUpstream();
+         await saveToCache(freshData);
+         resolve();
+       };
+
+       setTimeout(bgRefresh, 0);
+     }));
+   }
+   ```
+
+## Contribution Guidelines
+
+1. **Create an issue** first to discuss proposed changes or report bugs
+2. **Fork the repository** and create a feature branch for your changes
+3. **Write tests** for new features or bug fixes
+4. **Ensure code quality** by following the established style and patterns
+5. **Submit a pull request** referencing the original issue
+
+## Pull Request Process
+
+1. Ensure your code passes all tests and linting
+2. Update documentation if necessary
+3. Add yourself to the contributors list (if not already there)
+4. Submit your PR with a clear description of changes
+5. Address any feedback from code reviews
+
+## Code Style
+
+- Use modern JavaScript features
+- Follow the existing code structure and naming conventions
+- Document new functions and components with JSDoc comments
+- Maintain comprehensive test coverage
+
+Thank you for contributing to VSR!