Skip to content

WIP: some major improvements #12

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 31 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
956d27c
chore: remove dist, add build & secure db requests
darcyclarke Jan 22, 2025
c852255
wip: proxying
darcyclarke Mar 25, 2025
4265942
wip: tests
darcyclarke Mar 25, 2025
93605f9
wip: more tests & switch to drizzler
darcyclarke Mar 25, 2025
2f8fe17
wip: add drizzle studio
darcyclarke Mar 25, 2025
7978ade
wip: clean up db setup with drizzle
darcyclarke Mar 25, 2025
f049183
wip: tests update + contributing information
darcyclarke Mar 25, 2025
f11cd44
wip: store full manifest but serve slim
darcyclarke Mar 25, 2025
5d2ed28
wip: complete tests for slim manifests
darcyclarke Mar 25, 2025
0947131
wip: add integrity checking to tarball endpoints
darcyclarke Mar 25, 2025
ba7cef3
wip: updates open api docs
darcyclarke Mar 25, 2025
f12f639
wip: cache busting
darcyclarke Mar 26, 2025
b34c9c2
feat: semver range resolution for manifests
darcyclarke Apr 2, 2025
d222acd
wip: subtle additions alongside versioning
darcyclarke Apr 2, 2025
0bb19cc
wip: dist-tag support
darcyclarke Apr 2, 2025
235dfe7
wip: dist-tag validation
darcyclarke Apr 2, 2025
c6b935f
wip: add access endpoints
darcyclarke Apr 2, 2025
01d0092
wip: add extra dist-tag validation
darcyclarke Apr 2, 2025
5dcf0ec
chore: remove stars/starred behaivour as a todo
darcyclarke Apr 2, 2025
396c972
chore: move dist tags in api
darcyclarke Apr 18, 2025
899562c
feat: add version range to packuments
darcyclarke Apr 18, 2025
076e241
chore: add query endpoint
darcyclarke Apr 18, 2025
52b2d5b
chore: write tests for query endpoints
darcyclarke Apr 18, 2025
fbadc52
chore: clean up tests, routes, config & admin
darcyclarke Apr 18, 2025
893cd1c
chore: remove query endpoint for now
darcyclarke Apr 18, 2025
93660bb
chore: get rid of admin dashboard for now
darcyclarke Apr 18, 2025
47c6669
chore: clean up remnants
darcyclarke Apr 18, 2025
08537c1
wip: auth
darcyclarke May 1, 2025
1693ed4
wip
darcyclarke May 6, 2025
1725486
wip - auth
darcyclarke May 21, 2025
b40929e
wip: began ingesting gui
darcyclarke May 25, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .github/CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Contributing to VSR

Thank you for your interest in contributing to VSR (vlt serverless registry)!

Please see our [main CONTRIBUTING guide](../CONTRIBUTING.md) in the repository root for comprehensive information about:
- Development setup
- Project structure
- Testing procedures
- Deployment instructions
- Contribution guidelines
- Pull request process
- Code style guidelines

We look forward to your contributions!
36 changes: 36 additions & 0 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
---
name: Bug report
about: Create a report to help us improve VSR
title: "[BUG] "
labels: bug
assignees: ''
---

## Bug Description
A clear and concise description of what the bug is.

## Reproduction Steps
Steps to reproduce the behavior:
1. Run '...'
2. Call API endpoint '....'
3. Configure '...'
4. See error

## Expected Behavior
A clear and concise description of what you expected to happen.

## Actual Behavior
A clear description of what actually happened.

## Environment
- OS: [e.g. macOS, Windows, Linux]
- Node.js Version: [e.g. 18.15.0]
- NPM Version: [e.g. 9.5.0]
- VSR Version: [e.g. 0.1.1]
- Deployment: [e.g. local, Cloudflare Workers]

## Additional Context
Add any other context about the problem here. Include logs, screenshots, or related information.

## Possible Solution
If you have suggestions on how to fix the bug, please describe them here.
26 changes: 26 additions & 0 deletions .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
---
name: Feature request
about: Suggest an idea for VSR
title: "[FEATURE] "
labels: enhancement
assignees: ''
---

## Feature Description
A clear and concise description of the feature you're requesting.

## Problem / Motivation
Describe the problem or limitation that this feature would address.
Example: "I'm always frustrated when..."

## Proposed Solution
Describe how you envision this feature working. Be as specific as possible.

## Alternatives Considered
Describe any alternative solutions or features you've considered.

## Additional Context
Add any other context, screenshots, mockups, or examples about the feature request here.

## Impact
How would this feature benefit users of VSR? Is it something that could be generally useful for many users or specific to a particular use case?
37 changes: 37 additions & 0 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
## Description
<!-- Describe the changes you've made and the reason for the changes -->

## Related Issue
<!-- Link to the issue this PR addresses (if applicable) -->
Fixes #(issue)

## Type of Change
<!-- Check the applicable boxes -->
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (causes existing functionality to not work as expected)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactor (no functional changes)
- [ ] Test changes

## How Has This Been Tested?
<!-- Describe the tests you ran to verify your changes -->
- [ ] Unit tests
- [ ] Integration tests
- [ ] Manual testing
- [ ] Other (please describe)

## Checklist
<!-- Check the applicable boxes -->
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged and published

## Additional Context
<!-- Add any other context or screenshots about the PR here -->
35 changes: 35 additions & 0 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: Publish

on:
workflow_dispatch:
inputs:
ref:
description: 'the GitHub ref to checkout and publish'
type: string

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
lint:
name: Publish vlt serverless registry

runs-on: ubuntu-latest

steps:
- name: Checkout Repository
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref }}

- name: Install dependencies
run: npm install

- name: Build
run: npm run build

- name: Run Publish
run: cd dist && ls -la
# env:
# VLT_CLI_PUBLISH_TOKEN: ${{ secrets.VLT_CLI_PUBLISH_TOKEN }}
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -172,3 +172,6 @@ out
.dev.vars
.wrangler/
local-store

# build output
dist/
239 changes: 239 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,239 @@
# Contributing to VSR (vlt serverless registry)

Thank you for your interest in contributing to VSR! This document provides guidelines and instructions for development, testing, and contributing to the project.

## Table of Contents

- [Development Setup](#development-setup)
- [Project Structure](#project-structure)
- [Testing](#testing)
- [Running Tests](#running-tests)
- [Testing Background Refresh Functionality](#testing-background-refresh-functionality)
- [Testing Patterns](#testing-patterns)
- [Deployment](#deployment)
- [Local Deployment](#local-deployment)
- [Production Deployment](#production-deployment)
- [Caching Implementation](#caching-implementation)
- [Contribution Guidelines](#contribution-guidelines)
- [Pull Request Process](#pull-request-process)
- [Code Style](#code-style)

## Development Setup

1. **Clone the repository**:
```bash
git clone https://github.com/vltpkg/vsr.git
cd vsr
```

2. **Install dependencies**:
```bash
npm install
```

3. **Run the development server**:
```bash
npm run dev
```
This will start a local development server at http://localhost:1337 with the Cloudflare Workers environment simulated locally.

4. **Database setup**:
```bash
npm run db:setup
```
This initializes the local D1 database for development.

## Project Structure

- `src/` - Core application source code
- `db/` - Database models and migrations
- `routes/` - API route handlers
- `middleware/` - Middleware functions
- `utils/` - Utility functions
- `test/` - Test files and utilities
- `scripts/` - Helper scripts
- `bin/` - CLI entry point

## Testing

VSR has a comprehensive test suite that covers various aspects of the codebase, including the stale-while-revalidate caching pattern implemented with Cloudflare Workers' `waitUntil` API.

### Running Tests

```bash
# Run all tests
npm test

# Run specific test groups
npm run test:cache # Run caching-related tests
npm run test:improved # Run optimized `waitUntil` tests

# Clean up problematic test files
npm run test:cache:clean
```

The test suite is configured with the following npm scripts:

- `test`: Runs the full test suite with cleanup, setup, and execution
- `test:run`: Executes the tests using Vitest
- `test:cache`: Runs caching-related tests
- `test:improved`: Runs optimized tests for the `waitUntil` pattern
- `test:setup`: Sets up the test environment
- `test:cleanup`: Cleans up processes from previous test runs

### Testing Background Refresh Functionality

The registry implements a "stale-while-revalidate" caching pattern for faster responses:

1. When a request is received, we first check if the data is in our cache
2. If the data is in the cache, we return it immediately, even if it's stale (old)
3. If the data is stale, we queue a background task to refresh it from the upstream registry
4. The background task updates the cache with fresh data for future requests
5. This approach ensures users get a fast response (cached data) while keeping our cache up-to-date

This pattern is implemented in the following key files:
- `getPackagePackument`: For fetching package metadata
- `getPackageManifest`: For fetching specific package versions

#### How `waitUntil` Works

Cloudflare Workers provide a special API called `waitUntil` that enables background tasks to continue running after a response has been sent. Here's a simplified example:

```javascript
export default {
async fetch(request, env, ctx) {
// Return a response immediately
const response = new Response("Hello World");

// Queue a background task that continues after response is sent
ctx.waitUntil(
(async () => {
// This runs in the background after response is sent
await doLongRunningTask();
})()
);

return response;
}
};
```

### Testing Patterns

Testing `waitUntil` behavior can be tricky because of how JavaScript executes Immediately Invoked Function Expressions (IIFEs). The common mistake is to use code like this:

```javascript
// ❌ PROBLEM: This executes the function immediately!
c.waitUntil((async () => {
// Background work
await refreshCache();
})());
```

Our testing approach:

1. Correctly simulates the Cloudflare `waitUntil` API
2. Captures background tasks without executing them immediately
3. Allows precise control over when background tasks run

For detailed documentation on testing this pattern, refer to `test/waituntil-README.md`.

#### Test Files

- `waituntil-correct.test.js`: Demonstrates the core pattern with simple examples
- `hono-context.test.js`: Shows how to mock the Hono context with proper `waitUntil` support
- `route-with-waituntil.test.js`: Tests a route handler implementing the stale-while-revalidate pattern

## Deployment

### Local Deployment

For local usage, you can simply run:

```bash
npx -y vltpkg/vsr
```

This will start the registry locally at http://localhost:1337.

### Production Deployment

VSR is designed to be deployed on Cloudflare Workers. To deploy to production:

1. **Set up Cloudflare account and create necessary resources**:
- Create a Cloudflare account if you don't have one
- Create a new D1 Database
- Create a new R2 Bucket

2. **Configure your `wrangler.toml`**:
Make sure your configuration includes the correct bindings for D1 and R2.

3. **Deploy using Wrangler**:
```bash
npm run build # Create deployment bundle
npm run deploy # Deploy to Cloudflare Workers
```

4. **Post-deployment setup**:
- Initialize your database with the schema
- Create admin tokens as needed

## Caching Implementation

The stale-while-revalidate caching pattern is a core feature of VSR. It provides several benefits:

1. **Improved Response Times**: By returning cached data immediately, users experience fast response times.
2. **Reduced Load on Upstream Registry**: By caching package data, we reduce the number of requests to the upstream npm registry.
3. **Background Updates**: By refreshing stale data in the background, cache stays current without impacting user response time.

### Implementation Details

The caching logic is implemented as follows:

1. **Cache Check**: When a request comes in, we check if the data exists in our cache (D1 database).
2. **Freshness Check**: If the data exists, we check if it's fresh (typically less than 5 minutes old).
3. **Response Strategy**:
- If data is fresh: Return it immediately
- If data is stale: Return it immediately AND queue a background refresh
- If data doesn't exist: Fetch from upstream, cache it, then return

4. **Background Refresh**:
```javascript
// Example pattern (simplified)
if (isStale) {
c.executionCtx.waitUntil(new Promise(async (resolve) => {
const bgRefresh = async () => {
const freshData = await fetchFromUpstream();
await saveToCache(freshData);
resolve();
};

setTimeout(bgRefresh, 0);
}));
}
```

## Contribution Guidelines

1. **Create an issue** first to discuss proposed changes or report bugs
2. **Fork the repository** and create a feature branch for your changes
3. **Write tests** for new features or bug fixes
4. **Ensure code quality** by following the established style and patterns
5. **Submit a pull request** referencing the original issue

## Pull Request Process

1. Ensure your code passes all tests and linting
2. Update documentation if necessary
3. Add yourself to the contributors list (if not already there)
4. Submit your PR with a clear description of changes
5. Address any feedback from code reviews

## Code Style

- Use modern JavaScript features
- Follow the existing code structure and naming conventions
- Document new functions and components with JSDoc comments
- Maintain comprehensive test coverage

Thank you for contributing to VSR!
Loading