ShortURL Service

A high-performance URL shortener service built with Go, Gin framework, Redis caching, and Bloom filter for fault tolerance.

Features

Core Features

• URL Shortening: Convert long URLs into short, easy-to-share links
• High Performance: Optimized with Redis caching for fast access
• RESTful API: Clean and simple API endpoints
• Snowflake ID: Generate unique short IDs using Snowflake algorithm
• Database Support: Persistent storage with MySQL integration
• Configurable: YAML-based configuration for easy setup

Advanced Features

• Bloom Filter: Fast existence check to reduce database load
• Fault Tolerance: Graceful degradation when Redis is unavailable
• Data Consistency: Write protection when Redis is down
• Structured Logging: Production-grade logging with Zap
• OIDC Authentication: Secure user authentication

Tech Stack

Backend

• Go 1.25.5: Backend language
• Gin 1.11.0: HTTP web framework
• Redis: Caching layer and Bloom filter storage
• MySQL: Persistent database storage
• GORM: ORM library for database operations
• Zap: Structured logging
• Snowflake: Unique ID generation

Frontend

• React: UI framework
• UmiJS: Application framework
• TDesign: UI component library
• OIDC: Authentication

Architecture

Data Flow

Create Short URL

Request → Bloom Filter Check → Database Write → Response
         ↓
    Redis Down → Reject Request (Protect Data Consistency)

Get Origin URL

Request → Bloom Filter Check → Cache → Response
         ↓              ↓
    Redis Down → Database Query (Graceful Degradation)

Fault Tolerance Strategy

Scenario	Write Operation	Read Operation	Data Consistency
Normal	✅ Allowed	✅ Cache + Bloom Filter	✅ Consistent
Redis Down	❌ Rejected	✅ Database Only	✅ Protected
Database Down	❌ Rejected	❌ Failed	✅ Protected

Installation

Prerequisites

• Go 1.25.5+
• Redis 6.0+ (with Bloom Filter module)
• MySQL 5.7+
• Node.js 16+ (for frontend)

Backend Setup

1. Clone the repository:

git clone https://github.com/HuckOps/shorturl.git
cd shorturl

2. Install dependencies:

go mod download

3. Copy and configure the config file:

cp config.yaml.example config.yaml

Edit config.yaml with your settings:

mysql_dsn: user:password@tcp(localhost:3306)/shorturl?charset=utf8mb4&parseTime=True&loc=Local

redis:
  addr: localhost:6379
  password: ""
  DB: 0

machine_id: 1

service:
  bloomfilter_enable: true

shouturl_entrance:
  protocol: http
  host: localhost:8080

auth:
  provider: https://your-oidc-provider.com
  client_id: your-client-id
  client_secret: your-client-secret
  redirect_url: http://localhost:8080/callback

4. Initialize database:

# Create database
mysql -u root -p -e "CREATE DATABASE shorturl CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;"

# Run migrations (if using GORM AutoMigrate)
go run cmd/migrate/main.go

5. Run the service:

go run cmd/server/main.go --config=config.yaml

Frontend Setup

1. Install dependencies:

cd web
npm install

2. Start development server:

npm run dev

3. Build for production:

npm run build

Usage

API Endpoints

Create Short URL

curl -X POST http://localhost:8080/api/shorturl \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"origin": "https://example.com/very/long/url"}'

Redirect to Original URL

curl http://localhost:8080/m/abc123

List User's Short URLs

curl -X GET http://localhost:8080/api/shorturl/manage?page=1&pageSize=10 \
  -H "Authorization: Bearer YOUR_TOKEN"

Delete Short URL

curl -X DELETE http://localhost:8080/api/shorturl/{id} \
  -H "Authorization: Bearer YOUR_TOKEN"

Bloom Filter Management

Rebuild Bloom Filter

When you need to rebuild the Bloom filter (e.g., after data import or recovery):

go run cmd/bloomfilterRebuilder/main.go --config=config.yaml --chunk-size=10000

Options:

• --config: Path to config file (default: config.yaml)
• --chunk-size: Number of records to process per chunk (default: 10000)

Configuration

Bloom Filter Configuration

service:
  bloomfilter_enable: true # Enable/disable bloom filter

When bloomfilter_enable is set to false:

• Bloom filter checks are skipped
• All requests go directly to database
• No write protection when Redis is down

Redis Configuration

redis:
  addr: localhost:6379
  password: ""
  DB: 0

Note: Ensure Redis has the Bloom Filter module loaded (Redis Stack or RedisBloom).

Monitoring

Logs

The service uses structured logging with Zap. Logs include:

• Request/response details
• Bloom filter operations
• Cache hits/misses
• Error tracking

Example log entry:

{
  "time": "2024-01-15T10:30:00Z",
  "level": "debug",
  "msg": "Check from bloomfilter",
  "encoded": "abc123",
  "exists": true,
  "error": null
}

Metrics (Future Enhancement)

Consider adding Prometheus metrics for:

• Request rate
• Cache hit ratio
• Bloom filter false positive rate
• Response time percentiles

Troubleshooting

Redis Connection Issues

If Redis is down:

• Write operations: Will be rejected with "Service temporarily unavailable"
• Read operations: Will fall back to database (slower but available)
• Solution: Fix Redis connection or disable Bloom filter temporarily

Bloom Filter False Positives

If you encounter false positives:

• Cause: Bloom filter has a small false positive rate (typically < 1%)
• Impact: Minor - will check database and return appropriate error
• Solution: Rebuild Bloom filter with larger capacity

Data Consistency Issues

If you suspect data inconsistency:

1. Check logs for database write failures
2. Rebuild Bloom filter:

   go run cmd/bloomfilterRebuilder/main.go --config=config.yaml

3. Verify data in both database and Bloom filter

License

MIT

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Support

For issues and questions, please open an issue on GitHub.