Skip to content

Commit ca860d1

Browse files
hietthiett
authored
Merge pull request #8 from hiett/sh/response-encoding
Response encoding, and automated testing with @upstash/redis
2 parents 3bc171d + 23031cf commit ca860d1

File tree

15 files changed

+641
-89
lines changed

15 files changed

+641
-89
lines changed

‎.github/workflows/test.yml‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
name: Test @upstash/redis compatability
2+
on:
3+
workflow_dispatch:
4+
push:
5+
paths:
6+
- 'lib/**'
7+
schedule:
8+
- cron: '0 12 * * *'
9+
10+
env:
11+
SRH_TOKEN: example_token
12+
13+
jobs:
14+
container-job:
15+
runs-on: ubuntu-latest
16+
container: denoland/deno
17+
services:
18+
redis:
19+
image: redis/redis-stack-server:6.2.6-v6 # 6.2 is the Upstash compatible Redis version
20+
srh:
21+
image: hiett/serverless-redis-http:0.0.5-alpha
22+
env:
23+
SRH_MODE: env
24+
SRH_TOKEN: ${{ env.SRH_TOKEN }}
25+
SRH_CONNECTION_STRING: redis://redis:6379
26+
27+
steps:
28+
- name: Checkout code
29+
uses: actions/checkout@v3
30+
with:
31+
repository: upstash/upstash-redis
32+
33+
- name: Run @upstash/redis Test Suite
34+
run: deno test -A ./pkg
35+
env:
36+
UPSTASH_REDIS_REST_URL: http://srh:80
37+
UPSTASH_REDIS_REST_TOKEN: ${{ env.SRH_TOKEN }}

‎.gitignore‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,4 @@ srh-*.tar
2929

3030
*.iml
3131

32-
srh-config/
33-
34-
example/
32+
srh-config/

‎HOW_TO_BUILD.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
## Building the Docker image
2+
3+
To build both an amd64 image and an arm64 image, on an M1 Mac:
4+
5+
```
6+
docker buildx build --platform linux/amd64,linux/arm64 --push -t hiett/serverless-redis-http:0.0.5-alpha
7+
```

‎README.md‎

Lines changed: 117 additions & 74 deletions
Original file line numberDiff line numberDiff line change
@@ -1,102 +1,145 @@
1-
# SRH: Serverless Redis HTTP
1+
# Serverless Redis HTTP (SRH)
22

3-
---
3+
A Redis proxy and connection pooler that uses HTTP rather than the Redis binary protocol.\
4+
The aim of this project is to be entirely compatible with Upstash, and work with any Upstash supported Redis version.
45

5-
**TLDR: If you want to run a local Upstash-compatible HTTP layer in front of your Redis:**
6+
Use cases for SRH:
7+
- For usage in your CI pipelines, creating Upstash databases is tedious, or you have lots of parallel runs.
8+
- See [Using in GitHub Actions](#in-github-actions) on how to quickly get SRH setup for this context.
9+
- For usage inside of Kubernetes, or any network whereby the Redis server is not exposed to the internet.
10+
- See [Using in Docker Compose](#via-docker-compose) for the various setup options directly using the Docker Container.
11+
- For local development environments, where you have a local Redis server running, or require offline access.
12+
- See [Using the Docker Command](#via-docker-command), or [Using Docker Compose](#via-docker-compose).
613

7-
0) Have a locally running Redis instance - in this example bound to the default port 6379
8-
1) create a json file called tokens.json in a folder called srh-config (`srh-config/tokens.json`)
9-
2) paste this in:
10-
```json
11-
{
12-
"example_token": {
13-
"srh_id": "some_unique_identifier",
14-
"connection_string": "redis://localhost:6379",
15-
"max_connections": 3
16-
}
17-
}
18-
```
19-
3) Run this command:
20-
`docker run -it -d -p 8079:80 --name srh --mount type=bind,source=$(pwd)/srh-config/tokens.json,target=/app/srh-config/tokens.json hiett/serverless-redis-http:latest`
21-
4) Set this as your Upstash configuration
22-
```js
14+
## Differences between Upstash and Redis to note
15+
SRH tests are ran nightly against the `@upstash/redis` JavaScript package. However, there are some minor differences between Upstash's implementation of Redis and the official Redis code.
16+
17+
- The `UNLINK` command will not throw an error when 0 keys are given to it. In Redis, and as such SRH, an error will be thrown.
18+
- In the `ZRANGE` command, in Upstash you are not required to provide `BYSCORE` or `BYLEX` in order to use the `LIMIT` argument. With Redis/SRH, this will throw an error if not provided.
19+
- The Upstash implementation of `RedisJSON` contains a number of subtle differences in what is returned in responses. For this reason, **it is not advisable to use SRH with Redis Stack if you are testing your Upstash implementation that uses JSON commands**. If you don't use any JSON commands, then all is good :)
20+
- **SRH does not implement commands via paths, or accepting the token via a query param**. Only the body method is implemented, which the `@upstash/redis` SDK uses.
21+
22+
### Similarities to note:
23+
24+
Pipelines and Transaction endpoints are also implemented, also using the body data only. You can read more about the RestAPI here: [Upstash Docs on the Rest API](https://docs.upstash.com/redis/features/restapi)
25+
26+
Response encoding is also fully implemented. This is enabled by default by the `@upstash/redis` SDK. You can read more about that here: [Upstash Docs on Hashed Responses](https://docs.upstash.com/redis/sdks/javascriptsdk/troubleshooting#hashed-response)
27+
28+
## How to use with the `@upstash/redis` SDK
29+
Simply set the REST URL and token to where the SRH instance is running. For example:
30+
```ts
2331
import {Redis} from '@upstash/redis';
2432

2533
export const redis = new Redis({
2634
url: "http://localhost:8079",
2735
token: "example_token",
28-
responseEncoding: false, // IMPORTANT: Upstash has recently added response encoding, but SRH does not support it yet.
2936
});
3037
```
31-
---
3238

33-
A Redis connection pooler for serverless applications. This allows your serverless functions to talk to Redis via HTTP,
34-
while also not having to worry about the Redis max connection limits.
35-
36-
The idea is you host this alongside your Redis server, to minimise latency. Your serverless functions can then talk to
37-
this via HTTP.
38-
39-
## Features
40-
- Allows you to talk to redis via HTTP
41-
- Pools redis connections
42-
- Automatically kills redis connections after inactivity
43-
- Supports multiple redis instances, and you can configure unique tokens for each
44-
- Fully supports the `@upstash/redis` TypeScript library.
39+
# Setting up SRH
40+
## Via Docker command
41+
If you have a locally running Redis server, you can simply start an SRH container that connects to it.
42+
In this example, SRH will be running on port `8080`.
43+
44+
```bash
45+
docker run \
46+
-it -d -p 8080:80 --name srh \
47+
-e SRH_MODE=env \
48+
-e SRH_TOKEN=your_token_here \
49+
-e SRH_CONNECTION_STRING="redis://your_server_here:6379" \
50+
hiett/serverless-redis-http:latest
51+
```
4552

46-
## Client usage
47-
This will not work with regular Redis clients, as it is over HTTP and not the redis protocol.
48-
However, to try and keep the project as "standardised" as possible, you can use the `@upstash/redis` TypeScript library.
49-
You can read about it here: [Upstash Redis GitHub](https://github.com/upstash/upstash-redis)
53+
## Via Docker Compose
54+
If you wish to run in Kubernetes, this should contain all the basics would need to set that up. However, be sure to read the Configuration Options, because you can create a setup whereby multiple Redis servers are proxied.
55+
```yml
56+
version: '3'
57+
services:
58+
redis:
59+
image: redis
60+
ports:
61+
- '6379:6379'
62+
serverless-redis-http:
63+
ports:
64+
- '8079:80'
65+
image: hiett/serverless-redis-http:latest
66+
environment:
67+
SRH_MODE: env
68+
SRH_TOKEN: example_token
69+
SRH_CONNECTION_STRING: 'redis://redis:6379' # Using `redis` hostname since they're in the same Docker network.
70+
```
5071
51-
Soon I will add specific documentation for the endpoints so you can implement clients in other languages.
72+
## In GitHub Actions
73+
74+
SRH works nicely in GitHub Actions because you can run it as a container in a job's services. Simply start a Redis server, and then
75+
SRH alongside it. You don't need to worry about a race condition of the Redis instance not being ready, because SRH doesn't create a Redis connection until the first command comes in.
76+
77+
```yml
78+
name: Test @upstash/redis compatability
79+
on:
80+
push:
81+
workflow_dispatch:
82+
83+
env:
84+
SRH_TOKEN: example_token
85+
86+
jobs:
87+
container-job:
88+
runs-on: ubuntu-latest
89+
container: denoland/deno
90+
services:
91+
redis:
92+
image: redis/redis-stack-server:6.2.6-v6 # 6.2 is the Upstash compatible Redis version
93+
srh:
94+
image: hiett/serverless-redis-http:latest
95+
env:
96+
SRH_MODE: env # We are using env mode because we are only connecting to one server.
97+
SRH_TOKEN: ${{ env.SRH_TOKEN }}
98+
SRH_CONNECTION_STRING: redis://redis:6379
99+
100+
steps:
101+
# You can place your normal testing steps here. In this example, we are running SRH against the upstash/upstash-redis test suite.
102+
- name: Checkout code
103+
uses: actions/checkout@v3
104+
with:
105+
repository: upstash/upstash-redis
106+
107+
- name: Run @upstash/redis Test Suite
108+
run: deno test -A ./pkg
109+
env:
110+
UPSTASH_REDIS_REST_URL: http://srh:80
111+
UPSTASH_REDIS_REST_TOKEN: ${{ env.SRH_TOKEN }}
112+
```
52113
53-
## Installation
54-
You have to options to run this:
55-
- Via docker: `docker pull hiett/serverless-redis-http:latest` [Docker Hub link](https://hub.docker.com/r/hiett/serverless-redis-http)
56-
- Via elixir: `(clone this repo)` -> `mix deps.get` -> `iex -S mix`
114+
# Configuration Options
57115
58-
If you are running via Docker, you will need to mount the configuration file to `/app/srh-config/tokens.json`.\
59-
An example of a run command is the following:
116+
SRH works with multiple Redis servers, and can pool however many connections you wish it to. It will shut down un-used pools after 15 minutes of inactivity. Upon the next command, it will re-build the pool.
60117
61-
`docker run -it -d -p 8080:80 --name srh --mount type=bind,source=$(pwd)/srh-config/tokens.json,target=/app/srh-config/tokens.json hiett/serverless-redis-http:latest`
118+
## Connecting to multiple Redis servers at the same time
62119
63-
*Note that it is running on port 80*
120+
The examples above use environment variables in order to tell SRH which Redis server to connect to. However, you can also use a configuration JSON file, which lets you create as many connections as you wish. The token provided in each request will decide which pool is used.
64121
65-
To configure Redis targets:\
66-
Create a file: `srh-config/tokens.json`
122+
Create a JSON file, in this example called `tokens.json`:
67123
```json
68124
{
69125
"example_token": {
70126
"srh_id": "some_unique_identifier",
71127
"connection_string": "redis://localhost:6379",
72128
"max_connections": 3
73-
}
129+
}
74130
}
75131
```
132+
You can provide as many entries to the base object as you wish, and configure the number of max connections per pool. The `srh_id` is used internally to keep track of instances. It can be anything you want.
76133

77-
### Docker Compose
78-
You'll want the above `tokens.json` file but use this as your connection string:
79-
```json
80-
"connection_string": "redis://redis:6379"
81-
```
82-
docker-compose.yaml
83-
```yaml
84-
version: '3'
85-
services:
86-
redis:
87-
image: redis
88-
ports:
89-
- '6379:6379'
90-
serverless-redis-http:
91-
ports:
92-
- '8079:80'
93-
image: hiett/serverless-redis-http:latest
94-
volumes:
95-
- ./path/to/tokens.json:/app/srh-config/tokens.json
96-
```
134+
Once you have created this, mount it to the docker container to the `/app/srh-config/tokens.json` file. Here is an example docker command:
135+
136+
`docker run -it -d -p 8079:80 --name srh --mount type=bind,source=$(pwd)/tokens.json,target=/app/srh-config/tokens.json hiett/serverless-redis-http:latest`
137+
138+
## Environment Variables
97139

98-
Notes:
99-
- Srh_id can be anything you want, as long as it's a string, and unique.
100-
- `max_connections` is the maximum number of connections for the pool.
101-
- If there is inactivity, the pool will kill these connections. They will only be open while the pool is alive. The pool will re-create these connections when commands come in.
102-
- You can add more redis instances to connect to by adding more tokens and connection configurations. Based on the header in each request, the correct pool/connection info will be used.
140+
| Name | Default Value | Notes |
141+
| ---- | ------------- | ----- |
142+
| SRH_MODE | `file` | Can be `env` or `file`. If `file`, see [Connecting to multiple Redis servers](#connecting-to-multiple-redis-servers-at-the-same-time). If set to `env`, you are required to provide the following environment variables: |
143+
| SRH_TOKEN | `<required if SRH_MODE = env>` | Set the token that the Rest API will require |
144+
| SRH_CONNECTION_STRING | `<required if SRH_MODE = env>` | Sets the connection string to the Redis server. |
145+
| SRH_MAX_CONNECTIONS | `3` | Only used if `SRH_MODE=env`.

‎config/test.exs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
import Config

‎example/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
node_modules/
2+
dist/

‎example/package.json‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
{
2+
"name": "srh-example",
3+
"version": "1.0.0",
4+
"main": "index.js",
5+
"author": "Scott Hiett",
6+
"license": "MIT",
7+
"private": false,
8+
"scripts": {
9+
"start": "ts-node src/index.ts"
10+
},
11+
"dependencies": {
12+
"@upstash/redis": "^1.20.2"
13+
},
14+
"devDependencies": {
15+
"@types/node": "^18.15.11",
16+
"ts-node": "^10.9.1",
17+
"typescript": "^5.0.2"
18+
}
19+
}

‎example/src/index.ts‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
import {Redis} from "@upstash/redis";
2+
3+
const redis = new Redis({
4+
// The URL of the SRH instance
5+
url: "http://127.0.0.1:8080",
6+
7+
// The token you defined in tokens.json
8+
token: "example_token",
9+
10+
// Response encoding is supported (this is enabled by default)
11+
responseEncoding: true,
12+
});
13+
14+
(async () => {
15+
await redis.set("foo", "bar");
16+
const value = await redis.get("foo");
17+
console.log(value);
18+
19+
// Run a pipeline operation
20+
const pipelineResponse = await redis.pipeline()
21+
.set("amazing-key", "bar")
22+
.get("amazing-key")
23+
.del("amazing-other-key")
24+
.del("random-key-that-doesnt-exist")
25+
.srandmember("random-key-that-doesnt-exist")
26+
.sadd("amazing-set", "item1", "item2", "item3", "bar", "foo", "example")
27+
.smembers("amazing-set")
28+
.get("foo")
29+
.exec();
30+
31+
console.log(pipelineResponse);
32+
33+
const multiExecResponse = await redis.multi()
34+
.set("example", "value")
35+
.get("example")
36+
.exec();
37+
38+
console.log(multiExecResponse);
39+
})();

‎example/srh-config.json‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
{
2+
"example_token": {
3+
"srh_id": "some_unique_identifier",
4+
"connection_string": "redis://redis:6379",
5+
"max_connections": 3
6+
}
7+
}

0 commit comments

Comments
 (0)