Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: can't use API from workflows when IP Restriction is enabled #42

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

fix: can't use API from workflows when IP Restriction is enabled #42

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
2a00139
refactor: restructure Alb and IAlb
WinterYukky Mar 5, 2025
ffb32c3
fix: can't connect api from worker
WinterYukky Mar 5, 2025
3a80130
doc: add internal endpoint document
WinterYukky Mar 5, 2025
4730dac
test: update snapshot tests
WinterYukky Mar 5, 2025
577b296
refactor: albs rename to endpoint
WinterYukky Mar 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
161 changes: 89 additions & 72 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,27 +8,31 @@ Self-host [Dify](https://dify.ai/), an LLM app development platform, using AWS m

Key Features:

* Fully managed services requiring less maintenance effort
* Aurora servereless v2, ElastiCache, ECS Fargate, etc.
* Cost effective architectural decisions
* allow to use NAT instances instead of NAT Gateway, and Fargate spot capacity by default
* Easily integrate with Bedrock models and Knowledge Bases
- Fully managed services requiring less maintenance effort
- Aurora servereless v2, ElastiCache, ECS Fargate, etc.
- Cost effective architectural decisions
- allow to use NAT instances instead of NAT Gateway, and Fargate spot capacity by default
- Easily integrate with Bedrock models and Knowledge Bases

本リポジトリの使い方について、日本語で書かれた記事もあります:
* [AWS CDKでDifyを一撃構築](https://note.com/yukkie1114/n/n0d9c5551569f) ( [CloudShell版](https://note.com/yukkie1114/n/n8e055c4e7566) )
* [AWSマネージドサービスで Dify のセルフホスティングを試してみた](https://dev.classmethod.jp/articles/dify-self-hosting-aws/)
本リポジトリの使い方について、日本語で書かれた記事もあります:

- [AWS CDKでDifyを一撃構築](https://note.com/yukkie1114/n/n0d9c5551569f) ( [CloudShell版](https://note.com/yukkie1114/n/n8e055c4e7566) )
- [AWSマネージドサービスで Dify のセルフホスティングを試してみた](https://dev.classmethod.jp/articles/dify-self-hosting-aws/)

## Prerequisites

You must have the following dependencies installed to deploy this app:

* [Node.js](https://nodejs.org/en/download/) (v18 or newer)
* [Docker](https://docs.docker.com/get-docker/)
* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) and IAM profile with Administrator policy
- [Node.js](https://nodejs.org/en/download/) (v18 or newer)
- [Docker](https://docs.docker.com/get-docker/)
- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) and IAM profile with Administrator policy

## Deploy

You can adjust configuration parameters such as AWS regions by modifying [`bin/cdk.ts`](bin/cdk.ts). Please also check [`EnvironmentProps` interface](./lib/environment-props.ts) for all the available parameters.

> [!IMPORTANT]
>
> > If you are upgrading from Dify v0 to v1, please refer to [Upgrading Dify v0 to v1](#upgrading-dify-v0-to-v1).

Then you can run the following commands to deploy the entire stack.
Expand All @@ -51,6 +55,7 @@ The initial deployment usually takes about 20 minutes. After a successful deploy

Outputs:
DifyOnAwsStack.DifyUrl = https://dify.example.com
DifyOnAwsStack.DifyInternalUrl = http://internal-DifyOn-Alb-xxx.ap-northeast-1.elb.amazonaws.com
```

You can open the URL with a browser and get started!
Expand Down Expand Up @@ -102,17 +107,17 @@ Please also refer to this blog article for more details: [Using any Python libra
You can use the [External Knowledge Base feature](https://docs.dify.ai/guides/knowledge-base/connect-external-knowledge) to connect to [Amazon Bedrock Knowledge Bases](https://aws.amazon.com/bedrock/knowledge-bases/). Because the external knowledge API is deployed as a sidecar of Dify API, you can use the feature immediately with the following steps:

1. Click Dify -> Knowledge -> Add an External Knowledge API button.
* ![add external knowledge api](./imgs/add-external-knowledge-api.png)
- ![add external knowledge api](./imgs/add-external-knowledge-api.png)
2. Fill the form as below:
1. Name: any name as you like (e.g. `Bedrock Knowledge Bases`)
2. API Endpoint: `http://localhost:8000`
3. API Key: `dummy-key` (you can configure it by editing `BEARER_TOKEN` environment variable in [`api.ts`](./lib/constructs/dify-services/api.ts).)
1. Name: any name as you like (e.g. `Bedrock Knowledge Bases`)
2. API Endpoint: `http://localhost:8000`
3. API Key: `dummy-key` (you can configure it by editing `BEARER_TOKEN` environment variable in [`api.ts`](./lib/constructs/dify-services/api.ts).)
3. Click Dify -> Knowledge -> Create Knowledge -> Connect to an External Knowledge Base
* ![Connect to an External Knowledge Base](./imgs/connect-to-an-externa-lknowledge-base.png)
- ![Connect to an External Knowledge Base](./imgs/connect-to-an-externa-lknowledge-base.png)
4. Fill the form as below
1. External Knowledge Name / Knowledge Description: any string
2. External Knowledge API: the external API you created in the previous step
3. External Knowledge ID: The Bedrock Knowledge Base ID you want to use. The AWS region is us-west-2 by default, but you can override the AWS region by adding region prefix with colon, e.g. `us-east-1:QWERTYASDF`.
1. External Knowledge Name / Knowledge Description: any string
2. External Knowledge API: the external API you created in the previous step
3. External Knowledge ID: The Bedrock Knowledge Base ID you want to use. The AWS region is us-west-2 by default, but you can override the AWS region by adding region prefix with colon, e.g. `us-east-1:QWERTYASDF`.
5. Now you can use the knowledge from Dify tools.

For more information, please refer to this article: [Dify can also do RAG on documents with charts and graphs!](https://qiita.com/mabuchs/items/85fb2dad19ec441c870c)
Expand All @@ -124,16 +129,16 @@ Although this system is designed with infrastructure scalability in mind, there
The below are the list of configurable parameters and their default values:

1. ECS Task ([api.ts](./lib/constructs/dify-services/api.ts), [web.ts](./lib/constructs/dify-services/web.ts))
1. Size
1. api/worker: 1024vCPU / 2048MB
2. web: 256vCPU / 512MB
2. Desired Count
1. 1 task for each service
1. Size
1. api/worker: 1024vCPU / 2048MB
2. web: 256vCPU / 512MB
2. Desired Count
1. 1 task for each service
2. ElastiCache ([redis.ts](./lib/constructs/redis.ts))
1. Node Type: `cache.t4g.micro`
2. Node Count: 1
1. Node Type: `cache.t4g.micro`
2. Node Count: 1
3. Aurora Postgres ([postgres.ts](./lib/constructs/postgres.ts))
1. Serverless v2 maximum capacity: 2 ACU
1. Serverless v2 maximum capacity: 2 ACU

### Deploying to a closed network (a.k.a 閉域要件)

Expand All @@ -142,45 +147,54 @@ You can deploy the system on a closed network (i.e. a VPC without internet gatew
To deploy on a closed network, please follow the steps below:

1. Set configuration parameters in `bin/cdk.ts` as below:
```ts
export const props: EnvironmentProps = {
// set region and account explicitly.
awsRegion: 'ap-northeast-1',
awsAccount: '123456789012',

// Set your internal IP address ranges here.
allowedIPv4Cidrs: ['10.0.0.0/16'],
```ts
export const props: EnvironmentProps = {
// set region and account explicitly.
awsRegion: 'ap-northeast-1',
awsAccount: '123456789012',

// The below two flags must be set for closed network deployment.
useCloudFront: false,
internalAlb: true,
// Set your internal IP address ranges here.
allowedIPv4Cidrs: ['10.0.0.0/16'],

// If Docker Hub is not accessible from your vpc subnets, set this property and run copy-to-ecr script (see step#2)
customEcrRepositoryName: 'dify-images',
// The below two flags must be set for closed network deployment.
useCloudFront: false,
internalAlb: true,

// To let the CDK create a VPC with closed network, set this property.
vpcIsolated: true,
// Or, optionally you can import an existing VPC.
vpcId: 'vpc-12345678',
// If Docker Hub is not accessible from your vpc subnets, set this property and run copy-to-ecr script (see step#2)
customEcrRepositoryName: 'dify-images',

// Other properties can be configured as you like.
};
```
// To let the CDK create a VPC with closed network, set this property.
vpcIsolated: true,
// Or, optionally you can import an existing VPC.
vpcId: 'vpc-12345678',

// Other properties can be configured as you like.
};
```

2. Open [`python-requirements.txt`](lib/constructs/dify-services/docker/sandbox/python-requirements.txt) and remove all the dependencies from it
* This is **only required** if [PyPI](https://pypi.org/) is not accessible from your vpc subnets.
- This is **only required** if [PyPI](https://pypi.org/) is not accessible from your vpc subnets.
3. Copy all the dify container images in Docker Hub to an ECR repository by executing `npx ts-node scripts/copy-to-ecr.ts`.
* The script handles all the tasks required to copy images. You will also need to run `npm ci` before this.
* You can create an ECR repository with the name of `customEcrRepositoryName` by yourself, or the script creates one if it does not exist yet.
* This script must be executed in an environment that has access to the Internet.
* Please run the script every time you change `difyImageTag` or `difySandboxImageTag` property.
* This is **only required** if [Docker Hub](https://www.docker.com/products/docker-hub/) is not accessible from your vpc subnets.
- The script handles all the tasks required to copy images. You will also need to run `npm ci` before this.
- You can create an ECR repository with the name of `customEcrRepositoryName` by yourself, or the script creates one if it does not exist yet.
- This script must be executed in an environment that has access to the Internet.
- Please run the script every time you change `difyImageTag` or `difySandboxImageTag` property.
- This is **only required** if [Docker Hub](https://www.docker.com/products/docker-hub/) is not accessible from your vpc subnets.
4. If you are using an existing VPC (`vpcId` property), make sure the required VPC endpoints are provisioned before deployment.
* See [`vpc-endpoints.ts`](lib/constructs/vpc-endpoints.ts) for the list of required VPC endpoints.
* If you let CDK create a VPC (by setting `vpcIsolated: true`), all the endpoints are created automatically.
- See [`vpc-endpoints.ts`](lib/constructs/vpc-endpoints.ts) for the list of required VPC endpoints.
- If you let CDK create a VPC (by setting `vpcIsolated: true`), all the endpoints are created automatically.
5. Deploy the CDK project following the [Deploy](#deploy) section.
6. After the deployment, please configure Bedrock in Dify with the same AWS region as your VPC (see [setup section](#setup-dify-to-use-bedrock))
* This is **only required** if Bedrock API in other regions are not accessible from your vpc subnets.
- This is **only required** if Bedrock API in other regions are not accessible from your vpc subnets.

### Using API from Workflows when IP Restriction is enabled

When you use the Dify API in your workflows etc, you can configuration `DifyInternalUrl` that display at deployed as API endpoint. Using this endpoint, you don't need to add IP address of NAT Gateway to allow list. The following url is an example.

```
http://internal-DifyOn-Alb-xxx.ap-northeast-1.elb.amazonaws.com/v1/datasets/xxxx/retrieve
```

### Connect to Notion

Expand All @@ -189,6 +203,7 @@ You can connect to [Notion](https://www.notion.com/) data by the following steps
1. Obtain the Notion Secret Token: [Notion - Authorization](https://developers.notion.com/docs/authorization).

2. Create a Screts Manager secret for the token:

```sh
NOTION_INTERNAL_SECRET="NOTION_SECRET_REPLACE_THIS"
aws secretsmanager create-secret \
Expand All @@ -198,22 +213,23 @@ You can connect to [Notion](https://www.notion.com/) data by the following steps
```

3. Set `additionalEnvironmentVariables` in `bin/cdk.ts` as below:

```ts
export const props: EnvironmentProps = {
// ADD THIS
additionalEnvironmentVariables: [
{
key: 'NOTION_INTEGRATION_TYPE',
value: 'internal',
targets: ['api'],
targets: ['api'],
},
{
key: 'NOTION_INTERNAL_SECRET',
value: { secretName: 'NOTION_INTERNAL_SECRET'},
targets: ['api'],
value: { secretName: 'NOTION_INTERNAL_SECRET' },
targets: ['api'],
},
],
}
};
```

4. Deploy the stack by `cdk deploy` command.
Expand All @@ -226,6 +242,7 @@ You can let Dify send emails to invite new users or reset passwords. To enable t
After a successful deployment, you have to move out from SES sandbox to send emails to non-verified addresses and domains. Please refer to the document for more details: [Request production access (Moving out of the Amazon SES sandbox)](https://docs.aws.amazon.com/ses/latest/dg/request-production-access.html)

### Upgrading Dify v0 to v1

When you upgrade Dify from v0 to v1, you need to execute some migration steps described below.

1. Set `autoMigration: false` in lib/dify-on-aws-stack.ts (`ApiService` construct).
Expand All @@ -244,6 +261,7 @@ When you upgrade Dify from v0 to v1, you need to execute some migration steps de
6. After the commands run successfully, set `autoMigration: true`, and deploy CDK again. You should be now onboard with Dify v1.

## Clean up

To avoid incurring future charges, clean up the resources you created.

```sh
Expand All @@ -257,22 +275,20 @@ If you set `customEcrRepositoryName` and have run the `copy-to-ecr.ts` script, p

The following table provides a sample cost breakdown for deploying this system in the us-east-1 (N. Virginia) region for one month (when deployed using less expensive configuration).


| AWS service | Dimensions | Cost [USD/month] |
| --------------------| ----------------- | -------------------------------|
| RDS Aurora | Postgres Serverless v2 (0 ACU) | $0 |
| ElastiCache | Valkey t4g.micro | $9.2 |
| ECS (Fargate) | Dify-web 1 task running 24/7 (256CPU) | $2.7 |
| ECS (Fargate) | Dify-api/worker 1 task running 24/7 (1024CPU) | $10.7 |
| Application Load Balancer | ALB-hour per month | $17.5 |
| VPC | NAT Instances t4g.nano x1 | $3.0 |
| VPC | Public IP address x1 | $3.6 |
| Secrets Manager | Secret x3 | $1.2 |
| TOTAL | estimate per month | $47.9 |
| AWS service | Dimensions | Cost [USD/month] |
| ------------------------- | --------------------------------------------- | ---------------- |
| RDS Aurora | Postgres Serverless v2 (0 ACU) | $0 |
| ElastiCache | Valkey t4g.micro | $9.2 |
| ECS (Fargate) | Dify-web 1 task running 24/7 (256CPU) | $2.7 |
| ECS (Fargate) | Dify-api/worker 1 task running 24/7 (1024CPU) | $10.7 |
| Application Load Balancer | ALB-hour per month | $17.5 |
| VPC | NAT Instances t4g.nano x1 | $3.0 |
| VPC | Public IP address x1 | $3.6 |
| Secrets Manager | Secret x3 | $1.2 |
| TOTAL | estimate per month | $47.9 |

Note that you have to pay LLM cost (e.g. Amazon Bedrock ) in addition to the above, which totally depends on your specific use case.


## Security

See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information.
Expand All @@ -282,4 +298,5 @@ See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more inform
This library is licensed under the MIT-0 License. See the LICENSE file. You should also check [Dify's license](https://github.com/langgenius/dify/blob/main/LICENSE).

## Acknowledgement

This CDK code is heavily inspired by [dify-aws-terraform](https://github.com/sonodar/dify-aws-terraform).
Loading