Building Gateway Image with Policies from Policy Hub using apipctl

[renuka-fernando]

Building Gateway Image with Policies from Policy Hub using apipctl

apipctl Command

apipctl gateway build --image-tag v0.2.0-policy1

Format

apipctl gateway build \
  --image-tag <image-tag> \
  [-f <policy-manifest-yaml-file>]
  [--image-repository <image-repository>] \
  [--gateway-builder <gateway-builder-image>] \
  [--gateway-controller-base-image <gateway-controller-base-image>] \
  [--router-base-image <router-base-image>] \
  [--push] \
  [--no-cache] \
  [--platform <platform>] \
  [--offline] \
  [--output-dir <output_dir>]

Default values:

• f: policy-manifest.yaml
• image-repository: ghcr.io/wso2/api-platform
• gateway-builder: ghcr.io/wso2/api-platform/gateway-builder:0.2.0
• gateway-controller-base-image: (uses the default defined in the gateway-builder image)
• router-base-image: (uses the default defined in the gateway-builder image)
• platform: (uses the host platform)
• push: false
• no-cache: false
• offline: false
• output-dir: (no output)

When --offline is set:

• Policy Hub is NOT contacted
• Requires an existing policy-manifest-lock.yaml file
• All policies must exist in local cache at: ~/.apipctl/cache/policies/
• Build fails if any policy is missing from cache
• Error: "Policy basic-auth-v1.0.0 not found in cache. Run without --offline first."

Example

Run this command in a dir which contains the policy-manifest.yaml file.

apipctl gateway build \
  --image-tag v0.2.0-policy1

This will output the following Docker images:

• ghcr.io/wso2/api-platform/gateway-controller:v0.2.0-policy1
• ghcr.io/wso2/api-platform/router:v0.2.0-policy1
• ghcr.io/wso2/api-platform/policy-engine:v0.2.0-policy1

Policy.zip file name format

<policy-name>-v<policy-version>.zip

Here policy-name and policy-version are in kebab-case format.
For example, BasicAuth v1.0.0 will be downloaded as basic-auth-v1.0.0.zip.

Unzipping this file will create a directory named basic-auth/v1.0.0/ containing the policy source files.

Policy Cache

Location: ~/.apipctl/cache/policies/

Structure:

~/.apipctl/cache/
└── policies/
  ├── basic-auth-v1.0.0.zip
  ├── jwt-v2.1.0.zip 
  └── rate-limit-v1.5.2.zip

Internals of the apipctl gateway build Command

1. Read the policy-manifest.yaml file.
2. Send the JSON request to the Policy Hub by skipping local policies. Response from the Policy Hub contains the resolved versions and download URLs.
3. If the resolved version is in the local cache, skip the download.
4. Download other policies from the download URLs, checksum validation and cache them locally.
5. Cache local policies defined in the policy-manifest.yaml file.
6. Generate the policy-manifest-lock.yaml file in the same dir where policy-manifest.yaml is located.
7. Provide the policy-manifest-lock.yaml file to the gateway builder to build the gateway images.
8. Create a temp dir .
9. Copy the policy-manifest-lock.yaml to /policy-manifest-lock.yaml
10. Extract the downloaded policies to /policies/ which are defined in policy-manifest-lock.yaml.
11. Create dir /output.
12. Execute the following command to build policy engine binary.

    docker run --rm \
        -v <temp-dir>:/workspace \
        <gateway-builder-image>

13. Execute the following command to build gateway Docker images.

    cd <temp-dir>/output/policy-engine
    docker build -t <image-repository>/policy-engine:<image-tag> .
    cd <temp-dir>/output/gateway-controller
    docker build -t <image-repository>/gateway-controller:<image-tag> .
    cd <temp-dir>/output/router
    docker build -t <image-repository>/router:<image-tag> .

14. Clean up the temp dir . On failure, keep the temp dir and log "Build failed. Temp dir: /tmp/apipctl-build-123456"
15. Display the names of the built images.

Policy Manifest file and lock file discussion: #473

[piyumaldk]

Building Gateway Image with Policies from Policy Hub using apipctl

apipctl Command

apipctl gateway build --image-tag v0.2.0-policy1

Format

apipctl gateway build \

  --image-tag <image-tag> \

  [-f <policy-manifest-yaml-file>]

  [--image-repository <image-repository>] \

  [--gateway-builder <gateway-builder-image>] \

  [--gateway-controller-base-image <gateway-controller-base-image>] \

  [--router-base-image <router-base-image>] \

  [--push] \

  [--no-cache] \

  [--platform <platform>] \

  [--offline] \

  [--output-dir <output_dir>]

Default values:

• f: policy-manifest.yaml

• image-repository: ghcr.io/wso2/api-platform

• gateway-builder: ghcr.io/wso2/api-platform/gateway-builder:0.2.0

• gateway-controller-base-image: (uses the default defined in the gateway-builder image)

• router-base-image: (uses the default defined in the gateway-builder image)

• platform: (uses the host platform)

• push: false

• no-cache: false

• offline: false

• output-dir: (no output)

When --offline is set:

• Policy Hub is NOT contacted

• Requires an existing policy-manifest-lock.yaml file

• All policies must exist in local cache at: ~/.apipctl/cache/policies/

• Build fails if any policy is missing from cache

• Error: "Policy basic-auth-v1.0.0 not found in cache. Run without --offline first."

Example

Run this command in a dir which contains the policy-manifest.yaml file.

apipctl gateway build \

  --image-tag v0.2.0-policy1

This will output the following Docker images:

• ghcr.io/wso2/api-platform/gateway-controller:v0.2.0-policy1

• ghcr.io/wso2/api-platform/router:v0.2.0-policy1

• ghcr.io/wso2/api-platform/policy-engine:v0.2.0-policy1

Policy.zip file name format

<policy-name>-v<policy-version>.zip

Here policy-name and policy-version are in kebab-case format.

For example, BasicAuth v1.0.0 will be downloaded as basic-auth-v1.0.0.zip.

Unzipping this file will create a directory named basic-auth/v1.0.0/ containing the policy source files.

Policy Cache

Location: ~/.apipctl/cache/policies/

Structure:

~/.apipctl/cache/

└── policies/

  ├── basic-auth-v1.0.0.zip

  ├── jwt-v2.1.0.zip 

  └── rate-limit-v1.5.2.zip

Internals of the apipctl gateway build Command

1. Read the policy-manifest.yaml file.

2. Send the JSON request to the Policy Hub by skipping local policies. Response from the Policy Hub contains the resolved versions and download URLs.

3. If the resolved version is in the local cache, skip the download.

4. Download other policies from the download URLs, checksum validation and cache them locally.

5. Cache local policies defined in the policy-manifest.yaml file.

6. Generate the policy-manifest-lock.yaml file in the same dir where policy-manifest.yaml is located.

7. Provide the policy-manifest-lock.yaml file to the gateway builder to build the gateway images.

8. Create a temp dir .

9. Copy the policy-manifest-lock.yaml to /policy-manifest-lock.yaml

10. Extract the downloaded policies to /policies/ which are defined in policy-manifest-lock.yaml.

11. Create dir /output.

12. Execute the following command to build policy engine binary.

    docker run --rm \
    
        -v <temp-dir>:/workspace \
    
        -v <temp-dir>/output:/workspace/output \
    
        <gateway-builder-image>
    

13. Execute the following command to build gateway Docker images.

    cd <temp-dir>/output/policy-engine
    
    docker build -t <image-repository>/policy-engine:<image-tag> .
    
    cd <temp-dir>/output/gateway-controller
    
    docker build -t <image-repository>/gateway-controller:<image-tag> .
    
    cd <temp-dir>/output/router
    
    docker build -t <image-repository>/router:<image-tag> .
    

14. Clean up the temp dir . On failure, keep the temp dir and log "Build failed. Temp dir: /tmp/apipctl-build-123456"

15. Display the names of the built images.

Policy Manifest file and lock file discussion: #473

@renuka-fernando, I have few concerns on this newly proposed implementation.

1. What is the purpose of the platform flag? I thought APIPCTL is supposed to be independent, so it is not clear for me.

2. Do we really need offline and no-cache options?
   I feel like this adds complexity for users. Instead, we could always check the local directory first. If a policy is not found locally, then try to download it from PolicyHub. If Policy in PolicyHub is unavailable or it’s offline, we can either fail the build or continue with a warning. Since local policies are always preferred, this already behaves like offline mode when all required policies exist locally. With this approach, offline and cache flags may not be necessary.

3. Do we need a default manifest file? Providing a default manifest file at a fixed path may cause users to modify it instead of supplying their own file. Instead, we could provide a sample manifest file for reference and require users to explicitly pass their own manifest file or our sample when running the command. WDYT?

4. What is the purpose of push flag?

[renuka-fernando]

1. --platform: This is for the image platform. If it is confusing, we can name it as --image-platform.

   apipctl gateway build \
   --image-tag v0.2.0- \
   --platform linux/amd64,linux/arm64

2. --no-cache: This is for Docker build cache. If it is confusing, we can name it as --no-docker-cache.
   --offline: This is to indicate that the build should be done in offline mode without contacting the Policy Hub. So this will always use the local cache. Anyway, this command still requiresan internet connection to pull the base images and pull Go modules. It is not fully offline. But still, --offline is a common terminology, so we can keep it.

3. We can provide a default name for the manifest file as policy-manifest.yaml so that the user does not need to provide -f flag always. It is like Dockerfile default name for docker build command. So we can ask users to have directories for every gateway. This way, then can keep the policy-manifest.yaml file along with the lock file policy-manifest-lock.yaml file in the same dir. So my proposed folder structure is as follows:

   gateways/
   ├── ai-gateway/
   │   ├── policy-manifest.yaml
   │   └── policy-manifest-lock.yaml
   ├── ecommerce-gateway/
   │   ├── policy-manifest.yaml
   │   └── policy-manifest-lock.yaml
   

   So this way users are asked not to keep policy-manifest and lock file as ai-gateway-policy-manifest.yaml and ai-gateway-policy-manifest-lock.yaml.

4. --push: This is to indicate whether to push the built images to the image repository. If not set, the images will be only available in the local Docker daemon.

[piyumaldk]

@renuka-fernando,

Regarding 1, Ack. It self explained with the sample command you gave.

Regarding 2, +1 to go with --no-docker-cache. But regarding offline flag, is there any use really. Even without that bool flag, if someone add all the required policies to the .apipctl/policies, it's anyway offline, right?

Regarding 3, Ack. Let's show a message as well to say CLI is assuming file as policy-manifest.yaml when someone did't pass the file. Also, where do we assume the policy-manifest.yaml is, in default scenario?

Reagardin 4, Ack.

[piyumaldk]

@renuka-fernando, Also is it ok to have cache (~/.apipctl/cache/) in the naming even though we are permanently keeping the specific versions locally once we got it from hub to avoid downloading again and again similar to maven?

[piyumaldk]

Since, #516 (reply in thread)
I will keep the proposed flags as it is and if file is not given, try to get policy-manifest.yaml from the command location.

Also for now, I will use ~/.apipctl/cache/policies path. CLI is implemented in a way that its easy to change these paths later again if needed.

[renuka-fernando]

Shall we go with the following command apipctl gateway image build. This way it is clear that we are building images and no need to mention "image" in the flag names.

apipctl gateway image build \
  --tag <image-tag> \
  [-f <policy-manifest-yaml-file>]
  [--repository <image-repository>] \
  [--gateway-builder <gateway-builder-image>] \
  [--gateway-controller-base-image <gateway-controller-base-image>] \
  [--router-base-image <router-base-image>] \
  [--push] \
  [--no-cache] \
  [--platform <platform>] \
  [--offline] \
  [--output-dir <output_dir>]

And there are other commands after the gateway command as well, such as

apipctl gateway api get

So having the image makes this consistent.

apipctl command discussion: #356

BTW, I see some commands are not consistent. This should be discussed in the above GH discussion.

apipctl gateway add # add is a verb here
vs
apipctl gateway api list # api is a resource here

cc: @nuwand, @malinthaprasan, @piyumaldk

[piyumaldk]

@renuka-fernando

I am comfortable with both options. However, I +1 for adding image before build as you suggested, as it makes the command more explicit and informative.

Regarding the perceived inconsistency, this was an intentional decision and aligns with what we agreed on earlier. The command structure follows a clear principle: all nouns appear before the verb, and the noun immediately preceding the verb represents the subject of the command.

For example:

• apipctl gateway add → Adds a gateway in apipctl
• apipctl gateway api delete --id --confirm → Deletes an API from a gateway in apipctl

In my opinion, this approach is cleaner, more consistent, and easier to reason about. During development, I also followed the same hierarchy in the command folder structure, for example:

gateway
  |-- api
  |     |-- add.go
  |     |-- delete.go
  |-- add.go

This keeps the CLI’s internal code structure well organized and closely aligned with the user-facing command hierarchy.

That said, if we are open to revisiting the command naming, I believe it would be more accurate to change apipctl gateway apply to apipctl gateway resource apply, since the command applies a resource (MCP/API) rather than the gateway itself. I did not make this change earlier, as these commands were already discussed and documented. WDYT?

If we plan to revise the commands further, I suggest continuing the discussion in the original thread:#356, which was also shared with the architecture mailing list.

[piyumaldk]

@malinthaprasan, please note. Also, it I were to change apply command to resource apply for applying MCP, and APIs which I believe is better, can you give the +1 to proceed.

[renuka-fernando]

Let's keep the existing apipctl commands as they are. I had a small discussion with @malinthaprasan, and we can go with mixing these verbs and resources. In the help description, we can have two sub-sections for these verbs and sub-resources.

[piyumaldk]

Ack

[renuka-fernando]

When --offline is true

• apipctl is not calling the PolicyHub for version resolution.
• The lock file is required.
• Uses cache for policies.
• For local policies, just copy the policy to the

[piyumaldk]

Do you mean copying to temp? If so, from where? Like NPM, in offline mode, we don't read policy-manifest and read immediately the lock to get resolved finalized policies right?

[renuka-fernando]

Nope, policy-manifest.yaml is read too.

[piyumaldk]

Ack

[renuka-fernando]

Here is the temp dir structure discussed in the discussion above.

❯ tree .
.
├── output
├── policies
│   ├── basic-auth
│   │   ├── v1.0.0
│   │   │   ├── basicauth.go
│   │   │   ├── go.mod
│   │   │   └── policy-definition.yaml
│   │   └── v1.0.1
│   │       ├── basicauth.go
│   │       ├── go.mod
│   │       └── policy-definition.yaml
│   ├── content-length-guardrail
│   │   └── v0.1.0
│   │       ├── contentlengthguardrail.go
│   │       ├── go.mod
│   │       └── policy-definition.yaml
│   ├── json-schema-guardrail
│   │   └── v0.1.0
│   │       ├── go.mod
│   │       ├── jsonschemaguardrail.go
│   │       └── policy-definition.yaml
│   ├── jwt-authentication
│   │   └── v0.1.0
│   │       ├── go.mod
│   │       ├── go.sum
│   │       ├── jwtauth.go
│   │       ├── jwtauth_test.go
│   │       └── policy-definition.yaml
│   ├── mcp-auth
│   │   └── v0.1.0
│   │       ├── go.mod
│   │       ├── go.sum
│   │       ├── mcp-auth.go
│   │       ├── mcp-auth_test.go
│   │       └── policy-definition.yaml
│   ├── modify-headers
│   │   └── v1.0.0
│   │       ├── go.mod
│   │       ├── modifyheaders.go
│   │       └── policy-definition.yaml
│   ├── pii-masking-regex
│   │   └── v0.1.0
│   │       ├── go.mod
│   │       ├── piimaskingregex.go
│   │       └── policy-definition.yaml
│   ├── regex-guardrail
│   │   └── v0.1.0
│   │       ├── go.mod
│   │       ├── policy-definition.yaml
│   │       └── regexguardrail.go
│   ├── respond
│   │   └── v1.0.0
│   │       ├── go.mod
│   │       ├── policy-definition.yaml
│   │       └── respond.go
│   ├── sentence-count-guardrail
│   │   └── v0.1.0
│   │       ├── go.mod
│   │       ├── policy-definition.yaml
│   │       └── sentencecountguardrail.go
│   ├── url-guardrail
│   │   └── v0.1.0
│   │       ├── go.mod
│   │       ├── policy-definition.yaml
│   │       └── urlguardrail.go
│   └── word-count-guardrail
│       └── v0.1.0
│           ├── go.mod
│           ├── policy-definition.yaml
│           └── wordcountguardrail.go
└── policy-manifest-lock.yaml

Sample command

docker run --rm \
    -v $(pwd):/workspace \
    ghcr.io/renuka-fernando/api-platform/gateway-builder:0.1.0-SNAPSHOT

[renuka-fernando]

This is for the following command.

apipctl gateway image build \
  --tag my-tag \
  --gateway-builder ghcr.io/renuka-fernando/api-platform/gateway-builder:0.1.0-SNAPSHOT

[renuka-fernando]

Manifest Dir

gateways/
├── ai-gateway/
│   ├── policy-manifest.yaml
│   └── policy-manifest-lock.yaml
└── ecommerce-gateway/
    └── policy-manifest.yaml

[piyumaldk]

Ack

[piyumaldk]

This gateways/ai-gateway and gateways/commerce-gateway will be the structure of the path which the path is passed (optional) as --path in image build command.

[renuka-fernando]

@piyumaldk and I discussed to do the following.

• Instead of getting manifes file, read the directory that contains the the manifes file and the lock file.
• So these file names are then hard-coded.
• Default to the working directory

[piyumaldk]

In addition to that,

• In online mode, manifest is required in the path, and we we generate/update lock file after resolving policies.
• In offline mode, both lock and manifest is required and we only read manifest file to get the local policies and lock has already resolved policies.

[renuka-fernando]

New Suggestion for the Command

apipctl gateway image build \
  [--name <gateway-name>] \
  [-f <gateway-project-dir>]
  [--repository <image-repository>] \
  [--version <gateway-version>] \
  [--gateway-builder <gateway-builder-image>] \
  [--gateway-controller-base-image <gateway-controller-base-image>] \
  [--router-base-image <router-base-image>] \
  [--push] \
  [--no-cache] \
  [--platform <platform>] \
  [--offline] \
  [--output-dir <output_dir>]

Default values:

• name: current working dir name
• f: current working dir
• version: 0.2.0
• repository: ghcr.io/wso2/api-platform
• gateway-builder: ghcr.io/wso2/api-platform/gateway-builder:$version
• gateway-controller-base-image: (uses the default defined in the gateway-builder image)
• router-base-image: (uses the default defined in the gateway-builder image)
• platform: (uses the host platform)
• push: false
• no-cache: false
• offline: false
• output-dir: (no output)

Sample

gateways/
├── ai-gateway/
│   ├── policy-manifest.yaml
│   └── policy-manifest-lock.yaml
└── ecommerce-gateway/
    └── policy-manifest.yaml

If the following command is executed in gateways/ai-gateway dir the output images are

cd gateways/ai-gateway
apipctl gateway image build

ghcr.io/wso2/api-platform/ai-gateway-gateway-controller:0.2.0
ghcr.io/wso2/api-platform/ai-gateway-router:0.2.0
ghcr.io/wso2/api-platform/ai-gateway-policy-engine:0.2.0

[piyumaldk]

Ack. Lets use --path instead -f for gateway-project-dir

[renuka-fernando]

Internal Commands

docker run --rm \
    -v <temp-dir>:/workspace \
    <gateway-builder-image> \
    -gateway-controller-base-image <gateway-controller-base-image> \
    -router-base-image <router-base-image>

cd <temp-dir>/output/policy-engine
docker build -t [--no-cache] <image-repository>/<gateway-name>-policy-engine:<gateway-version> .
cd <temp-dir>/output/gateway-controller
docker build -t [--no-cache] <image-repository>/<gateway-name>-gateway-controller:<gateway-version> .
cd <temp-dir>/output/router
docker build -t [--no-cache] <image-repository>/<gateway-name>-router:<gateway-version> .

if [--push] is set then
  docker push <image-repository>/<gateway-name>-policy-engine:<gateway-version>
  docker push <image-repository>/<gateway-name>-gateway-controller:<gateway-version>
  docker push <image-repository>/<gateway-name>-router:<gateway-version>

if [--platform] is set then
  # --push should be true
  cd <temp-dir>/output/policy-engine
  docker buildx build --platform <platform> --push -t <image-repository>/<gateway-name>-policy-engine:<gateway-version> .
  cd <temp-dir>/output/gateway-controller
  docker buildx build --platform <platform> --push -t <image-repository>/<gateway-name>-gateway-controller:<gateway-version> .
  cd <temp-dir>/output/router
  docker buildx build --platform <platform> --push -t <image-repository>/<gateway-name>-router:<gateway-version> .

[piyumaldk]

Ack.

[renuka-fernando]

Policy Naming Convention

Policy Definition File

Name in manifest file can be in any format in ASCII.

name: Basic Auth
version: v1.0.0
description: ...
parameters:
  type: object
  properties: {}

Policy Manifest file

Name should be the same one used in the policy definition file.

version: v1
policies:
  - name: Basic Auth
    version: v1.0.0

Directory Structure

When unzipped, the policy source files should be in the following structure.

<policy-name-in-kebab-case>/<policy-version>/

basic-auth/v1.0.0/

basic-auth/
└── v1.0.0/
    ├── basicauth.go
    ├── go.mod
    └── README.md

[renuka-fernando]

I had a discussion with @malinthaprasan. We need a separate discussion for name, displayName (or title), and ID. For now, we can go as follows.

Let's go with the current naming. The policy name is not enforced. We recommend the kebab-case directory structure, but it is not enforced.

@piyumaldk I will provide the lock file, which should be in the temp dir.

[piyumaldk]

I had a discussion with @malinthaprasan. We need a separate discussion for name, displayName (or title), and ID. For now, we can go as follows.

Let's go with the current naming. The policy name is not enforced. We recommend the kebab-case directory structure, but it is not enforced.

@piyumaldk I will provide the lock file, which should be in the temp dir.

Is lock generated, provided when executing the command and workspace one different? Those should be same right. Only use of it is to use it in the workspace.

Also What do you mean by Policy Name is not enforced. Don't we need a fixed structure in workspace/policies?

[piyumaldk]

I had a discussion with @malinthaprasan. We need a separate discussion for name, displayName (or title), and ID. For now, we can go as follows.

Let's go with the current naming. The policy name is not enforced. We recommend the kebab-case directory structure, but it is not enforced.

@piyumaldk I will provide the lock file, which should be in the temp dir.

Is lock generated, provided when executing the command and workspace one different? Those should be same right. Only use of it is to use it in the workspace.

Also What do you mean by Policy Name is not enforced. Don't we need a fixed structure in workspace/policies?

Also, what's the use of filePath in workspace lock? Workspace is anyway a space we generate, hence we should know where to put policies only by it's name and version because once those are extracted from the source (local or hub) we only have its name/version to organize them in the temporary workspace.

cc: @malinthaprasan

[renuka-fernando]

Does this provide answers to your questions? #516 (comment)

[malinthaprasan]

we should know where to put policies only by it's name and version We'll need the path explicitly because this is tricky. We need to generate a path based on the name and version. We don't enforce any URL friendly name in name and it can contain anything. So, its easy if we can derive the path in one place and then pass it to the next steps in the flow.

Eg: When generating the workspace, CLI can go through the policy definition file -> name, then comeup with a folder path for the name/version by sanitizing it. Then enrich the policy manifest lock file with the correct path. Then the builder can continue with it without need to worry about path resolution from name/version.

[renuka-fernando]

Here are the files and format.

Policy Definition File

The name in the manifest file can be in any format in ASCII.

name: Basic Auth
version: v1.0.0
description: ...
parameters:
  type: object
  properties: {}

Policy Manifest file

The name should be the same one used in the policy definition file.

version: v1
policies:
  - name: Basic Auth
    version: v1.0.0

From the apipctl command, when it is copying to the temp dir, it adds a FilePath section in the policy-manifest-lock.yaml file which is in the temp dir. This FilePath is used by the gateway builder to locate the policy source files.

User's Project Dir (which commits to VCS)

gateways/
├── ai-gateway/
│   ├── policy-manifest.yaml
│   └── policy-manifest-lock.yaml # This lock file does not have FilePath entries.
└── ecommerce-gateway/
    ├── policy-manifest.yaml
    └── policy-manifest-lock.yaml # This lock file does not have FilePath entries.

Lock file:

version: v1/alpha1
policies:
  - name: Basic Auth
    version: v1.0.0
    checksum: sha256:abc123...
    source: hub
  - name: Basic Auth
    version: v1.0.1
    checksum: sha256:abc123...
    source: hub
  - name: Basic Auth
    version: v1.0.8
    checksum: sha256:abc123...
    source: hub
  - name: My Custom Policy
    version: v1.0.0
    checksum: sha256:abc123...
    source: local

Temp Dir

The directory names should not always be in the kebab-case format; it is an implementation detail in apipctl command.
The filePath should be included in the lock file.

❯ tree .
.
├── output
├── policies
│   ├── basic-auth
│   │   ├── v1.0.0
│   │   │   ├── basicauth.go
│   │   │   ├── go.mod
│   │   │   └── policy-definition.yaml
│   │   └── v1.0.1
│   │       ├── basicauth.go
│   │       ├── go.mod
│   │       └── policy-definition.yaml
│   ├── content-length-guardrail
│   │   └── v0.1.0
│   │       ├── contentlengthguardrail.go
│   │       ├── go.mod
│   │       └── policy-definition.yaml
└── policy-manifest-lock.yaml # This lock file has FilePath entries.

Lock file:

version: v1/alpha1
policies:
  - name: Basic Auth
    version: v1.0.0
    checksum: sha256:abc123...
    source: hub
    filePath: policies/basic-auth/v1.0.0
  - name: Basic Auth
    version: v1.0.1
    checksum: sha256:abc123...
    source: hub
    filePath: policies/basic-auth/v1.0.1
  - name: Basic Auth
    version: v1.0.8
    checksum: sha256:abc123...
    source: hub
    filePath: policies/basic-auth/v1.0.8
  - name: My Custom Policy
    version: v1.0.0
    checksum: sha256:abc123...
    source: local
    filePath: policies/my-custom-policy/v1.0.0

[malinthaprasan]

This is the flow we discussed cc: @renuka-fernando @piyumaldk

Gateway Build Command Flow

When the CLI gateway build command runs:

Phase 1: Initialization

1. CLI reads policy manifest file (user-provided file containing policies to include for the gateway)
2. CLI sends a request to PolicyHub to download checksum values of the policies in the manifest file (used to compare with local cache)
3. Create policy-manifest-lock.yaml based on the policy resolution from policy hub.

Phase 2: Caching

3. Create the repo cache location if not already done: <cacheRepo>

4. For each policy in the policy manifest file:

   If policy NOT in <cacheRepo>/manifest.yaml:

   1. Download policy from PolicyHub (e.g., name: "Basic Auth", version: "v1.0.0") → content.zip
   2. Validate checksum (from PolicyHub)
   3. content.zip structure:

      .
      ├── basicauth.go
      ├── go.mod
      └── policy-definition.yaml
      

   4. Generate folder name: Sanitize(name)/Sanitize(version)
   5. Create folder inside <cacheRepo> (e.g., <cacheRepo>/basic-auth/v1.0.0)
   6. Copy content.zip into the path (e.g., <cacheRepo>/basic-auth/v1.0.0/content.zip)
   7. Update <cacheRepo>/manifest.yaml with policy name, version mapped to file path:

      policies:
        "Basic Auth":
          "v1.0.0": basic-auth/v1.0.0
          "v1.1.0": basic-auth/v1.1.0
        "Rate Limit":
          "v2.1.0": rate-limit/v2.1.0

   If policy already in cache:

   1. Lookup the folder path inside <cacheRepo>/manifest.yaml
   2. Validate checksum of the policy content.zip

Phase 3: Prepare Workspace

5. Create a workspace folder for the user: <tempDir>
6. Create a folder to copy policies mkdir -p <tempDir>/policies
7. Create a folder to generate outputs from builder mkdir -p <tempDir>/output eg: Dockerfile, refined policy packages etc
8. Copy policy-manifest-lock.yaml into <tempDir>
9. For each policy in the policy manifest file:
   1. Lookup the policy path from <cacheRepo>/manifest.yaml → <policyPath>
   2. mkdir -p <tempDir>/policies/<policyPath>
   3. cp -r <cacheRepo>/<policyPath> <tempDir>/policies/<policyPath>
   4. cd <tempDir>/policies/<policyPath> && unzip content.zip
   5. Enrich policy-manifest-lock.yaml with the relative file path:

      version: v1
      policies:
        - name: basic-auth
          version: v1.0.0
          checksum: sha256:8ff50dbcb61af442df6e7accab83012be9ad6f494aad7ff0f52f3048438e57d2
          source: hub
          filePath: "policies/basic-auth/v1.0.0"

Phase 4: Build

10. Run the gateway build with the modified policy-manifest-lock.yaml

[malinthaprasan]

1. Download policy from PolicyHub (e.g., name: "Basic Auth", version: "v1.0.0") → content.zip
2. Validate checksum (from PolicyHub)
3. content.zip structure:

   .
   ├── basicauth.go
   ├── go.mod
   └── policy-definition.yaml
   

@DakshithaS currently as per @piyumaldk the downloaded policy zip file contains the version folder inside it and inside that we have the files. Can we zip the content without that folder?

[DakshithaS]

In this format, right?
https://policyhubstorage.blob.core.windows.net/policies/api-policies-local1/word-count-guardrail/word-count-guardrail-0.1.0.zip

[piyumaldk]

Yeah. That's better. Is this change done?

[renuka-fernando]

@piyumaldk
I've updated the new Gateway Builder Docker image: ghcr.io/renuka-fernando/api-platform/gateway-builder:0.1.0-SNAPSHOT-1

Here is a sample expected
builder-temp-dir.zip

❯ tree .
.
├── output
├── policies
│   ├── basic-auth
│   │   └── v1.0.0
│   │       ├── basicauth.go
│   │       ├── go.mod
│   │       ├── go.sum
│   │       └── policy-definition.yaml
│   ├── content-length-guardrail
│   │   └── v0.1.0
│   │       ├── contentlengthguardrail.go
│   │       ├── go.mod
│   │       ├── go.sum
│   │       └── policy-definition.yaml
│   └── jwt-authentication
│       └── v0.1.0
│           ├── go.mod
│           ├── go.sum
│           ├── jwtauth.go
│           ├── jwtauth_test.go
│           └── policy-definition.yaml
└── policy-manifest-lock.yaml

You can test it out with the following raw commands.

docker run --rm \
    -v $(pwd):/workspace \
    ghcr.io/renuka-fernando/api-platform/gateway-builder:0.1.0-SNAPSHOT-1

[DakshithaS]

As discussed with @malinthaprasan , we will enforce kebab-case (URL-friendly format) for the policy name. This policy name is the only identifier known to the Policy Engine.

From a UI perspective, we will maintain a separate policy display name. This display name is not part of the policy definition, and the Policy Engine is not aware of it, it operates solely based on the policy name.

The policy display name will be stored in the metadata.json file. When the Management Portal retrieves policy details from the Policy Hub, it can also obtain the corresponding policy display name.

For custom policies, the policy display name and other UI-related metadata (which the Policy Engine does not need to know) will be stored in the Management Portal policy database.

@renuka-fernando @piyumaldk @thivindu

[DakshithaS]

In the Policy Hub repository, policies are stored using the following directory structure:

/policies
├── rate-limiter
│   └── v1.0.0
│       ├── src
│       ├── docs
│       ├── policy-definition.yaml
│       └── metadata.json
├── jwt-validator
│   └── v2.1.0
│       ├── src
│       ├── docs
│       ├── policy-definition.yaml
│       └── metadata.json

[renuka-fernando]

Ok, let's go with this one.

[renuka-fernando]

APIPCTL Command

apipctl gateway image build \
  [--project-dir <project-dir>]
  [--name-prefix <image-name-prefix>] \
  [--repository <image-repository>] \
  [--tag <image-tag>] \
  [--gateway-builder <gateway-builder-image>] \
  [--gateway-controller-base-image <gateway-controller-base-image>] \
  [--router-base-image <router-base-image>] \
  [--push] \
  [--no-cache] \
  [--platform <platform>] \
  [--offline] \
  [--output-dir <output_dir>]

Step 1

Sample

docker run --rm \
    -v $(pwd):/workspace \
    ghcr.io/renuka-fernando/api-platform/gateway-builder:0.1.0-SNAPSHOT-2

Format

docker run --rm \
    -v <temp-dir>:/workspace \
    <gateway-builder-image>

Step 2

Sample

TEMP_DIR=`pwd`
cd ${TEMP_DIR}/output/policy-engine
docker build -t foo/policy-engine:v111 .
cd ${TEMP_DIR}/output/gateway-controller
docker build -t foo/gateway-controller:v111 .
cd ${TEMP_DIR}/output/router
docker build -t foo/router:v111 .

Format

cd <temp-dir>/output/policy-engine
docker build -t <image-repository>/<image-name-prefix>-policy-engine:<image-tag> .
cd <temp-dir>/output/gateway-controller
docker build -t <image-repository>/<image-name-prefix>-gateway-controller:<image-tag> .
cd <temp-dir>/output/router
docker build -t <image-repository>/<image-name-prefix>-router:<image-tag> .

[piyumaldk]

I just renamed flag names. Gateway name to name prefix and version to tag. No script changes.

@renuka-fernando If that's the case, and there is no functional requirements for the change, can we keep this as it is till we get this working. Later, I can change flag names if needed.

[renuka-fernando]

Please build the gateway first. Go to api-platform/gateway dir

Then
Make build

This will build this missing images

[piyumaldk]

0.175 ERROR: [Discovery] policy name mismatch: manifest declares 'JwtAuthentication' but policy-definition.yaml has 'JwtAuth' at /workspace/policies/jwt-auth/v0.1.0

Getting this in latest fetch.

[renuka-fernando]

@DakshithaS can you please correct the policy JWT Auth from Policy Hub side. please check the name in Policy Definition and Name in the RestAPI request.

Until then @piyumaldk please proceed without jwt auth policy by removing it from manifest file.

[piyumaldk]

Ack