Skip to content

Commit 37863dd

Browse files
norman-zonnorman-zon
committed
feat(terraform): add OCI protocol support for modules and providers
Adds support for Terraform modules and providers hosted in OCI-compatible container registries using the `oci://` protocol. Delegates OCI sources to Docker datasource for version resolution. Supports both version in separate field and embedded in URL. Closes #38927
1 parent 78ca9ea commit 37863dd

File tree

6 files changed

+472
-14
lines changed

6 files changed

+472
-14
lines changed

‎docs/development/oci-terraform-support.md‎

Lines changed: 248 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,248 @@
1+
# OCI Protocol Support for Terraform Modules and Providers
2+
3+
## Overview
4+
5+
This implementation adds support for the OCI (Open Container Initiative) protocol for Terraform modules and providers. Terraform supports pulling modules and providers from OCI-compatible registries using the `oci://` protocol prefix.
6+
7+
## Changes Made
8+
9+
### 1. Module Extraction (`lib/modules/manager/terraform/extractors/others/modules.ts`)
10+
11+
**Added:**
12+
13+
- `ociRefMatchRegex`: New regex pattern to detect OCI protocol sources
14+
- Pattern: `/^oci:\/\/(?<registry>[^/:]+)\/(?<repository>[^:]+?)(?::(?<tag>.+))?$/`
15+
- Captures the registry hostname, repository path, and optional tag/version
16+
- Supports both formats:
17+
- `oci://registry/repo` (version in separate field)
18+
- `oci://registry/repo:tag` (version in URL)
19+
20+
**Modified:**
21+
22+
- Imported `DockerDatasource` for handling OCI registry requests
23+
- Updated `analyseTerraformModule()` method to detect and handle OCI sources
24+
- OCI sources are now configured to use the Docker datasource
25+
- Registry URLs are extracted and set appropriately
26+
- Package names follow the format: `registry/repository`
27+
- **Version handling**: If a tag is present in the source URL, it takes precedence over the separate `version` field
28+
29+
**Example Detection:**
30+
31+
```hcl
32+
# Version in separate field
33+
module "vpc" {
34+
source = "oci://registry.example.com/terraform-modules/vpc"
35+
version = "1.2.3"
36+
}
37+
38+
# Version embedded in source URL
39+
module "storage" {
40+
source = "oci://docker.io/terraform-modules/storage:3.1.0"
41+
}
42+
43+
# Digest in source URL
44+
module "database" {
45+
source = "oci://registry.example.com/modules/db:sha256:abc123"
46+
}
47+
```
48+
49+
### 2. Provider Extraction (`lib/modules/manager/terraform/base.ts`)
50+
51+
**Added:**
52+
53+
- `ociRefMatchRegex` property to `TerraformProviderExtractor` class
54+
55+
**Modified:**
56+
57+
- Imported `DockerDatasource` for handling OCI provider sources
58+
- Updated `analyzeTerraformProvider()` method to check for OCI protocol before standard parsing
59+
- OCI providers delegate to Docker datasource
60+
- Registry URLs extracted from the OCI source URL
61+
- Maintains compatibility with existing provider source formats
62+
63+
**Example Detection:**
64+
65+
```hcl
66+
terraform {
67+
required_providers {
68+
custom = {
69+
source = "oci://registry.example.com/providers/custom"
70+
version = "1.0.0"
71+
}
72+
}
73+
}
74+
```
75+
76+
### 3. Test Coverage
77+
78+
**Test Fixture (`lib/modules/manager/terraform/__fixtures__/oci.tf`):**
79+
80+
- Created comprehensive test file demonstrating OCI usage
81+
- Includes both module and provider examples
82+
- Uses various OCI registries (ghcr.io, docker.io, custom registries)
83+
84+
**Unit Tests (`lib/modules/manager/terraform/extractors/others/modules.spec.ts`):**
85+
86+
- Added `ociRefMatchRegex` test suite
87+
- Tests various OCI URL formats:
88+
- Simple registry: `oci://registry.example.com/namespace/module`
89+
- GitHub Container Registry: `oci://ghcr.io/org/repo/module`
90+
- Docker Hub: `oci://docker.io/user/module`
91+
92+
**Integration Tests (`lib/modules/manager/terraform/extract.spec.ts`):**
93+
94+
- Added comprehensive test case for OCI module and provider extraction
95+
- Validates correct datasource assignment (docker vs terraform-module/provider)
96+
- Verifies registry URL extraction
97+
- Tests mixed configuration (OCI + traditional sources)
98+
99+
## How It Works
100+
101+
### OCI Module Sources
102+
103+
1. **Detection**: The module extractor checks if the source starts with `oci://`
104+
2. **Parsing**: Extracts registry hostname, repository path, and optional tag/version
105+
3. **Datasource Assignment**: Uses Docker datasource instead of Terraform Module datasource
106+
4. **Registry Configuration**: Sets `registryUrls` to the HTTPS version of the OCI registry
107+
5. **Version Handling**:
108+
- If tag is in the source URL (`oci://registry/repo:1.2.3`), it becomes `currentValue`
109+
- If tag is NOT in the URL, the separate `version` field is used
110+
- Supports both semver tags and digest references
111+
112+
### OCI Provider Sources
113+
114+
1. **Detection**: Provider extractor checks for `oci://` prefix before standard source parsing
115+
2. **Parsing**: Same pattern as modules - extracts registry, repository, and optional tag
116+
3. **Datasource Assignment**: Uses Docker datasource for version lookups
117+
4. **Version Handling**: Same as modules - tag in URL takes precedence over `version` field
118+
119+
## Datasource Delegation
120+
121+
When OCI protocol is detected:
122+
123+
- **Datasource**: `docker` (instead of `terraform-module` or `terraform-provider`)
124+
- **Registry URLs**: Extracted from the OCI URL and converted to HTTPS
125+
- **Package Name**: Full path including registry hostname
126+
- **Version Resolution**: Handled by Docker datasource using OCI registry APIs
127+
128+
## Benefits
129+
130+
1. **Registry Flexibility**: Users can host Terraform modules/providers in any OCI-compatible registry
131+
2. **Unified Tooling**: Leverage existing container registry infrastructure
132+
3. **Access Control**: Use existing registry authentication and authorization
133+
4. **Reuse Docker Datasource**: No need to implement OCI-specific logic; Docker datasource handles it
134+
135+
## Supported Registries
136+
137+
Any OCI-compatible container registry, including:
138+
139+
- GitHub Container Registry (ghcr.io)
140+
- Docker Hub (docker.io)
141+
- Google Container Registry (gcr.io)
142+
- Amazon Elastic Container Registry (ECR)
143+
- Azure Container Registry (azurecr.io)
144+
- Private/self-hosted registries
145+
- Harbor, Quay, Artifactory, etc.
146+
147+
## Example Configurations
148+
149+
### Module from GitHub Container Registry
150+
151+
```hcl
152+
module "networking" {
153+
source = "oci://ghcr.io/myorg/terraform-modules/networking"
154+
version = "2.0.0"
155+
}
156+
```
157+
158+
### Provider from Private Registry
159+
160+
```hcl
161+
terraform {
162+
required_providers {
163+
internal = {
164+
source = "oci://registry.company.com/providers/internal"
165+
version = "1.5.0"
166+
}
167+
}
168+
}
169+
```
170+
171+
### Mixed Configuration
172+
173+
```hcl
174+
# OCI module
175+
module "vpc" {
176+
source = "oci://registry.example.com/modules/vpc"
177+
version = "1.0.0"
178+
}
179+
180+
# Traditional module
181+
module "consul" {
182+
source = "hashicorp/consul/aws"
183+
version = "0.1.0"
184+
}
185+
186+
terraform {
187+
required_providers {
188+
# OCI provider
189+
custom = {
190+
source = "oci://ghcr.io/company/providers/custom"
191+
version = "2.0.0"
192+
}
193+
194+
# Traditional provider
195+
aws = {
196+
source = "hashicorp/aws"
197+
version = "5.0.0"
198+
}
199+
}
200+
}
201+
```
202+
203+
## Implementation Notes
204+
205+
### Regex Pattern Design
206+
207+
The OCI regex pattern is intentionally simple:
208+
209+
- Matches `oci://` prefix exactly
210+
- Captures registry as the first path segment (hostname)
211+
- Captures everything after as repository path
212+
- Doesn't validate specific URL components (delegated to Docker datasource)
213+
214+
### Datasource Choice
215+
216+
Using Docker datasource for OCI sources:
217+
218+
- OCI is the standard for container registries
219+
- Docker datasource already implements OCI registry protocol
220+
- Reduces code duplication
221+
- Terraform OCI modules/providers use the same registry APIs as container images
222+
223+
### Compatibility
224+
225+
- Fully backward compatible with existing Terraform configurations
226+
- OCI support is additive - no changes to existing functionality
227+
- Falls back to standard parsing if OCI pattern doesn't match
228+
229+
## Testing
230+
231+
Run the tests with:
232+
233+
```bash
234+
npm test -- terraform
235+
```
236+
237+
Specific test files:
238+
239+
```bash
240+
npm test -- lib/modules/manager/terraform/extractors/others/modules.spec.ts
241+
npm test -- lib/modules/manager/terraform/extract.spec.ts
242+
```
243+
244+
## References
245+
246+
- [Terraform Module Sources](https://developer.hashicorp.com/terraform/language/modules/sources)
247+
- [OCI Registry Specification](https://github.com/opencontainers/distribution-spec)
248+
- [Terraform Registry Protocol](https://developer.hashicorp.com/terraform/internals/module-registry-protocol)

‎lib/modules/manager/terraform/__fixtures__/oci.tf‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
module "vpc_oci" {
2+
source = "oci://registry.example.com/terraform-modules/vpc"
3+
version = "1.2.3"
4+
}
5+
6+
module "networking_oci" {
7+
source = "oci://ghcr.io/myorg/terraform-modules/networking"
8+
version = "2.0.0"
9+
}
10+
11+
module "storage_oci_tagged" {
12+
source = "oci://docker.io/terraform-modules/storage:3.1.0"
13+
}
14+
15+
module "database_oci_digest" {
16+
source = "oci://registry.example.com/terraform-modules/database:sha256:abc123"
17+
}
18+
19+
module "traditional" {
20+
source = "hashicorp/consul/aws"
21+
version = "0.1.0"
22+
}
23+
24+
terraform {
25+
required_providers {
26+
custom_oci = {
27+
source = "oci://registry.example.com/providers/custom"
28+
version = "1.0.0"
29+
}
30+
31+
another_oci = {
32+
source = "oci://ghcr.io/mycompany/providers/mycloud"
33+
version = "2.5.0"
34+
}
35+
36+
tagged_oci = {
37+
source = "oci://registry.example.com/providers/tagged:4.2.0"
38+
}
39+
40+
aws = {
41+
source = "hashicorp/aws"
42+
version = "4.0.0"
43+
}
44+
}
45+
}

‎lib/modules/manager/terraform/base.ts‎

Lines changed: 28 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import { isNonEmptyString } from '@sindresorhus/is';
22
import { regEx } from '../../../util/regex.ts';
3+
import { DockerDatasource } from '../../datasource/docker/index.ts';
34
import { TerraformProviderDatasource } from '../../datasource/terraform-provider/index.ts';
45
import type { ExtractConfig, PackageDependency } from '../types.ts';
56
import type { TerraformDefinitionFile } from './hcl/types.ts';
@@ -29,6 +30,9 @@ export abstract class TerraformProviderExtractor extends DependencyExtractor {
2930
sourceExtractionRegex = regEx(
3031
/^(?:(?<hostname>(?:[a-zA-Z0-9-_]+\.+)+[a-zA-Z0-9-_]+)\/)?(?:(?<namespace>[^/]+)\/)?(?<type>[^/]+)/,
3132
);
33+
ociRefMatchRegex = regEx(
34+
/^oci:\/\/(?<registry>[^/:]+)\/(?<repository>[^:]+?)(?::(?<tag>.+))?$/,
35+
);
3236

3337
protected analyzeTerraformProvider(
3438
dep: PackageDependency,
@@ -41,20 +45,31 @@ export abstract class TerraformProviderExtractor extends DependencyExtractor {
4145

4246
if (isNonEmptyString(dep.managerData?.source)) {
4347
// TODO #22198
44-
const source = this.sourceExtractionRegex.exec(dep.managerData.source);
45-
if (!source?.groups) {
46-
dep.skipReason = 'unsupported-url';
47-
return dep;
48-
}
49-
50-
// buildin providers https://github.com/terraform-providers
51-
if (source.groups.namespace === 'terraform-providers') {
52-
dep.registryUrls = [`https://releases.hashicorp.com`];
53-
} else if (source.groups.hostname) {
54-
dep.registryUrls = [`https://${source.groups.hostname}`];
55-
dep.packageName = `${source.groups.namespace}/${source.groups.type}`;
48+
const ociMatch = this.ociRefMatchRegex.exec(dep.managerData.source);
49+
if (ociMatch?.groups) {
50+
const { registry, repository, tag } = ociMatch.groups;
51+
dep.packageName = `${registry}/${repository}`;
52+
dep.registryUrls = [`https://${registry}`];
53+
dep.datasource = DockerDatasource.id;
54+
if (tag) {
55+
dep.currentValue = tag;
56+
}
5657
} else {
57-
dep.packageName = dep.managerData?.source;
58+
const source = this.sourceExtractionRegex.exec(dep.managerData.source);
59+
if (!source?.groups) {
60+
dep.skipReason = 'unsupported-url';
61+
return dep;
62+
}
63+
64+
// buildin providers https://github.com/terraform-providers
65+
if (source.groups.namespace === 'terraform-providers') {
66+
dep.registryUrls = [`https://releases.hashicorp.com`];
67+
} else if (source.groups.hostname) {
68+
dep.registryUrls = [`https://${source.groups.hostname}`];
69+
dep.packageName = `${source.groups.namespace}/${source.groups.type}`;
70+
} else {
71+
dep.packageName = dep.managerData?.source;
72+
}
5873
}
5974
}
6075
massageProviderLookupName(dep);

0 commit comments

Comments
 (0)