[BUG] imds.autoConfigureHopLimit not limited to VPC

[StephenKing]

Bug Description

even though the last resort is advertised to fall back to VPC-wide, it is actually trying to modify instances in the whole region, as there is no filter for the VPC:

aws-multi-eni-controller/pkg/aws/ec2.go

Lines 1141 to 1149 in ebc2c8e

	// Get all running instances in the region
	input := &ec2.DescribeInstancesInput{
	Filters: []types.Filter{
	{
	Name: aws.String("instance-state-name"),
	Values: []string{"running"},
	},
	},
	}

Additionally, the documentation lacks the permission ec2:ModifyInstanceMetadataOptions

Steps to Reproduce

1. Install it
2. See it attempting to reconfigure hop limit for whole region

Expected Behavior

To only modify resources that are part of the clusters

Actual Behavior

Luckily it fails due to missing IAM permissions

Screenshots

If applicable, add screenshots to help explain your problem.

Environment

• Kubernetes version: 1.33
• Controller version: 1.3.8
• Installation method: Helm

Logs

2025-09-02T10:01:03Z    DEBUG    controllers.NodeENI.aws-ec2-client    Looking up instance by private IP    {"privateIP": "10.0.19.11"}        
2025-09-02T10:01:03Z    DEBUG    controllers.NodeENI.aws-ec2-client    Found private IP from network interface    {"privateIP": "10.0.19.11", "interface": "eth0"}
2025-09-02T10:01:03Z    DEBUG    controllers.NodeENI.aws-ec2-client    Found private IP from network interface    {"privateIP": "10.0.19.11", "interface": "eth0"}
2025-09-02T10:01:03Z    DEBUG    controllers.NodeENI.aws-ec2-client    Looking up instance by private IP    {"privateIP": "10.0.19.11"
2025-09-02T10:01:03Z    INFO    controllers.NodeENI.aws-ec2-client    Attempting VPC-wide IMDS configuration as last resort
2025-09-02T10:01:04Z    INFO    controllers.NodeENI.aws-ec2-client    Configuring IMDS for instance    {"instanceID": "i-08066b3a291eba64d", "hopLimit": 2}
2025-09-02T10:01:04Z    INFO    controllers.NodeENI.aws-ec2-client    Updating IMDS hop limit for container compatibility    {"instanceID": "i-08066b3a291eba64d", "currentHopLimit": 1, "newHopLimit": 2}
2025-09-02T10:01:04Z    DEBUG    controllers.NodeENI.aws-ec2-client    Failed to configure IMDS for instance    {"instanceID": "i-08066b3a291eba64d", "error": "failed to modify IMDS hop limit: failed to modify instance metadata options: operation error EC2: ModifyInstanceMetadataOptions, https response error StatusCode: 403, RequestID: e665f979-fcce-4949-858d-23eac6815171, api error UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::XX:assumed-roleaws-multi-eni/1756807256366945716 is not authorized to perform: ec2:ModifyInstanceMetadataOptions on resource: arn:aws:ec2:eu-west-1:XXX:instance/i-08066b3a291eba64d because no identity-based policy allows the ec2:ModifyInstanceMetadataOptions action. Encoded authorization failure message:  ...."}
2025-09-02T10:01:04Z    INFO    controllers.NodeENI.aws-ec2-client    Configuring IMDS for instance    {"instanceID": "i-003daddd3e453fb72", "hopLimit": 2}

[traycerai]

Plan

Observations

The bug is in file:pkg/aws/ec2.go at the tryVPCWideConfiguration function (lines 1128-1203). When aggressive IMDS configuration is enabled, the function attempts to configure IMDS hop limits for all running instances in the entire AWS region without filtering by VPC. The DescribeInstancesInput at lines 1143-1149 only includes an instance-state-name filter but lacks a vpc-id filter. Additionally, the main README.md's IAM permissions section (lines 82-106) is missing the ec2:ModifyInstanceMetadataOptions permission, though it's documented in other files like file:docs/IRSA_SETUP.md and file:docs/imdsv2-support.md.

Approach

The fix requires two changes: (1) adding VPC filtering to limit IMDS configuration to instances within the same VPC as the controller, and (2) updating documentation to include the missing IAM permission. The VPC ID can be obtained by first looking up the current instance (which is already done in earlier fallback strategies) and extracting its VPC ID from the DescribeInstances response. This VPC ID should then be added as a filter in the tryVPCWideConfiguration function to ensure only instances in the same VPC are affected.

Implementation Steps

1. Add VPC ID Retrieval Method

File: file:pkg/aws/ec2.go

Add a new helper method to retrieve the VPC ID of the current instance. This should be placed near the other instance lookup methods (around line 1370):

• Create a method getVPCIDFromInstance(ctx context.Context, instanceID string) (string, error)
• Use DescribeInstances API to get the instance details
• Extract and return the VpcId field from the instance object
• Include proper error handling for cases where VPC ID is not available

2. Modify tryVPCWideConfiguration Function

File: file:pkg/aws/ec2.go (lines 1128-1203)

Update the tryVPCWideConfiguration function to filter by VPC:

• Before calling DescribeInstances (around line 1140), first determine the current instance's VPC ID
• Use the existing getPrivateIPFromNetworkInterface() and findInstanceByPrivateIP() methods to get the current instance ID
• Call the new getVPCIDFromInstance() method to retrieve the VPC ID
• Add a VPC ID filter to the DescribeInstancesInput at lines 1143-1149:

  {
    Name:   aws.String("vpc-id"),
    Values: []string{vpcID},
  }
  

• Add appropriate logging to indicate VPC-scoped configuration
• Handle errors gracefully if VPC ID cannot be determined (log warning and skip VPC-wide configuration)

3. Update Main README IAM Permissions

File: file:README.md (lines 82-106)

Add the missing ec2:ModifyInstanceMetadataOptions permission to the IAM policy example:

• Insert "ec2:ModifyInstanceMetadataOptions" in the Actions array after "ec2:ModifyNetworkInterfaceAttribute" (around line 100)
• Ensure proper JSON formatting with comma separation
• This aligns with the permissions already documented in file:docs/IRSA_SETUP.md (line 56) and file:docs/imdsv2-support.md (line 104)

4. Update Helm Chart Documentation

File: file:charts/aws-multi-eni-controller/README.md (if exists)

If there's a separate README in the Helm chart directory, ensure it also documents the ec2:ModifyInstanceMetadataOptions permission. If not, this step can be skipped as the main README is the primary documentation source.

5. Add Descriptive Logging

File: file:pkg/aws/ec2.go

Enhance logging in the tryVPCWideConfiguration function:

• Update the log message at line 1140 to include VPC ID: "Attempting VPC-wide IMDS configuration as last resort", "vpcID", vpcID
• Add a log message after successfully retrieving VPC ID: "Limiting IMDS configuration to VPC", "vpcID", vpcID
• Update the completion log at line 1201 to mention VPC scope: "VPC-wide IMDS configuration completed", "vpcID", vpcID, "configuredCount", configuredCount

6. Consider Environment Variable for VPC Override

File: file:pkg/aws/ec2.go

Optionally add support for manually specifying VPC ID via environment variable:

• Check for VPC_ID environment variable at the start of tryVPCWideConfiguration
• If set, use it directly instead of auto-detection
• This provides an escape hatch for edge cases where auto-detection might fail
• Document this in file:docs/imdsv2-support.md in the Configuration Options table (around line 74)

Sequence Diagram

sequenceDiagram
    participant Controller as ENI Controller
    participant EC2 as AWS EC2 API
    participant Instance as Current Instance
    
    Note over Controller: IMDS Configuration Triggered
    Controller->>Controller: tryVPCWideConfiguration()
    Controller->>Controller: Check IMDS_AGGRESSIVE_CONFIGURATION
    
    alt Aggressive Config Enabled
        Controller->>Controller: getPrivateIPFromNetworkInterface()
        Controller->>EC2: DescribeInstances(private-ip-address filter)
        EC2-->>Controller: Current Instance Details
        Controller->>Controller: Extract Instance ID
        Controller->>EC2: DescribeInstances(instanceID)
        EC2-->>Controller: Instance with VPC ID
        Controller->>Controller: Extract VPC ID
        
        Controller->>EC2: DescribeInstances(running + vpc-id filters)
        EC2-->>Controller: Instances in same VPC
        
        loop For each instance in VPC
            Controller->>EC2: getCurrentIMDSHopLimit(instanceID)
            EC2-->>Controller: Current hop limit
            
            alt Hop limit needs update
                Controller->>EC2: ModifyInstanceMetadataOptions(instanceID, hopLimit=2)
                EC2-->>Controller: Success/Failure
            end
        end
        
        Controller->>Controller: Log completion with VPC scope
    else Aggressive Config Disabled
        Controller->>Controller: Skip VPC-wide configuration
    end

Loading

Testing Considerations

While implementation is the focus, the following areas should be validated:

• Verify VPC ID is correctly extracted from instance metadata
• Confirm only instances in the same VPC are affected by IMDS configuration
• Test behavior when VPC ID cannot be determined (graceful degradation)
• Validate that the IAM permission is correctly documented
• Ensure logging clearly indicates VPC-scoped operations

Import In IDE

🤖 Prompt for AI Agents

## Observations

The bug is in `file:pkg/aws/ec2.go` at the `tryVPCWideConfiguration` function (lines 1128-1203). When aggressive IMDS configuration is enabled, the function attempts to configure IMDS hop limits for all running instances in the entire AWS region without filtering by VPC. The `DescribeInstancesInput` at lines 1143-1149 only includes an `instance-state-name` filter but lacks a `vpc-id` filter. Additionally, the main README.md's IAM permissions section (lines 82-106) is missing the `ec2:ModifyInstanceMetadataOptions` permission, though it's documented in other files like `file:docs/IRSA_SETUP.md` and `file:docs/imdsv2-support.md`.

## Approach

The fix requires two changes: (1) adding VPC filtering to limit IMDS configuration to instances within the same VPC as the controller, and (2) updating documentation to include the missing IAM permission. The VPC ID can be obtained by first looking up the current instance (which is already done in earlier fallback strategies) and extracting its VPC ID from the `DescribeInstances` response. This VPC ID should then be added as a filter in the `tryVPCWideConfiguration` function to ensure only instances in the same VPC are affected.

## Implementation Steps

### 1. Add VPC ID Retrieval Method

**File**: `file:pkg/aws/ec2.go`

Add a new helper method to retrieve the VPC ID of the current instance. This should be placed near the other instance lookup methods (around line 1370):

- Create a method `getVPCIDFromInstance(ctx context.Context, instanceID string) (string, error)`
- Use `DescribeInstances` API to get the instance details
- Extract and return the `VpcId` field from the instance object
- Include proper error handling for cases where VPC ID is not available

### 2. Modify tryVPCWideConfiguration Function

**File**: `file:pkg/aws/ec2.go` (lines 1128-1203)

Update the `tryVPCWideConfiguration` function to filter by VPC:

- Before calling `DescribeInstances` (around line 1140), first determine the current instance's VPC ID
- Use the existing `getPrivateIPFromNetworkInterface()` and `findInstanceByPrivateIP()` methods to get the current instance ID
- Call the new `getVPCIDFromInstance()` method to retrieve the VPC ID
- Add a VPC ID filter to the `DescribeInstancesInput` at lines 1143-1149:
  ```
  {
    Name:   aws.String("vpc-id"),
    Values: []string{vpcID},
  }
  ```
- Add appropriate logging to indicate VPC-scoped configuration
- Handle errors gracefully if VPC ID cannot be determined (log warning and skip VPC-wide configuration)

### 3. Update Main README IAM Permissions

**File**: `file:README.md` (lines 82-106)

Add the missing `ec2:ModifyInstanceMetadataOptions` permission to the IAM policy example:

- Insert `"ec2:ModifyInstanceMetadataOptions"` in the Actions array after `"ec2:ModifyNetworkInterfaceAttribute"` (around line 100)
- Ensure proper JSON formatting with comma separation
- This aligns with the permissions already documented in `file:docs/IRSA_SETUP.md` (line 56) and `file:docs/imdsv2-support.md` (line 104)

### 4. Update Helm Chart Documentation

**File**: `file:charts/aws-multi-eni-controller/README.md` (if exists)

If there's a separate README in the Helm chart directory, ensure it also documents the `ec2:ModifyInstanceMetadataOptions` permission. If not, this step can be skipped as the main README is the primary documentation source.

### 5. Add Descriptive Logging

**File**: `file:pkg/aws/ec2.go`

Enhance logging in the `tryVPCWideConfiguration` function:

- Update the log message at line 1140 to include VPC ID: `"Attempting VPC-wide IMDS configuration as last resort", "vpcID", vpcID`
- Add a log message after successfully retrieving VPC ID: `"Limiting IMDS configuration to VPC", "vpcID", vpcID`
- Update the completion log at line 1201 to mention VPC scope: `"VPC-wide IMDS configuration completed", "vpcID", vpcID, "configuredCount", configuredCount`

### 6. Consider Environment Variable for VPC Override

**File**: `file:pkg/aws/ec2.go`

Optionally add support for manually specifying VPC ID via environment variable:

- Check for `VPC_ID` environment variable at the start of `tryVPCWideConfiguration`
- If set, use it directly instead of auto-detection
- This provides an escape hatch for edge cases where auto-detection might fail
- Document this in `file:docs/imdsv2-support.md` in the Configuration Options table (around line 74)

## Sequence Diagram

```mermaid
sequenceDiagram
    participant Controller as ENI Controller
    participant EC2 as AWS EC2 API
    participant Instance as Current Instance
    
    Note over Controller: IMDS Configuration Triggered
    Controller->>Controller: tryVPCWideConfiguration()
    Controller->>Controller: Check IMDS_AGGRESSIVE_CONFIGURATION
    
    alt Aggressive Config Enabled
        Controller->>Controller: getPrivateIPFromNetworkInterface()
        Controller->>EC2: DescribeInstances(private-ip-address filter)
        EC2-->>Controller: Current Instance Details
        Controller->>Controller: Extract Instance ID
        Controller->>EC2: DescribeInstances(instanceID)
        EC2-->>Controller: Instance with VPC ID
        Controller->>Controller: Extract VPC ID
        
        Controller->>EC2: DescribeInstances(running + vpc-id filters)
        EC2-->>Controller: Instances in same VPC
        
        loop For each instance in VPC
            Controller->>EC2: getCurrentIMDSHopLimit(instanceID)
            EC2-->>Controller: Current hop limit
            
            alt Hop limit needs update
                Controller->>EC2: ModifyInstanceMetadataOptions(instanceID, hopLimit=2)
                EC2-->>Controller: Success/Failure
            end
        end
        
        Controller->>Controller: Log completion with VPC scope
    else Aggressive Config Disabled
        Controller->>Controller: Skip VPC-wide configuration
    end
```

## Testing Considerations

While implementation is the focus, the following areas should be validated:

- Verify VPC ID is correctly extracted from instance metadata
- Confirm only instances in the same VPC are affected by IMDS configuration
- Test behavior when VPC ID cannot be determined (graceful degradation)
- Validate that the IAM permission is correctly documented
- Ensure logging clearly indicates VPC-scoped operations

---

Execution Information

Branch: main
Commit: 2fda1d1

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.