howto-set-quilt-log-level-ecs.md

How do I debug back-end login or API call issues with a Quilt stack?

Tags

aws, ecs, logging, registry, login, api, cloudformation, troubleshooting

Summary

When login, OAuth, or API calls fail on a deployed Quilt stack, support may need more detailed back-end registry logs. Increase the registry container's QUILT_LOG_LEVEL by creating a new ECS task definition revision, then update the registry ECS service to run that revision. This is best treated as a temporary debugging change unless the stack template is also updated.

---

Symptoms

• Users cannot log in, OAuth setup fails, or back-end API calls return unexpected errors.
• Support needs more detailed registry logs from a deployed Quilt stack.
• Registry logs are present, but DEBUG messages are missing.
• The deployed ECS task definition shows QUILT_LOG_LEVEL=INFO or another non-debug level.

Root Cause

The registry application handles login, OAuth, and many back-end API requests. It reads QUILT_LOG_LEVEL from the container environment at process startup.

In current Quilt deployments, that environment variable is baked into the ECS task definition generated by the deployment template. It is not a user-editable CloudFormation parameter, and it cannot be changed directly on an already running ECS task.

Recommended Workaround

Create a new task definition revision for the registry service and update the service to run that revision.

AWS Console

1. Open the AWS ECS console in the stack's region.

2. Select the cluster for the Quilt stack.

3. Open the registry service, usually named <stack-name>-registry.

4. Note the active task definition, usually named <stack-name>-registry:<revision>.

5. Create a new revision of that task definition.

6. Edit the container named registry.

7. Add or update the environment variable:

   QUILT_LOG_LEVEL=DEBUG
   

8. Save the new task definition revision.

9. Update the registry ECS service to use the new revision.

10. Force a new deployment so the registry tasks restart with the new environment.

AWS CLI

Find the registry service:

aws ecs list-services \
  --cluster <stack-name> \
  --region <region>

Describe the service to find the active task definition:

aws ecs describe-services \
  --cluster <stack-name> \
  --services <stack-name>-registry \
  --region <region> \
  --query 'services[0].taskDefinition' \
  --output text

Describe the task definition and inspect the registry container environment:

aws ecs describe-task-definition \
  --task-definition <task-definition-arn> \
  --region <region> \
  --query "taskDefinition.containerDefinitions[?name=='registry'].environment"

Register a new task definition revision with the same settings except for the registry container environment.

Do not pass the raw describe-task-definition output directly to register-task-definition. It includes read-only fields that ECS rejects, such as task definition ARN, revision, status, compatibilities, and registration metadata.

The following example uses jq to strip those fields and set QUILT_LOG_LEVEL=DEBUG on the registry container:

aws ecs describe-task-definition \
  --task-definition <task-definition-arn> \
  --region <region> \
  --query taskDefinition \
  > task-definition.json

jq '
  del(
    .taskDefinitionArn,
    .revision,
    .status,
    .requiresAttributes,
    .compatibilities,
    .registeredAt,
    .registeredBy
  )
  | .containerDefinitions |= map(
      if .name == "registry" then
        .environment =
          (
            (.environment // [] | map(select(.name != "QUILT_LOG_LEVEL")))
            + [{"name": "QUILT_LOG_LEVEL", "value": "DEBUG"}]
          )
      else
        .
      end
    )
' task-definition.json > task-definition-debug.json

NEW_TASK_DEFINITION_ARN=$(
  aws ecs register-task-definition \
    --region <region> \
    --cli-input-json file://task-definition-debug.json \
    --query 'taskDefinition.taskDefinitionArn' \
    --output text
)

Update the service to run the new task definition revision:

aws ecs update-service \
  --cluster <stack-name> \
  --service <stack-name>-registry \
  --task-definition "$NEW_TASK_DEFINITION_ARN" \
  --force-new-deployment \
  --region <region>

Roll Back After Debugging

After collecting the needed logs, restore the previous log level to avoid noisy logs and unnecessary CloudWatch costs.

1. Create another task definition revision using the same process.
2. Set QUILT_LOG_LEVEL back to the original value, usually INFO.
3. Update the registry ECS service to the restored revision.
4. Force a new deployment so the registry tasks restart with the restored environment.

Important Notes

• Valid Python logging levels include DEBUG, INFO, WARNING, ERROR, and CRITICAL.
• ECS task definitions are immutable. Changing the environment variable always requires a new task definition revision and task restart.
• Manual ECS edits create CloudFormation drift. A later stack update may revert the service to the task definition generated by CloudFormation.
• Delete local task-definition.json files after use if they contain sensitive environment values.

Example

For a stack named quilt-staging, the registry service is:

quilt-staging-registry

The registry container is named:

registry

If the active task definition has:

QUILT_LOG_LEVEL=INFO

create a new revision with:

QUILT_LOG_LEVEL=DEBUG

then update quilt-staging-registry to use the new revision.