Optimizing AWS ECS Task Definition Environment Variables and Secrets Management with Terraform

Contents

Introduction

Are you managing multiple environments and services in AWS ECS? Do you often struggle with environment variables and secrets management?

"Defining the same environment variables repeatedly for development and production environments"
"Inconsistent methods for managing secret information"
"Repeating similar configuration tasks every time you add a new service"

If you’re facing these challenges, this article might help. I’ve experienced these same issues and have put together a method for centralizing environment variables and secrets management using Terraform.

Estimated reading time: 10 minutes
Technology stack: Terraform, AWS ECS, AWS SecretsManager, Container configuration management

Target Audience

Readers with a basic understanding of Terraform and AWS ECS concepts
Those with knowledge of container environment variables
Anyone managing configuration across multiple environments and services

This content is aimed at intermediate-level users.

Background

As our project grew, the number of environment variables and task definitions increased, making management increasingly complex. Specifically, we encountered the following issues:

Environment variables and secrets were mixed without clear distinction
The same configurations were repeatedly written across multiple task definitions
Environment-specific settings and common settings were jumbled together

These problems might be negligible in small-scale operations, but they become significant challenges as the number of services and environments increases.

Challenges to Solve

The specific challenges I tackled were:

Eliminating redundant environment variable management across task definitions
Clearly separating environment variables from secrets
Structuring environment-specific settings to make adding new environments easier

Solving these issues improves configuration management maintainability and significantly increases efficiency when adding new environments or services. It also enables proper handling of sensitive information, reducing security risks.

Implementation Approach

The approach I adopted consists of these three points:

Structuring environment variable template files
Explicitly separating environment variables and secrets
Dynamically assembling environment variables using Terraform local variables

I considered alternative approaches, such as creating completely separate files for each environment, but rejected this due to increased management complexity. I also considered managing all environment variables in SecretsManager but didn’t adopt this due to concerns about increased costs and reduced performance.

Project Structure

To understand the entire implementation, let’s look at the project directory structure and the role of each file.

Directory Structure

terraform-ecs-project/
├── main.tf              # Project entry point
├── variables.tf         # Variable definitions
├── outputs.tf           # Output definitions
├── environments/        # Environment-specific settings
│   ├── dev.tfvars       # Development environment variable values
│   ├── staging.tfvars   # Staging environment variable values
│   └── prod.tfvars      # Production environment variable values
├── ecs.tf               # ECS cluster and service definitions
├── secretsmanager.tf    # SecretsManager resource definitions
└── task_definitions/    # Task definition templates
    ├── common_env_vars.tftpl    # Common environment variable template
    ├── service_a.tftpl          # Service A task definition template
    └── service_b.tftpl          # Service B task definition template

Key Files and Their Roles

Let’s briefly explain each file’s role with actual code examples.

1. main.tf

This is the project’s entry point file.

provider "aws" {
  region = var.aws_region
}

locals {
  env = var.environment
}

# Other common settings...

2. ecs.tf

This file manages ECS clusters, services, and task definitions.

# ECS cluster definition
resource "aws_ecs_cluster" "main" {
  name = "${var.prefix}-cluster-${var.env}"

  setting {
    name  = "containerInsights"
    value = var.env == "prod" ? "enabled" : "disabled"
  }
}

# Task definition and ECS service definition (defined for each service)
resource "aws_ecs_task_definition" "service_a" {
  # Details in the implementation details section below
}

resource "aws_ecs_service" "service_a" {
  name            = "service-a"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.service_a.arn
  desired_count   = var.service_a_count

  # Other settings...
}

# Similarly for Service B

3. secretsmanager.tf

This file defines resources for secret management.

# Secret definition for each environment
resource "aws_secretsmanager_secret" "app_secrets" {
  name = "${var.env}/app-secrets"
  description = "Application secrets (${var.env} environment)"

  tags = {
    Environment = var.env
    Application = "my-app"
    ManagedBy   = "terraform"
  }
}

# Note: Actual secret values are set manually via AWS console for security reasons

4. task_definitions/common_env_vars.tftpl

This is a common template file for environment variables and secrets, defined in JSON format for each environment.

{
  "common": {
    "environment": [
      { "name": "APP_USER", "value": "user" },
      // Other common environment variables
    ],
    "secrets": [
      { "name": "DB_PASSWORD", "valueFrom": "${secrets_arn}/DB_PASSWORD" },
      // Other common secrets
    ]
  },
  "dev": {
    // Development environment specific settings
  },
  "prod": {
    // Production environment specific settings
  }
}

5. task_definitions/service_a.tftpl, service_b.tftpl

Container definition template files for each service.

[
  {
    "name": "service-a-${env}",
    "image": "${ecr_repo_url}:${image_tag}",
    "environment": ${common_env_vars},
    "secrets": ${common_env_secrets},
    // Service-specific settings
  }
]

Deployment Flow

Select the variable file for the target environment
```
terraform plan -var-file=environments/dev.tfvars
```
Environment variable templates are processed and incorporated into service settings
SecretsManager ARNs are inserted through template processing
Task definitions are generated and ECS services are deployed

Implementation Details

1. Structuring Environment Variable Template Files

{
  "common": {
    "environment": [
      { "name": "APP_USER", "value": "user" },
      { "name": "DB_USER", "value": "db_admin" }
    ],
    "secrets": [
      { "name": "DB_PASSWORD", "valueFrom": "${secrets_arn}/DB_PASSWORD" },
      { "name": "API_KEY", "valueFrom": "${secrets_arn}/API_KEY" }
    ]
  },
  "dev": {
    "environment": [
      { "name": "DB_ENDPOINT", "value": "db-dev.example.com" }
    ],
    "secrets": []
  },
  "prod": {
    "environment": [
      { "name": "DB_ENDPOINT", "value": "db-prod.example.com" }
    ],
    "secrets": []
  }
}

The reason for choosing this structure is simple: it reduces duplication of environment variables and makes it easy to see at a glance which values are used in which environments. By clearly separating common parts from environment-specific parts, it also makes it easier to understand the scope of changes.

2. Implementing Environment Variable Processing in Terraform

locals {
  # Regular environment variables are loaded and processed using the file function
  env_vars_tpl = file("task_definitions/common_env_vars.tftpl")

  # Environment variable joining process
  common_env_vars = jsonencode(concat(
    jsondecode(local.env_vars_tpl)["common"].environment,
    jsondecode(local.env_vars_tpl)[var.env].environment
  ))

  # Secret template processing - substituting ARNs
  common_env_secrets_tpl = templatefile(
    "task_definitions/common_env_vars.tftpl",
    {
      secrets_arn = aws_secretsmanager_secret.app_secrets.arn
    }
  )

  # Secret joining process
  common_env_secrets = jsonencode(concat(
    jsondecode(local.common_env_secrets_tpl)["common"].secrets,
    jsondecode(local.common_env_secrets_tpl)[var.env].secrets
  ))
}

There are two key points in this processing:

Regular environment variables: Using the file() function to load the template file and combine common and environment-specific variables.
Secret variables: Using the templatefile() function to directly substitute SecretsManager ARNs during loading. This enables dynamic generation of secret paths.

This two-stage processing allows efficient generation of settings from a single template file while properly separating non-sensitive and sensitive information. The templatefile() function enables variable substitution (like ${secrets_arn}) within the environment variable file.

3. Defining Secrets Manager Resources

resource "aws_secretsmanager_secret" "app_secrets" {
  name        = "${var.env}/app-secrets"
  description = "Application secrets (${var.env} environment)"

  # Deletion protection setting (enabled for production)
  recovery_window_in_days = var.env == "prod" ? 30 : 7

  tags = {
    Environment = var.env
    Application = "my-app"
    ManagedBy   = "terraform"
  }
}

For security reasons, we create a SecretsManager resource for each environment. By managing them separately from environment variables, we clearly distinguish the handling of sensitive information.

Note that the actual secret values are set directly from the AWS console, not through Terraform. This avoids the risk of storing sensitive information in code or version control systems.

4. Using Environment Variables Across Multiple Task Definitions

resource "aws_ecs_task_definition" "service_a" {
  family = "service-a-${var.env}"
  # ... other settings ...

  container_definitions = templatefile(
    "task_definitions/service_a.tftpl",
    {
      env                = var.env
      environment        = local.environment
      common_env_vars    = local.common_env_vars
      common_env_secrets = local.common_env_secrets
      # Pass the secret ARN reference directly
      secrets_arn        = aws_secretsmanager_secret.app_secrets.arn
      # Service-specific variables
      service_specific_var = "value-a"
    }
  )
}

Here, we pass the common environment variables and secrets we created earlier to the task definition template. We also pass the SecretsManager ARN directly to the template, enabling references in the form ${secrets_arn}/KEY_NAME within the template.

This approach allows multiple services to use the same set of environment variables, ensuring consistency across configurations.

5. Using Environment Variables in Task Definition Templates

[
  {
    "name": "service-${env}",
    "image": "123456789012.dkr.ecr.region.amazonaws.com/service-${env}",
    "environment": ${jsonencode(concat(jsondecode(common_env_vars), [
      {
        "name": "RAILS_ENV",
        "value": "${environment}"
      },
      {
        "name": "SERVICE_TYPE",
        "value": "service-a"
      },
      {
        "name": "SERVICE_SPECIFIC_VAR",
        "value": service_specific_var
      }
    ]))},
    "secrets": ${jsonencode(concat(jsondecode(common_env_secrets), [
      {
        "name": "SERVICE_API_KEY",
        "valueFrom": "${secrets_arn}/SERVICE_API_KEY"
      }
    ]))},
    "essential": true,
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "/ecs/service-${env}",
        "awslogs-region": "ap-northeast-1",
        "awslogs-stream-prefix": "ecs"
      }
    }
  }
]

In the task definition template, we dynamically add service-specific variables to the common environment variables. The process involves decoding the common environment variables with jsondecode(), combining them with service-specific variables, and then re-encoding them.

Common Pitfalls and Solutions

Here are some issues I encountered during implementation and their solutions:

JSON Format Issues
JSON syntax errors in templates have occasionally caused deployment failures. Pay special attention to trailing commas in arrays. I recommend validating templates with a JSON validator before deployment.
Incorrect Secret Reference Paths
If the valueFrom format is incorrect, ECS containers won’t start. Testing actual SecretsManager references with AWS CLI provides peace of mind.
Environment Variable Conflicts
When the same environment variable is defined in different places, one will overwrite the other. I use service name prefixes to prevent variable name duplications.
Undefined Template Variables
Referencing undefined variables in templates causes errors. Setting default values is helpful (e.g., ${var.name != "" ? var.name : "default"}).

Practical Steps

For those looking to implement a similar system, here are the practical steps:

Inventory and classify environment variables
List all environment variables and classify them into "common," "environment-specific," and "secrets."

Create environment variable template files

{
 "common": { "environment": [...], "secrets": [...] },
 "env1": { "environment": [...], "secrets": [...] },
 "env2": { "environment": [...], "secrets": [...] }
}

Implement Terraform locals block

locals {
 # Environment variable processing
 env_vars_tpl = file("path/to/template.tftpl")
 common_env_vars = jsonencode(concat(
   jsondecode(local.env_vars_tpl)["common"].environment,
   jsondecode(local.env_vars_tpl)[var.env].environment
 ))

 # Secret template processing - substituting ARNs
 common_env_secrets_tpl = templatefile(
   "path/to/template.tftpl",
   {
     secrets_arn = aws_secretsmanager_secret.app_secrets.arn
   }
 )

 # Secret joining process
 common_env_secrets = jsonencode(concat(
   jsondecode(local.common_env_secrets_tpl)["common"].secrets,
   jsondecode(local.common_env_secrets_tpl)[var.env].secrets
 ))
}

Update task definition templates

container_definitions = templatefile(
 "path/to/service_template.tftpl",
 {
   env = var.env,
   common_env_vars = local.common_env_vars,
   common_env_secrets = local.common_env_secrets,
   # Service-specific variables
   service_var = "value"
 }
)

Test and deploy
- Use terraform validate for syntax checking
- Review changes with terraform plan
- Deploy to the development environment and verify container startup

Q&A

Q1: How should I distinguish between environment variables and secrets?

A1: As a general rule, treat authentication information like API keys, passwords, and access tokens as secrets, and non-sensitive information like endpoint URLs, port numbers, and feature flags as regular environment variables. Manage secrets using specialized services like AWS SecretsManager and store them encrypted for safety.

Q2: What’s the procedure for adding a new environment?

A2: Simply add a new environment section to the environment variable template file and add the new environment name to the Terraform variables. Existing task definitions will automatically generate settings for the new environment through template processing.

Conclusion

I’ve shown how using Terraform and its template features can significantly improve ECS environment variable management. From my experience, proper structuring and separation streamline configuration management across multiple environments and services.

Looking forward, you might consider developing custom modules or validation tools for environment variable management, but I recommend starting with the approach outlined in this article.

If you’re working on similar challenges or have other solutions or ideas, please share them in the comments!

References