Introduction
I wrote this article because I solved the problem of passwords being displayed in plain text in Terraform and task_definitions using AWS SSM and Datadog’s auto-discovery template variable.
I was setting up Datadog database monitoring and needed to pass the RDS user password to the Datadog Agent.
The Agent was set up as a container on ECS.
We needed to pass the RDS user password to the Datadog Agent.
However, writing the password in task_definitions is not good from a security point of view, since the password will be visible in plain text from Terraform and the AWS ECS console.
Target
- One who want to avoid exposing Datadog Agent passwords
- One who want to know how to use Datadog’s template variables.
What you can get from this article
- How to pass passwords to Datadog Agent in Terraform & ECS
- How to use auto-discovery template variables
Solution
We took the following two steps to resolve this issue
- Save the password in AWS SSM and load it as an environment variable in the datadog-agent container
- Load the password set in the environment variable dynamically using a template variable
An illustration of AsIs and Tobe is shown below.
AsIs
ToBe
Explanation.
Store password in AWS SSM. Set it in the container’s environment variable.
The official Datadog documentation states that for Docker, database monitoring should be configured as follows.
https://docs.datadoghq.com/database_monitoring/setup_mysql/aurora/?tab=docker#command-line
export DD_API_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
export DD_AGENT_VERSION=7.36.1
docker run -e "DD_API_KEY=${DD_API_KEY}"
-v /var/run/docker.sock:/var/run/docker.sock:ro
-l com.datadoghq.ad.check_names='["mysql"]'
-l com.datadoghq.ad.init_configs='[{}]'
-l com.datadoghq.ad.instances='[{
"dbm": true,
"host": "<AWS_INSTANCE_ENDPOINT>",
"port": 3306,
"username": "datadog",
"password": "<UNIQUEPASSWORD>"
}]'
gcr.io/datadoghq/agent:${DD_AGENT_VERSION}
If this is defined in task_definitions, the “com.datadoghq.ad.instances” part would be set as follows.
...
},
"dockerLabels": {
"com.datadoghq.ad.check_names": "["mysql"]",
"com.datadoghq.ad.init_configs": "[{}]",
"com.datadoghq.ad.instances": "[{"dbm": true, "host": "※rds_endpoint※", "port": 3306,
"username": "datadog", "password": "hogehoge（rds パスワード)"}]"
}
...
In this case, the password would be managed in plain text on the Terraform.
If the Terraform is managed on GitHub, we do not want to push it to the repository as it is.
So, we register the password to AWS SSM, and from Terraform, we refer to the SSM path.
From the AWS Systems Manager parameter store, create a parameter.
For example, it looks like this.
Select “secure string” since this is confidential information.
Note that the role that executes ECS must also have the authority to Decrypt the Secret registered in SSM.
If the authority is insufficient, grant the necessary authority by referring to the following document.
https://docs.aws.amazon.com/ja_jp/AmazonECS/latest/userguide/specifying-sensitive-data-parameters.html
Use auto-detect template variables to dynamically read environment variables set in containers
Auto-discovery template variables are a feature of the Datadog Agent.
Variables described as auto-detect template variables will have their values dynamically reread by the Agent.
See the following document for details.
https://docs.datadoghq.com/agent/guide/template_variables/
This time, follow the description below to have the environment variable value (password) dynamically reread as seen by datadog-agent.
「”%%env_<ENV_VAR>%%”」
Here is the image.
This will allow the datadog agent to read the password without exposing it in the code or on the console.
For the previous configuration, fill in the task_definition as follows.
// Set as container environment variable
"secrets": [
{
"name": "PASSWORD",
"valueFrom": "/db/password/datadog"
}
],
"dockerLabels": {
"com.datadoghq.ad.check_names": "["mysql"]",
"com.datadoghq.ad.init_configs": "[{}]",
"com.datadoghq.ad.instances": "[{"dbm": true, "host": "※rds_endpoint※", "port": 3306, "username": "datadog", "password": "%%env_PASSWORD%%"}]"
}Once you have set this up, all you need to do is start datadog-agent.
Conclusion
In this article, I introduced how to use auto-detect template variables when dealing with confidential information in datadog-agent.
I think this is a problem that you will encounter when implementing Datadog Database Monitoring.
I would be happy if it would be of help to you.
If you would like to try Datadog Database Monitoring, please see the following article.
