Using AWS Secrets Manager as a Secrets Backend

Bruin supports using AWS Secrets Manager as a secrets backend for managing connection credentials. This is controlled via the --secrets-backend flag on the run command.

Enabling AWS Secrets Manager

To use AWS Secrets Manager as your secrets backend, pass the flag:

bruin run --secrets-backend aws

You can also set the backend via environment variable:

export BRUIN_SECRETS_BACKEND=aws
bruin run

Configuring AWS Connection

Bruin connects to AWS Secrets Manager using the default AWS credential provider chain. This means Bruin can use the same credentials that the AWS SDK normally finds, including:

• Environment variables such as AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN
• Shared AWS config and credentials files, including profiles selected with AWS_PROFILE
• IAM roles for Amazon EC2, ECS, EKS, Lambda, or other AWS runtime environments
• AWS SSO or other credential processes configured in your AWS profile

Set the AWS region through your normal AWS configuration, for example AWS_REGION, AWS_DEFAULT_REGION, your selected AWS profile, or the runtime environment.

Bruin also supports explicit BRUIN_ environment variables when you want to override the default AWS credential chain for this secrets backend:

• BRUIN_AWS_ACCESS_KEY_ID: Your AWS access key ID
• BRUIN_AWS_SECRET_ACCESS_KEY: Your AWS secret access key
• BRUIN_AWS_REGION: The AWS region where your secrets are stored (e.g., us-east-1, eu-west-1)

The following is optional when using temporary explicit credentials:

• BRUIN_AWS_SESSION_TOKEN: A session token for temporary credentials (e.g., when using AWS STS)

Example Setup

Use the AWS CLI or your cloud runtime to configure credentials as usual:

export AWS_PROFILE=my-profile
export AWS_REGION=us-east-1

bruin run --secrets-backend aws

Or set Bruin-specific credentials explicitly:

export BRUIN_AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export BRUIN_AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export BRUIN_AWS_REGION=us-east-1

bruin run --secrets-backend aws

Using Temporary Credentials

If you are using temporary credentials (e.g., from AWS STS AssumeRole), you can also set the session token:

export BRUIN_AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export BRUIN_AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export BRUIN_AWS_REGION=us-east-1
export BRUIN_AWS_SESSION_TOKEN=FwoGZXIvYXdzE...

bruin run --secrets-backend aws

Storing Secrets in AWS Secrets Manager

Bruin expects connection credentials to be stored in AWS Secrets Manager as JSON strings. Each secret should be named after the connection name and contain the connection details in a specific format.

Secret Format

The secret value must be a JSON string with two required fields:

• type: The connection type (must match a valid Bruin connection type)
• details: An object containing the connection-specific configuration

Example: PostgreSQL Connection

In AWS Secrets Manager, create a secret named my-postgres with this value:

{
  "type": "postgres",
  "details": {
    "host": "localhost",
    "port": 5432,
    "username": "myuser",
    "password": "mypassword",
    "database": "mydatabase",
    "schema": "public"
  }
}

Example: Snowflake Connection

In AWS Secrets Manager, create a secret named my-snowflake with this value:

{
  "type": "snowflake",
  "details": {
    "account": "my-account",
    "username": "myuser",
    "password": "mypassword",
    "warehouse": "my-warehouse",
    "database": "my-database",
    "schema": "my-schema"
  }
}

Example: Google BigQuery Connection

In AWS Secrets Manager, create a secret named my-bigquery with this value:

{
  "type": "google_cloud_platform",
  "details": {
    "project_id": "my-gcp-project",
    "service_account_file": "/path/to/service-account.json"
  }
}

Supported Connection Types

The type field must be one of the valid Bruin connection types. Common types include:

• postgres - PostgreSQL database
• mysql - MySQL database
• snowflake - Snowflake data warehouse
• google_cloud_platform - Google BigQuery
• redshift - AWS Redshift
• databricks - Databricks
• generic - Generic key-value secrets

For a complete list of supported connection types and their configuration schemas, see the connections documentation.

How It Works

When you run Bruin with --secrets-backend aws:

1. Bruin connects to AWS Secrets Manager using your credentials
2. For each connection referenced in your pipeline, Bruin fetches the corresponding secret by name
3. The secret is parsed and validated according to the connection type
4. The connection is established using the fetched credentials
5. Results are cached in memory for the duration of the run

Troubleshooting

AWS Credentials or Region Not Found

If you see an error like:

failed to initialize AWS Secrets Manager client: failed to load AWS config

Make sure AWS credentials and a region are available through either the default AWS credential chain or the Bruin-specific environment variables:

aws sts get-caller-identity
aws configure get region

If you are using Bruin-specific credentials, both BRUIN_AWS_ACCESS_KEY_ID and BRUIN_AWS_SECRET_ACCESS_KEY must be set together. BRUIN_AWS_REGION is optional when the region is already available through the default AWS configuration, but setting it explicitly is recommended when your secrets are in a specific region.

Secret Not Found

If you see an error like:

failed to read secret 'my-connection' from AWS Secrets Manager

Verify that:

1. The secret exists in AWS Secrets Manager with the exact name used in your pipeline
2. The secret is in the correct AWS region
3. Your AWS credentials have the secretsmanager:GetSecretValue permission for that secret

Invalid Secret Format

If you see an error like:

failed to parse secret as JSON

Verify that:

1. The secret value in AWS Secrets Manager is valid JSON
2. The JSON includes both type and details fields
3. The type value matches a supported connection type
4. The details object contains all required fields for that connection type

Secret Has No String Value

If you see an error like:

secret 'my-connection' has no string value

Make sure the secret is stored as a plaintext string (not binary) in AWS Secrets Manager.