S3 Platform

Amazon S3 (Simple Storage Service) is supported in Bruin for both data ingestion and as sensors for monitoring object availability.

S3 Sensors

S3 sensors allow you to monitor for the existence of specific objects in S3 buckets. The sensor waits for a file or object to become available before allowing downstream assets to proceed.

Connection Configuration

Sensors accept both aws and s3 connections in .bruin.yml. Existing AWS connections continue to work:

connections:
  aws:
    - name: "aws-default"
      access_key: "your-access-key"
      secret_key: "your-secret-key"
      region: "us-east-1"  # Optional - will be auto-discovered from bucket if not provided

S3-compatible connections

Use the same connections.s3 entry as S3 ingestion. Set the service endpoint, not an object URL; pass the bucket and key separately in the sensor asset. Bruin uses that endpoint for both object HEAD requests and wildcard listing, without AWS region discovery.

Field	Meaning
name	Connection name
access_key_id, secret_access_key	Static credentials; provide both or neither
endpoint_url	Custom HTTP(S) service URL, without credentials, query parameters, or fragment
region	Sensor signing region; falls back to the AWS environment/profile region, then us-east-1 for custom endpoints
url_style	Sensor addressing: path or vhost (same vocabulary as DuckLake). Defaults to path for custom endpoints and virtual-hosted addressing for AWS
use_ssl	Optional sensor transport assertion; must match the endpoint scheme. false requires an explicit http:// endpoint
ca_bundle	Path to a PEM CA bundle for sensors, resolved relative to the working directory
session_token	Optional session token accompanying static sensor credentials
bucket_name, path_to_file, layout	Ingestion destination settings; sensors use asset parameters instead

The region, addressing, TLS, and session-token options above configure the native sensor, not ingestr. Ingestion continues to use its existing endpoint and layout options.

When static sensor credentials are omitted, the standard AWS credential chain is used (environment, shared profile, workload/instance role). HTTPS always verifies certificates; use ca_bundle for a private CA rather than bypassing verification. Explicit HTTP endpoints remain supported for local development and print a warning. use_ssl never silently changes the endpoint scheme. Sensor output includes the validated endpoint and addressing mode, not credentials.

Example connection entries (place under your environment's connections):

connections:
  s3:
    - name: raw-b2
      access_key_id: your-b2-key-id
      secret_access_key: your-b2-application-key
      region: us-west-004
      endpoint_url: https://s3.us-west-004.backblazeb2.com
      url_style: path
      use_ssl: true

    - name: raw-r2
      access_key_id: your-r2-access-key
      secret_access_key: your-r2-secret-key
      region: auto
      endpoint_url: https://YOUR_ACCOUNT_ID.r2.cloudflarestorage.com
      url_style: path

    - name: local-minio
      access_key_id: your-minio-access-key
      secret_access_key: your-minio-secret-key
      region: us-east-1
      endpoint_url: http://localhost:9000
      url_style: path
      use_ssl: false  # Local development only

For HTTPS MinIO with a private CA, use an https:// endpoint, use_ssl: true, and ca_bundle: /path/to/minio-ca.pem. To use virtual-hosted addressing, set url_style: vhost and ensure your provider's DNS and certificate support BUCKET.ENDPOINT_HOST.

Sensor Configuration

Create a sensor asset in your pipeline:

name: "wait_for_s3_file"
type: s3.sensor.key_sensor
connection: aws-default
parameters:
  bucket_name: "my-data-bucket"
  bucket_key: "path/to/expected/file.csv"
  timeout: 1h

Parameters

• bucket_name (required): The name of the S3 bucket to monitor
• bucket_key (required): The key/path of the object to wait for. Supports wildcard patterns.
• timeout (optional): How long to wait before the sensor fails. Uses single-unit duration syntax (s, m, h, d, ms, ns), e.g. 1h or 90m. Defaults to 24h. See Sensor Timeout.

Metadata comparisons

Optional parameters apply to a specific key or to each wildcard candidate:

• etag: Exact, case-sensitive comparison with an opaque ETag (surrounding HTTP quotes are ignored). Multipart and provider-specific ETags are not interpreted as MD5 hashes.
• min_size: Minimum object size in bytes, inclusive; a non-negative integer.
• last_modified_after: Exclusive lower bound for the object's last-modified time, as an RFC3339 timestamp.

All supplied conditions must match the same object. Wildcards succeed when any candidate satisfies all conditions. Metadata filtering performs HEAD requests for matching keys; objects deleted between listing and HEAD are treated as missing. Permission errors are surfaced rather than treated as missing objects.

name: wait_for_manifest
type: s3.sensor.key_sensor
connection: raw-b2
parameters:
  bucket_name: raw-inputs
  bucket_key: manifests/*.json
  min_size: 1
  last_modified_after: "2026-01-01T00:00:00Z"
  poke_interval: 10
  timeout: 1h

Wildcard Patterns

The bucket_key parameter supports wildcard patterns for matching multiple objects:

• * matches any characters except / (does not cross directory boundaries)
• {a,b,c} matches any of the comma-separated alternatives

Examples

Pattern	Matches
exports/*.csv	Any CSV file directly under exports/
logs/2024-*/*.log	Log files in any 2024- prefixed subfolder
data/{foo,bar}.csv	data/foo.csv or data/bar.csv

When a wildcard pattern is used, the sensor lists objects under the common prefix and checks each key against the pattern. The sensor succeeds as soon as any matching object is found.

Sensor Modes

The sensor supports different modes, controlled via the --sensor-mode flag when running:

• once (default): Check once and fail if object doesn't exist
• wait: Continuously poll until object is found (default 24-hour timeout, configurable via timeout)
• skip: Skip sensor execution entirely

Running the Sensor

Execute the sensor using the bruin run command:

bruin run path/to/your/sensor.asset.yml --sensor-mode wait

Behavior

• For AWS endpoints, the region is discovered from the bucket only when neither the connection nor the AWS environment/profile specifies one. Custom endpoints never use AWS bucket-region discovery.
• In wait mode, the sensor polls every few seconds (configurable via poke_interval) as a parameter
• SDK retries remain provider-neutral and can be configured with the standard AWS_RETRY_MODE and AWS_MAX_ATTEMPTS environment variables. poke_interval controls repeated checks for missing objects or unmet metadata conditions; timeout bounds polling and requests.
• Default timeout is 24 hours for continuous polling. Override with the timeout parameter (single-unit duration syntax, e.g. 1h or 90m). See Sensor Timeout.
• Returns error if object is not found in once mode

S3 for Data Ingestion

Bruin also supports S3 as a data source and destination for ingestion workflows. For comprehensive documentation on using S3 for data ingestion, including reading from and writing to S3 buckets, see the S3 Ingestion Guide.