Bruin - Notion to Bigquery Template
This pipeline is a simple example of a Bruin pipeline that copies data from Notion to BigQuery. It demonstrates how to use the bruin
CLI to build and run a pipeline.
The pipeline includes two sample assets already:
raw.notion
: A simple ingestr asset that takes copies a table from Notion to BigQuerymyschema.example
: A simple SQL asset that creates a table in BigQuery.- Feel free to change the type from
bq.sql
to anything.
- Feel free to change the type from
Setup
The pipeline already includes an empty .bruin.yml
file, fill it with your connections and environments. You can read more about connections here.
Here's a sample .bruin.yml
file:
yaml
environments:
default:
connections:
google_cloud_platform:
- name: "gcp"
service_account_file: "/path/to/my/key.json"
project_id: "my-project-dev"
notion:
- name: "my-notion-connection"
api_key: "XXXXXXXX"
Running the pipeline
bruin CLI can run the whole pipeline or any task with the downstreams:
shell
bruin run assets/notion.asset.yml
shell
Starting the pipeline execution...
[2023-03-16T18:25:14Z] [worker-0] Running: dashboard.bruin-test
[2023-03-16T18:25:16Z] [worker-0] Completed: dashboard.bruin-test (1.681s)
[2023-03-16T18:25:16Z] [worker-4] Running: hello
[2023-03-16T18:25:16Z] [worker-4] [hello] >> Hello, world!
[2023-03-16T18:25:16Z] [worker-4] Completed: hello (116ms)
Executed 2 tasks in 1.798s
You can also run a single task:
shell
bruin run assets/hello.py
shell
Starting the pipeline execution...
[2023-03-16T18:25:59Z] [worker-0] Running: hello
[2023-03-16T18:26:00Z] [worker-0] [hello] >> Hello, world!
[2023-03-16T18:26:00Z] [worker-0] Completed: hello (103ms)
Executed 1 tasks in 103ms
You can optionally pass a --downstream
flag to run the task with all of its downstreams.
That's it, good luck!