Send survey responses to BigQuery

If your analysts already live in BigQuery, the useful thing is not another dashboard — it is the response table landing in the dataset they query, on the schedule you choose, without an export-and-upload step in between.

A Siamang project declares that as a task in siamang.yaml. The questionnaire, the analysis scripts and the export target all sit in the same repository, under version control, so how the data got to the warehouse is reviewable rather than folklore.

What this gives you

The warehouse is yours. The credential is yours, the dataset is yours, the bill is yours. Nothing needs to be re-exported from us later.
The pipeline is in the repository. The task block is committed alongside the questionnaire, so a change to where data goes shows up in a diff and gets reviewed like any other change.
Every run is recorded. Each connector run appears in the run history with its status, its log and the commit it ran from.

How to set it up

Create the service account. In Google Cloud, make a service account and grant it BigQuery Data Editor (to write the table) and BigQuery Job User (to run the load job) on the project holding your dataset. Download a JSON key.
Store the key as a project secret. In your project, open Secrets and add it under the key name BQ_SA. Secrets are encrypted at rest and write-only — after saving, the API returns key names only, never values.
Declare the task. Use Connectors → Add connector, which writes the block below into siamang.yaml and stages it as an ordinary diff for you to review and commit.
Run it and check the history. Trigger the connector and open the run: status, log and row count are all there. A failure shows the error string rather than failing silently.

The dataset has to exist already; the connector creates and replaces the table inside it, not the dataset.

Where this fits

Warehouse loading is the last step of a study that was version-controlled from the start. If you want the reasoning behind that, the reproducible survey research hub covers how a commit-pinned run and a declared export target combine into something you can hand to a reviewer a year later.

The siamang.yaml task

The Add-connector wizard writes this block and tells you which secret to create. It is committed to your repository like any other change, so a change to where responses go arrives as a reviewable diff.

tasks:
  responses_to_bigquery:
    type: connector
    target: bigquery
    direction: out
    table: clean_responses
    secret: BQ_SA
    config: { dataset: research, table: responses }

Limits and caveats

Full replace, not incremental. Each run loads the table with WRITE_TRUNCATE, so the destination is the current snapshot — there is no append or merge mode on export.
Every column lands as STRING. The connector does no type mapping, so dates, integers and booleans arrive as text and need casting in BigQuery.
100,000 rows per run. Larger tables need to be split or exported through a different route.
Manual trigger only. Schedules accept run_script and run_all, not connectors, so there is no built-in nightly sync — you run it, or you call the API from your own scheduler.
No automatic retry, and a five-minute job timeout. A failed run is recorded with its error in the run history and you re-run it yourself.
Column names must match ^[A-Za-z_][A-Za-z0-9_]{0,62}$. A variable with a dash or a dot fails the export rather than being silently renamed.

Questions

Do I need a Google Cloud project of my own?

Yes. The data lands in your dataset, in your project, billed to you — we never hold a copy in a warehouse of ours. You create a service account, grant it BigQuery Data Editor and Job User, and paste its JSON key in as a project secret.

Which BigQuery project does it write to?

By default the project the service-account key belongs to. Set config.project explicitly if the dataset lives in a different project from the service account.

Can it run automatically after each completed response?

No. Connector runs are triggered manually or through the API; there is no per-response streaming and no cron for connectors. If you need a nightly load, call the run endpoint from your own scheduler.

Why is everything a STRING?

The connector creates the destination table itself and does not infer types, which keeps a mid-fieldwork questionnaire change from breaking the load. Cast in a view or in dbt downstream — the codebook tells you what each column is.

Is BigQuery available on the free plan?

No connector is available on Free. BigQuery needs Pro; Google Sheets, Excel 365 and Supabase are on Plus.