Export conversations to your warehouse nightly

Load every conversation, with its transcript and Darwin’s grading, into BigQuery, Snowflake, Redshift or plain files each night. You need: a key with conversations:read. Leave out contacts:read if the warehouse should not hold names and emails: lead and booker fields then come back null (transcripts still hold whatever visitors typed).

export-conversations.mjs

import { createWriteStream } from "node:fs";

const API = "https://api.meetdarwin.ai/v1";
const headers = { Authorization: `Bearer ${process.env.DARWIN_API_KEY}` };

// A rolling three-day window: Darwin grades a conversation after it goes quiet,
// so re-export recent days and let the warehouse upsert by id.
const since = new Date(Date.now() - 3 * 86400_000).toISOString();
const file = createWriteStream(`conversations-${new Date().toISOString().slice(0, 10)}.ndjson`);

let cursor = null;
for (;;) {
  const url = new URL(`${API}/conversations`);
  url.searchParams.set("limit", "100");
  url.searchParams.set("created_since", since);
  url.searchParams.set("expand", "messages");
  if (cursor) url.searchParams.set("starting_after", cursor);
  const res = await fetch(url, { headers });
  if (res.status === 429) {
    await new Promise((r) => setTimeout(r, Number(res.headers.get("retry-after")) * 1000));
    continue;
  }
  if (!res.ok) throw new Error(`Darwin ${res.status}: ${(await res.json()).code}`);
  const page = await res.json();
  for (const conversation of page.data) file.write(JSON.stringify(conversation) + "\n");
  if (!page.has_more) break;
  cursor = page.next_cursor;
}
file.end();

Load the file into a table keyed on id, merging over yesterday’s rows. Useful columns:

Field	Use it for
`created_at`, `page_path`, `visitor.source`, `visitor.utm`, `visitor.ai_referred`	Where conversations come from
`outcome` (`meeting`, `lead`, `handoff`, `none`)	Results by page and source
`evaluation.intent`, `evaluation.buying_stage`, `evaluation.qualification_score`	Who is in market
`evaluation.understood`, `evaluation.failure_category`, `evaluation.recommended_improvement`	Where the agent needs better knowledge
`opening.experiment`, `opening.variant`	Which opening the visitor saw
`messages`	Full-text analysis

Add GET /v1/meetings?updated_since=… and GET /v1/analytics?range=7d to the same job for meetings and daily funnel numbers. It worked if the morning’s file has one line per conversation from the last three days and yesterday’s graded conversations have a non-null evaluation.

Get started

Guides

Export conversations to your warehouse nightly