Scraper pipeline

The pipeline turns raw ScrapeCreators responses into scored, deduplicated rows in your database. It lives in src/lib/pipeline.ts and is shared by the API route, the cron job, and server actions.

Stages

• Fetch — one API call per source. Trending endpoints per platform, or keyword search per niche.
• Normalize — platform-specific mappers produce one NormalizedVideo shape: metrics, author, caption, cover, preview URL, and the full raw JSON.
• Dedupe — by external_id within the batch, and by (platform, external_id) on upsert.
• Score — the batch runs through the viral score algorithm (see Viral score).
• Store — upsert into videos and log every run in scrape_runs with credits used.

Endpoints used

Every call costs 1 ScrapeCreators credit. Cached responses on their side are free. Response shapes are handled defensively: extractArray finds the items array, TikTok search items are unwrapped from aweme_info, and Instagram field variants (trending vs search) are merged in the mapper.

Running a job

POST /api/scrape/run
{
  "platforms": ["tiktok", "instagram", "youtube"],
  "keywords": ["food", "fitness"],   // optional, per-platform
  "trending": true                    // optional, defaults to true when no keywords
}

Response: per-source items, credits, stored counts, and the top 3 videos by score.