← back to catalog

commoncrawl hosted

CommonCrawl web + Wikipedia, April-2026 snapshot (frozen). General web lens, not a news or scholarly index.

tier

hosted

category

web

coverage

2026-04

ranking

bm25

knobs

q, limit, fetch_urls, fetch_snippets, min_quality, max_host_rank, min_length, max_per_host, host_boost, slug_boost, sort_by, blend_weight

Guidance

Default sort_by=bm25 here, but sort_by=quality ranks by authority NOT relevance (can surface high-authority off-topic pages); blend_weight mixes both. No date filtering on this corpus (crawl_date is metadata, not a filter knob). max_per_host=1 dedups hosts (by literal host string, so 'www.x.com' and 'x.com' count separately). text is a snippet (median_snippet), not the full page. Web index has no titles: title falls back to host.

Usage

curl -s https://api.bipsearch.ai/search \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"q": "climate policy briefing", "limit": 5}'