Scrape thousands of URLs. Run one batch.
Bulk web scraping in the background. Submit up to 100,000 URLs or crawl a large website, track the job, collect the results, and move on to building your product.
Powering agents and products with the world's data.
Thousands of pages. One job.
Hand off a URL list or a whole site, let it run in the background, and collect the results when it finishes.
Massive URL lists
Submit up to 100,000 URLs in one batch, with identifiers and metadata for your own records.
Full-site collection
Start from a URL or sitemap and set the scope of the crawl.
Collect the dataset
Track progress, page through results, and receive a completion webhook when the job finishes.
Submit the job. Let the batch run.
Pass your URL list to the batch API. Keep the returned job ID to inspect progress and retrieve results without holding open one long request.
npm install context.devimport ContextDev from 'context.dev';
const client = new ContextDev({ apiKey: 'YOUR_API_KEY' });
const urls = [
'https://example.com',
'https://example.com/pricing',
'https://example.com/docs',
];
const batch = await client.batch.submit({
input: {
mode: 'scrape',
data: {
formats: { markdown: true },
urls: urls.map((url) => ({ url })),
},
},
});
console.log(batch.id, batch.status);Go from pages to production datasets.
Collect content at the scale your knowledge base and application need.
Scrape anything
Turn every page in the batch into Markdown, HTML, structured fields, images, or screenshots.
Ground RAG in fresh content
Crawl a whole site into clean Markdown and send it straight into your embeddings, with no brittle parsers in between.
A few things to know.
More details in the API documentation.
How do I scrape a list of URLs?
Submit the list to Batches in one request, up to 100,000 URLs per batch. Keep the batch ID, then poll for progress, receive the batch.completed webhook, or download the result files when the job finishes.
How much does bulk scraping cost?
Batches cost 1 credit per successful URL, plus 1 per PDF page recovered by OCR. Credits are reserved when you submit, only successful pages are charged, and unused reserved credits are refunded when the batch settles.
How many URLs can I submit in a batch?
A scrape batch accepts up to 100,000 URLs. Use separate jobs when your dataset is larger, and apply the concurrency and credit limits for your account.
Can a batch discover pages for me?
Yes. Crawl mode starts from a URL or sitemap with controls for the scope of the job.
How do I collect a large result set?
Use the batch results API and follow its pagination cursor. Keep your item identifiers or metadata to connect the results back to your records.
Can I receive a webhook when it finishes?
Yes. Configure batch webhooks for completion and other supported final states, with optional retry settings.
Collect your first dataset.
Submit a list of URLs or a whole site and get the results back as one job. The same API key works for scraping, map, and monitors.