Sitemap extractor API. Find every page on a site.
Discover every URL on a domain from its sitemaps, with page titles and descriptions when available. Build the map before you collect the content.
Powering agents and products with the world's data.
Map the whole site. Then pick your pages.
Start from a domain and get back the pages on it, with titles and descriptions when available, so you only collect what you need.
Extract every sitemap
Read a domain’s sitemaps, including nested sitemap indexes, or fall back to homepage links when there is no sitemap.
Scope the discovery
Include subdomains and set a maximum number of links for your request.
Use available metadata
Read titles and descriptions when available. Newly discovered URLs can arrive before enrichment.
Map first. Then collect.
Call mapUrls as your sitemap scraper API. Pass the URLs you want into Scrape or Batches when you are ready to retrieve their content.
npm install context.devimport ContextDev from 'context.dev';
const client = new ContextDev({ apiKey: 'YOUR_API_KEY' });
const { urls } = await client.web.mapUrls({
domain: 'stripe.com',
includeSubdomains: true,
maxLinks: 100,
});
for (const page of urls) {
console.log(page.url, page.title, page.description);
}Turn a site map into a useful dataset.
Discover the pages, then build the content workflow that your product needs.
Scrape the pages you choose
Pass the URLs you want into Scrape and get Markdown, HTML, structured fields, images, or screenshots in one request.
Collect the whole map at once
Send thousands of mapped URLs to Batches in one job, then collect the results when the batch completes.
A few things to know.
More details in the API documentation.
How do I find all pages on a website?
Start with the site’s sitemaps, which list the pages it publishes. The Map API does that step for you: it reads the domain’s sitemaps, falls back to homepage links when there is no sitemap, and returns up to 100,000 URLs for 1 credit. Use a crawl for pages the sitemap leaves out.
Does Map return the content of every page?
Map returns discovered URLs and available metadata. Use Scrape for page content, or submit the discovered URLs to a batch for larger collection jobs.
Can I include subdomains?
Yes. Set includeSubdomains for the domain discovery request.
Can I limit the number of URLs?
Yes. Use maxLinks to bound the number of discovered links returned for your request.
Why might a title or description be missing?
URLs without stored enrichment can be returned immediately and queued for background HTML scraping. Later requests may include the additional metadata.
Start with a map of any site.
Discover the pages on a domain, then scrape the ones you need. The same API key works for scraping, batches, and monitors.