Introducing Highlights: the context that matters

Feed your RAG pipeline. Fresh web data, ready to embed.

Q: What is a RAG pipeline?

A RAG (retrieval-augmented generation) pipeline ingests documents, splits them into chunks, embeds them, and retrieves the most relevant chunks to ground an LLM’s answer. Context.dev handles ingestion for web content: it crawls sites and returns clean Markdown ready to chunk and embed.

Q: How do I get website data into a RAG pipeline?

Crawl the site with Context.dev to get main-content Markdown for every page. Split each page into chunks, embed them, and store the vectors with the page URL so answers can cite their source.

Q: How do I keep a RAG index up to date?

Create monitors for the pages or sitemaps you index. When a run detects a change, the webhook tells you which page changed, so you re-embed that page instead of re-crawling everything.

Q: Can I ingest PDFs as well as web pages?

Yes. Scrape and Crawl turn PDFs into Markdown alongside web pages, and the Parse API converts PDF and Office files you upload.

Turn docs, help centers, and whole websites into clean Markdown for your RAG pipeline, then keep the index current with monitors. No HTML parsers or cleanup scripts to maintain.

Start building free Read the crawl docs

crawl → markdown → your embeddings

Powering agents and products with the world's data.

Nick Khami

Engineering Manager @ Mintlify

“Onboarding couldn't be simpler. Self-serve sign-up gives you an API key immediately, the documentation is thorough, and we were integrating within 10 minutes.”

Read the case study

Pranav

Founder @ Chatwoot

“Happy to be a paid customer of @getcontextdev. It's great, gives us the best results with ms response times. We have tested a couple of products, and Context wins without a doubt.”

Raz

AI @ Similarweb

“My agent understood it perfectly. Setup was very easy. I was looking for a nice datapoint for my agents, and Context.dev was the best one in the marketplace.”

Read the case study

Wilson Wilson

Founder @ Ferndesk

“Now that context.dev has been integrated into Ferndesk, we've been able to ship this really beautiful onboarding that integrates the customer's company throughout the entire flow.”

Read the case study

Aaron Edwards

Founder @ DocsBot

“We're seeing much higher activation rates for our free trials and sign-ups because of it. It's made it a lot simpler for customers to get started with DocsBot and to get to that 'aha' value moment very quickly.”

Read the case study

Ido Shamun

Co-founder @ daily.dev

“We use Context.dev for our internal enrichment system to verify companies. The integration was straightforward. We set it up once and never had to touch it again. It just works.”

Read the case study

Christopher Price

COO @ Sixtyfour

“The API integration was seamless and expedient. My team was able to get it set up and testing with production workloads within an hour of hearing what Context.dev was!”

Read the case study

Lewis Carhart

Founder @ Comp AI

“Just used Context.dev to help Codex build an on-brand sales deck generator for us. Came out really well!”

Read the case study

Bhanu

Co-founder @ Squad.so

“It was great. Took 5 mins to integrate. We use Context.dev to give AI agents ability to scrape webpages, take screenshots, and also extract specific content from webpages.”

Read the case study

Teams feeding RAG with Context.dev

SiteGPT
Hyperspell
Agently
Sourcely
Notra
Polished.ad

“I installed the SDK, swapped the scraper service behind our Brain, and deleted the polling code the same day.”

Omar GhandourFounder, AgentlyRead the Agently story

for SiteGPT to move its chatbot knowledge bases: < 1 day
for Hyperspell’s web content and company data: 1 API

Ingest, clean, and keep it fresh.

Crawl a site into Markdown with only the main content, chunk and embed it with your own stack, and let a monitor tell you when a source page changes.

1IngestCrawl a site, map it, or scrape a URL list
2CleanMain-content Markdown, no nav or footers
3RefreshMonitors flag the pages that changed

$ npm install context.dev

Read the crawl API reference

1,000 free credits a month · no card required

ingest.ts

import ContextDev from 'context.dev';

const client = new ContextDev({ apiKey: 'YOUR_API_KEY' });

const crawl = await client.web.webCrawlMd({
  url: 'https://help.acme.com',
  maxPages: 500,
  useMainContentOnly: true,
});

for (const page of crawl.results) {
  // Chunk and embed with your own stack, keyed by the source URL
  console.log(page.metadata.url, page.markdown.length);
}

from context.dev import ContextDev

client = ContextDev(api_key="YOUR_API_KEY")

crawl = client.web.web_crawl_md(
    url="https://help.acme.com",
    max_pages=500,
    use_main_content_only=True,
)

for page in crawl.results:
    # Chunk and embed with your own stack, keyed by the source URL
    print(page.metadata.url, len(page.markdown))

require "context_dev"

client = ContextDev::Client.new(api_key: "YOUR_API_KEY")

crawl = client.web.web_crawl_md(
  url: "https://help.acme.com",
  max_pages: 500,
  use_main_content_only: true
)

crawl.results.each do |page|
  # Chunk and embed with your own stack, keyed by the source URL
  puts "#{page.metadata.url} #{page.markdown.length}"
end

<?php

require_once 'vendor/autoload.php';

use ContextDev\Client;

$client = new Client(apiKey: 'YOUR_API_KEY');

$crawl = $client->web->webCrawlMd(
    url: 'https://help.acme.com',
    maxPages: 500,
    useMainContentOnly: true,
);

foreach ($crawl->results as $page) {
    // Chunk and embed with your own stack, keyed by the source URL
    echo $page->metadata->url, ' ', strlen($page->markdown), PHP_EOL;
}

package main

import (
	"context"
	"fmt"

	contextdev "github.com/context-dot-dev/context-go-sdk/v2"
	"github.com/context-dot-dev/context-go-sdk/v2/option"
)

func main() {
	client := contextdev.NewClient(
		option.WithAPIKey("YOUR_API_KEY"),
	)

	crawl, err := client.Web.WebCrawlMd(context.TODO(), contextdev.WebWebCrawlMdParams{
		URL:                "https://help.acme.com",
		MaxPages:           contextdev.Int(500),
		UseMainContentOnly: contextdev.Bool(true),
	})
	if err != nil {
		panic(err)
	}

	for _, page := range crawl.Results {
		// Chunk and embed with your own stack, keyed by the source URL
		fmt.Println(page.Metadata.URL, len(page.Markdown))
	}
}

curl -X POST "https://api.context.dev/v1/web/crawl" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
  "url": "https://help.acme.com",
  "maxPages": 500,
  "useMainContentOnly": true
}'

What teams build on fresh web data.

RAG pipelines our customers run on Context.dev, from support bots to company brains.

Support chatbots
Train each customer’s bot on their own docs and help center, as clean Markdown.
SiteGPT
Company brains
Combine public website content with Slack, email, and Drive in one source of truth.
Hyperspell
Knowledge at signup
Read a new customer’s website during signup so the agent knows the business.
Agently
Research corpora
Crawl journals and PDFs into a corpus a research product can cite.
Sourcely
Brand knowledge bases
Feed ad generation with crawl-ready Markdown about each brand.
Polished.ad
Chatbots with live search
Pair a knowledge base with web search and fetch for what it doesn’t know.
Notra

Skip the ingestion code you’d have to babysit.

What a homegrown RAG ingestion pipeline needs before the first embedding, and what Context.dev handles.

	Homegrown ingestionbuild and maintain	Context.devone API key
Getting the pages	A crawler, headless browsers, and proxies	Crawl, map, or scrape through one API
Clean text	Per-site parsers to strip nav and footers	Main-content Markdown on every page
PDFs	A separate document parsing pipeline	Web pages and PDFs through the same API
Freshness	Re-crawl everything on a schedule	Monitors flag only the pages that changed
Cost	Servers, proxies, and engineering time	1 credit per page, failed pages are free

RAG in production, in their words.

All customer stories

RAG pipelines, answered.

Feed your pipeline today.

1,000 free credits every month, enough for 1,000 pages of Markdown. No credit card required.

1,000free credits

Every month. No credit card required.

Start for free Read the docs

Feed your RAG pipeline. Fresh web data, ready to embed.

Powering agents and products with the world's data.

Teams feeding RAG with Context.dev

Ingest, clean, and keep it fresh.

What teams build on fresh web data.

Support chatbots

Company brains

Knowledge at signup

Research corpora

Brand knowledge bases

Chatbots with live search

Skip the ingestion code you’d have to babysit.

RAG in production, in their words.

RAG pipelines, answered.

Feed your pipeline today.