---
title: "How to Extract Raw HTML from Any URL with a Single API Call"
description: "Learn how to extract raw HTML from any URL using a simple API call. Compare DIY scraping pitfalls with managed solutions and get working code examples in cURL, Python, JavaScript, and TypeScript."
url: "https://www.context.dev/blog/how-to-extract-raw-html-from-any-url-with-a-single-api-call"
---
# How to Extract Raw HTML from Any URL with a Single API Call
[
Yahia Bakour
@mynameisyahia
](/authors/yahia-bakour)·Apr 5, 2026
Sometimes you just need the HTML.
Not parsed Markdown or a structured JSON extraction or an AI summary. Just the raw, fully-rendered HTML of a webpage, the same DOM a real browser sees after JavaScript execution, redirects, and client-side rendering.
You might be feeding it into a custom parser, diffing two versions of a page, or building a monitoring tool that watches for DOM changes. Whatever the reason, "give me the HTML at this URL" sounds simple, and it's way more painful to do reliably than it should be.
This guide covers the practical ways to extract raw HTML from a URL, the problems you'll hit doing it yourself, and how a single API call can replace a lot of infrastructure code.
## Why "Just Fetch the HTML" Is Harder Than It Sounds
If you've ever written `fetch(url)` or `requests.get(url)` and called it a day, you've probably already been burned. The HTML you get back from a simple HTTP request is often not what you see in a browser. Here's why.
### JavaScript-Rendered Pages
The modern web runs on client-side JavaScript. React, Next.js, Vue, Angular: these frameworks render content after the initial page load. A basic HTTP request returns the bare shell:
Copy
```markup
```
That's not the HTML you wanted. The actual content (product listings, article text, pricing tables) is injected by JavaScript after the page loads. To get the real HTML, you need a headless browser that executes JavaScript and waits for the DOM to stabilize.
### Anti-Bot Protections
Cloudflare, Akamai, PerimeterX, DataDome: a growing number of sites use bot detection that blocks automated requests outright. Your `curl` command returns a CAPTCHA page or a 403. Your Python script gets rate-limited after three requests.
Getting past these protections requires rotating proxies, browser fingerprinting, TLS fingerprint matching, and sometimes residential IP addresses. This isn't a weekend project. It's ongoing infrastructure.
### Redirects, Cookies, and Authentication Walls
Many URLs don't serve content directly. They redirect through chains of shortened URLs, tracking pixels, and consent gates. Some pages require cookies from a previous visit. Others sit behind login walls or geo-restricted CDNs.
A simple GET request doesn't handle any of this. A headless browser with proper session management does.
### Dynamic Content and Lazy Loading
Even with JavaScript execution, some content only appears after scrolling, clicking, or waiting for async API calls to resolve. Images lazy-load. Infinite scroll pages only render the first batch. Tabs hide content until clicked.
Getting the "full" HTML means knowing when to stop waiting, and that's harder than it sounds.
## The DIY Approach: Headless Browsers
The standard solution for developers who need rendered HTML is to run a headless browser, typically Puppeteer (Chrome) or Playwright (Chrome, Firefox, or WebKit).
Here's a minimal Puppeteer example:
Copy
```typescript
import puppeteer from 'puppeteer';
const extractHtml = async (url: string): Promise => {
const browser = await puppeteer.launch({ headless: true });
const page = await browser.newPage();
await page.goto(url, { waitUntil: 'networkidle2' });
const html = await page.content();
await browser.close();
return html;
};
const html = await extractHtml('https://example.com');
console.log(html);
```
This works for a demo. It falls apart in production:
**Resource overhead.** Each browser instance consumes 100-300MB of RAM. If you're extracting HTML from 50 URLs concurrently, you need serious compute. Chrome processes leak memory over time, so you need restart logic.
**Startup latency.** Launching a browser takes 1-3 seconds. Navigating to a page, waiting for JavaScript to execute, and waiting for network idle adds another 3-10 seconds. For a "just get the HTML" use case, this is painfully slow.
**Anti-bot detection.** Headless Chrome is detectable. Sites check `navigator.webdriver`, the Chrome DevTools protocol fingerprint, and dozens of other signals. You need `puppeteer-extra-plugin-stealth` and constant updates to stay ahead.
**Infrastructure management.** You need to keep Chrome/Chromium updated, manage browser pools, handle crashes and timeouts, and set up retry logic. On serverless platforms, you need special Chrome builds (`chrome-aws-lambda`) because the standard binary is too large.
**Proxy management.** If you're scraping at any scale, you need rotating proxies to avoid IP bans. That means proxy providers, rotation logic, error handling for dead proxies, and billing management.
All of this to answer the question: "What's the HTML at this URL?"
## The API Approach: One Request, Done
The alternative is to let someone else manage the headless browsers, proxy infrastructure, and anti-bot bypasses, and just call an API. You send a URL, it renders the page in a real browser, deals with anti-bot protections, and hands back the complete HTML. One request, one response.
Here's what that looks like with [Context.dev Scrape with HTML output](https://docs.context.dev/api-reference/web-scraping/scrape):
### cURL
Copy
```bash
curl --request POST "https://api.context.dev/v1/web/scrape" \
--header "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
--header "Content-Type: application/json" \
--data '{"url":"https://example.com","formats":{"html":true}}'
```
An abbreviated response looks like this. Read `html.data` after checking `html.success`; other format envelopes and metadata are omitted here:
Copy
```json
{
"url": "https://example.com",
"html": {
"requested": true,
"success": true,
"data": "..."
}
}
```
The `url` field returns the resolved URL after any redirects, so you always know where the content actually came from.
### Python
Using the Context.dev Python SDK:
Copy
```python
from context.dev import ContextDev
client = ContextDev(api_key="YOUR_API_KEY")
response = client.web.scrape(
url="https://example.com",
formats={"html": True},
)
if response.html.data is None:
raise RuntimeError("HTML extraction failed")
print(response.html.data)
```
The SDK sends the same unified Scrape request and exposes the rendered document at `response.html.data`.
### JavaScript
Copy
```javascript
import ContextDev from 'context.dev';
const client = new ContextDev({ apiKey: 'YOUR_API_KEY' });
const response = await client.web.scrape({
url: 'https://example.com',
formats: { html: true },
});
if (response.html.data === null) throw new Error('HTML extraction failed');
console.log(response.html.data);
```
### TypeScript
Copy
```typescript
import ContextDev from 'context.dev';
const client = new ContextDev({ apiKey: 'YOUR_API_KEY' });
const scrapeHtml = async (url: string): Promise => {
const response = await client.web.scrape({ url, formats: { html: true } });
if (response.html.data === null) {
throw new Error(`HTML extraction failed for ${url}`);
}
return response.html.data;
};
const html = await scrapeHtml('https://example.com');
```
### Ruby
Copy
```ruby
require "context_dev"
client = ContextDev::Client.new(api_key: "YOUR_API_KEY")
response = client.web.scrape(
url: "https://example.com",
formats: {html: true},
)
raise "HTML extraction failed" if response.html.data.nil?
puts response.html.data
```
### Using Axios (No SDK)
If you prefer not to use an SDK, the REST API works with any HTTP client:
Copy
```typescript
import axios from 'axios';
const scrapeHtml = async (url: string) => {
const response = await axios.post(
'https://api.context.dev/v1/web/scrape',
{ url, formats: { html: true } },
{ headers: { Authorization: 'Bearer YOUR_API_KEY' }, timeout: 60000 },
);
if (!response.data.html.success) throw new Error('HTML extraction failed');
return response.data.html.data;
};
```
## What Happens Behind the Scenes
When you request `formats.html` from Scrape, Context.dev handles the rendering pipeline.
The URL gets loaded in a headless browser that executes all JavaScript and waits for the DOM to stabilize, so you get the rendered HTML, not the pre-render source. If a request is blocked, the system escalates through proxy tiers (datacenter, residential, mobile) until it gets through. All redirects are followed, and the final resolved URL comes back alongside the HTML so you know what page was actually rendered. Browser fingerprinting, TLS fingerprint matching, and session management are all automatic. No stealth plugins to configure on your end.
The result is the same HTML you'd see if you opened Chrome DevTools and copied the `` element, delivered in the JSON response to `POST /web/scrape`.
## Common Use Cases for Raw HTML Extraction
### Custom Parsing Pipelines
If you're building a scraper for a specific data format (say, extracting structured product data from e-commerce sites or pulling financial data from SEC filings), you need the raw HTML to run your own parsing logic. CSS selectors, XPath queries, and regex patterns all operate on HTML. An API that returns clean HTML lets you skip the browser infrastructure and focus on the parsing.
### DOM Diffing and Change Detection
Monitoring websites for changes (price drops, content updates, new job postings) requires comparing HTML snapshots over time. You need consistent, fully-rendered HTML for diffs to be meaningful. If your scraper sometimes returns JavaScript-rendered content and sometimes returns the raw shell, your diffs are useless.
### SEO Auditing and Analysis
SEO tools need to see what search engines see. That means fully rendered HTML with all meta tags, Open Graph data, structured data (JSON-LD), and canonical URLs intact. Extracting raw HTML is the first step in any programmatic SEO audit.
### Archiving and Compliance
Some industries require archiving web content for compliance. Financial services, legal, and healthcare organizations often need to store snapshots of web pages as they appeared at a specific point in time. Raw HTML is the most complete representation.
### Feeding Data into AI Pipelines
While [Markdown conversion](https://www.context.dev/data/web-scrape-markdown-api) is often better for LLM consumption, there are cases where you want the raw HTML: for example, when training models on web layout structure, building accessibility tools, or extracting data from HTML tables that lose structure in Markdown conversion.
## Going Beyond HTML: Other Scrape Formats and Map URLs
Once you're extracting HTML from URLs, you'll probably need related capabilities too. Select additional `formats` in the same Scrape request, or use Map URLs to discover pages first:
### URL to Markdown
If your goal is to feed web content into an LLM or RAG pipeline, raw HTML is usually more than you need. The [Markdown format](https://docs.context.dev/api-reference/web-scraping/scrape) returns GitHub Flavored Markdown. Use `sharedParams.mainContentOnly` to focus on the page body:
Copy
```python
response = client.web.scrape(
url="https://example.com/blog/post",
formats={"markdown": True},
shared_params={"main_content_only": True},
markdown_params={"include_links": True, "include_images": False},
)
if response.markdown.data is None:
raise RuntimeError("Markdown extraction failed")
print(response.markdown.data)
```
You can control whether links and images are preserved, useful for RAG pipelines where you want text-only content, versus agent workflows where links matter.
### Image Extraction
The [images format](https://docs.context.dev/guides/extract-page-images) returns page images in `images.data`, with `url` and `alt` fields. Optional enrichment can add dimensions, classification, and hosted file URLs:
Copy
```python
response = client.web.scrape(
url="https://example.com",
formats={"images": True},
)
if response.images.data is None:
raise RuntimeError("Image extraction failed")
for image in response.images.data or []:
print(image.url, image.alt)
```
### URL Discovery
Before scraping individual pages, you often need to know what pages exist. [Map URLs](https://docs.context.dev/api-reference/web-scraping/map) returns indexed URLs as objects with a `url` and optional page metadata. Bound the list with `maxLinks`:
Copy
```python
site = client.web.map_urls(domain="example.com", max_links=20)
for entry in site.urls:
page = client.web.scrape(url=entry.url, formats={"html": True})
if page.html.data is not None:
print(entry.url, page.html.data)
```
### Schema-Based Data Extraction
If you're extracting HTML just to parse specific fields, the [Scrape JSON format](https://docs.context.dev/api-reference/web-scraping/scrape) can return fields from that page using `jsonParams.schema`:
Copy
```python
response = client.web.scrape(
url="https://example.com/pricing",
formats={"json": True},
json_params={
"schema": {
"type": "object",
"properties": {"pricing": {"type": ["string", "null"]}},
"required": ["pricing"],
},
"instructions": "Extract the pricing shown on this page; use null if absent.",
},
)
if response.json_.data is None:
raise RuntimeError("JSON extraction failed")
print(response.json_.data)
```
Validate `response.json_.data` in Python before using it. JSON output adds 4 credits to the base Scrape charge when extraction succeeds. For research across multiple pages, use Answers with a task and an example `json_format` object.
## DIY vs. API: When to Use Which
Not every use case needs an API. Here's how I'd think about it:
**Use a simple HTTP request** (`requests.get` / `fetch`) when:
- The target page is server-rendered (no JavaScript frameworks)
- There's no bot protection
- You're scraping a small number of pages infrequently
- You control the target site
**Use a headless browser (Puppeteer/Playwright)** when:
- You need to interact with the page (click, scroll, fill forms)
- You're building a browser automation tool, not a data pipeline
- You're scraping your own application for testing
**Use an HTML extraction API** when:
- You need rendered HTML from JavaScript-heavy sites
- Target sites have anti-bot protections
- You're scraping at scale (dozens to thousands of URLs)
- You don't want to manage browser infrastructure
- Reliability matters. You need consistent results, not "works most of the time"
For most production use cases, the API wins on both developer time and operational cost. Running headless Chrome at scale gets expensive fast, and not just in compute. It's the engineering hours spent keeping it working that really add up.
## Getting Started
Context.dev offers SDKs for [TypeScript](https://docs.context.dev/quickstart), [Python](https://docs.context.dev/quickstart), [Ruby](https://docs.context.dev/quickstart), and [Go](https://pkg.go.dev/github.com/context-dot-dev/context-go-sdk/v2), plus a REST API that works with any HTTP client.
To start extracting HTML:
1. [Sign up for a free account](https://www.context.dev/signup)
2. Grab your API key from the dashboard
3. Make your first request:
Copy
```bash
curl --request POST "https://api.context.dev/v1/web/scrape" \
--header "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
--header "Content-Type: application/json" \
--data '{"url":"https://example.com","formats":{"html":true}}'
```
The response includes rendered HTML in `html.data`, the resolved `url`, and a per-format `html.success` flag. No browsers to manage, no proxies to configure, no stealth plugins to maintain. Just the HTML.
## Frequently Asked Questions
### Does the API execute JavaScript?
Yes. `formats.html` returns the rendered DOM. Set `sharedParams.waitFor` or browser actions when content needs more time or interaction before capture. For the original HTTP response instead, request `formats.bytes`.
### How does it handle pages that block scrapers?
Context.dev uses automatic proxy escalation and browser fingerprinting when a request is blocked. No universal success percentage is claimed without a dated benchmark and target definition.
### What's the rate limit?
Rate limits depend on your plan. The API returns standard HTTP 429 responses when limits are hit, with retry-after headers. Check the [pricing page](https://www.context.dev/pricing) for details.
### Can I scrape pages behind a login?
Scrape works with publicly accessible URLs. For pages behind authentication, you'd need to pass session cookies or use a headless browser with login automation.
### What's the difference between HTML and Markdown output?
Both are formats on `POST /web/scrape`. HTML preserves the rendered DOM; Markdown converts page content into readable text and links. Request both with `formats: { html: true, markdown: true }` and read `html.data` and `markdown.data`. Shared content filters apply to both.
### Is there a maximum page size?
Standard web pages work fine. Extremely large pages (multi-megabyte DOMs) may time out, but you can set a custom timeout up to 300 seconds (5 minutes) with `timeoutOpts.milliseconds`.
## Related articles
- [
### How Do Scraping APIs Handle JavaScript-Rendered Content?
Learn how scraping APIs execute JavaScript, wait for dynamic content, and turn rendered pages into HTML, Markdown, or structured data for AI pipelines.
Jul 20, 2026
](/blog/how-scraping-apis-handle-javascript-rendered-content)
- [
### How to Scrape a Company's Address from the Web
Complete guide to scraping company addresses from websites using TypeScript. Learn DOM parsing, regex patterns, and data validation techniques.
Jun 11, 2025
](/blog/how-to-scrape-a-company-s-address-from-the-web)
- [
### MCP Server vs REST API for Web Scraping: When to Use Each
Compare MCP servers and REST APIs for web scraping across setup, tool discovery, authentication, observability, batching, retries, latency, and portability, plus architecture examples for agent, pipeline, and hybrid deployments.
Oct 3, 2026
](/blog/mcp-server-vs-rest-api-for-web-scraping)
- [
### Browser Fingerprinting Explained: Headless Browsers and Anti-Detect Profiles
Learn how browser fingerprinting exposes automation, what anti-detect profiles solve, and when a managed scraping API fits your workflow.
Sep 27, 2026
](/blog/browser-fingerprinting-explained)
## Ship an agent that actually knows things.
Free tier, 10-minute integration, and the same API powering agents at Mintlify, daily.dev, and Propane. No credit card to start.
[
Get API Access
](/signup)[
Book a call
](https://cal.com/context-dev/30min?overlayCalendar=true&duration=20)