The secret to scraping React apps without a headless browser

Scraping SPAs is annoying. BeautifulSoup sees an empty shell. Puppeteer and Selenium work, but they're slow, pricey, and heavy on memory.

You often don't need them. Many React/Next apps dump initial state into a <script> tag for hydration. Pull that JSON with requests and a little parsing.

The trick

Make a standard GET request:

import requests
 
url = "https://example.com/spa"
response = requests.get(url)
html_content = response.text

Now extract the JSON. SPAs usually put it inside a specific script tag. Regex finds it; a slice pulls out the payload.

import re
import json
 
def extractor(html_content: str) -> str:
    react_app_script = re.search(
        "<script>document.getElementById.*</script>", html_content
    )
    # This slicing depends on the specific site's JS structure
    raw_json = react_app_script.group().split("{", 2)[2].split("};window.initilizeAppWithHandoffState", 1)[0]
    return "{" + raw_json + "}"
 
react_data = json.loads(extractor(html_content))

If the data is behind a login, use a requests.Session() for cookies and auth, then fetch the page.

session = requests.Session()
session.post('https://example.com/login', data={'user': 'me', 'pass': 'secret'})
response = session.get('https://example.com/protected_data')

That's it. No headless browser required.

I built a Chrome extension to check whether a page has this hidden JSON before I write a scraper. It's on the Web Store.