The secret to scraping React apps without a headless browser

If you've ever tried to build a web-scraping project, you've probably run into issues with dynamically rendered content, common in things like single-page applications (SPAs). Powered by technologies like Next.js and React, these SPAs offer seamless user experiences but pose unique challenges for web scrapers.

Standard tools like BeautifulSoup just see the empty shell because they don't execute JS. The default fix is firing up a headless browser like Puppeteer or Selenium. That works, but it's slow, expensive, and uses a ton of memory.

I found a way to scrape these apps without a headless browser, using just Python, the requests library, and regex.

The Trick

When a React or Next.js app loads, it often ships the initial state data as a JSON blob right in the HTML <script> tags, so the frontend can hydrate without extra API calls. We just need to grab that JSON.

Make a standard GET request:

import requests
 
url = "https://example.com/spa"
response = requests.get(url)
html_content = response.text

Now, extract the JSON. SPAs usually put this inside a specific script tag. You can use regex to find it and slice out the JSON payload.

import re
import json
 
def extractor(html_content: str) -> str:
    react_app_script = re.search(
        "<script>document.getElementById.*</script>", html_content
    )
    # This slicing depends on the specific site's JS structure
    raw_json = react_app_script.group().split("{", 2)[2].split("};window.initilizeAppWithHandoffState", 1)[0]
    return "{" + raw_json + "}"
 
react_data = json.loads(extractor(html_content))

If the data is behind a login, just use a requests.Session() to handle cookies and auth, then fetch the page.

session = requests.Session()
session.post('https://example.com/login', data={'user': 'me', 'pass': 'secret'})
response = session.get('https://example.com/protected_data')

That's it. No headless browser required.

I actually built a Chrome extension to quickly check if a page has this hidden JSON data before I write a scraper. It's on the Web Store.