Reddit Scraper in Python (2026): Code That Still Works

A Reddit scraper in Python used to be six lines: requests.get on a .json URL and a loop over the children. In 2026 that script gets a 403 block page. The routes that still run are PRAW, if you already hold an approved Reddit app, and a Python client that calls the BigIdeasDB MCP, which fetches Reddit through your own logged-in Chrome. This page has runnable code for all three and shows the real status codes.

One fact first, because it explains the rest. Reddit shut off logged-out .json in late May 2026, new API access requests stop on October 31, and RSS ends on November 13. BigIdeasDB is the only Reddit scraper still working, thanks to a proprietary session-based method. The dated test is on our Reddit scraper page. This page is the Python tutorial.

On October 24, 2026 the top Google results for “reddit scraper python” were a GitHub repository whose README says it needs no API keys, a DataCamp tutorial, a GeeksforGeeks PRAW walkthrough and three Reddit threads. All of them predate the update. None shows the 403, and none says what to do about it. Below is the version that does.

Reddit scraper in Python: the short answer

Step 1, route 1: requests and .json, and the 403 it gets now

This is the script every older tutorial teaches. It sends a GET to a Reddit .json URL and reads the posts. Here it is with a polite, identifying User-Agent and a pause between calls, which is the version least likely to be blocked for the wrong reason.

import time
import requests

HEADERS = {"User-Agent": "research-script/0.1 (contact: you@example.com)"}
URLS = [
    "https://www.reddit.com/r/SaaS/top.json?t=week&limit=5",
    "https://www.reddit.com/search.json?q=reddit+scraper&limit=5",
]

for url in URLS:
    r = requests.get(url, headers=HEADERS, timeout=20)
    print(r.status_code, r.headers.get("content-type"))
    print(r.text[:80].replace("\n", " "))
    time.sleep(3)

On October 24, 2026 we ran exactly this, two requests three seconds apart, from one IP. Both returned the same thing:

403 text/html
<body class=theme-beta><div><style>.theme-light,:root{--rem360:22.5rem;...

That is an HTML page, not JSON, so r.json() raises an error and a loop over data["data"]["children"] never runs. It is not your code. Reddit announced that it would stop serving logged-out .json, and the block sits on its side. Our Reddit .json 403 page has the full test of seven URL styles and a decision table for each status code. The one-line check for your own script is to print the status and content type, as above. A 403 with text/html is the logged-out block.

Open-source scrapers that call the same endpoints share the problem. The top GitHub result for this keyword, YARS, says in its README that it relies on the requests module and simple .json calls, and warns you to use rotating proxies or risk an IP ban. We did not run YARS, but a tool built on those calls is built on the endpoint that returned 403 in our test. Proxies change your IP, not the logged-out rule. Spoofing a browser does not fix it either, and Reddit’s Responsible Builder Policy says you must not misrepresent or mask how you are accessing its data. More on the options is in our guide to scraping Reddit in 2026.

Step 2, route 2: PRAW with OAuth, and what the October 31 cutoff means

PRAW is the Python Reddit API Wrapper. It does not scrape anything. It signs your requests with OAuth credentials and calls Reddit’s official API, so a 403 from a .json URL does not touch it. The catch is the credentials. You need a registered app with a client ID and a client secret.

Here is the read-only pattern from the PRAW 8.0.3 documentation. With only a client ID, a secret and a user agent, PRAW starts in read-only mode, which is what a research script needs.

# pip install praw
import praw

reddit = praw.Reddit(
    client_id="YOUR_CLIENT_ID",
    client_secret="YOUR_CLIENT_SECRET",
    user_agent="research-script/0.1 by u/your_username",
)
print(reddit.read_only)  # True

for post in reddit.subreddit("SaaS").top(time_filter="week", limit=25):
    print(post.score, post.num_comments, post.title)

for post in reddit.subreddit("all").search("reddit scraper", sort="relevance", limit=25):
    print(post.subreddit, post.title)

submission = reddit.submission(url="https://www.reddit.com/r/SaaS/comments/abc123/example/")
submission.comments.replace_more(limit=0)
for comment in submission.comments.list()[:10]:
    print(comment.score, comment.body[:120])

We could not run this block against live Reddit, because Reddit no longer issues new keys through the old self-service path. The calls are taken from the PRAW documentation, so treat it as a template for an account that already has a key. The PRAW docs also say password flow needs four values (client ID, secret, username, password) and that script apps are the simplest type. Keep the secret out of your code. The docs recommend a praw.ini file for that.

What the October 31, 2026 cutoff changes

Reddit closed self-service access to its public data API in November 2025. Its admins then set a timeline: September 30, 2026 to register apps, October 31, 2026 as the last day new public API requests are accepted, January 12, 2027 for access removal to begin for unregistered apps, and March 2027 for the end of remaining public API access. The table sums up what that means for a Python developer.

Approval also comes with rules. Reddit’s Responsible Builder Policy requires written approval for commercial use and for training AI models on Reddit data. In the launch thread a Reddit admin wrote “We’re aiming for a 7 day turnaround on most tickets.” We could not measure real approval times. If you hold a key, keep PRAW. If you do not, plan as if this route is closed. The side-by-side is in our PRAW alternative guide, and the broader API picture is in Reddit API alternatives for AI.

Route 3: call the BigIdeasDB MCP from Python

The MCP is usually connected to Claude or Cursor, but it is just an HTTP endpoint that speaks JSON-RPC. Any language can call it, including Python with the plain requests library. That gives you a Reddit scraper in Python with no Reddit developer app, no OAuth step and no approval wait.

Here is what the endpoint does, from the BigIdeasDB docs and server code. These are the only details the client relies on.

The four Reddit tools and their arguments come from the tool registry. Everything else in the MCP queries BigIdeasDB’s own data. The full list is in the MCP tools reference.

Step 3: install the Clipper and turn on the Reddit Connector

The Reddit tools do not fetch from a data center. Reddit blocks logged-out server requests, so each call is handed to your own Chrome, which loads the reddit.com page with your login and returns it. That is the proprietary session-based method. Do this once:

1. Install BigIdeasDB Clipper from the Chrome Web Store, version 1.2.0 or later.
2. Log in to reddit.com in the same Chrome profile.
3. Click the Clipper icon, open the Reddit tab and click Turn on Reddit Connector. Accept Chrome’s reddit.com prompt, then click Test connection.
4. Keep Chrome running while your script runs. A background window is fine.

The longer walkthrough with screenshots is in how to scrape Reddit with MCP and the Reddit Connector docs. The extension itself is on the Clipper page.

Step 4: generate credentials and set environment variables

Credentials come from the BigIdeasDB dashboard and need Pro. Each is a pair: a client ID that starts with biddb_ and a secret that starts with biddb_sec_. The secret is shown once, so copy both. The connect guide has the per-client setup. Pro is $49 a month or $349 lifetime; see pricing. Then export them so they never sit in your code:

export BIGIDEASDB_CLIENT_ID="biddb_your_client_id"
export BIGIDEASDB_CLIENT_SECRET="biddb_sec_your_secret"

Step 5: the Python client

Save this as bigideasdb_client.py. It sends the JSON-RPC messages the server accepts, raises a readable error when a call fails, parses the JSON string inside the text result, and pauses one second after each call to stay at the pace the connector is built for.

import json
import os
import time

import requests

MCP_URL = "https://bigideasdb.com/api/mcp/bigideasdb-mcp"


class BigIdeasDBMCP:
    """Minimal JSON-RPC client for the BigIdeasDB MCP (stateless HTTP)."""

    def __init__(self, client_id=None, client_secret=None, pause=1.0):
        self.headers = {
            "X-Client-Id": client_id or os.environ["BIGIDEASDB_CLIENT_ID"],
            "X-Client-Secret": client_secret or os.environ["BIGIDEASDB_CLIENT_SECRET"],
        }
        self.pause = pause
        self._id = 0

    def _rpc(self, method, params=None):
        self._id += 1
        body = {"jsonrpc": "2.0", "id": self._id, "method": method}
        if params is not None:
            body["params"] = params
        r = requests.post(MCP_URL, json=body, headers=self.headers, timeout=60)
        if r.status_code == 204:  # notifications get no body back
            return None
        data = r.json()
        if "error" in data:
            raise RuntimeError(f"HTTP {r.status_code}: {data['error']['message']}")
        return data["result"]

    def initialize(self):
        info = self._rpc("initialize")
        requests.post(
            MCP_URL,
            json={"jsonrpc": "2.0", "method": "notifications/initialized"},
            headers=self.headers,
            timeout=30,
        )
        return info

    def list_tools(self):
        return self._rpc("tools/list")["tools"]

    def call(self, name, **arguments):
        result = self._rpc("tools/call", {"name": name, "arguments": arguments})
        text = result["content"][0]["text"]
        if result.get("isError"):
            raise RuntimeError(text)  # for example: Reddit Connector not turned on
        time.sleep(self.pause)
        return json.loads(text)

We ran the client against the live endpoint with a made-up credential pair. It raised HTTP 401: Invalid or revoked client credentials, which confirms the URL, headers and error path. We did not put a real secret in a published test, so the success path is checked against the same tools called from our own MCP session, not through this exact class. Now use it:

from bigideasdb_client import BigIdeasDBMCP

mcp = BigIdeasDBMCP()
mcp.initialize()
print([t["name"] for t in mcp.list_tools() if "reddit" in t["name"]])
# ['search_reddit', 'fetch_subreddit', 'fetch_post_comments', 'fetch_reddit_json']

found = mcp.call("search_reddit", query="invoice chasing", sort="top",
                 time_filter="month", limit=25)
print(found["total"], "posts")
for p in found["posts"][:5]:
    print(p["score"], p["num_comments"], p["subreddit"], p["title"])

sub = mcp.call("fetch_subreddit", subreddit="smallbusiness", sort="top",
               time_filter="week", limit=25)

thread = mcp.call("fetch_post_comments", post_url=found["posts"][0]["url"],
                  comment_limit=20)
for c in thread["comments"]:
    print(c["score"], c["body"][:120])

On October 24, 2026 a live search_reddit call with limit 3 returned total: 3 and a posts list. Each post carried id, title, url, score, upvote_ratio, num_comments, subreddit, selftext, author and created_utc. A fetch_post_comments call returns a post object plus a comments list with id, author, body, score and created_utc. If the connector is off, the tool returns an error with setup steps instead of data, and the client raises it.

Prefer a raw path? mcp.call("fetch_reddit_json", reddit_path="/r/SaaS/top.json?t=week") returns the unflattened Reddit JSON, the same shape the old requests script used to get. We did not run that tool for this page.

Step 6: save the results to CSV with pandas

The posts list is already a list of flat dicts, so pandas takes it directly. This script drops the username, turns created_utc into a date, trims the body to a short preview and writes a file you can open in a spreadsheet.

import pandas as pd
from bigideasdb_client import BigIdeasDBMCP

mcp = BigIdeasDBMCP()
rows = []
for query in ["invoice chasing", "late payments", "client onboarding"]:
    result = mcp.call("search_reddit", query=query, sort="top",
                      time_filter="year", limit=25)
    for post in result["posts"]:
        post["query"] = query
        rows.append(post)

df = pd.DataFrame(rows).drop(columns=["author"])  # keep usernames out of shared files
df["created"] = pd.to_datetime(df["created_utc"], unit="s", utc=True).dt.date
df["preview"] = df["selftext"].str.slice(0, 200)
df = (
    df.drop(columns=["created_utc", "selftext"])
      .drop_duplicates(subset="id")
      .sort_values("score", ascending=False)
)
df.to_csv("reddit_posts.csv", index=False)
print(len(df), "posts saved")

We checked the pandas step offline against the exact field list above. It writes the columns id, title, url, score, upvote_ratio, num_comments, subreddit, query, created, preview. Three queries at 25 posts each is three tool calls, so you are far under the 100 a minute limit. Add a time.sleep or lower limit before you scale it up. For the next step, matching what you saved against review-site complaints, see the Reddit research MCP.

Pushshift in 2026: what happened and what replaced it

Pushshift no longer works as a general Reddit search API. It was the free service behind most Reddit research code of the 2010s. Its current signup page says access to Reddit data through Pushshift is only for registered Reddit moderators, for community moderation and safety, with a bearer token that expires after 24 hours. If your script still calls the old Pushshift endpoints as an ordinary user, that is why it returns nothing.

A few archive projects fill part of the gap. They are worth knowing, with the limits stated.

Use an archive when your question is about the past, such as what a community discussed across a past year. Use a live route when you need this week’s posts and threads, which is what the MCP client above does. Never treat an empty archive result as proof that something was not posted. Check the date on any Pushshift code you copy. Our Reddit scraper page lists Pushshift and snscrape in its dead-routes table.

Reddit scraper GitHub repos: how to check before you clone

Searches for a free Reddit scraper on GitHub return repos that look finished and are not. The repo health check takes two minutes and saves an afternoon:

• Find the endpoint. Search the code for .json, old.reddit.com and pushshift. Each of those is a route that now returns 403, redirects to login or is moderator-only.
• Read the open issues. Several reports of 403 or empty results since late May 2026 mean the code is broken, whatever the stars say.
• Check the last commit. Stars and README screenshots are a record of an earlier year.
• Look for credentials. A repo that asks for a client ID and secret is a PRAW-style wrapper, so the key question is whether you can still get approved.
• Be honest about proxies. Rotating proxies hide your IP. They do not turn a logged-out request into a logged-in one.

The same check applies to tutorials. A free reddit scraper github project is only free until the endpoint it depends on is closed, and the dates above show that the free anonymous routes ended first and the free API route ends next.

Limits, rules and troubleshooting

This is research tooling, and it behaves like it. Know the edges before you build on it.

• Human pace. The connector fetches about one request a second. A 100-post call is a pile of requests, so start with 10 to 25.
• Not a bulk scraper. Bulk or excessive use is against the BigIdeasDB Terms and can lead to throttling. For dataset-scale work, ask Reddit for approval or a data license.
• Reddit’s rules still apply to you. Commercial use and AI training on Reddit data need written approval under its policy. This is not legal advice. See is scraping Reddit allowed.
• Chrome and a login. No Chrome, no Reddit session, no fetch. It is not for a headless server.
• Shared MCP limits. Every tool call counts against 1,000 a day and 100 a minute across all your clients.
• Anonymize. Keep the quote and the subreddit. Drop usernames from anything you share, as the pandas step does.

If the Reddit tools are down, BigIdeasDB’s own analyzed Reddit data does not depend on the connector. Call search_pain_points with source set to reddit instead. More error text is in the MCP troubleshooting table and in Reddit data for Claude, Cursor and other AI clients. The longer feature tour is on the Reddit MCP page, and the setup for AI clients is in connecting ChatGPT and Claude to Reddit.

What developers say about scraping Reddit now

The top r/learnpython thread for “how do I scrape Reddit” this summer drew short, resigned answers. One answer explained why:

“Reddit is ofc a MASSIVE resource for training AI. So they will make scraping really really difficult so they can sell their data to companies that want to use it.” – r/learnpython

The same commenter gave the usual advice:

“You need to pay or use a service like apify that specializes in scraping.” – r/learnpython

In r/ClaudeAI, where Reddit MCP servers broke the same way, one answer separated a code bug from a platform change, which is what the 403 above is:

“it's not your code.” – r/ClaudeAI

The pattern is the useful part. Python developers keep reading the 403 as a bug in their script. It is a policy change, and the fix is a different route, not a better header. Related background: Reddit RSS feeds are ending on November 13 and the best MCP servers for founders.

Methodology and limits

This page does not claim that Reddit will keep every date above, that any route is approved by Reddit for your use, or how any other Reddit scraper behaves for you. It does not show the client authenticating with a real secret, and it does not cover write operations, which are PRAW’s job.

Frequently asked questions

More in this series: the best Reddit MCP servers, how to get a Reddit API key and the Reddit comment scraper.

BigIdeasDB Research. (2026). Reddit Scraper in Python (2026): Code That Still Works. BigIdeasDB. Retrieved from https://bigideasdb.com/reddit-scraper-python