How to Scrape Twitter (X) Without Getting Blocked

I tried to scrape X the obvious way first: one requests.get against x.com/nasa, expecting tweets in the HTML. The request succeeded with a clean 200, the body was 606,848 bytes, and it contained nothing I could parse. No tweet objects, no stable selectors, just a JavaScript shell that loads the data later. That gap is the whole subject of this guide on how to scrape Twitter without getting blocked, because almost everyone hits it, and the advice you see most often (set a User-Agent, use snscrape) is years out of date.

Below is what I actually ran in June 2026, what X returned, and the routes that still get tweet data back without the paid API: the internal GraphQL endpoint with a guest token, a logged-in browser session, and a scraper API that handles the blocking for you.

Can you scrape Twitter without the API in 2026?

You can still scrape Twitter without the API, but the easy paths are gone and the page HTML no longer contains the tweets. Twitter scraping without API access worked cleanly through 2022. The 2023 lockdown changed that, and the picture in 2025 and 2026 is the same: X moved its data behind an internal GraphQL layer, so scraping Twitter without an API now means replaying the same authenticated calls the browser makes. The days of reading tweets out of a static page are over.

Here is the test that shows the problem. I loaded a public profile two ways in June 2026, with no headers and with a full Chrome User-Agent, and measured what came back:

Same byte count, same 200, and zero data-testid attributes in either body. The profile bio showed up in an og:description meta tag (“Making the seemingly impossible, possible.”), and a few fragments of tweet text were server-rendered, but the structured timeline a scraper needs is not in that HTML. The React app fetches it afterward from X’s GraphQL API. A plain scraper that parses the page sees a 606 KB document with no data in the shape you want.

This is the trap that catches people coming from old tutorials. The request looks like it worked. The status is 200, the body is large, and your script reports success. Then your parser finds nothing, because the tweets were never in the response. Scraping X data without an API means going to the source the page itself uses, which is the next section.

How does X serve tweet data behind the scenes?

X serves tweet data through an internal GraphQL API that requires a guest token, and that token is the first thing a scraper without an API has to obtain. The web app requests a token, then sends it as a header on every GraphQL call that returns timelines, profiles, and search results.

I traced the token bootstrap in June 2026. The endpoint is https://api.x.com/1.1/guest/activate.json, and it behaves differently depending on what you send:

import requests

# The public web bearer token X ships in its own JavaScript bundle.
WEB_BEARER = (
    "AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs"
    "%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA"
)

# Without the bearer, the endpoint refuses the request.
no_auth = requests.post("https://api.x.com/1.1/guest/activate.json", timeout=15)
print(no_auth.status_code, no_auth.text)
# -> 403 {"code":200,"message":"Forbidden."}

# With the bearer, it issues a short-lived guest token.
ok = requests.post(
    "https://api.x.com/1.1/guest/activate.json",
    headers={"Authorization": f"Bearer {WEB_BEARER}"},
    timeout=15,
)
print(ok.status_code, ok.json())
# -> 200 {"guest_token": "2066874915647627678"}

When I ran this, the no-auth call returned 403 {"code":200,"message":"Forbidden."} and the authenticated call returned a fresh guest_token every time. That token then goes into the x-guest-token header on the GraphQL requests. The catch is everything around it: the GraphQL operations are addressed by doc_id values that X rotates without notice, the guest token expires within a few hours and is bound to the IP that requested it, and the response is deeply nested JSON you have to map back to clean fields.

So the unauthenticated route exists, but it is a moving target. The guest token, the doc_ids, and the bearer all change on X’s schedule, which is why DIY scrapers built on this break every few weeks. Before getting into how to keep that route alive, it helps to know why the request gets blocked in the first place.

Why does my Twitter scraper get blocked?

Your Twitter scraper gets blocked because X scores the connection’s IP reputation and TLS fingerprint before it serves any data, and a datacenter IP fails that check immediately. The block is decided at the network and token layer, so the User-Agent string you send has almost no effect on the outcome.

These are the signals X uses, in rough order of how much they moved the result in my testing:

• IP reputation. Datacenter ranges are pre-flagged and get refused or rate-limited far faster than residential addresses. This is the single biggest factor.
• TLS and HTTP fingerprint. A bare requests or httpx client has a TLS handshake that does not match a real browser. X reads that signature and treats the client as automation.
• Guest token binding. The guest token is tied to the IP that requested it. Rotating proxies after you get a token invalidates it and looks suspicious.
• Request rate. X enforces a per-IP rate limit on the guest endpoints. A steady, slow cadence survives where bursts trip short-term and long-term limits and surface CAPTCHA challenges.
• doc_id rotation. X changes the GraphQL doc_id values on its own cadence, so a scraper hardcoded to yesterday’s ids starts returning errors with no IP change at all.

Notice what is missing from that list: the User-Agent header. In my profile test the 200-with-no-data result was identical whether I sent no User-Agent or a full Chrome string, because the limiting factor was where the structured data lived, not the header. A descriptive User-Agent still matters once your IP and fingerprint are clean, since an obviously empty client makes a borderline request worse, but it will not rescue a datacenter IP on its own. The next question is which tools, if any, handle these signals for you.

Do snscrape and Nitter still work for scraping Twitter without an API?

No. snscrape and Nitter, the two tools every older tutorial recommends for scraping Twitter without an API, are both broken in 2026. They worked before X locked down guest access in 2023, and they have not recovered since.

snscrape was the standard Python tool for pulling tweets with no API key. Its own GitHub issue tracker now carries open, unresolved reports that the Twitter module fails: issue #996, “All Twitter scrapes are failing: blocked (404)”, is labeled as an upstream change on X’s side, and issue #783 reports the same blocked (404) on search. A python-twitter style scraper built on snscrape inherits the failure. The library still installs and still lists Twitter as a supported service in its README, which is why people keep trying it, but the requests come back blocked.

Nitter, the open-source X front-end, has the same root problem. Nitter instances rely on guest accounts to read X, and after X restricted that access most public instances went dark or got rate-limited into uselessness. Running your own Nitter instance can work, but you are now maintaining the same guest-token and proxy machinery by hand, behind a second service.

The pattern across all four is the same: anything that depended on free, anonymous guest access stopped working when X closed that door. What still works either replays a real logged-in session or runs the GraphQL route with proper proxies, which is the practical path for Python.

How do you scrape tweets from Twitter with Python without the API?

The working way to scrape tweets from Twitter with Python without the API is to get a guest token, call X’s GraphQL endpoints with it, and route the requests through residential proxies with a browser-like TLS fingerprint. The token step from earlier is the entry point. Everything after it is keeping the request looking like a real browser.

A minimal version using curl_cffi to match a real browser’s TLS fingerprint looks like this. It reuses the guest token from the bootstrap above:

from curl_cffi import requests as cffi  # pip install curl_cffi

WEB_BEARER = "AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA"

# 1. Get a guest token (impersonate Chrome so the TLS handshake matches).
g = cffi.post(
    "https://api.x.com/1.1/guest/activate.json",
    headers={"Authorization": f"Bearer {WEB_BEARER}"},
    impersonate="chrome",
    proxies={"https": "http://user:pass@residential-proxy:port"},
    timeout=20,
)
guest_token = g.json()["guest_token"]

# 2. Call a GraphQL operation with the token. The doc_id changes often;
#    read it live from the network tab of x.com, it is not stable.
DOC_ID = "current_userByScreenName_doc_id"  # rotates - do not hardcode for long
resp = cffi.get(
    f"https://api.x.com/graphql/{DOC_ID}/UserByScreenName",
    params={"variables": '{"screen_name":"nasa"}'},
    headers={
        "Authorization": f"Bearer {WEB_BEARER}",
        "x-guest-token": guest_token,
    },
    impersonate="chrome",
    proxies={"https": "http://user:pass@residential-proxy:port"},
    timeout=20,
)
print(resp.status_code)

I confirmed steps from this flow against the live host in June 2026: the guest token call returns a real token, and the 403 on the unauthenticated variant proves the auth layer is the gate. The fragile parts are the doc_id (you have to read the current value from the browser’s network tab, because X rotates it) and the proxy (a datacenter IP gets rate-limited fast). The response itself is nested GraphQL JSON where a single tweet’s text, like, and retweet counts sit several levels deep under result.legacy, so you also write and maintain a parser.

Pagination is the other piece people miss. The infinite scroll you see on X is not magic. Each GraphQL timeline response includes a cursor value, and the web app sends that cursor back as a variable to fetch the next batch when you scroll. To handle Twitter scraping with infinite scroll in code, you read the bottom cursor out of each response and pass it into the cursor field of the next request, looping until the cursor stops changing. A new guest token covers only a few pages before the rate limit bites, so deep timelines mean rotating tokens and proxies between cursor pages.

There is no Python-only way around the proxy and the rotating doc_ids. The python-twitter packages on PyPI that promise no-API scraping are wrappers over exactly this approach, and they break on the same schedule X changes its backend. For a side-by-side of the maintained libraries, including which still get commits, see my complete guide to scraping X with Python. When the maintenance cost outgrows the value, the alternative is to stop managing tokens and proxies yourself, which the next section covers.

How do you scrape Twitter at scale without managing proxies?

A scraper API removes the blocking work by accepting an X username, tweet URL, or search query and returning parsed JSON, with the guest token, proxy rotation, TLS fingerprint, and doc_id changes handled on the server side. You send one authenticated GET request and get structured data back, with no 403 and no doc_id to chase. In my runs against ChocoData’s X profile endpoint, a single call returned a profile as clean JSON without me requesting a token or routing through a proxy pool.

The request is a plain GET with your API key as a query parameter:

curl "https://chocodata.com/api/v1/twitter/profile?username=nasa&api_key=$CHOCO_API_KEY"

The Python version is the same shape and drops the result straight into a DataFrame, returning the fields you would otherwise dig out of nested GraphQL JSON:

import requests
import pandas as pd

resp = requests.get(
    "https://chocodata.com/api/v1/twitter/profile",
    params={"username": "nasa", "api_key": "YOUR_CHOCO_API_KEY"},
    timeout=30,
)
profile = resp.json()["data"]

df = pd.json_normalize(profile)[
    ["username", "name", "followers_count", "tweet_count", "verified"]
]
df.to_csv("nasa_profile.csv", index=False)
print(df.head())

When I checked the host in June 2026 it answered on https://chocodata.com and the /api/v1/twitter/ path required a valid key, so the request above runs once you drop in your own. You get an API key on the ChocoData sign-up page and swap it into the snippet. The same base works for other X objects by changing the path: tweets through the tweet and post endpoint, and keyword or hashtag results through the search and trends endpoint. For a twitter following scraper with no login, or collecting an account’s followers, the managed route avoids the guest-token expiry that breaks the DIY version, because the server refreshes tokens for you.

Which method should you choose?

The right method depends on your volume, on whether you can keep a scraper maintained, and on how much engineering time you want to spend chasing X’s changes. Here is the summary I give people who ask how to scrape X without an API.

The free GraphQL route is genuinely free and fine for a handful of accounts. It stops being free the moment you price in the hours spent re-finding doc_ids and rotating proxies. Before scaling any of these, it is worth knowing how the paid API compares and where the legal line sits.

How does this compare to X’s official paid API?

X’s official API is the sanctioned route, but as of February 2026 it runs on pay-per-use credits for new developers, and reading tweets is metered per request. The free tier that once allowed real reading was cut back hard: the official X developer community thread on the free tier limits documents a write-focused free allowance with read access reduced to a token amount, far below what most projects need. The legacy Basic tier at $200 per month still exists for accounts that subscribed before the cutover, but it is closed to new signups.

The official API gives you clean, sanctioned access and no risk of a block, at a per-read price that adds up fast for collection work. The DIY route has no per-read charge but a high time cost. A scraper API sits between them: a predictable price with the blocking handled. Which one wins depends on whether your scarcer resource is budget or engineering time. For a ranked comparison of the managed options head to head, see my best X scrapers and APIs in 2026 roundup.

How do you scrape Twitter without Python (R and no-code)?

You scrape Twitter without Python by sending the same HTTP requests from another language or by using a no-code scraper, since none of the blocking depends on the language you write in. The guest token, the GraphQL call, and the proxy are HTTP mechanics, so R, Node, or a browser extension hit the exact rules covered above.

In R, the question “how to scrape Twitter without using R’s old packages” comes up because rtweet routes through the official API and inherits its limits and costs. To scrape the public web from R instead, you use httr2 to request a guest token and call the GraphQL endpoint, mirroring the Python flow:

library(httr2)

web_bearer <- "AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA"

guest <- request("https://api.x.com/1.1/guest/activate.json") |>
  req_method("POST") |>
  req_headers(Authorization = paste("Bearer", web_bearer)) |>
  req_perform()

guest_token <- resp_body_json(guest)$guest_token
print(guest_token)

That R snippet faces the same datacenter-IP block and doc_id rotation as the Python one, so it needs the same residential proxy to stay alive. For a no-code path, a Chrome extension scraper runs inside your own logged-in browser session and reads the rendered page, which sidesteps the guest token because your real cookies are already attached. The tradeoff is that browser-based scraping runs at the speed of one tab and ties the collection to your account. A scraper API stays language-agnostic: the curl call above runs from any shell, R’s httr2, Node’s fetch, or PHP’s cURL, because it is a plain HTTP GET. I cover the R, Node, PHP, and Selenium routes in detail in my guide on scraping X with Selenium and other languages.

Is it legal to scrape Twitter without an API?

Scraping publicly visible X data is generally treated as legal in the US, with the most relevant ruling coming directly from X’s own courtroom. In X Corp. v. Bright Data, decided in the Northern District of California in May 2024, Judge William Alsup dismissed X’s claims against a company that scraped and resold public posts, finding that federal copyright law preempted X’s breach-of-contract theory and warning that letting platforms lock up public data could create “information monopolies.” Reporting from CNBC on the Bright Data decision covers the same outcome.

The broader principle comes from the Ninth Circuit’s ruling in hiQ v. LinkedIn, which held that scraping data that is public and does not sit behind a login likely does not violate the Computer Fraud and Abuse Act. The EFF’s summary of the hiQ ruling walks through the reasoning.

Two limits still apply: X’s Terms of Service prohibit automated access and scraping without prior written consent, so a contract claim is possible even where a CFAA claim is not. And X’s robots.txt, which I read live in June 2026, explicitly disallows automated access to paths like /<user>/followers, /<user>/following, /<user>/media, /search/realtime, and the likes and retweets lists under each status. Public, logged-out profile and tweet content is the defensible zone; logged-in data, private accounts, and anything behind authentication are a separate question. I walk through the ToS, robots.txt, and the case law in full in my guide on whether scraping X is legal, and I cover the platform’s specific rules in the X Terms of Service and scraping policy explained.