What Is Twitter (X) Scraping?

I run X data pipelines for a living, and the question I get asked most by people new to it is the simplest one: what is scraping on Twitter, actually? They want the plain definition and a clear picture of what is happening under the hood, ahead of the legal debate or the tooling. This article answers the definition first, then walks out from the definition into how X scrapers really work in 2026, what data they pull, why X fights it, and where the law lands.

Everything below is grounded in how I extract X data day to day, plus the primary sources: X’s own Terms of Service, the court ruling in X Corp v. Bright Data, and X’s published API pricing.

What is scraping on Twitter?

Scraping on Twitter, now called X, is the automated extraction of public data from the platform using software instead of manual copying. A scraper sends requests to X, reads what comes back, and saves the fields you care about (tweet text, author, timestamps, like counts) into a structured format like JSON or CSV. The same data a person could read one tweet at a time gets collected at volume by a program.

The phrase has a few interchangeable names. “Twitter data scraping,” “web scraping on Twitter,” “twitter-scrape,” and “data scraping on Twitter” all describe this one activity: pulling public X data with code. A piece of software built to do it is a twitter scraper.

The term covers a range of targets. People use “Twitter scraping” to mean pulling any public surface on X:

Scrapfly’s technical breakdown of X scraping lists the same scrapable set: “public tweets and profiles,” including “tweet text and full content,” engagement counts, “view count,” and “media URLs (images, videos).” Anything a logged-out or guest session can see is reachable by a scraper. Anything behind a login or a privacy setting is off limits.

The word “scraping” used to imply parsing raw HTML out of a page. On X that picture is outdated, and the difference matters for understanding everything that follows. The next section covers how a modern X scraper actually gets data.

How does Twitter scraping work technically?

A Twitter scraper works by replaying the same private API calls that x.com makes in your browser. It does not read the page HTML. X is a React application that loads minimal HTML and then fetches all its data through a private GraphQL backend. When you scroll your timeline, the web app fires GraphQL requests to X’s servers, and a scraper imitates those exact requests.

To do that, the scraper has to handle three moving parts that the browser handles invisibly:

1. Guest token. Every call to X’s GraphQL backend needs a guest token, a temporary credential that marks the request as coming from a session. Scrapfly notes these tokens “expire every 2-4 hours” and are “tied to your IP address.” When a token expires, the scraper stops until it fetches a fresh one.
2. doc_ids. X’s GraphQL queries are identified by doc_ids, operation identifiers that tell the backend which query to run (timeline, profile, search, and so on). They are undocumented and rotate. Scrapfly reports you “typically need to track 8-12 different IDs simultaneously” and that they “rotate every 2-4 weeks” with “no pattern since they’re essentially random identifiers.”
3. Rate and IP handling. X enforces roughly “300 requests per hour per IP address,” and datacenter IPs face “instant blocking” within “1-2 requests,” per the same source. So a scraper needs residential proxies and a steady request cadence to survive.

Put together, the loop a scraper runs looks like this:

This is why a Twitter scraper is a maintenance project. X “modifies these methods every few weeks to break reverse-engineered scrapers,” as Scrapfly puts it, so the guest-token routine and the doc_ids your code depends on are a moving target. Open-source projects track this constantly: the actively maintained twscrape library on GitHub exists specifically to keep up with X’s authorization changes across search, profile, and tweet endpoints, and its commit history reads like a log of X breaking things.

Because the GraphQL plumbing shifts so often, many teams skip building it and send the target to a managed scraper API instead, which I cover further down. First, the distinction that trips up most newcomers: scraping versus the official API.

What is the difference between a Twitter scraper and the Twitter API?

A Twitter scraper and the official X API both return X data through different doors with different costs. The X API is a paid, authenticated service that X sells, with documented endpoints and read caps per pricing tier. A Twitter scraper is third-party software that reads public data through the same private GraphQL backend the website uses, without paying X for an API key.

The gap is mostly about price and limits. X’s official API got dramatically more expensive after 2023. As of early 2026, X moved to pay-per-use pricing at $0.005 per post read, capped at 2 million reads per month, and discontinued the old free tier. The legacy fixed tiers that survive for existing subscribers run $200/month (Basic) and $5,000/month (Pro), with Enterprise starting around $42,000/month.

For a hobby project that reads a few thousand posts a month, pay-per-use API access is cheap and clean. For volume that the API prices out, or data the tiers gate behind Enterprise, scraping the public GraphQL backend is the route people take. That economic gap is what makes a Twitter scraper API attractive: it returns scraped public data on tap with no per-read API bill and no token maintenance. Which raises the obvious next question: if the data is public, why does X push back so hard?

Why is data scraping bad for Twitter?

X argues data scraping is bad for two reasons it has stated publicly: server load and lost revenue. High-volume scrapers send huge numbers of requests, which X says taxes its servers and degrades the experience for real users. And scraped public data can train AI models or feed ad targeting, value X would rather capture by selling API access itself.

The server-load argument went public in July 2023. X temporarily capped how many posts users could read per day, and Elon Musk framed it as an emergency measure against scraping. ITV’s report records the original limits: verified accounts capped at 6,000 posts per day, unverified at 600, and new unverified at 300, raised repeatedly over the following hours. Musk said the move addressed “extreme levels of data scraping & system manipulation.”

Days later, X put the argument in court. It sued four unnamed defendants in Dallas County, identified only by IP address, seeking more than $1 million in damages. The complaint claimed their requests “severely taxed X Corp.’s servers and impaired the user experience for millions of X Corp.’s customers.” The stated harm in both episodes was infrastructure strain plus the commercial value of data leaving for free.

There is a strong counter-argument, and a court has accepted it. The “bad for X” framing rests on X treating public posts as its property, and the ruling in the next section rejected exactly that premise. Whether scraping is “bad” depends on whose interest you weigh: X’s server bill and business model, or the public’s access to public information.

Is scraping Twitter data legal?

Scraping public Twitter data is broadly lawful in the United States, and X’s Terms of Service still prohibit it. Both statements are true at once, and the distinction between “lawful” and “permitted by X” is the whole story.

The legal anchor is X Corp v. Bright Data. In May 2024, the US District Court for the Northern District of California dismissed X’s claims against the data provider Bright Data. Judge William Alsup held that X failed to state a claim over access to its public site, and that any claim based on copying public posts was preempted by the Copyright Act. The court reasoned that the rights to the content belong to X’s users themselves, and warned that letting platforms lock down public data could create “information monopolies.” Morrison Foerster’s analysis walks through the preemption reasoning in detail.

X’s Terms of Service say the opposite about permission. The current terms state that “crawling or scraping the Services in any form, for any purpose without our prior written consent is expressly prohibited”. X removed the older clause that allowed crawling under robots.txt rules. The 2025 and 2026 terms also set liquidated damages at $15,000 per 1,000,000 posts accessed in any 24-hour period.

So the practical reading is:

Treat this as a plain-language summary for orientation, since it is not legal advice and the line shifts with jurisdiction and use case. I go deeper into the case law and the contract terms in my guide on whether scraping Twitter is legal, and in the breakdown of the X Terms of Service scraping policy. The posture most teams adopt: collect only public data, stay off private content, and keep request volume reasonable.

How do you scrape Twitter without building all of this yourself?

The way to scrape Twitter without managing guest tokens, doc_ids, and proxies yourself is to send the target to a scraper API that handles X’s backend and returns parsed JSON. You make one authenticated request with the username, tweet ID, or search term you want, and the parsing, token rotation, and IP handling run on the server side.

In my runs against ChocoData’s X endpoints, a single GET returned a profile as clean JSON, with no guest token to refresh and no proxy pool to rent. The request is a plain call with your API key as a query parameter:

curl "https://chocodata.com/api/v1/twitter/profile?username=nasa&api_key=$CHOCO_API_KEY"

The Python shape is the same, and it returns the profile fields you would otherwise have to pull out of a GraphQL response:

import requests

resp = requests.get(
    "https://chocodata.com/api/v1/twitter/profile",
    params={"username": "nasa", "api_key": "YOUR_CHOCO_API_KEY"},
    timeout=30,
)
profile = resp.json()
print(profile["username"], profile["followers"])

Swapping the endpoint changes the data type while the request shape stays identical, which is the appeal of the managed route. The same pattern covers a tweet and post scraper, a search, hashtag and trends scraper, and the other public surfaces. You can get an API key on the ChocoData sign-up page and drop it into the snippet above.

The tradeoff is the one I weigh on every project. Doing it yourself with Python and a proxy pool is cheapest at low volume and gives you full control. A scraper API costs per request and removes the token-and-doc_id treadmill, which is the part that eats engineering time once X changes its backend again. For the hands-on language detail, I have a full walkthrough on how to scrape Twitter with Python, and a broader comparison of managed tools in my best Twitter scrapers in 2026 roundup.