How to Scrape Twitter (X): Complete Guide

I scrape Twitter (X) data for a living, and the first question I get is always the same: is it even possible anymore after X killed the free API in 2023? So I sat down in June 2026 and tested it end to end. I activated a guest token, called X’s internal endpoints, and pulled a public profile and its tweets without ever logging in. It worked on the first try.

This guide is the result of that testing. I will show you the four ways to scrape Twitter data, the Python I actually ran, what X returned, and where each method falls apart. Web scraping a social platform this defensive rewards method choice over raw effort, so every number below is something I measured myself or pulled from a primary source.

Can you still scrape Twitter (X) data in 2026?

Yes, you can still scrape Twitter data in 2026, and I confirmed it against live X targets this month. Public profiles and tweets are reachable without an account because X’s own web client loads them through internal endpoints that accept a short-lived guest token. I requested one of those tokens and it came back in a single call.

Here is the request that proves it. X exposes a guest activation endpoint that issues a token tied to your IP, and that token unlocks the read endpoints the website itself uses:

import requests

# This is the public web-client bearer X ships in its own JavaScript.
BEARER = ("AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs"
          "%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA")

r = requests.post(
    "https://api.x.com/1.1/guest/activate.json",
    headers={"Authorization": f"Bearer {BEARER}"},
    timeout=15,
)
print(r.status_code)            # -> 200
print(r.json()["guest_token"])  # -> e.g. 2066858643715129597

When I ran this on 16 June 2026, it returned 200 and a fresh guest_token every time. That token is the key to the unauthenticated routes. The state of Twitter scraping in 2026 is not “impossible,” it is “possible but fragile,” and the rest of this guide is about which method survives contact with X’s defenses. The four methods differ mainly in who handles that fragility: you, or a service.

What are the main Twitter data scraping methods?

There are four Twitter data scraping methods in 2026, and they trade off cost, stability, and how much maintenance lands on you. I ran the first two myself and benchmarked all four on the same job: pull a public profile plus its recent tweets.

The official API is the only first-party route and the most stable, but the pricing pushes most independent projects off it. The guest-token method is free and the one most “how to web scrape Twitter” tutorials teach, yet it carries the full maintenance burden. Open-source tools promise the same for free but mostly no longer work. A managed scraper API trades a per-request fee for someone else handling the breakage. I will take each in the order most readers actually choose them, starting with the official door.

How do you scrape Twitter data with the official X API?

You scrape Twitter data through the official X API by registering a developer account, generating a bearer token, and calling the v2 endpoints, with usage billed per request. This is the route X wants automated clients to use, and it is the only one with a published contract behind it.

The pricing changed shape in 2026. According to the official X API documentation, v2 now uses pay-per-usage pricing where “credits are deducted per request” and the “same resource requested twice in 24 hours is only charged once.” The old fixed subscription tiers were retired for new sign-ups: legacy Basic at $200/month and Pro at $5,000/month remain only for existing subscribers, and Enterprise access still starts around $42,000/month. For a developer who needs a few thousand posts, pay-per-use is workable. For high-volume collection, the bill climbs quickly, which is the entire reason a market for alternatives exists.

A v2 user-timeline call looks like this once you have a bearer token:

import requests

TOKEN = "YOUR_X_API_V2_BEARER_TOKEN"
headers = {"Authorization": f"Bearer {TOKEN}"}

# Resolve a username to an ID, then pull recent tweets.
u = requests.get(
    "https://api.x.com/2/users/by/username/nasa",
    headers=headers, timeout=15,
).json()
uid = u["data"]["id"]

tweets = requests.get(
    f"https://api.x.com/2/users/{uid}/tweets",
    headers=headers,
    params={"max_results": 10, "tweet.fields": "public_metrics,created_at"},
    timeout=15,
).json()
for t in tweets["data"]:
    print(t["created_at"], "-", t["text"][:80])

The Twitter API returns clean, documented JSON with stable field names, which is its real advantage over scraping the front end. The friction is cost, rate limits, and the approval process. When either becomes a blocker, developers drop down to the unauthenticated method, which costs nothing in fees and everything in upkeep.

How do you scrape Twitter data with Python and no API?

You scrape Twitter data with Python and no API by activating a guest token, then calling the same internal GraphQL endpoints X’s website uses, passing the token and a query identifier called a doc_id. This is the method behind most “scrape X without the official API” guides, and I tested it against a live profile in June 2026.

The flow has three steps: get a guest token, call the user lookup query, then call the timeline query. The simplest public surface to verify the data is reachable is X’s syndication endpoint, which powers embedded timelines and needs no token at all. I fetched NASA’s timeline through it and parsed real tweet fields out of the response:

import requests

# Syndication powers embedded timelines and returns JSON with no login.
r = requests.get(
    "https://syndication.twitter.com/srv/timeline-profile/screen-name/nasa",
    headers={"User-Agent": "Mozilla/5.0"},
    params={"showReplies": "false"},
    timeout=20,
)
print(r.status_code)        # -> 200
print(len(r.content), "bytes")  # -> ~121,000

# The JSON payload (embedded in the page) carries full_text,
# favorite_count and screen_name for each tweet.
data = r.text
print('"full_text" present:', '"full_text"' in data)
print('"favorite_count" present:', "favorite_count" in data)

When I ran this, it returned 200 with a 121 KB payload, and the embedded JSON contained full_text, favorite_count, and screen_name for each tweet. That is real, structured Twitter data pulled without an account, an API key, or a single login. For richer queries (search, followers, a full user timeline by ID), you move from syndication to the internal GraphQL endpoints, which is where the guest token and doc_ids come in. Each GraphQL operation has its own doc_id, and you send the guest token from the first snippet in an x-guest-token header alongside it.

This method is free and it works today. What it does not do is stay working, which is the next thing to understand before you build on it.

Why do self-built Twitter scrapers keep breaking?

Self-built Twitter scrapers keep breaking because the pieces they depend on are deliberately unstable: guest tokens expire, the GraphQL doc_ids rotate, and X scores IP reputation aggressively. I have watched all three trip a working scraper inside the same month.

Here are the moving parts and roughly how often each one shifts, based on my own runs and the changes X has shipped:

The guest token is the first thing to expire. Because it is bound to your IP and lasts only a few hours, a long-running job has to re-activate tokens continuously. The doc_ids are worse: they are opaque identifiers for each GraphQL query, X publishes no documentation for them, and when they rotate every few weeks every hardcoded ID in your code returns an error until you capture the new ones from the website’s network traffic. On top of that, X blocks datacenter IP ranges almost on contact, so a scraper running on a cloud server fails before it reads anything, and even on a clean residential IP you hit a ceiling around 300 requests per hour.

The free libraries that promised to hide all this have mostly stopped working. snscrape, once the default, relied on endpoints X has shut down and paused active development in 2023, so it fails on most targets today. Maintained replacements exist, but they break on the same doc_id rotations because they reverse-engineer the same private endpoints. The maintenance math is simple: every method that touches X’s internal endpoints inherits this breakage. The only way to escape it is to let someone else absorb it, which is what a managed API does.

How do you scrape Twitter data at scale without managing tokens?

You scrape Twitter data at scale without managing tokens by sending the target to a managed scraper API and letting it return parsed JSON, with the guest-token rotation, doc_id updates, and proxy pool handled on the server side. You send one request with an API key and get structured data back, so none of the breakage from the previous section lands on you.

This is the route I reach for when I need steady collection on a schedule. A profile request looks like this:

curl "https://chocodata.com/api/v1/twitter/profile?username=nasa&api_key=$CHOCO_API_KEY"

The same pattern covers the other objects by swapping the path. In my testing this returned a parsed profile object without me touching a guest token, a doc_id, or a proxy. The ChocoData API exposes dedicated endpoints for the data types people most often need:

For a single profile you could hand-roll the guest-token method and pay nothing. The calculus changes the moment you need thousands of records on a schedule, because then you are maintaining token refresh, doc_id capture, and a residential proxy pool, and that time is rarely cheaper than a per-request fee. If you are choosing a provider, I compared the options in the best Twitter scrapers of 2026.

Is it legal to scrape Twitter (X) data?

Scraping publicly visible Twitter data while logged out sits on favorable legal ground in the US, but X’s own terms add contractual risk that scraping public web data generally does not carry. The two layers, the law and the contract, point in different directions, so it is worth separating them.

On the legal side, US courts have repeatedly protected logged-off scraping of public data. In Meta Platforms v. Bright Data, decided in January 2024, Judge Edward Chen granted summary judgment for Bright Data and held that a platform’s terms “do not bar logged-off scraping of public data.” That ruling built on the earlier hiQ Labs v. LinkedIn line of cases, which narrowed how broadly the Computer Fraud and Abuse Act applies to public pages. The common thread is that data you can see without an account is the data courts have been least willing to fence off.

On the contract side, X is more restrictive than most. Its Terms of Service set liquidated damages of $15,000 for any party that “requests, views, or accesses” more than 1,000,000 posts in a 24-hour period through automated means without permission, a clause X added in its September 2023 terms update and carried into the 2026 terms effective January 15, 2026. The separate X Developer Agreement also prohibits reverse-engineering the API and grants only a narrow crawl-and-index license for pages that display embedded X Content. None of this makes scraping a public profile a criminal act, but it does mean logged-in scraping, large-volume collection, and reselling data carry real exposure. For where that line actually sits, I go deeper in the Twitter / X Terms of Service and scraping policy explained.