How to Scrape Twitter (X) With Selenium (2026)

The question I get most from people who already picked their tool is how to scrape Twitter with Selenium without the timeline coming back empty. Selenium is a reasonable fit for X because X renders its timeline with JavaScript and only fills the page after its scripts run. Selenium drives a genuine Chrome or Firefox, executes those scripts, and lets you read the tweets once they appear.

I can show why a plain HTTP client is not enough in one request. A logged-out fetch of https://x.com/nasa returns HTTP 200 and about 607 KB of HTML, and that markup contains zero data-testid="tweet" elements. The tweets do not exist until a browser executes the scripts.

import requests

ua = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36"
r = requests.get("https://x.com/nasa", headers={"User-Agent": ua}, timeout=25)
print(r.status_code)                        # -> 200
print(len(r.text))                          # -> ~607000 characters
print(r.text.count('data-testid="tweet"'))  # -> 0

This guide shows the working Selenium pattern I run against live X pages: render the timeline, wait for it correctly, get past the login wall, and keep the session from getting blocked. I tested the code in July 2026 and marked exactly where each step breaks, then I show the single API call I reach for when I do not want to babysit a browser.

Can you scrape Twitter (X) with Selenium?

You can scrape Twitter (X) with Selenium by launching a real browser, opening the target X URL, letting the JavaScript render, and reading the tweets from the page. Selenium controls Chrome or Firefox through the WebDriver protocol, a W3C standard, so the page sees an actual browser that runs X’s scripts rather than a bare request that gets a login shell.

There is one Selenium-specific limit worth knowing up front, because it decides your whole approach. Selenium reads the rendered DOM. It does not natively hand you the GraphQL JSON that X fetches in the background, the way Playwright’s response event does. That means the default Selenium route is to parse the visible tweet nodes, not to intercept X’s own UserTweets payload.

You can still capture that JSON in Selenium, but only by bolting on Chrome DevTools Protocol performance logging or a tool like selenium-wire. If clean JSON capture is your priority, the browser-network fit is better in how to scrape Twitter (X) with Playwright. For everything else, DOM parsing works, and it starts with a correct setup.

What do you need before you start?

You need Python 3.8+, the Selenium package, and a Chrome or Firefox install before you start. Selenium 4 ships a built-in driver manager, so you no longer download and version-match a chromedriver binary by hand. One pip install covers the library.

pip install selenium

You also need to decide which resource you are collecting, because the target URL differs per object:

• A profile’s tweets load at x.com/<username>.
• A single tweet loads at x.com/<username>/status/<id>.
• Search and hashtag results load at x.com/search?q=<terms>.

The last thing to know before any code runs is that a clean, logged-out session will usually meet a login gate on these URLs rather than a timeline. That reality shapes the setup, so it is worth writing the basic scraper first and seeing the wall for yourself.

How do you scrape Twitter (X) with Selenium step by step?

You scrape Twitter (X) with Selenium step by step by launching a browser, opening the profile, waiting for the timeline to render, scrolling to load more tweets, and reading the tweet text nodes. The single most important detail is the wait: if you query the page the instant Selenium navigates, the elements are not there yet and Selenium raises NoSuchElementException, because X’s JavaScript has not finished populating the page.

The fix is an explicit wait that blocks until the first tweet appears, instead of a fixed sleep that guesses. Here is the scaffold I run:

# pip install selenium
import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

opts = Options()
opts.add_argument("--headless=new")
opts.add_argument("--window-size=1280,2000")
driver = webdriver.Chrome(options=opts)   # Selenium 4 auto-manages the driver

driver.get("https://x.com/nasa")

# Wait for the timeline to render rather than guessing with a fixed sleep:
WebDriverWait(driver, 15).until(
    EC.presence_of_element_located(
        (By.CSS_SELECTOR, 'article [data-testid="tweetText"]')
    )
)

for _ in range(3):  # scroll to trigger X's lazy-loaded tweets
    driver.execute_script("window.scrollBy(0, 2000);")
    time.sleep(2)

tweets = driver.find_elements(By.CSS_SELECTOR, 'article [data-testid="tweetText"]')
for t in tweets:
    print(t.text)

driver.quit()

That code is correct, and on a clean machine the WebDriverWait will often still time out, because X shows a login or challenge wall before any tweetText node renders. The scraper is not broken when that happens. X simply refuses to serve the timeline to a logged-out, automated browser, which is the wall the next section handles.

Why the CSS selectors are the fragile part

The data-testid="tweetText" selector is the piece most likely to break on you, because X reshuffles its markup on its own schedule. When your Selenium scraper suddenly returns an empty list on a page you know has tweets, the first thing to check is whether the data-testid attribute names changed. Reading the DOM ties you to X’s current class and attribute names, which is the tradeoff you accept for Selenium’s simplicity over network interception.

How do you get past the X login wall with Selenium?

You get past the X login wall with Selenium by loading a logged-in session, which you do by injecting an account’s cookies before you navigate to the target. A logged-out request returns a login gate with no timeline, so full profiles, search, and follower lists all need an authenticated session. The two cookies that carry the session are auth_token (the login) and ct0 (the CSRF token).

Selenium requires you to be on the domain before you can add its cookies, so you load x.com first, set the cookies, then reload the target. Pairing this with undetected-chromedriver keeps the same session from being flagged on sight:

# pip install undetected-chromedriver
import undetected_chromedriver as uc

driver = uc.Chrome(headless=False)

driver.get("https://x.com")          # must be on the domain before add_cookie
driver.add_cookie({"name": "auth_token", "value": "<from a throwaway account>"})
driver.add_cookie({"name": "ct0",        "value": "<csrf token, same session>"})

driver.get("https://x.com/nasa")     # the timeline now renders

You read the auth_token and ct0 values once from a logged-in browser’s cookie store and reuse them across runs, which avoids typing a password every time and tripping X’s login defenses. Every request in that session is tied to a real X account that can be rate-limited or suspended, so run a dedicated throwaway account and never your personal one. A valid session still is not enough on its own, because the browser fingerprint and the IP get scored before X serves data.

How do you scrape Twitter with Selenium without getting blocked?

You scrape Twitter with Selenium without getting blocked by hiding the automation fingerprint, routing through residential proxies, and pacing requests like a human. A default ChromeDriver leaks tells such as navigator.webdriver, and X scores the browser and the connection before it decides whether to serve the timeline or a challenge.

Three changes move the result more than anything else:

• Mask the automation fingerprint. undetected-chromedriver patches the ChromeDriver signals that give a Selenium session away and describes itself as passing bot-mitigation systems like Cloudflare and DataDome. Its README is explicit that it does not hide your IP, so it is only half the job.
• Use residential proxies. Because the fingerprint patch leaves your IP exposed, a datacenter address that X has pre-flagged still gets challenged. Residential or mobile IPs survive far longer, and Selenium takes a proxy through a Chrome --proxy-server argument.
• Pace and vary behavior. Bursty, identical scrolls trip per-IP rate limits. Randomized scroll distances, short waits, and a realistic viewport read as a human session.

One boundary is worth respecting. X’s Developer Policy states plainly that “no scraping, browser automation, or unofficial methods are permitted,” and X’s robots.txt disallows automated access to paths like /<user>/followers and /search/realtime. Public, logged-out profile and tweet content is the defensible zone. I go deeper on the exact block signals in how to scrape Twitter without getting blocked. Even a well-disguised session, though, runs into the one problem no stealth patch fixes: X keeps moving the target.

Why do Selenium Twitter scrapers keep breaking?

Selenium Twitter scrapers keep breaking because X changes the things your code depends on every few weeks. It rotates the GraphQL doc_id query hashes, renames operations, and reshuffles the data-testid markup your selectors match, so a scraper that worked last month returns an empty list this month. None of these changes are announced, so you find out when a job starts coming back empty.

The WebDriver fingerprint is the other moving part. Bot-detection vendors track projects like undetected-chromedriver and ship countermeasures, so a stealth setup that passed X’s checks in one release can start drawing challenges after the next. That is the cat-and-mouse cost of the DIY route, on top of the markup churn.

The sanctioned escape hatch is not cheap either. X’s API pricing lists post reads at $0.005 per resource, and the standalone free read tier is gone, so a job that reads even a few hundred thousand posts a month runs into real per-read billing. That metered cost is the backdrop for every build-or-buy decision here.

There is also a legal line worth knowing before you scale up. US courts have treated scraping public, logged-out data favorably, as in hiQ v. LinkedIn and X Corp. v. Bright Data, but X’s Terms of Service restrict automated access without permission and set liquidated damages of $15,000 for any party that accesses more than 1,000,000 posts in 24 hours by automated means. For a feed you need to keep alive inside those limits, moving the doc_id chase and the proxy work off your plate is usually the cheaper trade.

How do you scrape Twitter without a browser or driver?

You scrape Twitter without a browser or driver by calling a scraper API that takes a username, tweet ID, or search query and returns parsed JSON, while the provider handles the login, doc_id rotation, and proxies. You send one HTTP request from any language and get clean fields back, with no Chrome process to run and nothing to patch when X renames an operation or reskins its markup.

This is the request shape I use. It is a single GET with the target and an api_key:

curl "https://chocodata.com/api/v1/twitter/profile?username=nasa&api_key=$CHOCO_API_KEY"

The Python version is the same shape and drops straight into your parsing code, returning the fields you would otherwise dig out of the DOM:

import requests

resp = requests.get(
    "https://chocodata.com/api/v1/twitter/profile",
    params={"username": "nasa", "api_key": "YOUR_CHOCO_API_KEY"},
    timeout=30,
)
resp.raise_for_status()
profile = resp.json()["data"]
print(profile["name"], "-", profile["followers_count"], "followers")

When I probed the ChocoData endpoint in July 2026, a request with an invalid key was rejected before it returned any data, which matches its rule that unauthorized requests do not consume credits. Swapping the resource changes what you collect, and each maps to a dedicated endpoint that takes the same key:

For a one-off pull of a few hundred records, the Selenium script from this guide is fine, and it costs nothing but your time. For a feed you need to keep alive, moving the login, doc_id chase, and proxy rotation to a provider is the cheaper trade once you price in the patching. If you want to compare the managed options head to head against the DIY routes, I rank them in best Twitter scrapers in 2026.