Async scraping is where proxy setups quietly fall apart. With requests you rotate one IP per call and move on; with asyncio you suddenly have 200 requests in flight, all needing their own exit, their own session affinity, and their own retry when a target throws a 403. Get it wrong and you either hammer one IP into a ban or shred every login session by rotating mid-flow.
This guide shows the patterns that actually hold up at concurrency: per-request rotation in httpx and aiohttp, a bounded worker pool, sticky sessions for stateful flows, and block-aware retries — with working code you can paste in.
A synchronous scraper touches one proxy at a time, so "rotate on each request" is trivially correct. Async breaks three assumptions at once:
httpx.AsyncClient binds one proxy per client, so to rotate you pick the proxy per request and route through a small pool of clients keyed by exit:
import asyncio, itertools, httpx
PROXIES = [
"socks5h://USERNAME:[email protected]:913",
"socks5h://USERNAME:[email protected]:913",
"socks5h://USERNAME:[email protected]:913",
]
pool = itertools.cycle(PROXIES)
# One reusable client per exit (connection pooling stays intact)
clients = {p: httpx.AsyncClient(proxy=p, timeout=30) for p in PROXIES}
async def fetch(url):
proxy = next(pool)
r = await clients[proxy].get(url)
return r.status_code, r.text
async def main(urls):
results = await asyncio.gather(*(fetch(u) for u in urls))
for client in clients.values():
await client.aclose()
return resultsReusing one client per exit keeps HTTP/2 and connection pooling alive instead of paying a fresh handshake on every request. Note httpx>=0.28 renamed proxies= to proxy= on the client.
aiohttp is simpler here — a single ClientSession takes a proxy= argument per request, so you rotate inline:
import asyncio, itertools, aiohttp
pool = itertools.cycle(PROXIES) # http:// or socks5h:// (needs aiohttp-socks for SOCKS)
async def fetch(session, url):
proxy = next(pool)
async with session.get(url, proxy=proxy, timeout=aiohttp.ClientTimeout(total=30)) as r:
return r.status, await r.text()
async def main(urls):
async with aiohttp.ClientSession() as session:
return await asyncio.gather(*(fetch(session, u) for u in urls))aiohttp speaks HTTP proxies natively; for socks5h:// add aiohttp-socks and pass a ProxyConnector.
Unbounded gather is the fastest way to get every IP in your pool flagged at once. Bound it so each exit carries a human-plausible number of parallel requests:
sem = asyncio.Semaphore(10) # at most 10 in flight
async def fetch_capped(session, url):
async with sem:
return await fetch(session, url)Rule of thumb: keep concurrency at or below the number of distinct exits you have, so you're not stacking many parallel requests on a single residential IP.
Rotation is for getting in; sticky sessions are for staying in. A login, a cart, anything cookie-bound must hold one exit for the whole sequence. Pin the proxy per task instead of per request:
async def run_account(account, proxy):
# Same exit IP for every step of this account's flow
async with httpx.AsyncClient(proxy=proxy, timeout=30) as client:
await client.post("https://target.site/login", data=account.creds)
await client.get("https://target.site/dashboard")
await client.post("https://target.site/cart", json=account.order)
async def main(accounts):
# One sticky exit per account, accounts run concurrently
await asyncio.gather(*(run_account(a, p)
for a, p in zip(accounts, itertools.cycle(PROXIES))))Use a provider that supports sticky residential sessions so the same exit is held for the lifetime of the task, not silently rotated under you.
At concurrency you cannot trust a 200 — anti-bot systems return a 200 with a challenge body. Detect blocks and retry the single failed task on a fresh exit:
BLOCK_MARKERS = ("just a moment", "/cdn-cgi/challenge-platform",
"datadome", "px-captcha", "access denied")
def looks_blocked(status, text):
return status in (403, 429, 503) or any(m in text[:4000].lower() for m in BLOCK_MARKERS)
async def fetch_retry(url, attempts=4):
for _ in range(attempts):
proxy = next(pool)
async with httpx.AsyncClient(proxy=proxy, timeout=30) as c:
r = await c.get(url)
if not looks_blocked(r.status_code, r.text):
return r
await asyncio.sleep(0.5)
raise RuntimeError(f"blocked after {attempts} exits: {url}")One catch async doesn't solve: httpx and aiohttp both sit on Python's TLS stack, so they send a JA3 no real browser produces. A perfect rotation pool on a clean residential IP still gets fingerprinted as automation on the handshake. Pair async rotation with TLS impersonation — see Bypass TLS Fingerprinting with curl_cffi, and verify your exit's JA3 + ASN at check.jibaoproxy.com.
New users get 500MB free traffic instantly, plus an extra first-deposit reward — limited-time offer.