Python Proxy IP Rotation: 3 Common Methods

The essence of a proxy is to “change the exit IP of a request from your local machine to a controllable transit exit.” “Rotation” means avoiding the repeated use of the same exit IP for multiple requests. This article will explain common proxy rotation implementation methods and demonstrate how to quickly complete the integration configuration and practical application based on kookeey‘s dynamic residential proxy service (including directly runnable code examples ).

Want to avoid the hassle of maintaining your own proxy pool?

Using kookeey’s gateway proxy directly: After obtaining the host/port/username and password, you can paste the code to initiate a request.

Know This First: Proxy Rotation Has Two Types

Many people copy “rotating proxy” code but it doesn’t work because the proxy you buy often doesn’t “give you a bunch of IPs to switch between,” but rather “gives you a fixed gateway, and the backend automatically assigns exit points.” Both are called rotating, but the implementation methods are different:

① Client rotation (self-provided proxy list)

You have multiple proxy URLs (A/B/C), and each request picks one randomly or in sequence.

• Advantages: Fully controllable; allows for weighted allocation and elimination of bad proxies.
• Disadvantages: You need to maintain the proxy pool (availability, health checks, replenishment/removal).

② Gateway rotation (service provider managed pool)

You only connect to one host:port, and the same entry point is assigned different exit IPs by the service provider in the background (or kept on a session basis).

• Advantages: Less maintenance, easier to “use right out of the box” after purchase.
• Note: Whether to change the IP address each time depends on your session/rotation policy settings.

When Should You Use Proxies?

• The target website has strict frequency limits on single IP addresses (common 429/CAPTCHA/403 errors).
• You need to access/verify regional content in multiple regions.
• You need to perform concurrent data collection and want to distribute requests across multiple outputs.

A friendly reminder: Proxy is not a magic bullet. Stability usually comes from: timeout settings, concurrency limits, retry policies, and session policies, not just “changing IPs”.

Automatic Rotation Using kookeey Dynamic Residential Proxy

Go to the kookeey website, create an account, and log in.

1. Choose a package and purchase : Select a dynamic housing based on your business.

• Obtain access information from the console : find your proxy gateway’s Host , Port , username , and password (or whitelist authentication).
• First, run a connectivity test : use https://httpbin.org/ip to verify whether the request goes through a proxy and whether the outgoing IP matches expectations.

You need to remember four parameters: host, port, username, and password . All subsequent code examples will revolve around these.

It is recommended to first copy the host/port/username/password from the console, and then paste it into the code below.

Minimize connectivity testing (it’s recommended to run this test immediately after purchasing).

import requests

proxy_host = "http-dynamic.kookeey.com" # Host in the console

proxy_port = "8888" # The port in the console

username = "your_username" # Your username

password = "your_password" # Your password

gateway = f"http://{username}:{password}@{proxy_host}:{proxy_port}"

proxies = {"http": gateway, "https": gateway}

r = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=10)

r.raise_for_status()

print(r.text)

If the returned IP address is not your local public IP address, it means the proxy link is established. If the IP address doesn’t change, it could be because you have session persistence enabled, or request reuse is causing the same outgoing route to be observed for a short period.

Three Basic Syntaxes: Requests / Aiohttp / Scrapy

Method 1: requests (the simplest, suitable for single requests or low concurrency)

import requests

proxy_host = "http-dynamic.kookeey.com"

proxy_port = "8888"

username = "your_username"

password = "your_password"

gateway = f"http://{username}:{password}@{proxy_host}:{proxy_port}"

proxies = {"http": gateway, "https": gateway}

try:

r = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=10)

r.raise_for_status()

print(r.json())

except requests.exceptions.RequestException as e:

print("request failed:", e)

Note: In gateway rotation mode, you don’t need to maintain a list of proxies; all requests go through the same gateway entry point.

Method 2: aiohttp (asynchronous concurrency, suitable for higher throughput)

import aiohttp

import asyncio

proxy_host = "http-dynamic.kookeey.com"

proxy_port = "8888"

username = "your_username"

password = "your_password"

proxy = f"http://{username}:{password}@{proxy_host}:{proxy_port}"

async def fetch(session, i):

try:

async with session.get("https://httpbin.org/ip", proxy=proxy) as resp:

text = await resp.text()

print(f"[{i}] status={resp.status} body={text}")

except Exception as e:

print(f"[{i}] failed: {e}")

async def main():

timeout = aiohttp.ClientTimeout(total=15, connect=10, sock_connect=10, sock_read=10)

connector = aiohttp.TCPConnector(limit=50) # Limit the connection pool to avoid unlimited concurrency

async with aiohttp.ClientSession(timeout=timeout, connector=connector) as session:

await asyncio.gather(*[fetch(session, i) for i in range(1, 6)])

if __name__ == "__main__":

asyncio.run(main())

Concurrency is not necessarily better the higher it is. Even the best proxy can’t withstand mindlessly maximizing concurrency: remember to set timeouts and connection pool limits.

Method 3: Scrapy (unified proxy settings via middleware)

Suitable for web crawler engineering: inject the proxy uniformly into the Downloader Middleware, and trigger retries when encountering exceptions/common status codes.

# middlewares.py

import random

from scrapy.exceptions import NotConfigured

class RotateProxyMiddleware:

def __init__(self, proxy_url_list):

self.proxy_url_list = proxy_url_list

@classmethod

def from_crawler(cls, crawler):

proxy_url_list = crawler.settings.getlist("PROXY_URL_LIST")

if proxy_url_list:

raise NotConfigured("PROXY_URL_LIST is empty")

return cls(proxy_url_list)

    def _set_proxy(self, request):

        request.meta["proxy"] = random.choice(self.proxy_url_list)

    def process_request(self, request, spider):

        if not request.meta.get("proxy"):

            self._set_proxy(request)

    def process_exception(self, request, exception, spider):

        self._set_proxy(request)

        request.dont_filter = True

        return request

# settings.py

proxy_host = "http-dynamic.kookeey.com"

proxy_port = "8888"

username = "your_username"

password = "your_password"

PROXY_URL_LIST = [

f"http://{username}:{password}@{proxy_host}:{proxy_port}",

]

DOWNLOADER_MIDDLEWARES = {

"yourproject.middlewares.RotateProxyMiddleware": 543,

"scrapy.downloadermiddlewares.retry.RetryMiddleware": 550,

}

RETRY_ENABLED = True

RETRY_TIMES = 3

DOWNLOAD_TIMEOUT = 20

Five Small Habits for Greater Stability (More Important than “Changing Your IP Address”)

• Set timeouts for connect/read/total to prevent faulty connections from crashing the program.
• Limit concurrency : aiohttp uses connectors/semaphores; Scrapy adjusts concurrency and download latency.
• There should be a strategy for retrying : retry for 429/5xx/timeouts, and change the session or re-initiate the connection if necessary.
• To observe whether “rotation is in effect,” use the correct method : make multiple requests to httpbin and combine this with the rotation/session strategy you selected in the console to make a judgment.
• Compliance and Risk Control : Adhere to the target site’s terms and local laws; the proxy is a tool, not a disclaimer of liability.

Frequently Asked Questions (Quick Troubleshooting)

• 407 Proxy Authentication Required : Incorrect username/password, or you selected whitelist authentication but did not add your local IP address.
• Consecutive timeouts : The proxy is unavailable, the network is down, or excessive concurrency has exhausted the connections.
• The outgoing IP address remains unchanged : This may be due to session stickiness being enabled, or short-term connection reuse causing it to appear unchanged.

Final step: Enter the console parameters into the code and run httpbin.

After the test is successful, it will be deployed to the actual target site, and concurrency limits and retry strategies will be gradually increased.

Click to try it out and receive a new user data package.

Note: The code example in this article uses the “gateway rotation” approach (ready to use after purchase). If you are using a “self-provided proxy list”, simply replace the gateway with your multiple proxies and perform random/sequential selection on the client side.