Proxy API for Web Scraping
By the Proxy11 team. Updated .
Pull a filtered list from the API instead of parsing a proxy table. Then test each proxy against the site you actually scrape, because that is the only test that counts.
Get a list
GET https://proxy11.com/api/proxy.json?key=YOUR_API_KEY&type=anonymous&limit=50
Or use the Python SDK:
import os
from proxy11 import ProxyClient
client = ProxyClient(os.environ["PROXY11_API_KEY"])
candidates = client.get(
limit=50,
country="us",
proxy_type="anonymous",
speed=1.0,
)
for proxy in candidates:
print(proxy["ip"], proxy["port"], proxy["time"])
Choose filters that help your task
| Filter | Example | Use it when |
|---|---|---|
country | country=us | The target shows different content by region |
type | type=anonymous | The target should not see your real IP |
speed | speed=1 | You want proxies that respond within a second |
port | port=8080 | Your firewall or software only allows certain ports |
limit | limit=50 | You want a smaller batch. Your plan sets the maximum |
A narrow filter can return nothing, so handle an empty list and loosen the filter if your job allows it.
A proxy list is not a reliability guarantee
A proxy that works in general can still be blocked by your target, and the speed value says nothing about your target either.
So test before you depend on one:
import requests
def works(proxy, url):
address = "http://{0}:{1}".format(proxy["ip"], proxy["port"])
try:
response = requests.get(url, proxies={"http": address, "https": address}, timeout=(5, 10))
except requests.RequestException:
return False
return response.ok
usable = [proxy for proxy in candidates if works(proxy, "https://example.com/")]
Check the site's terms and robots rules, keep your request rate reasonable, and never send account details through a public proxy.