Proxy Integration with Scrapy Proxy Middleware

A residential proxy serves as a mediator, allowing users to get an IP address from an authentic desktop or mobile device connected to a local network. Due to its origin, residential proxies are a perfect match for overcoming geo-restrictions, bypassing CAPTCHAs, managing multiple accounts, and conducting web testing with the CapSolver platform.

Smartproxy offers top-notch residential proxies with an extensive IP pool of over 55M IPs across 195+ locations. With an unparalleled responsive rate, clocking in at under 0.6 seconds, a success rate of 99.47%, and an affordable entry point with Pay As You Go, Smartproxy is a great deal for hustlers and fast-growing companies.

Log in to your Smartproxy dashboard, find residential proxies by choosing Residential under the Residential Proxies column on the left panel, and select a plan that suits your needs. Then, follow these steps:

1. Open the Proxy setup tab.
2. Navigate to the Endpoint generator below.
3. Configure the parameters. Set your authentication method, location, session type, and protocol. 
4. Select the number of proxy endpoints you want to generate (default – 10). 
5. Copy the endpoints by clicking the Copy button.

1. Open the settings.py file in your project folder.
2. Edit the file by adding the following properties at the bottom:

DOWNLOADER_MIDDLEWARES = { 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110, 'yourprojectname.smartproxy_auth.ProxyMiddleware': 100, } SMARTPROXY_USER = 'username'## Smartproxy username SMARTPROXY_PASSWORD = 'password'## Smartproxy password SMARTPROXY_ENDPOINT = 'gate.smartproxy.com'## Endpoint you'd like to use SMARTPROXY_PORT = '7000'## Port of the endpoint you're using

3. Change the yourprojectname in the above to the name of your project folder.

4. Enter the Smartproxy credentials and endpoint information you’ve received from the dashboard.

5. Your web crawlers will now be going through Smartproxy proxies.