【Smart Mode】【Flowchart Mode】How to set IP Rotation & Delay | Web Scraping Tool | ScrapeStorm

It contains IP Rotaion, automatic settings and manul settings. This part of the function is mainly used to avoid various website blocking problems that may be encountered.

In the editing task interface, click the “Start” button to open the running settings, and click “IP Rotation & Delay”.

 

1. IP Rotation

(1) IP Type

ⅰ. Bright Data Zone

Bright Data Zone is an external proxy, you need to buy ip on the official website.

How to Set up Bright Data Zone

ⅱ. Custom IP

If you need to use your own IP, please click “Set Up” and set it as required. (Note: the custom IP will be switched in sequence)

(2) Switch IP when the following condition is met:

ⅰ. Interval

IP switches based on time. For example, if you set the switching condition to “Interval: 3 minutes”, then the IP will be switched every 3 min and one IP will be consumed at the same time.

ⅱ. Text appears on the page

Switch according to the text. For example, if you set the switching condition to “Text appears on the page: error”, then when the corresponding text appears on the web page, the IP will be switched once and one IP will be consumed at the same time.

 

2. Automatic

For general scraping tasks, just follow the default automatic settings.

 

3. Manual

If the webpage encountered is special, the automatic setting does not work. At this time, you can set the manual setting.

ⅰ. Delay(s)

Some webpages open slowly and sometimes affect the scraping effect. You can set a delay to effectively improve the quality of scraping. The default delay of the system is 1 second, which can be modified according to your needs.

ⅱ. Detect CAPTCHA During the Scraping

Software generally detects captcha automatically. In special cases, you can manually set to detect captcha when encountering specific text.

ⅲ. Detect Login Prompt During the Scraping

Websites that need to log in to scrape data may be logged out during the running process, making it impossible to scrape data, or some websites will be prompted to log in after scraping a certain amount of data. Select this function. When you log out or need to log in, you will be prompted to log in.

ⅳ. Only Extract Visible Elements

Some websites mix invalid data with valid data. When scraping data, many invalid characters appear, and these invalid characters are hidden. In this case, we can choose this setting and only scrape visible elements.

Note: If the website has no setting to hide invalid characters, selecting this option will cause incomplete data or cannot be scraped.

ⅴ. Load Page Info one by one

Some websites need to scroll to a certain position before the content can be displayed, otherwise the data cannot be scraped, and this function can be selected at this time. However, it should be noted that when this function is checked, the scraping speed will be affected.

ⅵ. Switch Browser Regularly

Switching browser regularly can solve the blocking problem of some websites and achieve anti-blocking effect.

You can set the interval for switching browser versions. Set the interval to choose from 30 seconds to 10 minutes. The software will automatically switch various browser versions according to the interval.

ⅶ. Clear Cookies Regularly

Clearing cookies regularly can solve the blocking problem of some websites and achieve anti-blocking effect.

You can set the interval for clearing cookies. Set the interval to choose from 30 seconds to 10 minutes. The software will automatically clearing cookies according to the interval.