【Smart Mode】【Flowchart Mode】How to use URLs Generator | Web Scraping Tool | ScrapeStorm

ScrapeStorm supports URLs Generator, which is to automatically batch generation of URLs you need by setting certain rules.

The settings of Smart Mode and Flowchart Mode are the same, as shown in the following figure:

URLs Generator is generally used for multi-URLs scraping scenarios, which can be divided into two cases:

1. There is no next-page button on the page.

In this case you cannot scrape it by clicking the next-page button. The URLs of each page of this kind of website generally only have some different parameters, such as page number. So you can use the URLs Generator to generate URLs for all pages at once, and then scrape them by multiple URLs.

2. When you need to scrape multiple URLs that are compliant with certain rules, then can use the URLs Generator to generate URLs without having to manually fill them out.

In general, the basic structure of the URL is:

fixed URL + change parameters + fixed URL

Here we use Wikipedia as an example, the URL example is as follows:

https://en.wikipedia.org/wiki/100_(number)

https://en.wikipedia.org/wiki/101_(number)

https://en.wikipedia.org/wiki/102_(number)

……

https://en.wikipedia.org/wiki/110_(number)

We can see that the URLs are the same except for the red part numbers.

……

Now let’s introduce how to use URLs Generator.

Step 1: Enter the first part of the fixed URL.

https://en.wikipedia.org/wiki/

Step 2: Click Add Param

Step 3: Set the first param and set the parameter type to number. Then configure parameters such as start value,end value, and step size.

Step 4: Click Add Param, add the second parameter, set the parameter type to customize, and then paste the second half of the fixed URL into it, that is: _(number)

We can see the resulting URLs in the URLs preview.

P.S. This article only describes the parameters of digital type as an example. The parameters supported by ScrapeStorm also include letters and time. You can refer to this article for setting.