【Flowchart Mode】How to set up paging | Web Scraping Tool | ScrapeStorm

In the process of setting up the scraping task, users will encounter a variety of web pages, which can be generally divided into three types: one is a web page with a next page button, one is a web page that is scrolled to load, and one is next page button + scroll to load.

1. Next Page Button

This kind of web page is the most common type of pagination. After extracting all list elements, the software will prompt whether you need automatic pagination. If you only want to scrape the current page data, there is no need to set a paging loop.

After clicking to set the paging loop, the software will prompt whether the identified paging is correct. If you click “Yes”, the software will configure the loop component.

The details are shown below:

If the paging detected by the software is incorrect, you can click “No, reset paging”, reselect the paging type, and then manually click the next page button on the page.

If the correct xpath is still not recognized by manually selecting the next page button, you can manually modify the xpath in the paging loop.

2. Scroll to Load

After the list data is extracted, since there is no next page button on the page, the operation tips will prompt that paging is not detected.

At this point, you can click the “Set aging” and select “Scroll to Load”, and the software will automatically set the paging loop.

3. Scroll to Load + Next Page Button

(1) The next page button that needs to scroll to appear

The next page button of some web pages does not appear on the interface of the first screen, it needs to scroll the page to appear. After we extract all list elements and determine that a paging loop needs to be set, the operation tips will prompt that paging is not detected.

At this point, you can click the “Set aging” and select “Scroll to Load + Next Page Button”, and the software will automatically set the paging loop.

If the correct xpath is still not recognized by manually selecting the next page button, you can manually modify the xpath in the paging loop.

(2) Next page button can be detected but need to add scroll

Some webpages can directly recognize the next page button, but you need to add scroll to scrape the data. After we extract all list elements and detect pagination, set “No, reset paging”.

Click the “Set aging” and select “Scroll to Load + Next Page Button” in the new operation box, and the software will automatically set the paging loop.

Select “Scroll Loading + Paging Button” in the operation tips, and then the software will jump to the new operation tip.

If the correct xpath is still not recognized by manually selecting the next page button, you can manually modify the xpath in the paging loop.