【Smart Mode】【Flowchart Mode】How to use Webhook | Web Scraping Tool | ScrapeStorm

ScrapeStorm supports Webhook. By using this function, ScrapeStorm can publish the collected data to the user’s HTTP address in real time, and the user needs to develop the Webhook receiving end code.

This function can be set in Run settings.

After the Webhook function is enabled, the scraped data will be sent in JSON format. A notification will also be sent through webhook when the task has been finished.

The HTTP Header is "Content-Type: application/json; charset=utf-8".

The result of the md5 operation using Webhook Token and Timestamp can be verified against the “sign” signature of the JSON data to verify whether the data is legal.

Example of sending data:

{
    "task_id": 3920415,                 // Scraping task ID, find it by "View All Tasks"
    "task_name": "Scraping task name",  // Scraping task name
    "type": "data",                     // This is a webhook of scraped data
    "urls": [                           // The entries of the scraping task
        "http://www.*****.com/list",
        "http://www.*****.com/list2"
    ],
    "sign": "**********************",   // md5(webhook_token+timestamp)
    "timestamp": 1555326657,            // current timestamp
    "data_list": [                      // Scrape data list
        {
            "_id": "0000000000001",     // data ID
            "data": {
                "title": "Landscape",
                "url": "http://www.*****.com/scenery/"
            }                           // Scrape data field content
        }
    ]
}

Example of finish notification:

{
    "task_id": 3920415,                  // Scraping task ID, find it by "View All Tasks"
    "task_name": "Scraping task name",   // Scraping task name
    "type": "finish",                    // This is a webhook of finish notification
    "urls": [                            // The entries of the scraping task
        "http://www.*****.com/list",
        "http://www.*****.com/list2"
    ],
    "sign": "**********************",    // md5(webhook_token+timestamp)
    "timestamp": 1555326657              // current timestamp
}