Scraping website, encypted traffic

ompare prices. I am using Google Web Scraper which needs a number of links to scrape the page. So I am trying to get the web pages for each part.

this part number:

ge:

e part page, so I cannot just generate links from the part number. Also, th e http request seems to be incrypted (I used Flipper4 to check that), so no way to see how I need to arrange the query

that in the search box and click the search button

Digging deeper, I might have found a way to do it

Just use google search:

The link to the page comes up, and I can use scraping to go down that link

You can GET a JSON document resolving the search term at

The network tab of browser dev tools (e.g. context menu -> Inspect Element in Firefox) is quite helpful for things like this.

? David

page:

the part page [?]

Thanks David for the hint :-)

sure why not! they only have a quarter million parts.

What's "Flipper4"

It's encrypted because it's HTTPS, but the payload prior to encryption is not obfuscated, I used firefox "Web console"