1. Home
  2. Electronics Design
  3. Scraping website, encypted traffic

Scraping website, encypted traffic

Hi

I am working on a pricelist of lcsc.com, so I can select components and com pare prices. I am using Google Web Scraper which needs a number of links to scrape the page. So I am trying to get the web pages for each part.

Site:

formatting link

I have a list of parts, about 30k partnumbers from the site, for example th is part number:

C4410

I type this part number into the search box and the website lists this page :

formatting link

But, there is no link bewteen the number typed into the search box and the part page, so I cannot just generate links from the part number. Also, the http request seems to be incrypted (I used Flipper4 to check that), so no w ay to see how I need to arrange the query

But, maybe I could make the part list, and use a automatic tool to place th at in the search box and click the search button

But then I need to integrate some third party tool with Web Scraper

Or I could find a web scraper that supports this input field

Any hints?

Regards

Klaus

Reply to
Klaus Kragelund
Loading thread data ...

ompare prices. I am using Google Web Scraper which needs a number of links to scrape the page. So I am trying to get the web pages for each part.

this part number:

ge:

e part page, so I cannot just generate links from the part number. Also, th e http request seems to be incrypted (I used Flipper4 to check that), so no way to see how I need to arrange the query

that in the search box and click the search button

Digging deeper, I might have found a way to do it

Just use google search:

formatting link

The link to the page comes up, and I can use scraping to go down that link

Reply to
Klaus Kragelund

You can GET a JSON document resolving the search term at

formatting link

The network tab of browser dev tools (e.g. context menu -> Inspect Element in Firefox) is quite helpful for things like this.

? David

Reply to
David Nadlinger

page:

the part page [?]

Thanks David for the hint :-)

Reply to
Klaus Kragelund

sure why not! they only have a quarter million parts.

What's "Flipper4"

It's encrypted because it's HTTPS, but the payload prior to encryption is not obfuscated, I used firefox "Web console"

formatting link

Reply to
Jasen Betts

ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.