Skip to content
Advertisement

scraping website for info when the URL has product id’s instead of true values

Im guessing its php cURL, but Whats the best way to make a loop to scrape the DOM for info from a webpage that uses id’s in the URL Query like (?ProductId=103) There is about 1200 pages. I need to find the innerHTML of the 9th span on each page. This info will just get stored in a mySQL table (id->value) for future scraping of this site.

Advertisement

Answer

Well curl might be faster (not sure), but if it is a one off thing, then I would just use file_get_contents

for($x=0;$x<1200;$x++){
  $f = file_get_contents(URL . '?productId='.$x);
  #do stuff to $f
 }
User contributions licensed under: CC BY-SA
6 People found this is helpful
Advertisement