Comments on: Should scrAPIs be open source? http://thormuller.com/netpositive/2006/03/03/should-scrapis-be-open-source/ In defense of irrational exuberance Thu, 21 Aug 2008 17:17:08 +0000 http://wordpress.org/?v=2.6 By: Thor http://thormuller.com/netpositive/2006/03/03/should-scrapis-be-open-source/#comment-148 Thor Thu, 23 Mar 2006 15:32:54 +0000 http://thormuller.com/netpositive/2006/03/04/should-scrapis-be-open-source/#comment-148 Sean, Good scrapers should have error checking just as you describe, but given how quick-and-dirty so many scrapers are I'd guess that most don't. One of the advantages of a scrAPI initiative would be to popularize these best practices and make reusable code readily available. We need to remove all barriers to doing this. Sean,
Good scrapers should have error checking just as you describe, but given how quick-and-dirty so many scrapers are I’d guess that most don’t. One of the advantages of a scrAPI initiative would be to popularize these best practices and make reusable code readily available. We need to remove all barriers to doing this.

]]> By: seanohagan http://thormuller.com/netpositive/2006/03/03/should-scrapis-be-open-source/#comment-147 seanohagan Thu, 23 Mar 2006 02:01:46 +0000 http://thormuller.com/netpositive/2006/03/04/should-scrapis-be-open-source/#comment-147 I've always wanted to implement an automated way for a scraper to catch site changes and to notify the developer. Each time a site is scraped, certain static features of the site should always be checked. This could include text or images that always occurs in the same place; text that always has the same format (ie. a date, a time, a price, etc.); other static HTML markers; etc. As soon as the scraper detects that the site's "skeleton" has changed, it alerts the developer so that the proper modifications can be made. Besides being run each time a scrape is executed, this check could be made hourly or at some other regular interval. Perhaps this is already a standard feature of most scrapers? I’ve always wanted to implement an automated way for a scraper to catch site changes and to notify the developer.

Each time a site is scraped, certain static features of the site should always be checked. This could include text or images that always occurs in the same place; text that always has the same format (ie. a date, a time, a price, etc.); other static HTML markers; etc.

As soon as the scraper detects that the site’s “skeleton” has changed, it alerts the developer so that the proper modifications can be made.

Besides being run each time a scrape is executed, this check could be made hourly or at some other regular interval.

Perhaps this is already a standard feature of most scrapers?

]]> By: Labnotes » Blog Archive » What exactly is a scrAPI? http://thormuller.com/netpositive/2006/03/03/should-scrapis-be-open-source/#comment-146 Labnotes » Blog Archive » What exactly is a scrAPI? Sun, 05 Mar 2006 02:01:10 +0000 http://thormuller.com/netpositive/2006/03/04/should-scrapis-be-open-source/#comment-146 [...] 8217;s also raising the point that there could be an ecosystem of scrAPIs, including unifying services and open source lib [...] [...] 8217;s also raising the point that there could be an ecosystem of scrAPIs, including unifying services and open source lib [...]

]]>