how do you integrate a web spider into your php site? please help me...?

Answers

1. Answer Man

   On 2006-08-01 08:22:23

   
   
   To answer your question I will have to make assumption that your search engine will be specialized and has a narrow focus. Otherwise it would require a huge data base if it was covering a general subject area. The way to integrate you spider is to have the spider set to periodically search for new material and then up date a data base. You should have it run from a cron job at the web server. Your search engine would then use the data base to fulfill the searches done by the users. The two programs operate on the same data base but would never interact with each other. Now, if you intend to eliminate the data base and have the spider search every time a user performs a search or if you intend to not have a limited focus on the search content of your search engine, then a different approach is necessary. In that case, I would suggest that you set it up as a meta-search engine and only spider other search engines. You can find a very good implementation of a meta-search engine in PHP here: http://turn-k.net/k-search Answer Man