||In the intorduction, you'll learn how I started writing webbots and spiders in 1996, what to expect from the book, tools you'll need (all open source) and coding standards.
|Part I: Fundamental Concepts and Techniques|
||What's in It for You?
||Describes webbots can uncover the Internet's true potential
at the No Starch Press website.
||Ideas for Webbot Projects
Where do ideas for webbots come from?
at the No Starch Press website.
||Downloading Web Pages
||Explores techniques for downloading web pages with PHP built-in functions and PHP/CURL
||Teaches how to effectively parse data from web pages.
||Advanced Parsing with Regular Expressionsnew
||This chapter shows how Regular Expressions can be used to parse data. It also describes when best--and best not, to use Regular Expressions.
||Automating Form Submission
||Explains how to write webbots that automatically fill out forms and upload data to remote web servers
||Managing Large Amounts of Data
||Describes how to organize and store large amounts of data with compression, tag removal and thumbnailing
|Part II: Projects|
||Shows how to write webbots that monitor prices at online stores
||Describes a project that downloads all the images from a web page
||Explores a project that verifies all the links on a web page
||Explores a webbot that determines the search engine ranking of a web page
||Explains how to write webbots that combine information from multiple resources, including RSS feeds
||Explains how webbots can use FTP as an online resource
||Webbots That Read Email
||Describes methods webbots can use to read email from POP3 Mail Servers
||Webbots That Send Email
||Explores methods webbots can use to send email to SMTP Mail Servers
||Converting a Website into a Function
||Identifies ways to convert an online service into a PHP function your webbots can call
|Part III: Advanced Technical Considerations|
||A study of spider theory, with a simple spider project
||Procurement Webbots and Snipers
||Explores how webbots automatically buy things from online stores and how snipers bid on online auctions.
||Webbots and Cryptography
||Learn how to communicate with websites that use encryption.
||Discover various authentication methods and how webbots can auto authenticate into various websites.
||Advanced Cookie Management
||Master reading and writing cookies with webbots.
||Scheduling Webbots and Spiders
||Learn how to make webbots and spiders launch and run automatically.
||Scraping Difficult Websites with Browser Macrosnew
||Learn how to programatically modify iMacros macros with PHP/MySQL for added functionality.
||Deployment and Scalingnew
||This chapter describes how to deploy large-scale webbot projects. (Or, how to write a botnet.)
|Part IV: Larger Considerations|
||Designing Stealthy Webbots and Spiders
||Learn when and why its important for your webbots to run without detection. Then learn how to achieve stealth with your webbots.
||Learn the various types of proxies, how they're used and what advantages they offer.
||Writing Fault-Tolerant Webbots
||Discover how to write webbots and parse routines that are "less affected" by changes to the web pages you target.
||Designing Webbot-Friendly Websites
||Master Search Engine Optimization as well as methods for communicating data with websites, including light-weight interfaces and SOAP
||Gain an understanding of techniques web developers use to discourage the use of automated browsing agents.
||Keeping Webbots out of Trouble
||Uncover the dangers of writing disreputable webbots and spiders
||A handy reference for using PHP/CURL
||A list of HTTP and NNTP status codes
||SMS Email Addresses
||Address and tips for sending text messages through email