Web scraping is to extract or harvest data from websites using programs to automatically fetch/extract data after per-determined intervals. It is quite similar to a search engine bot crawling a website, the only difference here being that we'd be looking for specific data. We may scrape websites to fetch data into specific formats, or just to have data automatically available when required, or may be to automated a specific process. For example once I had written a small script to scrape the Indian Railways' website to check the status of my PNR and intimate me via email in case there is any change in the status. In the article we'll be look at scraping web pages using the Ruby language and we'll be using the Ruby module Mechanize. Mechanize make the underlying task of following links, submitting forms, etc. very easy so that you may concentrate on the logic of data extraction. Installing Mechanize Installing Mechanize is very easy, just issue the following command on a terminal as root. Code: $ gem install mechanize Using Mechanize Say, we'd like to search distrowatch.com for all available Linux distributions and save the data. First, let's see how can we submit forms with Mechanize. Code: #!/usr/bin/ruby require 'rubygems' require 'mechanize' mech = Mechanize.new mech.get('http://distrowatch.com/') do |page| # Submit the search form my_page = login_page.form_with(:action => 'table.php') do |field| field.distribution = 'ubuntu' end.click_button end We can click on links, programatically and set custom user-agent, some websites do not allow programmatic access. Let's see how. Code: #!/usr/local/bin/ruby require 'rubygems' require 'mechanize' mech = Mechanize.new mech.user_agent_alias = 'Mac Safari' mech.get('http://distrowatch.com/') do |page| # Submit the search form my_page = page.form_with(:action => 'table.php') do |field| field.distribution = 'ubuntu' end.click_button # print the current URL print my_page.uri.to_s # goto link which has ostype parameter next_page = my_page.link_with(:href=>/^?ostype=/).click # print the new URL print next_page.uri.to_s end You can use Nokogiri to parse HTML and traverse the DOM to extract data, I have written about it in [THREAD=29471]HTML Parsing in Ruby[/THREAD]. Enjoy scraping the web.