The Code

The following code uses the same nokogiri strategies in the past three lessons. But here are the specific considerations that we have to make for Pfizer’s list:

• The base url is: http://www.pfizer.com/responsibility/working_with_hcp/payments_report.jsp?enPdNm=All&iPageNo=1 The most interesting parameter, iPageNo, is bolded. If you replace ‘1’ with any number, you’ll see you can progress through the list. There appears to be 486 pages.
• So each page has a table of data with id #hcpPayments. The rows of data aren’t very normalized. For example, each “Entity Paid” can have many services/activity listed, with each of those items having another name attached to it. Then there are “cash” and “non-cash” values, which may or may not be numeric (“—” apparently means 0) There’s no easy css selector to grab each entity…but it seems that we can safely assume that if the first table column has a name (and the second and third contain city and state) that this is a new entity.

These are the steps we’ll take:

• Download pages 1 to 486 of the list (each page has 10 entries)
• Run a method that gathers all the doctor names from the pages we just downloaded on to our hard drive)
• From that list of doctors, query the Pfizer site and gather the individual payments to every doctor.

At the top, I’ve written a few convenience methods to deal with strings. Also included are: get_doc_query is a function we call to extract the doctor name from the links on the site.

puts_error is a quick function to log any errors we might have

						# Some general functions to deal with strings
					class String

					  alias_method :old_strip, :strip

					  def strip
						  self.old_strip.gsub(/^[\302\240|\s]*|[\302\240|\s]*$/, '').gsub(/[\r\n]/, " ")
					  end

					  def strip_for_num
					    self.strip.gsub(/[^0-9]/, '')
					  end

					  def blank?
						respond_to?(:empty?) ? empty? : !self
					  end
					end
					
					
					END_PAGE=486
					BASE_URL=''
					DOC_QUERY_URL='http://www.pfizer.com/responsibility/working_with_hcp/payments_report.jsp?hcpdisplayName='


					def get_doc_query(str)
					  str.match(/hcpdisplayName\=(.+)/)[1]
					end

					def puts_error(str)
					  err = "#{Time.now}: #{str}"
					  puts err
					  File.open("pfizer_error_log.txt", 'a+'){|f| f.puts(err)}
					end
					
					
						

I found it easiest to download all the pages onto the hard drive first, using something like CURL, and then run the following code on it.

process_local_pages is a method that will iterate through every page (you can set BASE_URL to either your hard drive if you’ve downloaded all the pages yourself, or to the Pfizer page), run process_row, and store all the doctor names and payees into separate files, as well as hold all the total amounts

The three resulting files that you get are:

• pfizer_doctors.txt – Every doctor name listed. We will use this in the next step to query each doctor individual on Pfizer’s site
• pfizer_entities.txt – A list of every payment made to Entities
• pfizer_entity_totals.txt – A list of the total payments made to Entities

						def process_row(row, i, current_entity, arrays)  

						  tds = row.css('td').collect{|r| r.text.strip}

						   if !tds[3].blank? 
						     if !tds[1].blank?
						     # new entity
						     puts tds[0]
							     current_entity = {:name=>tds[0],:city=>tds[1], :state=>tds[2], :page=>i, :services=>[]} 
							     arrays[:entities].push(current_entity) if arrays[:entities]
						  	   current_class = row['class']
							   end

						     if tds[3].match(/Total/)
						       arrays[:totals].push([current_entity[:name], tds[4].strip_for_num, tds[5].strip_for_num].join("\t")) if arrays[:totals]

						     else
						        # new service
						   	   services_td = row.css('td')[3]
						   	   service_name = services_td.css("ul li a")[0].text.strip 
						   	   puts "#{current_entity[:name]}\t#{service_name}" 
						   	   current_entity[:services].push([service_name, tds[4].strip_for_num, tds[5].strip_for_num]) 

						   	   arrays[:doctors].push(services_td.css("ul li ul li a").map{|a| get_doc_query(a['href']) }.uniq) if arrays[:doctors]
						     end
						   elsif tds.reject{|t| t.blank?}.length == 0
						     #blank row
						   else
						     puts_error "Page #{i}: Encountered a row and didn't know what to do with it: #{tds.join("\t")}"
						   end

						   return current_entity
						end





						def process_local_pages

						  doctors_arr = []
						  entities_arr = []
						  totals_arr =[]

						  for i in 1..END_PAGE
						    begin
						  	   page = Nokogiri::HTML(open("#{BASE_URL}#{i}.html"))

						    	 count1, count2 = page.css('#pagination td.alignRight').last.text.match(/([0-9]{1,}) - ([0-9]{1,})/)[1..2].map{|c| c.to_i}
						    	 count = count2-count1+1

						    	 puts_error("Page #{i} WARNING: Pagination count is bad") if count < 0
						    	 puts("Page #{i}: #{count1} to #{count2}")

						    	 rows = page.css('#hcpPayments tbody tr')

						    	 current_entity=nil

						    	 rows.each do |row|  	   
						    	   current_entity= process_row(row, i, current_entity, {:doctors=>doctors_arr, :entities=>entities_arr, :totals=>totals_arr})
						       end

						     rescue Exception=>e
						  	   puts_error "Oops, had a problem getting the #{i}-page: #{[e.to_str, e.backtrace.map{|b| "\n\t#{b}"}].join("\n")}"
						     else


						     end
						  end

						  File.open("pfizer_doctors.txt", 'w'){|f|
						    doctors_arr.uniq.each do |d|
						        f.puts(d)
						    end
						  }

						  File.open("pfizer_entities.txt", 'w'){|f|
						    entities_arr.each do |e|
						      e[:services].each do |s|
						        f.puts("#{e[:name]}\t#{e[:page]}\t#{e[:city]}\t#{e[:state]}\t#{s[0]}\t#{s[1]}\t#{s[2]}")
						      end  
						    end
						  }


						  File.open("pfizer_entity_totals.txt", 'w'){|f|
						    totals_arr.uniq.each do |d|
						        f.puts(d)
						    end
						  }
						end

					

process_doctor is what we run after we’ve compiled the list of doctor names that show up on the Pfizer list. Each doctor has his/her own page with detailed spending. The data rows are roughly in the same format as the main list, so we reuse process_row again

.

						def process_doctor(r, time='')
						  begin
						    url = "#{DOC_QUERY_URL}#{r}"
						    page = Nokogiri::HTML(open("#{url}"))
						  rescue
							   puts_error "Oops, had a problem getting the #{r}-entry: #{[e.to_str, e.backtrace.map{|b| "\n\t#{b}"}].join("\n")}"
						  end

						  rows = page.css('#hcpPayments tbody tr')
						  entities_arr = []
						  current_entity=nil

						   rows.each do |row|  	   
						     current_entity= process_row(row, '', current_entity, {:entities=>entities_arr})
						   end


						   name = r.split('+')
						   puts_error("Should've been a last name at #{r}") if !name[0].match(/,$/)
						   name = "#{name[0].gsub(/,$/, '')}\t#{name[1..-1].join(' ')}"

						   vals=[]
						   entities_arr.each do |e| 
						     e[:services].each do |s|
						       vals.push("#{name}\t#{e[:name]}\t#{e[:page]}\t#{e[:city]}\t#{e[:state]}\t#{s[0]}\t#{s[1]}\t#{s[2]}\t#{url}\t#{time}")
						    end
						   end

						  vals.each{|val| File.open("pfizer_doctor_details.txt", "a"){ |f| 
						    f.puts val
						  }}

						  puts vals
						  return vals
						end


					

process_doctor_pages is just a function that calls process_doctor for each name in the pfizer_doctors.txt we previously gathered

The final result is pfizer_doctor_details.txt, which contains a line for every payment to every doctor.

						def process_doctor_pages
						  time = Time.now

						  File.open("pfizer_doctors.txt", 'r'){|f|
						     f.readlines.each do |r|
						        vals = process_doctor(r, time)
						     end 
						  }
						end		

					

The Results

After Googling the top-Pfizer-paid-doctor on the list (Gerald Michael Sacks for ~$150K), I came across the Pharma Conduct blog, which had already posted partial aggregations of the list, including the top 5 doctors, complete with profiles and pics.

As Pharma Conduct has already been on the ball, I’ll defer to its analysis. It has some good background here on how lame pharma companies have been in past releases of data. Overall, Pharma Conduct is less-than impressed with Pfizer:

Despite reporting more information than some its peers, Pfizer’s interface is still very limited. For one, to use the search filtering, you must know a physician’s first name and last name, as well as the state where the payment was made. Also, the data cannot be sorted by payment amount, which is a big limitation. Pfizer should be given credit for releasing the information and being so thorough. However, by releasing it in a format that is not really amenable to data analysis and is more suited to simply looking up results one physician at a time, I echo John Mack’s sentiment, namely, that this data is translucent, but not transparent.

Ruby Mechanize

Go the the mechanize library homepage to learn how to install it as a Ruby gem. It requires that nokogiri is installed, which you should’ve done if you’ve made it this far into my tutorials.

There are some basic examples on the project page, but you’re going to have to read some of the technical documentation to learn some of mechanize’s commands.

Here’s a code example we’ll be using:

search_form['txtXref']='00112233'
result_page_form = search_form.submit

search_form refers to a mechanize Form object. In that HTML form is a textfield with a name of ‘txtXref’. The array notation we used above is setting that textfield to the value ‘00112233’.

Then, using mechanize’s Form object’s submit method, we submit the form just as if we had clicked the “Submit” button on a webpage.

That’s the basic theory.

The Code

Note: The following code works, if you have an inmates.txt file from the last lesson (use this one if you don’t; keep in mind that the last names and birthdates have been changed/redacted). However, it’s very rudimentary, with no error-checking at all. Still, it’ll give you a couple tab-delimited files that will list an inmate’s past charges and past sentences served, with XREF being the key that links those files to inmates.txt.

Remember that you’re accessing a live site here. This script pauses for 2 seconds after each access…there should be no reason to be more frequent about it.

This tutorial will be updated in the future.

require 'rubygems'
require 'mechanize'
search_url='https://services.saccourt.com/indexsearchnew/CriminalSearchV2.aspx'
xrefs = File.open("inmates.txt", 'r').readlines().map{|x| x.split("\t")[7].match(/[0-9]+/).to_s}.uniq

# open datafile


a = Mechanize.new { |agent|
  agent.user_agent_alias = 'Mac Safari'
}

search_page = a.get(search_url) 
search_form = search_page.form_with(:name=>'frmCriminalSearch')

#show the fieldnames
search_form.fields.map {|f| f.name}
#=> ["__EVENTTARGET", "__EVENTARGUMENT", "__VIEWSTATE", "txtLastName", "txtFirstName", "txtDOB", "txtXref", "txtCaseNumber", "lstCaseType"]

search_form.buttons.map{|m| m.name}
# => ["btnFindByName", "btnFindByNumber"]


xrefs.each do |xref|
  puts "\nFinding info for xref: #{xref}"
  search_form['txtXref']=xref
  search_form.field_with(:name=>'lstCaseType').options[1].select
  result_page_form = search_form.submit.forms.first
  case_buttons = result_page_form.buttons[1..-2]

  puts "There are #{case_buttons.length} cases to check:"
  case_buttons.each do |cb|
    file_page = result_page_form.click_button(cb)
    file_page = file_page.parser
  
    charges_arr = []
    sentences_arr =[]
    charge_rows = file_page.css('#dgDispositionCharges tr')
  
    if charge_rows.length > 0
    puts "Charges: "
      charge_rows[1..-1].each do |cr|
        ctd = cr.css('td').map{|td| td.text}
        charges_arr << {:plea=>ctd[1], :charge=>ctd[2], :date=>ctd[4], :severity=>ctd[5]}
        puts "\t - #{charges_arr.last.collect().join("\t")}"
      end  
    end
  
    sentence_rows = file_page.css('#dgSentenceSummary tr')
  
    if sentence_rows.length > 0
      puts "Sentences: "
      sentence_rows[1..-1].each do |sr|
        sentences_arr << sr.css('td').map{|td| td.text}.join("\t")
        puts "\t - #{sentences_arr.last}"
      end
    end
    
    
    File.open("court_charges.txt",'a+'){ |f|

      charges_arr.each do |c|
        f.puts("#{xref}\t#{c[:plea]}\t#{c[:charge]}\t#{c[:date]}\t#{c[:severity]}")
      end
    }

    File.open("sentences.txt", 'a+'){ |f| 
      sentences_arr.each do |c|
        f.puts("#{xref}\t#{c}")
      end
    }
    
    
    
  
  end #done checking a case entry
  
  puts "Done with #{xref}, sleeping"
  sleep 1
  
  
end  

 

 
 

The Cops Reporter and the Log

If you’re a daily cops reporter, calling the police station to ask for the list of last night’s arrests is probably part of your job. Because many papers have some kind of cops blotter where arrested suspects are listed…and online and in print, this is usually one of a paper’s top features. The St. Petersburg Times has a modern version of the feature, complete with mugshots and stats summaries.

Arrest logs have sometimes been criticized for being little more than voyeurism (here’s a discussion over the St. Pete’s mugshot site). But knowing who your law officers are arresting, and why, is essential to a nice, free society (and for a fair and efficient police force). And the more data you have as a reporter, the better you’ll be able to cover your beat.

Most pro-active police departments will announce when they’ve made high-profile arrests. But relying on the police to tell you what the most noteworthy arrests are kind of begs the question, and doesn’t tell the whole picture of arrest activity. Most states consider arrest logs to be public information (not that that stops some jurisdictions from hiding them). But a paper list or a PDF is hard to analyze. Luckily, some police departments are putting their work on the Web They might be willing to send you a spreadsheet of arrest activity, but what if you wanted up-to-the-hour information, so that you could be aware of:

1. Suspected crimes that fall between egregious and infamous (non-fatal assaults, robberies, car jackings, etc.)
2. An abnormally large number of arrests at a given time
3. Unusual types of suspected crimes at a given time

This is where the web-scraping you learned in my last tutorial gets useful. You’re going to have an automated way of collecting the latest arrests news, in an ordered fashion (so that you could, for example, find the inmate with the largest bail at a given time), and you’ll save yourself and your friendly police PIO tedious paper shuffling and typing.

I’m going to base my lesson on this sheriff department’s jail system. I’ve mirrored a snapshot of their site here (zip file here), so I recommend you run your scripts on my mirror (root directory: https://danwin.com/static/jail-list/)before doing a real-world test.

The jail web site has these characteristics:

• At this page is a list of every person booked in the last 24 hours
• The list typically has 100 to 200 inmates at a time
• Most entries in that list contain a link to an inmate’s page containing data including name, DOB, bail, charges, booking time.
• Each inmate has a unique identifying number called X-REF
• Not all entries have a link; inmates who have been released have only their names listed

The site is pretty useful and user-friendly. However, it’s hard to quickly glean any useful information from the main list. You have to click through each individual entry to find out why someone was jailed. The purpose of the following lesson is to automate that process so you can efficiently get the big picture of a jail’s activity.

Program flow will go something like this:

1. Create two text files: one to store the list of inmates (inmates.txt), one to store the list of charges (charges.txt)
2. Open the inmate listing page
3. Collect each list entry
4. If list entry is not a link (i.e. inmate has been released)
1. Fetch first name, middle name, last name, intake time and release date
5. Else If list entry that is a link, open it
1. Fetch first name, middle name, last name, xref, intake time, and DOB of an inmate
2. Fetch and parse list of charges
3. Fetch the bail amount
6. In an each loop, for each inmate entry we collected above:
1. Output inmate information, in tab-delimited format, into inmates.txt, including the XREF.
2. Output the charges associated with the inmate into charges.txt. Each charge will take up one line, and the XREF of the inmate will also be included as to provided a key to the associated inmate

File I/O

We didn’t cover opening and writing to an external text file in the last lesson. So here’s how it goes briefly: Using Ruby’s IO class, we’re going to create two files, inmates.txt and charges.txt, and write to them what we find on the jail’s website. We’ll be using the variables inmates_file and charges_file to refer to the external files.

To open the the files and set the variables, use the IO class’s new method, which takes in two parameters: a string designating the file name, and a string
designating the mode…which in this case, will be “a”: write-only (read about the various modes here).

inmates_file = File.new('inmates.txt', 'a')
charges_file = File.new('charges.txt', 'a')

If these files don’t already exist, they will now. If they did, the ‘a’ mode will append new content to the end of the file.

To write something to the file, use the puts method, which writes whatever string you supply to it as one line in the file (we’ve used this method without the IO class, in which case it outputs to the screen):

charges_file.puts("Adding a new line of text to the charges file.")

While we’re setting up, let’s create an array of hashes, with each hash object holding an inmate and his/her information. We don’t have to do this…we could just output to the file each inmate record as we get to it, but this will allow us some flexibility later. All we have to do is initialize the array:

inmates_array = []

Open the inmate listing page

Now let’s fetch the inmates listing. We’ll be using Nokogiri in the same fashion we did in the last lesson, beginning by requiring the nokogiri and open-uri libraries, then using the Open-URI’s open method to fetch the page, and then Nokogiri’s HTML class to wrap up the page in a parsable format.

require 'rubygems'
require 'nokogiri'
require 'open-uri'
		
base_url='https://danwin.com/static/jail-list/' # all links on the list will be relative to this address		
inmate_listing = Nokogiri::HTML(open("#{base_url}current_listing.cfm.html"))

Let’s visit the page with a browser and examine the structure. The list is an HTML table, with each row containing several columns, the first column being the inmate’s full name and, if the inmate hasn’t been released, a link to his/her booking page.

If you inspect the HTML closely, you’ll see that this page is composed of several tables. What we want is the table contained inside the <td> element with a class of “content.gsub(/\302\240/, ‘ ‘).”

So we’ll collect all the table rows, using Nokogiri’s xpath method, and iterate through them using an each loop. We’re going to use a variation of an each loop called each_index, which provides the numerical index of the current iteration we’re on.

	inmate_rows = inmate_listing.xpath("//td[@class='content']/table")[0].xpath(".//tr").collect[1..-1]

The XPath syntax here is looking for a td element with class=’content’, then the table inside of that. There’s more than one, but the first one on the page has the data. From that, we gather all the rows (tr) within that. We call the collect method to convert the result into an array since Nokogiri’s xpath method returns a NodeSet, which won’t have the each_index method. each_index loops through an array, just like each, but it provides the index of the current iteration.

	inmate_rows.each_index do |i|
		inmate_row = inmate_rows[i]
		inmates_array[i] = {}
		inmate = inmates_array[i]

		# each row has a set of columns with the inmate info
		list_columns = inmate_row.xpath('./td')

Because we know we’re on the ith row, we can also initialize the ith index in inmates_array as a hash to store the ith inmate’s information. Remember that each element in the inmates_array is going to be a hash of information.

Lets use the variable named inmate as a shorthand way to refer to this position in the inmates_array .Each time we iterate through the loop, inmate will refer to the next spot in the inmates_array.

This is easier to type out 10 times than inmates_array[i]

Before we get to visiting the individual inmate pages, let’s just collect the name and other information readily available here

Each name consists of a String in this format: last_name, first_name middle_name

So let’s use the String split method. First to split the string by comma; this will give us an array with the first element being what’s on the left side of the comma. Splitting the second element of that array, with a space, will give us another array, consisting of a first name and middle name.

		
		
		
		# remember that you need to call Nokogiri's content method to get the text, as a String, between a tag	
		the_inmate_name =  list_columns[0].content.gsub(/\302\240/, ' ').strip.split(',')
		
		inmate['last_name'] = the_inmate_name[0]					# the name before the comma
		inmate['first_name'] = the_inmate_name[1].split(' ')[0]		# the name after the comma, but before the next space
		inmate['middle_name'] = the_inmate_name[1].split(' ')[1..-1]
		
		

I’m going to be using this method call after each use of content: gsub(/\302\240/, ‘ ‘).strip

Not all entries have a middle name. So we use the if the_inmate_name.length > 2 conditional statement to tell Ruby to skip this line if the_inmate_name

		
		# Moving on to the next table cell, which will be the 1 spot in list_columns
		inmate['sex'] = list_columns[1].content
		
		
		# next cell, DOB
		inmate['dob'] = list_columns[2].content
			
		# next cell, booking time
		inmate['intake_time'] = list_columns[3].content
		
	
		
		
		# let's go back to the first column to see if it contained a link
		if list_columns[0].xpath('./a').length == 0  # if there was no link, there would be 0 links returned
			
			# No link to visit, so this must have been a released inmate. Let's grab his/her release date 
			# which comes in the pattern "Released mm/dd/yyyy"...so we'll split the string and capture the second term

			inmate['release_date'] = list_columns[4].content.gsub(/\302\240/, ' ').split(' ')[1]
			
		else
		
			# visit link
			# we'll get to this subroutine in the next section
			
			
		end
	end

OK, that should’ve given you a refresher on arrays, hashes, XPath, and string manipulation. Now we’ll handle the case of when the first list_column array item does contain a link. It will involve fetching the page from that link and then more XPathing to pick out the wanted data.

At this time, go to the inmate list page and click on one of the inmate pages in the browser.

There’s a lot more information here; what will be most relevant to us right now is the X-Reference Number, charges, and bail. This next section of code will fit into the else branch of our previous section of code.

	# visit link (remember that the xpath method returns an array, so we have to explicitly refer to
	# the 0th index to get the link)
	inmate_link = list_columns[0].xpath('./a')[0]["href"] 
	
	# remember that we set base_url to contain the site's base address. we append 
	# inmate_link to it to get the absolute address to the inmate page
	inmate_page = Nokogiri::HTML(open("#{base_url}#{inmate_link}"))
	
	# everything is inside a <td> with a class="content" attribute, so let's set a variable
	# to hold the table rows inside
	
	content_table_rows = inmate_page.xpath("//td[@class='content']/table/tbody/tr")
	
	# the xref number appears to be in the third row and in the third cell
	# again, we're still using the inmate variable to hold the data associated with an inmate
	
	inmate["xref"] = content_table_rows[2].xpath("./td")[2].content.gsub(/\302\240/, ' ').strip
	# the strip method removes characters that are just space, such as tabs and carriage returns
	
	inmate['booking_number'] = content_table_rows[3].xpath("./td")[2].content.gsub(/\302\240/, ' ').strip
	inmate['arresting_agency'] = content_table_rows[13].xpath("./td")[2].content.gsub(/\302\240/, ' ').strip
	inmate['total_bail'] = content_table_rows[16].xpath("./td")[2].content.gsub(/(\302\240)|\s|\n|\r/, ' ').strip

OK, so we collected the basic info about each inmate. Now, we want to collect the charges leveled against them. This is a little bit trickier. If you inspect the table-cell containing the charges, you’ll see that the charge listing itself is a table. The first row of the table lists the case number and type of arrest (warrant, or fresh pickup). Below that is a list of charges, with each charge taking up two rows, like so:

For most of the inmate listings, this is immediately followed by another row listing the bail amount.

However, there are a few inmates who are held on more than one charge. And there are some who are being held from multiple charges stemming from multiple warrants, such as this person here, who appears to have racked up a number of public nuisance accusations, including evading ticket fare and prohibited public drinking. In his case, the charge listing is one row after another, and each row could either mention the case, the agency that issued the warrant, the charge, or the bail amount per warrant.

My point here is that you won’t be able to predict that the third row, for instance, always contains the charge code and severity. But using Inspect Element, we see that the table cells containing the code, severity, and description have class attributes “cellTopLeft”, “cellTopMiddle” and “cellBottom”, respectively. The bail amount per case is in the cell with class “cellBail”…but we’re not interested in bail per case, so we’ll ignore it.

We’re going to loop through rows inside this table, and if that row contains a td cell of class “cellTopLeft”, we know that each this row will contain the code and severity of a charge. We’re going to assume that the row immediately following it has a cell with class “cellBottom,” which contains the description.

Processing this sub-table of charges will require its own loop. And since each inmate could have more than one charge, we need to store “charges” inside our inmate hash…charges will point to an array. And each item in the charges array will itself be a hash, with keys of “code”, “severity”, and “description.”

Confusing? Well, here’s a quick diagram of what we have so far, in terms of variables:

inmates		=> an array of Hashes...
				inmate = inmates[index] (each inmate is a Hash)
			=> inmate['first_name'] => inmate's first name
			=> inmate['last_name']  => inmate's last name
			=> inmate['xref'] 		=> inmate's xref
			... all the other attributes
			=> inmate['charges']  =>  an array of hashes
						charge = inmate['charges'][charge_index] (each charge is a Hash)
						charge['code']			=> charge's code
						charge['severity']			=> charge's severity
						charge['description']	=> charge's description

The loop to fill out that charge array is as follows:

	# first, grab the entire table of charges that exists in the 16th row of the main content table
	table_of_charges = content_table_rows[15].xpath("./td")[2]
	
	# and give this inmate an array of charges
	inmate['charges'] = []
	
	# Now, collect all rows that have a td with class "cellTopLeft"
	charge_1st_rows =  table_of_charges.xpath(".//tr[td[@class='cellTopLeft']]")
	
	# Now, collect all rows that have a td with class "cellBottom"
	charge_2nd_rows = table_of_charges.xpath(".//tr[td[@class='cellBottom']]")
	
	# OK, you should do some basic error checking here. We expect the arrays of charge_1st_rows and charge_2nd_rows to have
	# equal length, since each charge has a code, severity and description, right?
	
	# If not, that means our assumption was wrong, and you should do something...like exit the script and re-examine your
	# datasource and assumptions about it. But I'll skip that for now
	
	charge_1st_rows.collect.each_index do |charge_row_index|
	
		# we found a row with a charge, so let's create a new hash that will hold the charge's attributes
		hash_of_inmate_charge = {}
		
		charge_1st_row = charge_1st_rows[charge_row_index]
		hash_of_inmate_charge['code'] = charge_1st_row.xpath('.//td')[0].content.gsub(/\302\240/, ' ').strip
		hash_of_inmate_charge['severity'] = charge_1st_row.xpath('.//td')[1].content.gsub(/\302\240/, ' ').strip
			
		# we assume that the row, with the same index in the charge_2nd_rows array will be the description of the charge
		# listed in charge_1st_rows
			
		hash_of_inmate_charge['description'] = charge_2nd_rows[charge_row_index].xpath('.//td')[1].content.gsub(/\302\240/, ' ').strip
		
		
		# push this hash on to the array of inmate charges:
		inmate['charges'] << hash_of_inmate_charge	
		
	end

Well, we've collected all the relevant inmate information, and if our assumptions were right, each of the inmate's charges. We've reached the end of the loop that examines each row in the main inmate listing. Our script will go onto the next inmate and collect his/her info. And so on until it has reached the end of the list. Here's all the code so far:

		require 'rubygems'
		require 'nokogiri'
		require 'open-uri'
		inmates_array = []
		base_url='' 		
		inmate_listing = Nokogiri::HTML(open("#{base_url}current_listing.cfm.html"))

		inmate_rows = inmate_listing.xpath("//td[@class='content']/table")[0].xpath(".//tr").collect[1..-1]
		inmate_rows.each_index do |i|
			inmate_row = inmate_rows[i]		
			inmates_array[i] = {}
			inmate = inmates_array[i]


			list_columns = inmate_row.xpath('./td')		
			the_inmate_name =  list_columns[0].content.gsub(/\302\240/, ' ').strip.split(',')
			inmate['last_name'] = the_inmate_name[0]					# the name before the comma

			inmate['first_name'] = the_inmate_name[1].split(' ')[0]		# the name after the comma, but before the next space
			inmate['middle_name'] = the_inmate_name[1].split(' ')[1..-1] if  the_inmate_name.length > 2	



			inmate['sex'] = list_columns[1].content		
			inmate['dob'] = list_columns[2].content

			inmate['intake_time'] = list_columns[3].content


			if list_columns[0].xpath('./a').length == 0 
				inmate['release_date'] = list_columns[4].content.gsub(/\302\240/, ' ').split(' ')[1]
			else

				inmate_link = list_columns[0].xpath('./a')[0]["href"] 
				inmate_page = Nokogiri::HTML(open("#{base_url}#{inmate_link}"))
				content_table_rows = inmate_page.xpath("//td[@class='content']/table/tr")

		    if content_table_rows.length > 0


			  	inmate["xref"] = content_table_rows[2].xpath("./td")[2].content.gsub(/\302\240/, ' ').strip
		  		inmate['booking_number'] = content_table_rows[3].xpath("./td")[2].content.gsub(/\302\240/, ' ').strip
		  		inmate['arresting_agency'] = content_table_rows[13].xpath("./td")[2].content.gsub(/\302\240/, ' ').strip
		  		inmate['total_bail'] = content_table_rows[16].xpath("./td")[2].content.gsub(/\302\240/, ' ').gsub(/\s|\n|\r/, ' ').strip

		  		table_of_charges = content_table_rows[15].xpath("./td")[2]
		  		inmate['charges'] = []

		  		charge_1st_rows =  table_of_charges.xpath(".//tr[td[@class='cellTopLeft']]")
		  		charge_2nd_rows = table_of_charges.xpath(".//tr[td[@class='cellBottom']]")

		  		charge_1st_rows.collect{|x| x}.each_index do |charge_row_index|

		  			hash_of_inmate_charge = {}

		  			charge_1st_row = charge_1st_rows[charge_row_index]
		  			hash_of_inmate_charge['code'] = charge_1st_row.xpath('.//td')[0].content.gsub(/\302\240/, ' ').strip
		  			hash_of_inmate_charge['severity'] = charge_1st_row.xpath('.//td')[1].content.gsub(/\302\240/, ' ').strip
		  			hash_of_inmate_charge['description'] = charge_2nd_rows[charge_row_index].xpath('.//td')[0].content.gsub(/\302\240/, ' ').strip

		  			# push this hash on to the array of inmate charges:
		  			inmate['charges'] << hash_of_inmate_charge	

		  		end
				end # end if content_table_rows

			end
		end
	

Storing your Data into a File

At this point in your script, all your carefully collected data is in memory. When the script finishes execution, it disappears. That defeats the purpose of any way of tracking data. So let's store it in a persistent way...my choice would be in some kind of database, like MySQL or SQLite. But for our purposes, we can quickly learn the methods to store this information in a tab-delimited file that can be opened as an Excel spreadsheet.

We will be using Ruby's File class:

			##write to file
			File.open("inmate.txt", 'w'){ |f| 

				f.write("first_name\tmiddle_name\tlast_name\tsex\tdob\tintaketime\trelease_date\txref\tbooking_number\tarresting_agency\ttotal_bail\n")

				inmates_array.each do |inmate|

			f.write("#{inmate['first_name']}\t#{inmate['middle_name']}\t#{inmate['last_name']}\t#{inmate['sex']}\t#{inmate['dob']}\t#{inmate['intake_time']}\t#{inmate['release_date']}\t#{inmate['xref']}\t#{inmate['booking_number']}\t#{inmate['arresting_agency']}\t#{inmate['total_bail']}\n")

				end
			}

A quick explanation. The File class has the open method, to which we pass in two arguments: the name of the file we want to write to, and the mode. In this case, we're using 'w', which stands for "write" mode. The curly-braces sets off the code that gets executed while this File is open, with the variable f referring to the actual file.

File also has an instance method called write, which takes in a String as an argument to write to the open file.

Backslash-t will write a tab, and backslash-n will write a newline character.

The next block of code is similar to the first...but it refers to a "charges.txt" file. Remember that each inmate could have more than one charge to his/her name. The following file lists every charge, but also lists the xref key to tie back into inmates.txt. For convenience sake, we're also going to print out the inmate name and the inmate's total bail on each line.

			File.open("charges.txt",'w'){ |f|
			  f.write("name\txref\ttotal_bail\tcode\tseverity\tdescription\n")

			  inmates_array.each do |inmate|	  
				  if inmate['charges']
				    inmate['charges'].each do |charge|
			  	    f.write("#{inmate['first_name']} #{inmate['last_name']}\t#{inmate['xref']}\t#{inmate['total_bail']}\t#{charge['code']}\t#{charge['severity']}\t#{charge['description']}\n")
			      end
			    end
				end

			}
		

Printing out the inmate's name and total bail, although redundant, allows us to quickly skim the list to see if there were any unusual crimes connected to unusual amounts of bail (note that the jail site does not breakdown bail amounts per charge).

Putting it all together for the real world

The above code, put all together, will execute cleanly and compile some nice text files for you, especially if you've saved the package of HTML files onto your hard drive. But in the real world, you'll be targeting an internet server, which may not like you hitting it at a rate of five times per second. Or, may intermittently fail.

To deal with this, I've added a call to Ruby's sleep method, which pauses script execution for a given number of seconds. I've also thrown in some error-handling. Here's the basic structure:

		# some code
		begin
			# risky code here
			# The Ruby interpreter will watch the code that gets executed within the begin branch...if something goes wrong, it's going to execute code in the following rescue branch
	
		rescue
			# the begin-branch messed up, time to run some other code
			puts "An error happened!"
		else
			# this code gets executed if the begin-branch worked fine
		ensure
			# this code in the ensure branch (which is optional) runs no matter what.
			puts "We're done with our error handling"
		end
	
		

Read more about error-handling here.

And finally, I'm going to make a few alterations to the script to make it so that it'll run repeatedly for every half hour (essentially, by sleeping a half hour after going through the list). This is the crudest way to schedule a script, but it'll work for now. It will also use another instance method of File: readlines.

Each half hour, it's likely that the list of inmates will be the same. So a crude way to reduce the number of repeat listings is to check the inmates.txt file (using the match method) to see if a given inmate's xref number is in there. This gets slower as inmates.txt grows. Like I said, it's crude. I prefer using a database, which is a topic outside the scope of this tutorial.

So I've taken the code above and split it into five parts:

1. the process_inmate_row method - This method takes in a single row from the list of inmates and reads the basic information, including name, sex, and date of birth. It takes in as its second argument the entire text of inmates.txt and sees if inmate.txt already contains the name. If so, it will return a hash of inmate data. If not, it will return nil

   Note: As said previously, constantly searching the entire inmates.txt file is incredibly inefficient. And, what happens if two John Smiths are arrested in the same time period? The name-check will fail to differentiate inmates of similar names (an even better match method would involve using the date of birth). But I leave it as an exercise for you to develop a more efficient method, which could involve a database. Or storing the name columns of inmates.txt into an array.

   But the reason why we're doing the name-check is to save us the time of entering an inmate's page. And, of course, to not fill the inmates.txt file with duplicate entries.

2. the process_inmate_page_link method - The code that fetches an inmate's individual page and then processes the extra data, including the total bail amount and charges, is done here. It returns a hash of the inmate data.
3. the write_to_file method - This code invokes the File.open methods and, for each inmate and charge, writes a tab-delimited line to the inmates.txt and charges.txt files
4. the check_the_site method - This is the master method. It retrieves the list of inmates from the jail site and then on each row of inmate data, calls all the previously defined methods. IT also has some basic error handling. If something happens, like your internet connection drops in the middle of a page retrieval, it skips the current inmate and moves on. This is better than just crashing.
5. The main execution loop - All the code previously written out as methods will do nothing unless you actually invoke the methods. So we initialize a variable, called hours, to zero and while that is less than 24, we run the check_the_site method. After check_the_site finishes, hours is incremented and the script sleeps for an hour (3600 seconds).

require 'rubygems'
require 'nokogiri'
require 'open-uri'


def process_inmate_row(inmate_row, inmate_text)
  
	list_columns = inmate_row.xpath('./td')		
	inmate = {}
	the_inmate_name =  list_columns[0].content.gsub(/\302\240/, ' ').strip.split(',')
	inmate['last_name'] = the_inmate_name[0]					# the name before the comma  
	inmate['first_name'] = the_inmate_name[1].split(' ')[0]		# the name after the comma, but before the next space
	inmate['middle_name'] = the_inmate_name[1].split(' ')[1..-1] if  the_inmate_name.length > 2	
  
  # at this point, we can determine if the inmate is already in our textfile
  
  name_to_match="#{inmate['first_name']}\t#{inmate['middle_name']}\t#{inmate['last_name']}"  
   # remember that in the text file, we tab-delimited the name, so we have to match that pattern
	
  
   if inmate_text.match(name_to_match)
     puts "NOT adding inmate #{name_to_match} to inmates txt, as it already exists"
     inmate = nil
     # the method that invoked process_inmate_row will only add the inmate if it is not nil
     # we DON'T want this inmate added, so that's why we're setting it to nil
		
   else  
     
     	puts "Adding inmate #{name_to_match} to inmates txt"
 		  inmate['sex'] = list_columns[1].content		
   	  inmate['dob'] = list_columns[2].content
   	  inmate['intake_time'] = list_columns[3].content
	    puts "Basic info of inmate: #{inmate['first_name']} #{inmate['last_name']}: #{inmate['dob']}"
       
  end
  
  
  return inmate
  
end


def process_inmate_page_link(inmate_link)
  	inmate_page = Nokogiri::HTML(open(inmate_link))
		content_table_rows = inmate_page.xpath("//td[@class='content']/table/tr")

    more_inmate_stuff = {}
    
    if content_table_rows.length > 0
      
	  	more_inmate_stuff["xref"] = content_table_rows[2].xpath("./td")[2].content.gsub(/\302\240/, ' ').strip
  		
  		more_inmate_stuff['booking_number'] = content_table_rows[3].xpath("./td")[2].content.gsub(/\302\240/, ' ').strip
  		more_inmate_stuff['arresting_agency'] = content_table_rows[13].xpath("./td")[2].content.gsub(/\302\240/, ' ').strip
  		more_inmate_stuff['total_bail'] = content_table_rows[16].xpath("./td")[2].content.gsub(/\302\240/, ' ').gsub(/\s|\n|\r/, ' ').strip

  		puts "Found more inmate info, total-bail: #{more_inmate_stuff['total_bail']} arresting-agency: #{more_inmate_stuff['arresting_agency']}"


  		table_of_charges = content_table_rows[15].xpath("./td")[2]
  		more_inmate_stuff['charges'] = []

  		charge_1st_rows =  table_of_charges.xpath(".//tr[td[@class='cellTopLeft']]")
  		
  		puts "Number of charges: #{charge_1st_rows.length}"
  		charge_2nd_rows = table_of_charges.xpath(".//tr[td[@class='cellBottom']]")

  		charge_1st_rows.collect{|x| x}.each_index do |charge_row_index|

  			hash_of_inmate_charge = {}

  			charge_1st_row = charge_1st_rows[charge_row_index]
  			hash_of_inmate_charge['code'] = charge_1st_row.xpath('.//td')[0].content.gsub(/\302\240/, ' ').strip
  			hash_of_inmate_charge['severity'] = charge_1st_row.xpath('.//td')[1].content.gsub(/\302\240/, ' ').strip
  			hash_of_inmate_charge['description'] = charge_2nd_rows[charge_row_index].xpath('.//td')[0].content.gsub(/\302\240/, ' ').strip

  			# push this hash on to the array of inmate charges:
  			more_inmate_stuff['charges'] << hash_of_inmate_charge	
    
        puts hash_of_inmate_charge.collect.join(" | ")
  		end
  		
  	else
  	  "Could not find more inmate info"	
		end # end if content_table_rows
  
    return more_inmate_stuff
end

def write_to_file(inmate)

    ##write to file
    puts "Writing to inmates.txt"
    
    # note that we use the 'a' mode here, which will append new input onto the end of an existing file (or create a new one if it doesn't exist), instead of overwriting it
    # Obviously, we don't want to keep overwriting inmates.txt if we intend it to be a persistent record of the inmate log
    
    File.open("inmates.txt", 'a+'){ |f| 
	f.write("first_name\tmiddle_name\tlast_name\tsex\tdob\tintaketime\trelease_date\txref\tbooking_number\tarresting_agency\ttotal_bail\n") unless File.size(f) >= 0 
      # we don't want to repeatedly print the column headers    
      f.write("#{inmate['first_name']}\t#{inmate['middle_name']}\t#{inmate['last_name']}\t#{inmate['sex']}\t#{inmate['dob']}\t#{inmate['intake_time']}\t#{inmate['release_date']}\t#{inmate['xref']}\t#{inmate['booking_number']}\t#{inmate['arresting_agency']}\t#{inmate['total_bail']}\t#{Time.now}\n")
   
    }
   
    puts "Writing to charges.txt"

    File.open("charges.txt",'a+'){ |f|
      f.write("name\txref\ttotal_bail\tcode\tseverity\tdescription\n") unless File.size(f) >= 0 
      # we don't want to repeatedly print the column headers
      
  	  if inmate['charges']
  	    inmate['charges'].each do |charge|
  	      puts "Writing charge: #{charge['description']}"
    	    f.write("#{inmate['first_name']} #{inmate['last_name']}\t#{inmate['xref']}\t#{inmate['total_bail']}\t#{charge['code']}\t#{charge['severity']}\t#{charge['description']}\t#{Time.now}\n")
        end
      end
  
    }
end
  
  

def check_the_site(base_url, index_url)
  # read the contents of inmates.txt into a variable so that we can check to see if an inmate already exists
   inmate_text = File.exists?("inmates.txt") ? File.open("inmates.txt", 'r').readlines().join() : ''
   inmates_added_count = 0 # just a piece of info we want to keep track of. We'll increment this number on each successful add
   
    
    begin	
      inmate_listing = Nokogiri::HTML(open("#{base_url}#{index_url}"))
    rescue Exception=>e
      puts "Oops, had a problem getting the inmates list at #{Time.now}"
      return nil #get out of here.
    end
      
    inmate_rows = inmate_listing.xpath("//td[@class='content']/table")[0].xpath(".//tr").collect[1..-1]
    puts "There are #{inmate_rows.length} rows to process"
    inmate_rows.each_index do |i|
  
      puts "\nProcessing inmate row: #{i}"
      inmate_row = inmate_rows[i]
      
      begin
        # The following code is potentially risky; we're making calls to process_inmate_row and process_inmate_page_link, two methods that could potentially throw an error if the data is improperly formatted or if the website refuses to send data
        
        # I've set up some rudimentary error handling to notify you of an error, but to keep chugging along to the next row
        
        inmate = process_inmate_row(inmate_rows[i], inmate_text)
        
        # process_inmate_row will return a hash of inmate data 
        # BUT, it will reutrn nil if it turns out this inmate already exists
        # so here's another if branch to check for that
        
        if inmate.nil?
          # do nothing
        else  
          # inmate was not blank, so let's continue
          list_columns = inmate_row.xpath('./td')		
        	if list_columns[0].xpath('./a').length == 0 
        		inmate['release_date'] = list_columns[4].content.gsub(/\302\240/, ' ').split(' ')[1]
        		puts "inmate was released on #{inmate['release_date']}"
        	else
        	  inmate_link = list_columns[0].xpath('./a')[0]["href"] 
        	  inmate_link = "#{base_url}#{inmate_link}"
        	  puts "Fetching: #{inmate_link}"
            more_inmate_attributes = process_inmate_page_link(inmate_link)
            inmate.merge!(more_inmate_attributes)
          end
    
          
        end # end of the if inmate.blank? branch
      rescue Exception=>e
        puts "Oops, had a problem getting data from inmate row #{i}, Error: #{e}"
      rescue Timeout::Error => e 
        puts "Had a timeout error: #{e}"
        sleep(10)
      else
         # got all the info for the inmage, so lets add him/her to the file
        
        unless inmate.nil?
          write_to_file(inmate) unless inmate.nil? 
  	      # an inline conditional: remember that inmate was set to nil if it already existed in the text file
  	      # we don't want to add it to the main array in such a case, hence the 'unless'
  	      inmates_added_count+=1
  	        puts "We successfully queried the site, so let's sleep a second"
          	sleep 1
  	    end

        
        
      end
    
    end
	
	  # reached the end, let's print a summary:
	  puts "#{Time.now}: Out of #{inmate_rows.length}, we added #{inmates_added_count} inmates"
	
end

   


hours = 0
BASE_URL='https://danwin.com/static/jail-list/'

while(hours < 24)
  puts "Checking the site (#{hours} out of 24 times):"
  puts "***********************"
  check_the_site(BASE_URL, 'current_listing.cfm.html')
  #run the code that hits the site and processes the links...this method also returns an array of all the inmates
  
  
  
  hours += 1 # increment the counter, or this will run forever...
  puts "sleeping till next iteration"
  
  sleep_count = 0
  while(sleep_count < 1800)
    sleep(1) #sleep for an hour
    sleep_count +=1
    puts "Will check again in #{(1800-sleep_count)/60} minutes" if sleep_count%60==0
  end
  
  
end

    

Who this post is for

His Girl Friday

You’re a journalist who knows almost nothing about computers beyond using them to connect to the Internets, email, and cheat on Facebook scrabble. This is not entirely trivial; if you’re able to do this without typing your password and SSN into a phishing site, you’re (sadly) a step ahead of most of the Internet populace. OK, it’ll also help if you’re familiar enough with your operating system (Windows or Mac…I’m assuming anyone using Linux won’t even need this tutorial) to know how to install programs.

Anyone who has taken a semester of computer science will scoff at how I’ve simplified even the basic fundamentals of programming…and they’d be right…but my goal is just to get you into the basics to write some useful code immediately. You’re going to have to make the effort yourself to learn the topics in-depth.

Thankfully, coding is something that provides immediate success and failure. You hit Ctrl-R, your script runs, and in five seconds or less, you’ll learn if you did right. The more you fumble, the more you learn. And getting around an error no longer requires owning a reference library.

The roadmap

This tutorial aims to walk you through the bare essentials of HTML, programming theory and tools so that you can do something very practical: build an automatic process to gather data from websites. I made this lesson into one giant page so you can see for yourself, in one glance, the number of words (about 9,000) it takes to touch upon what is essentially one semester in a first-level computer science course. Also, I have no ads to sell.

Here’s what will happen if you read this entire page:

1. Learn a little HTML
2. Install Firefox+Fire Bug
3. Install Ruby, a programming language
4. Learn some programming theory
5. Write a script
6. Execute the script

Jump to the table of contents or read some more blab.

What is web-scraping and how it’s important to journalists

Web-scraping (also called screen-scraping) is the automated process of collecting the *useful* data off of a webpage. This is made possible because of the design of HTML, which, when done right, puts this data in as predictable a format as an Excel spreadsheet…sans the convenient interface, keyboard shortcuts, and Clippy. So you have to write your own tool tailored to the structure of a webpage.

The importance of data collection should be obvious to a journalist. Used to be, if you wanted a set of data…such as the list of restaurant inspections so you could do a regression analysis of failed tests with respect to neighborhood income levels, you’d ask them for the data, sue them if they said no, and if you were on the right side of the law, they’d grudgingly hand you a chunk of ordered text that you could eventually put into a spreadsheet.

But now, it’s possible that a public-information officer will just point you to the public website and say, there it is. And it’s not always a case of them being ignorant/disdainful of laws that oblige them to give the dataset, in electronic form, that backs the website. From their viewpoint, the information is there for any idiot with an Internet connection to ask for, so what are you whining about?

At this point, you can either go through a weeks-long argument through emails and phone messages that ends with their legal counsel compelling the PI officer to hand over the data. Or, if keeping your story idea secret isn’t a priority, you could explain what your intent is, and why you need a whole dataset to see if a trend exists. Either way, you almost might have another week or so of waiting for the PIO to successfully wrangle their tech people (and legal staff, who need to vet the released data for any confidential info) to giving you the data in a nice comma-delimited format.

So, if their website already has the information you need (although, often, the web display omits record keys and such that are useful), why not write a script in 15 minutes to grab it? Also, even if data is released willingly, it’s not always at a convenient pace. If a website is updated faster than a PIO can send you email attachments, then scraping the website on a nightly basis will save both of you headaches.

And some types of information is just not FOIA-able. My former colleague Brian Boyer, now news-apps chief at the Tribune, created ProPublica’s ChangeTracker, built on a web-scraping service, to check when and how the White House changes its website. The request, “Hey, can you tell me all the times you’ve changed text on your website, what the text originally was, and what you changed it to” is not something a PIO could, even if he/she wanted to, could easily fulfill.

Web-scraping sometimes has bad connotations…because this is how various members of royalty find your email address in order to tell you that they are a distant family relation with $10,000,000US that they desperately want to give to you. So yes, you could use it for ugly purposes. My response is that if that’s your ultimate goal, you are way behind the game, and you will probably suffer a humiliating karmic fate, either in your online or real life.

On the other hand, there are innnumerable sets of public, useful data that no one has gotten around to mapping out and collecting, in a useful format. So let’s get to it.

The task

Thomas Jefferson lived to be 83, according to Wikipedia

When you get through this tutorial, you will be able to answer the question: According to Wikipedia, what is the average age of U.S. Presidents whose last names have more than six characters? Not an important question, but it is on the same order of difficulty as, say, scraping a county jail’s booking list to find the inmates with the largest bail amount and charge list, and how many are repeat-offenders…which are the second and third lessons.

Table of Contents

1. The basics of HTML
   • Tags
   • Hyperlinks
   • Install Firefox and Firebug
2. Programming Basics
   • Installing Ruby
   • The Ruby Interactive Prompt
   • Strings
   • Variables
   • Comparison Operators
   • Conditional Branches
   • Methods
3. Writing Your Script
   • Nokogiri – a Ruby parser of HTML
   • Running scripts from Text Editors or the Command Line
   • Step 1: Fetch the Presidents List
   • XPath
   • Step 2: From a Table of Data, Fetch the President’s Name
   • Step 3: Determine if the Last Name Is Longer Than 6 Characters
   • Step 4: If So, Fetch President’s Page
   • Step 5: Grab the age from the president’s page
   • Regular Expressions
   • Step 6: Add up the data you gathered
   • The complete script

HTML

HTML is what makes web pages not just a stream of characters. Why did that “not” in the previous sentence appear bold?. Because I wrapped the word “not” in tags. The raw code is: <b>not</b>

The design and theory of HTML are topics that could consume the rest of your waking life. For now, it’s relevance to us is that with HTML, web pages have structure. And with structure, a web-scraper can reliably collect the useful bits of data as it would from columns of a spreadsheet.

W3Schools is the best place to get a primer on HTML.

Tags

Tags are themselves contained in angle brackets (< and >) and come in pairs. The end tag is denoted by a forward slash: /.

So, anything between these tags – <i> </i> – will appear in italics.

To make something a headline, use <h1> </h1> tags. You can replace that ‘1’ with numbers 2 through 6, with ‘1’ being the most prominent kind of headline.

Here is a h1 headline

Here is a h4 headline

OK, one more critically important thing about tags. They can have attributes.

Let’s say I wanted to make something not only be a headline (i.e. bold large text), but the color red. There are many ways to do this, but let me show you the most simple (if not totally standards-compliant) way to illustrate the simplest form of an attribute:

An attribute consists of: the name of the attribute, an equals sign, and then the value of that attribute enclosed in quotation marks. Like so: attribute=”this_is_the_attributes_value”

<h1 color="red">This is a headline</h1>

In that starting tag – <h1> – is where attributes go– after the tagname, h1, and before the closing right-angle-bracket. The name of the attribute, color is followed by an = sign. Then quotation marks (or single quotes; either way, they have to match, as they would when you write down someone’s quote, or someone quoting a quote) enclosing the value of the attribute. In this case, red.

HTML Errors

Couple of things to keep in mind. Tags come in pairs. When things look funny on a hand-coded webpage, usually it’s because the coder didn’t provide a closing tag to his starting tag. Here’s a properly tagged sentence:

<b>This sentence is meant to be bold.</b> <i>This sentence is just in italics.</i>

Results in: This sentence is meant to be bold. This sentence is just in italics.

In this sentence, I didn’t provided a closing bold tag, and so the bold part overlaps into the italics sentence, making a bold AND italicized sentence:

<b>This sentence is meant to be bold. <i>This sentence is just in italics.</i>

Results in: This sentence is meant to be bold. This sentence is just in italics.

Also, close the tags in the order they come in…I don’t know how to concisely explain this point, but the following is not properly-structured HTML. The part in red denotes how the closing-bold-tag should NOT come after the opening italics tag:

<b>This sentence is meant to be bold. <i></b>This sentence is just in italics.</i>

Sometimes browsers will compensate for coder-error and interpret this in a way that doesn’t look awful. But you just need to know that this violates a principle of HTML…and pages that you scrape that aren’t well structured may give strange results even if you’ve written a logically-designed scraper.

Hyperlinks

Hyperlinks are those (depending on a website’s style) underlined words that, upon clicking, send you to a whole different page. They are nothing more than special tags with an important attribute.

The tagged hyperlink makes the word “link” a clickable link that goes to Google. The href attribute describes where the link sends you:

This <a href="http://google.com">link</a> has many answers

Results in:

This link has many answers.

Want to try some tags and hyperlinks yourself? Use W3Schools interactive editor.

Firefox and Firebug

As I wrote earlier, HTML structures the data you want. But you need to know how it’s structured, and so you need to know the designer’s blueprint. Not to get in a browser war, but just to make things easier on me, you can’t go wrong by first downloading Firefox, the free open-source browser by Mozilla.

Now go to any website, right click on an empty space, and click “View Source” in the submenu. You’ll likely see something like this:

That’s the raw HTML. You might eventually get to the point where HTML is what the Matrix is to Neo. But let’s make it as painless as possible. Firefox has many plugins, including one called Firebug, which makes it very easy to dissect code. Get it here.

Firebug, a plugin for Firefox

Double-click on one of the sample headlines in this tutorial to highlight it. Then right-click to open the submenu, then click “Inspect Element“. This should bring up a Firebug panel that lets you see the HTML that made that headline. This saves you from having to search through the entire source to find that headline, just to see the tags that wrap it.

Like I said, in order to successful web-scrape, you’re going to have to know how the elements – the paragraphs, headlines, and links – were structured. Firebug is a tool that helps pinpoint the elements you want to know about.

Programming Basics

A good way to annoy a programmer is to say something like, “Yeah, I have some programming experience: I’ve been writing HTML for two weeks now.” Writing HTML is not programming, any more than operating a stereo equalizer makes you a classically-trained guitarist. HTML is a way to describe and present content, but you’re not running any kind of computerized task.

So, I went through the basics of HTML so you’d be familiar with the content that you’d be collecting. Now we’ll learn the basics of how to program a script that will actually collect that content.

Installing Ruby

What is Ruby? It’s a programming language. And like a spoken language, once you’ve learned one, you’ve learned the fundamentals (i.e. the concepts of verbs, nouns, sentences, etc.) that allow you to try out all the other ones. Ruby is also the basis for Ruby on Rails, a very popular framework that many developers use to build data-driven websites. But right now, we’re collecting data from websites, not building them.

Installation instructions for Ruby are here…if you’re on a Mac OS X with Leopard or better, you should be good to go. Hopefully, the one-click installer for Windows should be easy enough to install (check the Enable RubyGems and SciTE boxes).

The One-Click Ruby Installer for Windows

The Ruby Interactive Prompt (IRB)

If you belong to the target-audience of this tutorial, you probably have been able to get your computer to perform tasks (such as, ‘Open my web browser’) with your mouse-clicking. Programming means you’re going to be typing out lines of code that executes tasks. Your web-scraping is essentially going to be a sequence of such commands, i.e. a script.

But why wait until you get a complete script when we can start executing commands right now? This is where Ruby’s Interactive Prompt (IRB) comes in. In its simplest form of operation, the IRB waits for you to type in a line of code, then for you to hit “Enter/Return”, and then it will run your command, provided it makes sense.

On Windows, go to your menu and type ctrl-R to bring up the Run… prompt. Type in ‘cmd’. Then type in ‘irb’. On the Mac, go to Applications=>Terminal. At the command line, type in ‘irb’.

Interactive Ruby prompt

Now that you’re here, type in the following:

1+6
	#result: 7

Congrats. You just wrote a one-line script to figure out what one plus six is.

Let’s also learn a common Ruby command: puts. It simply outputs what comes after it (actually, not quite that simple, but you’ll learn soon in the next section)…I’ll be using this in the script to output results.

		puts "Hello World"
		#result: Hello World

Read more about the command-line interpreter.

Strings

Let’s say you want to be a little more narrative about the above 1+6 calculation. Try writing out those numbers and enclosing them in quotation marks. Like so:

	"One"+"Six"
	# result: "OneSix"

Your answer won’t be “Seven”, but “OneSix”. Why? To human eyes and ears, 1+6 and “One”+”Six” might be the same. But in Ruby, and most other programming languages, the computer interprets the latter command to be joining two words, i.e. strings together.

In the programming-world, “six” is fundamentally different than 6. “Six” is what Ruby considers a String. 6 is a Number.

So what happens when you try to add “Six”, the string, to 6, the number?

"Six"+6

TypeError: can't convert Fixnum into String
	from (irb):2:in `+'
	from (irb):2
	from /usr/local/bin/irb:12:in `'

Congrats, it’s your first of many, many times of making the Ruby interpreter choke. In the case of numbers and strings, it only knows how to add like items together.

The takeaway from this is that, for our purposes, anything in quotation marks is a string. Even a number in quotation marks is no longer a number. You’ll get the same above error if you try:

"6"+6

The quotation marks make all the difference, just as they do in the journalism world. For example:

The governor is a scumbag who molests staffers on taxpayer-dime
by Dan Nguyen, Newswire, Inc.

Whistleblower: “The governor is a scumbag who molests staffers on taxpayer-dime”
by Dan Nguyen, Newswire, Inc.

Variables

OK, you now know that you shouldn’t add strings to numbers, and you’re perfectly content to add strings to create results like “eightzero”. What if you tire of typing quotation marks?

	eight+zero
	# NameError: undefined local variable or method `eight' ...

What happened here? Well, without quotation marks, eight and zero are no longer considered strings. In their unquoted form, they are considered variables that hold some kind of value.

Think back to algebra when you were asked to solve “x+1=6″. You weren’t supposed to interpret that as:
the letter x added to the number 1 equals 6

The x is a stand-in for the value 5. x could’ve been a, b or y.

So, to make eight+zero understandable by the Ruby interpreter, you must assign those two terms values. So, try:

eight=8
zero=0
eight+zero
# result: 8

Now, eight+zero is the same as 8+0.

Enter the following into the IRB:

zero=1
eight+zero
# result: 9

You should get 9 as the result. The variable eight is still 8. But you assigned zero the value of 1. Therefore, you were asking the interpreter to execute 8+1.

Here’s what you should grok by now: unquoted words are considered to be variables, and they are empty unless you’ve assigned them a value. And the name of the variable is completely independent and unrelated to its actual value. Thus, nine=”nine” makes as much sense in Ruby as this_variable_has_a_value_that_is_not_nine_dang_it=”nine”

Obviously, since you can name your variables just about anything (stick to a series of lowercase letters and numbers with no spaces or hypens), name them something that is related to their actual value, so that your code is more readable.

At this point, we’ve run through a lot of programming concepts. But if you don’t understand how the above examples, and the following:

one = 1
one = 2  # assigning the variable named one to another value
one + one
# result: 4

…then pause for a moment. It’s not a trivial topic, but it is critical to understand it at least at this level. Go here for more discussion on variables.

Comparison operators

Let’s say you’ve written a bunch of code and forgot whether you set the variable eight to “eight” or 8. How to test that? Well…typing in eight and hitting ‘Enter’ is the easy way…but now’s a good time to learn the concept of a comparison.

We already know that =, the equals sign, is something that assigns a value: what’s on the right of the = is set as the value of the variable on the left side.

So what’s a double equals sign == mean?

Write this sequence of code:

eight="eight"
eight==8
# result: false

The second line of code, translated into English, is you telling the interpreter:

The value of the variable named eight is the number 8

To which the computer responds: false

Here, Ruby is telling you that the string “eight”, to which the variable eight was assigned, is not equal to the number 8.

Which we, from vainly trying to add “eight”+8, know is how Ruby interprets things. Evaluating eight==”eight” will yield the value of true

Arrays

Think of an Array as something that contains a sequence of other variables and values. In Ruby, and most other languages, arrays are set off by square brackets, [ and ].

Here’s the easiest way to initialize an Array:

an_empty_array = []
array_with_numbers=[1,2,3,4]

Above, I’ve assigned two variables the values of two different arrays. The first, an_empty_array, is empty. The second, array_with_numbers, is filled with four numbers. You could’ve written out four lines of code, assigning four different variables respectively with the numbers 1 through 4. With an array, you essentially have one variable referring to 4 values.

How do you access the individual values? Use the name of the variable, and then the index. Consider the index as an address) of the element you want, set off by square brackets (in this fashion, the square brackets denotes the variable they follow is an array, while the value inside them is the index/address). Such as:

array_with_numbers[0]

In Ruby, the first element of an array has an index of 0. So the above line would give you the value of 1. array_with_numbers[3] would get you 4. The index 4 in array_with_numbers would get you an empty (nil) value.

Arrays can contain other variables too, like so:

an_empty_array = []
array_with_numbers=[1,2,3,4, an_empty_array]

array_with_numbers[4] would now yield [], an empty array, which is the value of the variable named an_empty_array

More about Arrays here.

Hashes

OK, I’m going to make another vast simplification of a programming object: Hashes can be considered Arrays in which the indexes are strings, not numbers. Hashes are denoted by curly brackets.

a_hash = {"one"=>1, "two"=>2, "three"=>3}

Note the convention of => which assigns a value to an index (the correct term, actually, is key) of the hash. So:

a_hash["two"]
# result: 2

It’s not important right now to understand the full differences and capabilities of Arrays and Hashes, but you’ll be seeing this notation in the script we write.

Read more about Hashes here.

Conditional Branches

So far, we’ve been typing in single line commands. Your final script is going to be a long list of commands telling the computer to:

1. Go to Wikipedia’s listing of each U.S. President’s page (i.e. a list of links to each page)
2. Visit, via hyperlink, each page belonging to a president whose last name is longer than six letters
3. Grab the president’s age from each individual page, if that president is dead
4. Average those ages

Our criteria for inclusion means we have to come up with some way to not visit, say, John Adams’s Wikipedia page. And to not include a living president’s age. So inside our script, there’s going to be a section of code telling the computer to go into a webpage…but that code should only execute if the length of a President’s last name is greater than 6.

That’s where the if conditional branch comes in. Without getting too far past the basics, here’s the simplified code:.

president = "John Adams"
last_name_length = 5  # I manually set this variable for now; in your actual script, you'll find this value programmatically 

if last_name_length > 6
 # then go to his wikipedia page...and while we're in this branch of code, let's print something
 puts "Entering a page"
else
 #OK, don't go there. But let's print out a statement
 puts "This name is too short"
end

# result: "This name is too short"

What the above section of code is essentially saying is that if the value of the variable last_name_length is greater than 6, then do what was in between if and else. Otherwise, completely skip what was there and go to what’s between the else and end

The else is optional…if you want, you could do nothing if the conditional statement (if last_name_length > 6) isn’t satisfied. The end is required; it tells Ruby that that’s the end of that optional branch of code that started with the if.

Up till now, our series of commands have been straight-forward: the interpreter executes one line after another. Introducing the if statement has introduced a fork in the road; if the condition in the if statement isn’t met, the interpreter skips past that if block.

The if statement is the simplest of such conditional branches. All you need to know for now is that there’s a way to tell the Ruby interpeter to execute a certain bit of code if a condition is met. Read more about it here.

Methods

I’m really going to be brief here. Think of methods as a set of commands that are useful enough to run more than once.

Let’s say I need to take two numbers, multiply them together, subtract 5 from the product, and then add the result to itself. In code, that would be:

#initialize the variables:
a = 10
b = 20

#now make each step its own line
c = a * b
c = c - 5
c = c+c
# result: 390

Well, that could’ve been one line, without using the placeholder variable named c, like so:

(a*b)-5 + (a*b)-5

If I need to run this more than once, it’s a bit annoying to type out each time we want to run that series of commands, so let’s define a function called my_funny_equation

def my_funny_equation (first_argument, second_argument)
  answer = (first_argument*second_argument)-5 + (first_argument*second_argument)-5
  return  answer
end

Inside the parentheses, following my_funny_equation are the arguments, the values that you want the method to work with.

The takeaway here is that I’ve encapsulated my series of commands into a block of code. The variable names, arbitrarily named first_argument, second_argument, and answer, are references that only exist within that block of code which defines the method my_funny_equation.

Now that this method is defined, I can do:

my_funny_equation(10, 12)
230

my_funny_equation( 4, 5)
# result: 30

answer+10
# result: (Ruby will choke here)

Why does the third command choke? Again, answer exists only within the little world defined in the my_funny_equation method, between the def and end lines. It has no value outside of the method definition. This is called function scope, a topic outside of, well, the scope of this simplified tutorial. Read more about scope here.

OK, the above was just introducing you to the concept of a method/function. The kind of methods we’ll be dealing with in our script are called instance methods. These methods belong to something…an actual number, for example. 6 is an instance of a Number. “Six” is an instance of a String

Example:

The number 2.67 is considered by Ruby to be of the class Float…that is, a number with a floating decimal point.

More specifically, 2.67 is an instance of a Float. So is 4.777. And so is 8.999.

What if I wanted to go about rounding a Float number? Well, luckily, Ruby has built in instance methods that do this. The basic structure is the instance, followed by the method’s name…as follows:

instance.method_name

The method for rounding a Float is called “round”. So, to round 2.67, we do:

2.67.round
>>2

This is a little confusing because of the two periods. Just be faithful that the Ruby interpreter knows the difference; it sees the first “dot” as a decimal point defining the number. The second “dot” tells it that we want to access the built-in Float method called round.

One more example, let’s work with arrays.

Let’s say we have:

an_array= [1,2,3,4,5,6]

I want to make an array that consists of the first three elements of *any* array. Luckily, Ruby arrays has a built in method called slice.

an_array.slice(0,3)
>>[1,2,3]

So, slice is the name of an instance method of things that are Arrays. Inside the parentheses are two arguments, the first denotes the element to start out at (in this case, 0, since we want the first element), the second denotes how many elements to include in this sub-array (3).

What was the point of all of this? In our final code, you’ll be seeing calls to methods. Someone already wrote the method that, say, collects all the text of a webpage and stores it into a variable for you. But you need to know the name of that method and how to invoke it.

Writing Your Script

OK, now we get past the fundamentals and into things that will really solve your problems. It wasn’t important to have intimate knowledge of the previous concepts, but just to know they exist.

But how can you, knowing just the basics, do something as complicated as connect to a series of web pages, collect their content, pick the exact points of needed data, and arrange them in a useful structure? Because other programmers have abstracted all these functions in such a way that we could do this series of tasks in just a few lines.

I’m going to write out an extremely-verbose way of performing these tasks to make each step clear…but as you get better, you’ll find ways to minimize your typing.

Here’s the list of steps we’ll be doing, in somewhat plain English:

• 1) Grab the contents of the presidents list
• 2) From that list, grab each president’s name
• 3) Determine if the last name is longer than 6 characters
• 4) If so, fetch the link to the president’s page and open it
• 5) Grab the age from the president’s page
• 6) Add up the data you gathered

Nokogiri

I won’t get into what “gems” are in relation to the Ruby programming language; just think of them as pre-packaged functions and code that you can easily download and re-use for your own scripts.

Complete instructions can be found at the nokogiri homepage. You may run into a lot of errors…my advice is to copy part of that error and Google it with that and “nokogiri,” and hopefully you’ll get an answer.

Hopefully, it’s as simple as going into your command-line console (exit the interpreter if you’re in there) and typing:

>> gem install libxml-ruby
>> gem install nokogiri

What is nokogiri? It’s a library of code that makes it easy to parse a webpage. Remember when you right-clicked on a webpage to view source, and how painful of a task it would be to collect, say, what the text of the third headline is…on 100 different pages? Nokogiri essentially allows you to do this with a couple lines of code. Check out the homepage here.

Step One: Fetch the Contents From the Presidents List

Let’s try Nokogiri out. Open your ruby interpreter and type in the following commands; these first lines invoke the method require, which will give your script access to the required libraries of code, including nokogiri:

require 'rubygems'
require 'nokogiri'
require 'open-uri'

This next line will fetch the contents of Wikipedia’s list of U.S. Presidents

	list_of_presidents = Nokogiri::HTML(open('http://en.wikipedia.org/wiki/List_of_Presidents_of_the_United_States'))

I’m going to quickly deconstruct this line:

• Nokogiri::HTML specifies that we want a method that exists in the Nokogiri library, and more specifically, in its class named HTML.
• open is the name of the method we want. Now you see why we had to specify the above…there are lots of libraries and contexts that have methods named open. We want Nokogiri’s.
• ‘http://en.wikipedia.org/wiki/List_of_Presidents_of_the_United_States’ is the string that holds the address of the page we want. The open method needs this to…well…know what to open.
• list_of_presidents is the variable to which open will spit its contents into.

OK, that one line, maybe the most complicated line we’ve written so far, just did a whole lot for you.

Using a method in the Nokogiri library called open (which takes in a web page address as an argument), it opened a connection with Wikipedia, performed the Internet protocols necessary to exchange information, copied the content of the target page, and wraps it all up in a Nokogiri data structure for later manipulation. We are pointing to this data structure with the variable list_of_presidents

Let’s try to grab the contents of the second h2 tag (i.e. the second, secondary headline)

list_of_presidents.xpath('//h2')[1].content
=>Presidents

Running scripts from Text Editors or the Command Line

Running Ruby commands from the Interactive Ruby prompt is nice and all, for quick feedback. But from here on out, we’ll be writing a full-on script with a few dozen lines of code. So, it’ll be easier if you create a new text file with a file extension of .rb … something like, myfirstscript.rb to put your code in.

You should be using a text-editor for this…something better than Notepad, at least.

For Macs, there’s the free and excellent TextWrangler. If you’re willing to spend some money, TextMate is what I use and it’s worth the $55. A free 30-day trial can be \”downloaded here.

For Windows, the one-click Ruby installer includes the free SciTE4. Also, there’s the free Komodo Edit. For $35, there’s the “Textmate on Windows”, E-TextEditor (free trial here)

Some of these text editors have a shortcut-key that allows you to run the script. For example, SciTE uses F5. Note how the output is conveniently displayed to the side:

Writing a Ruby script in SciTE for Windows

There’s also the old-fashioned command line, from which you ran IRB from. Navigate to the directory that you saved your file in. Then type “ruby whatever_your_file_name_is.rb“:

Running a script from the Windows command line

XPath

XPath is a syntax used to address parts of HTML documents. It allows you, for example, to find all text that’s between headline, italics, paragraph, or whatever tags you want. You could also do something as specific as “Find the third link in every paragraph.”

From Zvon.org, how to select all 'BBB' nodes using XPath

It’s another field of knowledge in which you could spend your life memorizing. For our purposes, you just need to know that it’s a way to pinpoint an element, or a set of elements, in an HTML document.

list_of_presidents.xpath('//h2')[1].content
#result: "Presidents"

Let’s dissect the above nokogiri command. list_of_presidents was a variable holding a Nokogiri data structure…essentially, the entirety of the Wikipedia page in a format that the Nokogiri library can understand.

xpath, then, is an instance method of this data structure, that takes a string as an argument. That string contains XPath syntax.

The string, in the above example, is “//h2″. In XPath syntax (check out W3Schools for a primer), the double-slashes // tells the parser to look anywhere in the document. h2 is the specific tag – a level-2 headline – that we want. And [1] denotes that the result of the xpath method is an array, of which we want the value at the 1st index (technically, the second value of that array…remember that an array’s index starts at the 0th index). And content is an instance method of what was in that 1st index: a nokogiri data structure. content, in this case, pulls what was in those h2 tags: “Presidents“.

The above line could’ve been broken down into:

a = list_of_presidents
a = a.xpath('//h2')
a = a[1]
a = a.content
#result: "Presidents"

That was a very simple XPath query. Another one could be:

list_of_presidents.xpath('//p/a[4]')

This will find the 4th hyperlink in each paragraph. If you try it out, you’ll get an array containing two elements…which makes sense, as there are only two paragraphs on this page (therefore, there can only be two fourth-in-a-paragraph hyperlinks)

Step 2: From a Table of Data, Fetch the President’s Name

At this time, it’s worth looking at how Wikipedia lists its presidents:

Wikipedia's List of Presidents of the United States

This is an HTML table. Each row appears to contain one president (there are sub-rows, which we’ll ignore, corresponding to each term). In the third column (the second column is the actual image file) are two important pieces of data for us: the president’s name and a link to that president’s Wikipedia page.

Remember that we wanted the age of each president. Unfortunately, that’s not listed on this table, so we’ll have to visit each page, where, presumably, an age is listed.

Visit w3Schools for a quick primer on HTML tables. But to be brief: tr designates a row and td designates a column. Let’s put our installation of Firefox’s Firebug to use. Let’s confirm that the info we want – a president’s name – is indeed in the third column.

Right click on the hyperlink of John Adams and select Inspect Element. The Firebug panel should pop-up like so, showing that the third <td> element contains “John Adams”. More specifically, it contains the text “John Adams” in between <a> tags, which we learned marks off a hyperlink. This will be important in the next step…

Using Firebug to find out the element containing "John Adams"

Adapting from our previous line of code using XPath, let’s try this:

those_columns = list_of_presidents.xpath("//tr/td[3]")

That XPath notation will find us every third <td&rt; (column) that is enclosed in a <tr> tag (row). That should spit out a large array of Nokogiri elements (as many as there are presidents).

We want the first of those, which is addressed in the 0th-index of that array…

those_columns[0]
# result is: "George Washington[2][3][4][5]"

So we got a name…but what’s with the bracketed numbers? If you look at the Wikipedia list again, you’ll see that those numbers are links to footnotes. Useful, but not to us. So how to extract just the name? Remember that each president’s name is enclosed in a a (hyperlink) tag. And it’s the first hyperlink. So let’s make our previous XPath a little more complex:

george_washingtons_name = list_of_presidents.xpath("//tr/td[3]/a[1]")[0]
=>"George Washington"

We’re now asking for the 1st (a[1], in XPath notation, is asking for the first a tag) hyperlink, in the third column (td), in each row (tr). The result is the string “George Washington”.

Step 3: Determine if the Last Name Is Longer Than 6 Characters

OK, now we have a name; how do we programmatically determine the length of the last name (remember, our goal is to search all presidents with last names with more than 6 letters)?

The split and length methods of String

First, let’s get the last name. It’s reasonable to assume that the last word in each string (“Bush” in “George W. Bush”) is the last name. Each word is set off by a space. So we are going to use a String instance method called split, which will take a string and divide it into separate pieces, using a character we specify. The result is an Array of strings.

So:

the_last_name = george_washingtons_name.split(' ')[-1]
# Result: "Washington"

1. The above line can be described as thus: Take the string inside the variable george_washingtons_name
2. Split it at every instance of a space
3. Return the last element (the -1 index of an array returns the last element. -2 would return the second-to-last)

The result is: “Washington” from the string “George Washington” is assigned to the variable the_last_name

Now, this is when we finally use the conditional branch statement if

the_last_name.length > 6
# result: true

if the_last_name.length > 6
 puts("Yep, greater than 6")
end
# result: Yep, greater than 6

length is an instance method of Strings. In the first bit of code, we basically asked: is the length of the_last_name greater than 6. The interpreter says, true

In the second bit of code, we defined a branch statement, saying to print “Yep, greater than 6″ if the condition in the if statement (the_last_name.length > 6) was true. It was.

Step 4: If So, Fetch the Link to the President’s Page and Open It

Here’s the code, in verbose form, that we’ve taken to get here…plus a few more lines that flesh out how we want the script to actually execute.

	# open the required libraries
	require 'rubygems'
	require 'nokogiri'
	require 'open-uri'

	# Using nokogiri, fetch Wikipedia's list of presidents page
	list_of_presidents = Nokogiri::HTML(open('http://en.wikipedia.org/wiki/List_of_Presidents_of_the_United_States'))

	# Using another nokogiri method, grab the third column from every row, and from those, grab the first hyperlink (which contains the prez's name)
	an_array_of_links = list_of_presidents.xpath("//tr/td[3]/a[1]")

So we dealt with George Washington’s name…but we want to deal with an array of presidential names. On each element, we want to execute the same operation (see if length of last name is greater than 6 letters, if so, fetch the link).

We’re going to use something called an each loop.

		count = 0

		an_array_of_links.each do |link_to_test|
		# This above statement can be read as: for each element in an_array_of_links, do
		# the following code (until the end line)
		# And as you go through each element, the variable use to reference the element will be named "link_to_test"

		   last_name = link_to_test.content.split(' ')[-1]   #remember that between the <a> tags was the president's name, with the last word being the last  name
			if last_name.length > 6
				the_link_to_the_presidents_page = link_to_test["href"]
				# We'll get to this part in the next section...
			end

		end
		# OK, we're at the end of the each loop. Go back to the top

I’m not going to dissect this. It’s enough to know that each is a method of an Array, and the code inside each do and end is executed for each element of an Array.

OK, using the code above, we are looping through all the presidents’ names and page links. On each name, we’re testing the length of the last name. And if the last name is longer than 6 letters…we’re going to open the link and grab the president’s age.

So:

	if last_name.length > 6
		the_link_to_the_presidents_page = link_to_test["href"] 

		# OK, the value of href is going to be something like "/wiki/George_Washington". That's an address relative to the Wikipedia site
		# so we need to prepend "http://en.wikipedia.org" to have a valid address...

		the_link_to_the_presidents_page = "http://en.wikipedia.org"+the_link_to_the_presidents_page

		# now let's fetch that page

		the_presidents_page = Nokogiri::HTML(open(the_link_to_the_presidents_page))

		# ... OK, now what?

	end

Step 5: Grab the age from the president’s page

All right, so the_presidents_page now holds all the html inside one of the president’s page. We need to scope it out to find the XPath necessary to fetch the age of the president.

Let’s take a look at George Washington’s page. More specifically, look at the sidebar to the right, which contains his vital statistics:

George Washington's Wikipedia Sidebar

As you can see, the age is listed, next to the “Died” line.

Using Firebug to check out the structure tells us that the sidebar is a table, and the death date is in the <td> cell that immediately follows the <th> cell containing the text “Died”.

Firebug Inspection of George Washington Sidebar

OK, were going to have to use XPath to target those specific cells. Let’s test it out on George Washington’s page. I’m just going to provide you the XPath syntax; you’re welcome to read W3School’s tutorial to figure why it works:

	george = Nokogiri::HTML(open('http://en.wikipedia.org/wiki/George_Washington'))
	death_date = george.xpath("//th[contains(text(), 'Died')]/following-sibling::*")[0].content 

	# => "December 14, 1799 (aged 67)Mount Vernon, Virginia,\nUnited States"

(Some references to the syntax above: contains, following-sibling

Well, death_date contains more than we wanted. How do we just get the 67 from the aged 67 part? There’s no html tag that sets 67 off (our job would have been so easy if it had been <age>67</age>).

The last new topic you’ll learn in order to complete the task is regular expressions.

Regular Expressions, aka regexes

Again, like HTML and XPath, regular expressions aren’t “programming”, but it’s a universe of syntax that requires entire books to describe. Put simply, regular expressions allow you to grab strings of text that match a pattern.

From regular-expressions.info, how to match HTML tags

In this case, the pattern I want is: a number, either two-to-three digits long, that is after the word “aged “

I won’t go into the specifics here…I’ve found that you can learn regular expressions with a little reading and trial and error. In this case, the pattern I want, in regex terms, is /aged.+?([0-9]+)/ (note: although the text on the Wikipedia page reads something like “aged 67″, the space in between is a special HTML character, hence, the .+? used to capture it in the reg ex…don’t worry, that last sentence will make perfect sense when you someday understand reg exes.).

In descriptive English, this pattern is going to capture (what’s in the parentheses) any digits from 0-9 that follow the character sequence aged. The forward-slashes denote the beginning and end of the regex.

Again, a regular expression is a syntax, not an actual programming function. So we need to call Ruby’s instance method, match, which executes a text-search based on the syntax of regular expression that you passed into it. Like so:

death_date = george.xpath("//th[contains(text(), 'Died')]/following-sibling::*")[0].content
age_at_death = death_date.match(/aged.+?([0-9])/)[1]

As you can guess, match returns an array of elements. I don’t want to explain the match method in full here, but the 0th element contains the entire match, which would be “aged 67″, and the 1st element returns what was in between the parentheses of my regular expression…the pattern for a multi-digit number, i.e. 67. Again, you just have to learn about reg exes for this to make more sense.

Bookmark regular-expressions.info and save yourself a lot of time in learning about reg exes.

Step 6: Add up the data you gathered

So now we’ve gotten to our goal: retrieving a president’s age from his Wikipedia page. Now we just need to add it all up and take the average.

Here’s the remaining things we have to do, in narrative form:
Before we go into each president’s page, we need a variable to hold the sum of all the ages (total_age). And we’ll need a variable to keep track of how many president’s ages we’ve retrieved (prez_count). However, not every page is going to have an age…since not all former presidents have passed away. So, if the “age” datapoint exists, add it to the total_age variable. And increment prez_count. If not, then do nothing, and go onto the next president until we’ve gone through all the presidents.

Once we’ve finished looping through the pages of presidents, divide total_age by prez_count. And we’re done.

The complete script

The final code is as follows (I’ve added several puts statements to notify you where in the execution the script is…it should take less than 2 minutes):

	require 'rubygems'
	require 'nokogiri'
	require 'open-uri'

	list_of_presidents = Nokogiri::HTML(open('http://en.wikipedia.org/wiki/List_of_Presidents_of_the_United_States'))

	an_array_of_links = list_of_presidents.xpath("//tr/td[3]/a[1]")

	## These two variables will be added to throughout the execution of the script
	## At the end, they'll have the answers

	prez_count = 0
	total_age = 0

	an_array_of_links.each do |link_to_test|  

	   last_name = link_to_test.content.split(' ')[-1]   

		if last_name.length > 6
			the_link_to_the_presidents_page = link_to_test["href"]
			the_link_to_the_presidents_page = "http://en.wikipedia.org" + the_link_to_the_presidents_page
			prez_page = Nokogiri::HTML(open(the_link_to_the_presidents_page))

			puts "Entering the page: #{the_link_to_the_presidents_page}"

			death_date = prez_page.xpath("//th[contains(text(), 'Died')]/following-sibling::*")

      if death_date && death_date[0]
        # Doing something like `if some_variable_name` is basically asking, "Does some_variable_name have any value?".
        # It will return false if some_variable_name has been set to false or if it had never been set to anything at all, not even 0 or an empty string (both of which would return true)
        # The double ampersand && functions as an "AND", requiring that two conditional tests be true before entering the if-statement's true branch

			  age_at_death = death_date[0].content.match(/aged.+?([0-9]+)/)[1]
  	  		if age_at_death
  	  		  # we only get here if there was a "Died" table cell AND a text pattern similar to: "aged XX"
  	  		  puts "Age of #{link_to_test.content} is: #{age_at_death}"
  	  			total_age += age_at_death[1].to_i  # technically, age_at_death[0] is a String. to_i will make it a Number so we can safely add it to total_age
  	  			prez_count += 1
  	  		end #end of the if age_at_death
  	  end # end of the if death_date...
	  else
	    # we reach this branch of code if last_name was shorter than 6. Let's print a debug message to notify us:
	    puts "#{last_name} is not longer than 6 letters"
		end #end of the if last_name.length > 6

	end # OK, we're at the end of the each loop. Go back to the top

	# if we got here, we're out of the loop, and total_age and prez_count have the right values. So:
	the_final_value = total_age/prez_count.to_f  # to_f converts an integer to a decimal number, so we'll get partial years for the average
	puts "#{prez_count} presidents were counted, their age totaling: #{total_age}."
	puts "The average of their ages is #{the_final_value}"

As of Feb. 2010, running that script produces this output:

Entering the page: http://en.wikipedia.org/wiki/George_Washington
Age of George Washington is: 67
Adams is not longer than 6 letters
Entering the page: http://en.wikipedia.org/wiki/Thomas_Jefferson
Age of Thomas Jefferson is: 83
Entering the page: http://en.wikipedia.org/wiki/James_Madison
Age of James Madison is: 85
Monroe is not longer than 6 letters
Adams is not longer than 6 letters
Entering the page: http://en.wikipedia.org/wiki/Andrew_Jackson
Age of Andrew Jackson is: 78
Buren is not longer than 6 letters
Entering the page: http://en.wikipedia.org/wiki/William_Henry_Harrison
Age of William Henry Harrison is: 68
Tyler is not longer than 6 letters
Polk is not longer than 6 letters
Taylor is not longer than 6 letters
Entering the page: http://en.wikipedia.org/wiki/Millard_Fillmore
Age of Millard Fillmore is: 74
Pierce is not longer than 6 letters
Entering the page: http://en.wikipedia.org/wiki/James_Buchanan
Age of James Buchanan is: 77
Entering the page: http://en.wikipedia.org/wiki/Abraham_Lincoln
Age of Abraham Lincoln is: 56
Entering the page: http://en.wikipedia.org/wiki/Andrew_Johnson
Age of Andrew Johnson is: 66
Grant is not longer than 6 letters
Hayes is not longer than 6 letters
Entering the page: http://en.wikipedia.org/wiki/James_A._Garfield
Age of James A. Garfield is: 49
Arthur is not longer than 6 letters
Entering the page: http://en.wikipedia.org/wiki/Grover_Cleveland
Age of Grover Cleveland is: 71
Entering the page: http://en.wikipedia.org/wiki/Benjamin_Harrison
Age of Benjamin Harrison is: 67
Entering the page: http://en.wikipedia.org/wiki/Grover_Cleveland
Age of Grover Cleveland is: 71
Entering the page: http://en.wikipedia.org/wiki/William_McKinley
Age of William McKinley is: 58
Entering the page: http://en.wikipedia.org/wiki/Theodore_Roosevelt
Age of Theodore Roosevelt is: 60
Taft is not longer than 6 letters
Wilson is not longer than 6 letters
Entering the page: http://en.wikipedia.org/wiki/Warren_G._Harding
Age of Warren G. Harding is: 57
Entering the page: http://en.wikipedia.org/wiki/Calvin_Coolidge
Age of Calvin Coolidge is: 60
Hoover is not longer than 6 letters
Entering the page: http://en.wikipedia.org/wiki/Franklin_D._Roosevelt
Age of Franklin D. Roosevelt is: 63
Truman is not longer than 6 letters
Entering the page: http://en.wikipedia.org/wiki/Dwight_D._Eisenhower
Age of Dwight D. Eisenhower is: 78
Entering the page: http://en.wikipedia.org/wiki/John_F._Kennedy
Age of John F. Kennedy is: 46
Entering the page: http://en.wikipedia.org/wiki/Lyndon_B._Johnson
Age of Lyndon B. Johnson is: 64
Nixon is not longer than 6 letters
Ford is not longer than 6 letters
Carter is not longer than 6 letters
Reagan is not longer than 6 letters
Bush is not longer than 6 letters
Entering the page: http://en.wikipedia.org/wiki/Bill_Clinton
Bush is not longer than 6 letters
Obama is not longer than 6 letters
21 presidents were counted, their age totaling: 1398.
The average of their ages is 66.5714285714286

The End?

Well, congratulations…you accomplished a trivial task, but you learned a set of methods that you can apply to much more important goals. If you’re a complete newbie to programming, hopefully this tutorial has given you a glimpse of what’s involved. And how, once you firm up your programming fundamentals, you can get real work done.

But I need to stress that this tutorial simplified things as much as possible…at the cost of best-practices programming. I chose Wikipedia as a target because it’s a reasonably well-structured, high-traffic site that has an ethos of making volumes of information available for the public good.

The script that we just wrote is a naive, little child, that gets what it wants as fast as it wants. In the real world, many sites that you attempt to scrape will not be so forgiving. Some sites will block you, or fail to connect, if you try to read a hundred pages at once. Some sites will have horrific HTML that will require much more complicated XPath and regular expression syntax. Sometimes, your internet connection might drop. All of this will cause the above script to halt to a ugly and premature death. Or even worse: collect bad data that you won’t know was erroneous.

All of these problems are solvable, but like any task, it takes experience that comes from trying and failing. Hopefully, this tutorial at least shows you how easy it is to try.

Other resources:

• Tutorials on HTML, CSS, XPath and a bunch of other useful topics, from W3Schools
• About.com’s guide to Ruby
• Installing Ruby
• The Little Book of Ruby – A free e-book from SaphireSteel Software
• Ruby Standard Library Documentation
• Nokogiri Tutorial
• Video tutorial of Nokogiri Screen Scraping from Railscasts
• Ruby Examples from Wikipedia
• XPath tutorial from Zvon
• Regular-Expressions.info, pretty much the best regular expression resource online
• A printable cheat-sheet for regular expressions