Comments on: Coding for Journalists 104: Pfizer’s Doctor Payments; Making a Better List https://danwin.com/2010/04/pfizer-web-scraping-for-journalists-part-4-pfizers-doctor-payments/ Words, photos, and code by Dan Nguyen. The 'g' is mostly silent. Sun, 07 Dec 2025 04:13:29 +0000 hourly 1 https://wordpress.org/?v=4.2.39 By: Code, Don’t Tell: Programming as an Essential Journalism Skill | Dan Nguyen pronounced fast is danwin https://danwin.com/2010/04/pfizer-web-scraping-for-journalists-part-4-pfizers-doctor-payments/#comment-2045 Wed, 22 Feb 2012 19:58:23 +0000 https://danwin.com/?p=643#comment-2045 […] At that point, I knew virtually nothing about the issue. But what I did see was that Pfizer’s site seemed unnecessarily cumbersome. Though the disclosures were mandated, Pfizer’s site made it difficult to do simple analyses such as finding the professionals who had received substantial amounts or even the sum of the database’s payments. So I wrote a scraper and published the code and data for others to use. […]

]]> By: dataist blog: An inspiring case for journalists learning to code | Dan Nguyen pronounced fast is danwin https://danwin.com/2010/04/pfizer-web-scraping-for-journalists-part-4-pfizers-doctor-payments/#comment-928 Wed, 16 Feb 2011 13:02:01 +0000 https://danwin.com/?p=643#comment-928 […] Dollars for Docs project originated in part from this Pfizer-scraping lesson I added on to my programming tutorial: I needed a timely example of public data that wasn’t […]

]]> By: amul https://danwin.com/2010/04/pfizer-web-scraping-for-journalists-part-4-pfizers-doctor-payments/#comment-848 Thu, 09 Dec 2010 20:14:33 +0000 https://danwin.com/?p=643#comment-848 really really awesome that you did this.. I think if there were more GUI oriented tools built into the browser itself via plugins and panels this practice of large data-set anaylsis could really catch on.. especially with these cool examples.

On a side note these ceo’s siting on multiple boards is fascinating..

]]> By: Dan Nguyen’s coding for journalists 101 https://danwin.com/2010/04/pfizer-web-scraping-for-journalists-part-4-pfizers-doctor-payments/#comment-846 Tue, 07 Dec 2010 20:01:55 +0000 https://danwin.com/?p=643#comment-846 […] Click here for this page’s table of contents. Or jump to the the theory lesson. Or to the programming exercise. Or, if you already know what a function and variable is, and have Ruby installed, go straight to two of my walkthroughs of building a real-world journalistic-minded web scraper: Scraping a jail site, and scrapingPfizer’s doctor payment list. […]

]]> By: Joseph https://danwin.com/2010/04/pfizer-web-scraping-for-journalists-part-4-pfizers-doctor-payments/#comment-389 Wed, 12 May 2010 17:10:03 +0000 https://danwin.com/?p=643#comment-389 Dan

Where is ‘process_row’? I can’t see it defined anywhere? Thanks

Joe

]]> By: Pfizer Data Redux | Danwin: Dan Nguyen, in short https://danwin.com/2010/04/pfizer-web-scraping-for-journalists-part-4-pfizers-doctor-payments/#comment-379 Wed, 28 Apr 2010 14:22:43 +0000 https://danwin.com/?p=643#comment-379 […] the code and results to my guide on how to scraper Pfizer’s list of payments to doctors. It now contains a more normalized file that has a line for every doctor and payment. The aggregate […]

]]> By: Pharma Conduct Guy https://danwin.com/2010/04/pfizer-web-scraping-for-journalists-part-4-pfizers-doctor-payments/#comment-339 Tue, 06 Apr 2010 15:20:42 +0000 https://danwin.com/?p=643#comment-339 Hi Dan,

What an excellent write-id. I was pondering the idea of whether or not my readers would be interested in tutorial for how I aggregated the data so quickly. Thus, by posting such an excellent overview of web-scraping, you have saved me a tremendous amount of time. I’m a PHP/MySQL user myself, but anyone with a modicum of programming skill should be able to follow your example. Posting the raw data, which I had hoped to do later this week (I’m traveling at the moment), was a nice thing to do. I didn’t want to post the full set until I’d had a chance to perform some serious QC on the data. Looking over your numbers, we seem to have found most of the same things, as well as some of the same problems.

For example, regarding Duke, I think I understand where the discrepancy lies. I alluded to this briefly in a previous post (see Who were the top 5 recipients of money from Pfizer during the period 2009 Q3-Q4?), but I have a more complete post in the works. Thanks again for writing such an excellent tutorial, for making the data available, and for posting on my blog. In one of my next posts, I’ll be sure to spotlight this post.

-Eric

]]> By: Credigy Receivables and Steve Stewart | SAP Business One – How To – A/R Down Payments https://danwin.com/2010/04/pfizer-web-scraping-for-journalists-part-4-pfizers-doctor-payments/#comment-338 Tue, 06 Apr 2010 15:13:48 +0000 https://danwin.com/?p=643#comment-338 […] Coding for Journalists 104: Pfizer's Doctor Payments; Making a … […]

]]> By: Coding for Journalists 101 : A four-part series | Danwin: Dan Nguyen, in short https://danwin.com/2010/04/pfizer-web-scraping-for-journalists-part-4-pfizers-doctor-payments/#comment-337 Tue, 06 Apr 2010 14:32:16 +0000 https://danwin.com/?p=643#comment-337 […] Danwin: Dan Nguyen, in short The 'g' is mostly silent Skip to content About Dan NguyenThe Great Snowball Fight in Times SquareDepGal, a Pretty, Free, and Fast Flash Photo Gallery « Coding for Journalists 104: Pfizer’s Doctor Payments; Making a Better List […]

]]>