Updated the code and results to my guide on how to scraper Pfizer’s list of payments to doctors. It now contains a more normalized file that has a line for every doctor and payment. The aggregate totals changed marginally.
AskReddit is just a great place to kill time and learn a lot about life, from people for whom anonymous handles means more about being honest and less about trolling. This thread asking what was the nicest thing you’ve ever done without telling anyone is a tear jerker. Whether or not these were real secrets, it’s interesting/amazing what people can keep to themselves.
Some of my favorites:
mightyelf 212 points
I used to pick up pretty much any hitch hiker I saw. I would even pick up people at the bus stop. It just seemed to me that since I had a car and some free time that it was no big deal to help someone out. I never asked for money or anything like that. It was nice to see the people smile. One day I pickup up a guy about my age( I was 19 at the time) and he was very thankful that I I did. He was telling me about all the stuff he had to do that day and he knew that he would only get 2 or 3 things done because the buses. I had to be at work soon so I asked him if he knew how to drive standard. He looked at me funny and said yes. I pulled into my work and told him to be back here in 12 hours. I gave him $20 for gas and said bye. The guys at work asked me who he was but I didn’t even remember his name from the short time we rode together. They urged me to call the cops but I told them the kid would be back. After my shift ended my car was there. The kid had the biggest smile on his face and as I drove him home he told me how he was able to get everything done and that I had saved him atleast a full day of running around. I saw him from time to time after that. It was nice to hear that because I helped him out he would go out of his way to help others out.
MonkeySpanker 53 points 4 months ago[-]
I won a lawsuit and got about 25k in the bank. I dropped about $8k on my debt, $2k on toys for me (laptop), and the rest went in to paying off debt for friends and family. $8k went to my sister’s credit cards, $2k went to dental work and other kinds of stuff needed for her. $3-5k on a friend who was about to lose his house.
Besides, I’m horrible with my money… I probably would have wasted it on stupid shit anyways.
Line6 27 points 4 months ago* [-]
One day my best friend’s wife told me that she is planning to leave her husband. She wanted me to know in advance so that I can be there for him. Now my friend is one of the decent person I’ve ever known in my life. He loved her like crazy and would have done almost anything. Her wife’s problem was that she was not feeling like a princess. I don’t judge her for that but the guy was working hard to provide his family with some financial stability.
I told her then and there, if she is planning to leave him because she doesn’t love him anymore that can easily be fixed. Just let him know somehow and he’ll make you fall in love with him again but if you don’t want to love him anymore just drop the bomb and leave. She didn’t say anything and left. Next morning I got a text from her that said “Thank You I know now”. They are still married.
euryalus0 31 points 4 months ago[-]
When it looked like he didn’t have much time left, I gave my best friend who had AIDS all my money so that he could take a trip across the country to SF and be with his boyfriend. Only about $1500, but we were both pretty broke at the time. The good news is that after about 10 pretty rough years full of very close calls he is now in pretty good health thanks to the latest meds. (The bad news is we had a pretty major break-up of our friendship about 8 months ago and he hasn’t spoken to me since.) Don’t regret it at all however.
JustATroll 190 points 4 months ago[-]
One of my best friends lost $800 in overdraft fees and didn’t get paid for 2 weeks. They called me crying, I told them to call the bank and see if they can sort it out. Meanwhile, I went to the bank and deposited the amount he said he was missing. To this day he still thinks the bank reversed the overdraft fees. This was like a year ago, and (at least from what he tells me) he’s never overdrafted since.
kr 202 points 4 months ago[-]
I pickup tissues and trash that coworkers (and other folks in the building) throw on the floor in the men’s room before the caretaker goes in to clean.
She caught me once.
atinasutherland 789 points 4 months ago[-]
When I got my settlement check for getting my finger cut off at work I kept $2000 dollars and put the rest in the bank. That night after dinner and drinks I was coming home and saw a homeless man (25-30) that I’ve seen several times before, posted up against a wall near the intersection shivering in the cold. Since there were 3 hotels at that intersection, I stopped, rented a room for a week on my debit card then took the key out of the envelope, replaced it with $1700 and walked over to the gentleman to hand him the key and cash.
No bullshitting, I saw him a month or so later working at a gas station, clean, shaved and nice hair. Im not sure if he recognized me but I’m glad because I recognized him and he appeared to be happy and doing well which said enough. I haven’t seen him in several years but I like to think he’s back on his feet, maybe a family, a house, whatever really but just doing well.
johnbc5 985 points 4 months ago[-]
When I lived in the city an older lady about 90 got her apt robbed in my building. They went in a stole all her cash and took some valuables that she had. She did not have a bank account so the thieves took about 30K the ladies life savings. She was afraid of being evicted for the apt because she wouldn’t have the rent money and did not want to end up in a state run nursing home. I called the landlord and paid her rent in full for the rest if the year, five months worth and told the landlord not to tell her it was me. I also had groceries delivered to her once a week for the next two months until she had some money saved from her social security checks. I never told anyone what I had done for her and I don’t think she even knew my name because the apt building had about 50 apartments in it. The landlord was I only one who knew and he wanted to tell her what I was doing but I told him that I would deny it. I did not want her to feel indebted to me. She posted a letter in the lobby of the building to thank who ever had helped her. I took the letter down and kept it. The landlord still writes to me every few months to tell me how she is doing. She is still living in the apt seven years later. I never told any one.
I’d like to think I’ve done equally selfless things and that I don’t remember them off the top of my head because I don’t have anyone else who knows to refresh my memory. One time I drove a drugged-out homeless man who was making a scene at Starbucks to the nearest Methadone clinic, but that’s mostly because I didn’t have anything to do that morning.
The baby products label, Pampers, put out a free app called “Hello Baby” which lets you input your baby’s info, get a week-by-weeky depiction of what it looks like in the momma, and then probably reminds you on the baby’s birthdate to buy Pampers.
It’s great, actually, to see the real-life size of a four-week old embryo. But bringing the iPad to the bar and showing everyone the real-size of a 20+ weeks-old fetus, that might prevent a lot of unwanted (and wanted) pregnancies from even starting.
DailyFinance’s Jeff Bercovici makes a compelling case that Gizmodo opened itself to criminal and civil penalties by paying someone $5,000 for the “found” iPhone prototype. He argues that California law compelled the finder, and Gizmodo, to make good-faith efforts to find the owner. Gizmodo’s efforts to return the device (before taking it apart and making millions of page hits from it) were at best, lazy and uninterested, and worst, nominal for the sake of covering-their-asses in a legal suit, Bercovici writes:
At heart is the question of whether the person who found the phone made “reasonable and just efforts to find the owner and to restore the property to him,” as required by the California penal code. In its account of what happened, Gizmodo says the finder “asked around” the bar where he found it. And after realizing it was an Apple prototype, he called several numbers at the company.
What he never did, however, was notify anyone who worked at the bar, according to its owner, Volcker Staudt. That would have been the simplest way to get the phone back to the Apple employee who lost it, who “called constantly trying to retrieve it” in the days afterward, recalls Volcker. “The guy was pretty hectic about it.”
Nor did the finder report it to the Redwood City Police Department, says Sgt. Dan Mulholland. To be fair, no one from Apple told the police the phone was lost, either. I contacted a company spokeswoman to ask why not but never heard back.
And make no mistake: In this case, it was up to Gawker to establish that the seller legally possessed the property. Paul J. Wallin, a founding partner at the California law firm Wallin & Klaritch, offers an analogy. “If you purchase a Rolex watch at a swap meet for $200, a reasonable person would be put on notice that it might be stolen goods,” he says. The buyer would thus be required to take extra measures to determine that it wasn’t.
When I asked Denton what steps his company took to ensure that the seller had, in fact, made a good-faith effort to return the phone to Apple before shopping it around, he redirected the question. “We weren’t convinced the phone was even a genuine prototype until the weekend [ie. after Gizmodo bought and dismantled it],” he said. “And we didn’t discover the name of the Apple engineer who lost it until Monday. We called him and — after Apple officials got back to us — we returned the device to them.”
After further reflection, Bercovici is even more committed in his stance:
I understand the moral calculus they used. We all feel intuitively that picking up something that someone else left behind is not as bad as seizing it by force, stealth or deception. But in the eyes of the law, it’s still stealing. And buying stolen goods is a crime. In those rare cases where a journalist commits a crime and receives the benefit of prosecutorial discretion, it’s usually because he can demonstrate there was a compelling public interest at stake. There is no such interest here. The only parties who benefited from Gizmodo’s story are Gawker Media and Apple’s competitors.
It’s hard for me to pick a horse in this race. I’m a frequent reader of Gawker, though Gizmodo turned me off for awhile with their remotely-tampering-with-CES-displays stunt, and I most definitely read through their iPhone dissection (I also thought Giz had the best iPad app coverage). I think Bercovici is right, but if Giz is the purveyor of stolen info, then I definitely didn’t take the moral high ground by avoiding it.
On the other hand, I hope that if this disclosure can be shown not to have hurt Apple’s bottom line, I hope they ease up on their infamous, and now-tiringly-overdone commitment to secrecy. Not for the sake of its info-hungry fans, but for the workers employed by police-state-like distributors.
Pulitzer-Prize winning cartoonist Mark Fiore, who won this year’s Pulitzer for his editorial cartoons, says he tried to make an iPhone app but it was just too politically hot for Apple’s guidelines. From Nieman Labs (h/t Poynter):
In December, Apple rejected his iPhone app, NewsToons, because, as Apple put it, his satire “ridicules public figures,” a violation of the iPhone Developer Program License Agreement, which bars any apps whose content in “Apple’s reasonable judgement may be found objectionable, for example, materials that may be considered obscene, pornographic, or defamatory.”
Apple attached screenshots of the offending material, including an image depicting the White House gate crashers interrupting an Obama speech. Two other grabs include images referencing torture, Balloon Boy, and various political issues.
It’s been a huge last few days for ProPublica. My colleagues Jesse Eisinger and Jake Bernstein unveiled the result of 7+ months of reporting, a much anticipated collaboration with “This American Life” on how the hedge fund Magnetar Capital helped prolong the housing bubble by betting against risky investments that it advocated for. Also, our story on private jet owners hiding in public airspace, uncovered by Michael Grabell (after our lawyers’ successful litigation), was one of our most viewed, thanks to it getting top play by USA Today and Yahoo.
Those both alone would’ve made it one of ProPublica’s most prominent weeks, but then Sheri Fink won the Pulitzer for Investigative Reporting for her massive investigation, published in the NYT magazine, on how a hospital’s doctors, post-Katrina, reportedly put patients to death under the guise of mercy and grace under chaos. Sheri’s win is extremely gratifying, because her subject had a lot of things going against it: Katrina was a four-year-old painful, chaotic memory that most Americans wanted to forget. And for N.O. residents, it seemed that the overwhelming sentiment was for the doctors and other authorities who did what they could. Anna Pou, the doctor at the center of Sheri’s story, had been exonerated (and the prosecutor who went after her was removed). And after Sheri’s story, no new charges have been made against her.
The story itself is a long-read. In addition to the factors above going against it, it also doesn’t deliver an immediate payoff for the ADD-afflicted reader. It’s only until the end that you can appreciate the light that Sheri shed on a universally important, yet opaque topic: who deserves life in a time of crisis? I think Sheri’s story, and subsequent follow-ups related to swine flu preparations, raised the alarm that not even our medical professionals are on the same page, and moved the ball in such a way that her findings would shock even the most cynical skeptics of the medical profession.
Also, congrats to my colleagues Charles Ornstein and Tracy Weber for being finalists in the Public Service category for their exposure of California’s broken nursing board. For them to even be considered for that prize, considering they won it recently before in the same area (lax oversight of medical care) is a testament to how thorough their work was again, and how much impact their stories had (Gov. Schwarzenegger immediately sacked or forced out a majority of the board afterwards).
I think our office felt confident our work was as good as any Pulitzer contender and it wouldn’t be a shock to win, even though we would be the first online-only organization (and possibly the youngest, at two years old) to do it. The drama was less about whether if we would win but which one of our reporters would win. For example, T. Christian Miller and his work on defense contractors was, in my mind, as deserving as any. Like Sheri, he shed light, in an exhaustive, dogged fashion, on a subject that most people would rather not care about: the treatment of civilians who are injured in warzones while working as contractors. With the bad rep of Blackwater, it’s proof of T’s herculean reporting and writing efforts that he got lawmakers to make some real moves into an easily overlooked (for political reasons) but essential area of our national security (in terms of prizes though, T already brought home the Selden Ring).
And of course, all those stories above would’ve had a harder hill to climb without the collaboration of all our great editors and research staff. And in my own department, Krista Kjellman and Jeff Larson put in just as much dedication and deliberation to further illuminate the stories in their online presentation (and in the process, often provided research and work important to the stories themselves).
Congrats to the other Pulitzer winners. I haven’t had time to look through all their work. I did put WaPo’s Gene Weingarten’s winning feature on the hellish punishment of parents who left their children to die in overheated cars on my iPad’s Instapaper. I got about a fourth-way through before I had to put it away so I wouldn’t be crying in the subway car.
Update, January 2012: Everything…yes, everything, is superseded by my free online book, The Bastards Book of Ruby, which is a much more complete walkthrough of basic programming principles with far more practical and up-to-date examples and projects than what you’ll find here.
I’m only keeping this old walkthrough up as a historical reference. I’m sure the code is so ugly that I’m not going to even try re-reading it.
So check it out: The Bastards Book of Ruby
Update, Dec. 30, 2010: I published a series of data collection and cleaning guides for ProPublica, to describe what I did for our Dollars for Docs project. There is a guide for Pfizer which supersedes the one I originally posted here.
So a little while ago, I set out to write some tutorials that would guide the non-coding-but-computer-savvy journalist through enough programming fundamentals so that he/she could write a web scraper to collect data from public websites. A “little while” turned out to be more than a month-and-a-half. I actually wrote most of it in a week and then forgot about. The timeliness of the fourth lesson, which shows how to help Pfizer in its mission to more transparent, compelled me to just publish them in incomplete form. There’s probably inconsistencies in the writing and some of the code examples, but the final code sections at the end of each tutorial do seem to execute as expected.
As the tutorials are aimed at people who aren’t experienced programming, the code is pretty verbose, pedantic, and in some cases, a little inefficient. It was my attempt to think how to make the code most readable, and I’m very welcome to editing changes.
DISCLAIMER: The code, data files, and results are meant for reference and example only. You use it at your own risk.
Tutorial 1: Go from knowing nothing to scraping Web pages. In an hour. Hopefully – A massive, sprawling tutorial that attempts to take you from learning what HTML is, to the definition of an “if
Tutorial 2: Scraping a County Jail Website to Find Out Who’s in Jail – This uses all the concepts from the first tutorial and applies them to something that a cops reporter might actually want to try out.
Tutorial 3: Who’s Been in Jail Before: Cross-checking the jail logs with the court system with Ruby’s Mechanize – This lesson introduces you to another Ruby library that allows you to automate the filling-out of forms so that you can access online databases, in this case, California criminal case histories to see if current inmates are repeat-alleged-offenders.
Tutorial 4: Improving Pfizer’s Dollars-to-Doctors Pay List – Last week, Pfizer released a list of nearly 5,000 doctors and medical institutions that it made $35 million in consulting and expense payments. Fun. Unfortunately, the list, as it initially existed online, is just about useless to anyone wanting to examine trends. This tutorial provides a script to make the list more interesting to journalists.
Update (12/30): So about an eon later, I’ve updated this by writing a guide for ProPublica. Heed that one. This one will remain in its obsolete state.
Update (4/28): Replaced the code and result files. Still haven’t written out a thorough explainer of what’s going on here.
Update (4/19): After revisiting this script, I see that it fails to capture some of the payments to doctors associated with entities. I’m going to rework this script and post and update soon.
So the world’s largest drug maker, Pfizer, decided to tell everyone which doctors they’ve been giving money to to speak and consult on its behalf in the latter half of 2009. These doctors are the same ones who, from time to time, recommend the use of Pfizer products.
Pfizer, the world’s largest drug maker, said Wednesday that it paid about $20 million to 4,500 doctors and other medical professionals for consulting and speaking on its behalf in the last six months of 2009, its first public accounting of payments to the people who decide which drugs to recommend. Pfizer also paid $15.3 million to 250 academic medical centers and other research groups for clinical trials in the same period.
A spokeswoman for Pfizer, Kristen E. Neese, said most of the disclosures were required by an integrity agreement that the company signed in August to settle a federal investigation into the illegal promotion of drugs for off-label uses.
So, not an entirely altruistic release of information. But it’s out there nonetheless. You can view their list here. Jump to my results here
Not bad at first glance. However, on further examination, it’s clear that the list is nearly useless unless you intend to click through all 480 pages manually, or, if you have a doctor in mind and you only care about that one doctor’s relationship. As a journalist, you probably have other questions. Such as:
- Which doctor received the most?
- What was the largest kind of expenditure?
- Were there any unusually large single-item payments?
None of these questions are answerable unless you have the list in a spreadsheet. As I mentioned in earlier lessons…there are cases when the information is freely available, but the provider hasn’t made it easy to analyze. Technically, they are fulfilling their requirement to be “transparent.”
I’ll give them the benefit of the doubt that they truly want this list to be as accessible and visible as possible…I tried emailing them to ask for the list as a single spreadsheet, but the email function was broken. So, let’s just write some code to save them some work and to get our answers a little quicker.
Going to Court
In the last lesson, we learned how to write a script that would record who was in jail at a given hour. This could yield some interesting stories for a crime reporter, including spates of arrests for notable crimes and inmates who are held with $1,000,000 bail for relatively minor crimes. However, an even more interesting angle would be to check the inmates’ prior records, to get a glimpse of the recidivism rate, for example.
Sacramento Superior Court allows users to search by not just names, but by the unique ID number given to inmates by Sacramento-area jurisdictions. This makes it pretty easy to link current inmates to court records.
However, the techniques we used in past lessons to automate the data collection won’t work here. As you can see in the above picture, you have to fill out a form. That’s not something any of the code we’ve written previously will do. Luckily, that’s where Ruby’s mechanize comes in.