Monthly Archives: June 2012

The Bastards Book of Photography

Instead of fixing up the yet-to-be-revised Bastards Book of Ruby, I decided to do one on Photography, not least because I have plenty of photos to use as fodder.

It was a nice change of pace; I realized as soon as I released the Ruby book that it needed to be radically reorganized and tightened. So I went into the Photography book with a goal of keeping things short (including making the font-size a bit too large).

More importantly, I got more acquainted with the awesome Octopress platform…which will most definitely replace the moronic rake script I use to compile the Ruby book. I can’t say enough how essential Octopress was to making it possible for the Photography book to be put together in two weeks.

So check it out: a beginner’s guide to photography

The Entrepreneurial Drive in Mexico’s Drug Trade; “Cocaine, Inc.”

Patrick Radden Keefe, writing for the NYT Magazine, takes a different tack than I’ve usually seen in stories covering Mexico’s Drug War: examining the business practices and supply-chain logistics of the illegal drug trade rather than its violent consequences.

If the subject didn’t have the tragic context of 50,000 deaths in six years, it’d be a guide for entrepreneurs on how nimble and simple can outrun top-heavy opponents.

From the article, “Cocaine, Incorporated“:

Moving cocaine is a capital-intensive business, but the cartel subsidizes these investments with a ready source of easy income: marijuana. Cannabis is often described as the “cash crop” of Mexican cartels because it grows abundantly in the Sierras and requires no processing. But it’s bulkier than cocaine, and smellier, which makes it difficult to conceal. So marijuana tends to cross the border far from official ports of entry. The cartel makes sandbag bridges to ford the Colorado River and sends buggies loaded with weed bouncing over the Imperial Sand Dunes into California.

Michael Braun, the former chief of operations for the D.E.A., told me a story about the construction of a high-tech fence along a stretch of border in Arizona. “They erect this fence,” he said, “only to go out there a few days later and discover that these guys have a catapult, and they’re flinging hundred-pound bales of marijuana over to the other side.” He paused and looked at me for a second. “A catapult,” he repeated. “We’ve got the best fence money can buy, and they counter us with a 2,500-year-old technology.”

Read the rest of the longform piece here.

Humble Bundle payments: Just because you use open software doesn’t mean you’re cheap

Humble Bundle V payment breakdown stats

Humble Bundle V payment breakdown stats

Just purchased the Humble Bundle V indie game pack, which is notable for being one of their best deals yet, with such star games as Psychonauts (Double Fine) and Sword and Sworcery (Superbrothers) among the collection.

Even better is that all the games are cross-platform, which is great news for Linux and Mac users including myself. I think this has been the case(?) for past bundles, but this is the first one I’ve actually bought.

At the bottom of the downloads page is a chart showing the revenue breakdowns by platform. Bundle V has surpassed $3.2M in payments. As expected, Windows users account for more purchases than Linux and Mac users combined. But check out the average payment by platform; I’ve added the percentage over/under the average for reference’s sake:

Average (+/- average)
Overall $7.87 0.0%
Windows $7.27 -7.6%
Mac $9.46 20.2%
Linux $12.24 55.5%

Looks like the Linux users are using the money saved from free software for a good cause…and the Mac users, I guess we just like to spend extra 😉

[insert pithy statement about how a spirit of generosity and sharing begets more generosity that I’m too busy playing these games to write]

* Edit: As an astute HN user points out, the Linux average is likely skewed by Notch, who apparently chipped in $9999.99 and presumably did so from a Linux system. His outlier effect is further exaggerated given that the Linux userbase is relatively small.

What the Windows userbase has going for it is that it presumably has more game developers, who you think would be more sympathetic to Humble Bundle’s pro-developer ideals than the average gamer. However, it’s unlikely they make up enough of the overall Windows userbase to have a large effect on the average.

(as for the Mac users, they probably really are more affluent or are more used to paying a premium for most software – see the anchor effect.)

This is yet another reminder why you should use the median if your other option is a simple average.

The greatest sneaker commercial, according to the greatest This American Life episode

“He would dribble from morning till night, watching his reflection in a basement window. Trying to repeat exactly the same moves again and again. His neighbors, who were often kept awake by his dribbling, thought he was out of his mind.”
– Joel Lovell, This American Life, describing Luis Da Silva.

Everytime I think I’ve heard the greatest This American Life episode yet, I hear another one.

The new favorite, “Crispy with the Rock”, is about the “world’s greatest sneaker commercial.”…or maybe “the greatest thing that has ever been on television. It kicks the moon landing’s ass,” according to reporter Joel Lovell.

Lovell is referring to the famous Nike “freestyle” commercial campaign. The TAL segment originally aired back in 2001. The commercials were in heavy rotation during the NBA season. But this was before YouTube, so Lovell was forced to become an obsessive watcher of the NBA:

I found myself spending a lot of hours watching games I didn’t really care about. Games involving the Indiana Pacers, for instance. Just so I could see the commercial.

Here’s one of the three spots in “Freestyle”, featuring NBA and WNBA pros with streetballers.

Lovell focuses on Luis Da Silva, one of the streetballers featured in the ensemble commercial. Luis grew up in Elizabeth, N. J., in an area dangerous enough that his mother wouldn’t let him go to the park. So, Luis dribbled in his own backyard, “a concrete slab about the size of a Twister mat”

He would dribble from morning till night, watching his reflection in a basement window. Trying to repeat exactly the same moves again and again. His neighbors, who were often kept awake by his dribbling, thought he was out of his mind.

Luis: “When my father called me in, he’s like, son, it’s time to come in now. I’m like, what time is it? He’s like, it’s 2:30. I was like, wow. Time just came by. Time just flew.

“And I would wake up the next day during sometime like 9, 10 o’clock and get like four or five hours of sleep. Back up. But I wanted it so bad. I wanted it so bad.”

“I was just in my backyard. That’s how much I loved it. I mean, nothing else mattered. Nothing else mattered…”

For all of his practice, Luis didn’t even make it as a starter on his high school team. Then his friend told him of a Nike commercial filming the next day in Manhattan.

There are three versions of the freestyle ad. Two of them are ensembles. The third features just Luis:

The rest of the episode is a must-listen, which includes an explanation of “crispy with the rock” and an encounter at the Taipei airport.

The story ends with Lovell recounting his attempt at going 1-on-1 with Luis:

This wave of relief and giddiness comes over me. There’s something reassuring in the idea that someone, through sheer determination and will, can become so impossibly blow-your-mind good. And there’s something so comforting about being in the presence of such goodness.

I expected to Google “Luis Da Silva” and not find anything, as his commercial aired back in 2001 and seemed to be something that should have consumed the entirety of his 15 minutes of fame (and more). As it turned out, he’s doing well. Well enough, at least, to have his own Wikipedia entry, and even a movie career.

Check out the episode here, which I found through TAL’s excellent iOS app. Completely worth a few bucks. I’m sometimes skeptical about verbal storytelling’s ability to capture the visual, but TAL makes be a believer every time.

A defense of web-scraping as a vital tool for journalists

Another day, another overlong justification of programming. On the NICAR mailing list, someone asked how people are coping with the shutdown of Needlebase, a web-scraping tool maintained by Google. This led to the inevitable debate of whether a out-of-luck reporter should just learn enough code to web-scrape herself or to continue the search for more scraping, push-button tools.

Ed. B, who’s no technical slouch himself, wrote the list saying that scraping is “rarely necessary.”

The underlying tools in ScraperWiki, at least the Python and Ruby ones, are about as easy as you can get. You can build something on ScraperWiki in Ruby, Python or PHP, then run it on your own machine. Recursive “wget” followed by local parsing is another option, at least for some kinds of authentication.

Personally I’m not a huge fan of scraping for a number of reasons:

1. It’s time-consuming and error-prone, neither of which is a characteristic compatible with journalism. If you want to be fast and correct, scraping is a bad way to start. 😉

2. It’s very often a violation of someone’s terms of service.

3. It’s easy to collect mountains of irrelevant bits when the real story can be uncovered much more effectively by human-to-human interactions.

I think scraping is *rarely* necessary. Sure, not every site provides an API or downloadable spreadsheets, but we are nowhere near as effective at using the sites *with* APIs and spreadsheets as we could be.

My problem with this argument is that it makes scraping seem like a straighforward exercise of data-collection. This is not at all the case. The most obvious rebuttal is technical; there are a near infinite combination of design patterns and configurations – the vast majority of them resulting in slop when it comes to government sites – that no commercial program or third-party script can anticipate the variations. Learning to code gives you the power to adapt efficiently, rather than trying to wrangle someone else’s program into something barely useful for the situation.

But my main concern about being code-oblivious is that you cannot know, as Don Rumsfeld said, what you don’t know. This is why I compare scraping to conducting interviews without speaking the same language. It’s quite possible, with an interpreter, to get the single piece of data that you need:

  • Reporter: “Did you see the man who killed the child?”
  • Interpreter: “¿Has visto al hombre que mató al niño”
  • Subject: “Si”
  • Interpreter: “Yes”

But if an interview is meant to explore – that is, you don’t really know what the subject might tell you and where he will lead you – then you are at a huge disadvantage if you use an interpreter, at least in my brief experience with non-English speakers. You not only miss whatever nuance (or, in some cases, actual meaning) that the translator misses, you miss the ability to tell at which point in a long statement that the subject’s eyes looked down, as if trying to hide something. You lose the ability for give-and-take, because each of you has to wait for the interpreter to finish a translation (and this repetition, of course, effectively halves the length of time that you have for an interview).

Learning a new language is a real investment but it’s hard to imagine being effective without it in a foreign country. And to belabor the metaphor, we’re all entering a new country with the advent of our digitized society. Here, too, it is worth being acquainted with the new language.

My response to Ed on the NICAR list:

I feel these complaints about scraping either miss the point of scraping or make unreasonable expectations. Scraping (whether from web, pdf, or other non-normalized formats) is just information gathering and has all the same limitations as all other forms of reporting and the same benefits from being skilled at it.

All of your complaints could be made about the method of interviewing, something which is not inherently required to write a worthwhile journalistic story: it’s *extremely* time-consuming and error-prone. It violates people’s sense of comfort and privacy and – at times – official disclosure policies. And it frequently nets you mountains of cumbersome handwritten notes that you’ll never use.

The underlying theme in most data stories, in fact, is that information coming from officials’ interviews are mostly bullshit, which is why we turn to data to find the fuller picture. And single-source data is frequently bullshit, which is why we include interviewing, observation, and other data sources for the fuller picture.

There’s rarely an API to access data on a website in the same way that human interview subjects rarely issue (useful) pre-written questions for you to ask. Information found on the web is as diverse in formats and content as it is elsewhere and no prebuilt software solution will be able to deal with all of it at an acceptable standard.

Whether it’s worth it to learn to code custom scrapers* is definitely still a debate. It certainly is more justifiable than it was even 5 years ago, given the much-lower barrier to entry and the much higher amount of digital information. I agree with Aron Pilhofer that a day of Excel training for every journalist would bring massive benefits to the industry as a whole. But that’s because journalism has been in overdue need of spreadsheet-skills, not because spreadsheets are in themselves, alone, useful sources of information.

(* of course, learning enough programming to make custom scrapers will as a bonus let you do every other powerful data-related task made possible through the mechanism and abstract principles of code)

I think it’s ironic that programming advocates are sometimes suspected of putting too much faith into scraping/the Internet/computers writ large. It’s usually the people with minimal insight about scraping who are dangerously naïve about the flaws inherent to information gathered online.

At some point, it’s worth evaluating whether the time and energy (and money) you spend in clumsily navigating a website and figuring out how to wrangle pre-built solutions could be better spent becoming more technical. Just like you don’t get a practical appreciation of “if your mother says she loves you, check it out” until you’ve actually done some reporting.

…just to be clear, I’m not saying that it’s either learn-to-code or stay-off-the-computer. The solutions so far mentioned in the thread are good and in some cases, may be all is needed. But you could say the same for a Mexico bureau chief who hires a full-time translator instead of trying to learn conversational Spanish. There’s plenty of yet unrealized benefits in moving past the third-party solutions.

Feynman’s Clock

Richard and Arlene Feynman

Richard and Arlene Feynman

You, dead, are so much better than anyone else alive.
Richard Feynman, to his wife, Arlene

Richard Feynman is famous for his ceaseless questioning of life and the universe, to a degree unexpected even for being a Nobel Prize quantum physicist. His dogged investigation of the Challenger explosion is the most generally well-known demonstration of his mindset.

And then there was the time – while working on the atomic bomb at Los Alamos – when he got on all fours and put his nose on the carpet because he wouldn’t just accept at face value that bloodhounds had better senses of smell than do humans:

From his memoir, Surely You’re Joking, Mr. Feynman!:

(I’ve noticed that my dog can correctly tell which way I’ve gone in the house, especially if I’m barefoot, by smelling my footprints. So I tried to do that: I crawled around the rug on my hands and knees, sniffing, to see if I could tell the difference between where I walked and where I didn’t, and I found it impossible. So the dog is much better than I am.)

– Feynman, Richard P.; Ralph Leighton (2010-06-28). “Surely You’re Joking, Mr. Feynman!”: Adventures of a Curious Character (p. 105). Norton. Kindle Edition.

In his biography of Feynman, James Gleick writes that Feynman devoted his logical mind to plenty of non-quantum-physics hobbies, including teaching himself “how to play drums, give massages, to tell stories, to pick up women in bars…[and] he had no difficulty learning to make an impromptu xylophone by filling water glasses, nor had he any shyness about playing them, all evening, at a dinner party for an astonished Niels Bohr.”

However, Gleick writes, Feynman “despised philosophy as soft and unverifiable.” Philosophers, Feynman said, “are always on the outside making stupid remarks.”

But there are no atheists in foxholes, right?

We can’t be sure of what Feynman thought at his own deathbed. But we do know, through his biographers, a little of how his mind worked at the deathbed of Arlene, his first wife.

The two met as teenagers and had fallen so in love that Feynman promised to marry her after graduating from college. But Arlene fell ill and was eventually diagnosed with terminal tuberculosis and had just a few years to live. Feynman’s family tried to convince him that he should break his promise. They even brought the family doctor over to warn him of the dangers of contracting tuberculosis. Feynman responded, “Just tell me how it’s transmitted, and we’ll figure it out.”

Richard and Arlene (note: her name is also spelled “Arline” in the various biographies) decided that she should stay at hospital near Princeton and the couple got married on the drive there.

By June of 1945, Feynman was at Los Alamos working on the atomic bomb. His group was focused on measuring the energy output of the first (and only) trial of the bomb (July 16, 1945) before Hiroshima.

He was at work in the computing room when the call came from Albuquerque that Arline was dying. He had arranged to borrow Klaus Fuchs’s car. When he reached her room she was still. Her eyes barely followed him as he moved. He sat with her for hours, aware of the minutes passing on her clock, aware of something momentous that he could not quite feel. He heard her breaths stop and start, heard her efforts to swallow, and tried to think about the science of it, the individual cells starved of air, the heart unable to pump.

Finally he heard a last small breath, and a nurse came and said that Arline was dead. He leaned over to kiss her and made a mental note of the surprising scent of her hair, surprising because it was the same as always.

The nurse recorded the time of death, 9:21 P.M. He discovered, oddly, that the clock had halted at that moment—just the sort of mystical phenomenon that appealed to unscientific people.

Then an explanation occurred to him. He knew the clock was fragile, because he had repaired it several times, and he decided that the nurse must have stopped it by picking it up to check the time in the dim light.

Gleick, James (2011-02-22). Genius: The Life and Science of Richard Feynman (Kindle Locations 3604-3612).

At a moment when just about anybody would shove aside rational thought – even at least as a helpless reaction to mindnumbing grief – and favor that the universe’s chaos stopped just then to offer a sign of divine significance, Feynman instead sticks to cold hard logic, even if it means sterilizing the moment of his wife’s death. Some might find this application of scientific method off-putting. But maybe it was just an unavoidable feature of one of our most interesting and brilliant modern minds.

In his memoir, What Do You Care What Other People Think? (the title comes from Arlene badgering him over what Christmas cards to send), Feynman says upon kissing Arlene one last time, he was “surprised to discover that her hair smelled exactly the same” and, after thinking about it, thought “there was no reason why hair should smell different in such a short time.”

When asked the next day at the mortuary whether he would like to see Arlene one last time before she’s cremated, Feynman responds: “What kind of a – no, I don’t want to see her, no! I just saw her!” He instead packs Arlene’s things and heads back to Los Alamos:

I didn’t know how I was going to face all my friends at Los Alamos. I didn’t want people with long faces talking to me about the death of Arlene.

Somebody asked me what happened.

“She’s dead. And how’s the program going?” I said.

However, don’t assume that Feynman fit the maligned stereotype of the asocial, cold-hearted scientist. Read, well, any of his memoirs. Or read at least this letter he addressed to Arline, the year after her passing, unsent and undiscovered until his death in 1988 (Source Perfectly Reasonable Deviations from the Beaten Track: The Letters of Richard P. Feynman, h/t Letters of Note):

October 17, 1946


I adore you, sweetheart.

I know how much you like to hear that — but I don’t only write it because you like it — I write it because it makes me warm all over inside to write it to you.

It is such a terribly long time since I last wrote to you — almost two years but I know you’ll excuse me because you understand how I am, stubborn and realistic; and I thought there was no sense to writing.

But now I know my darling wife that it is right to do what I have delayed in doing, and that I have done so much in the past. I want to tell you I love you. I want to love you. I always will love you.

I find it hard to understand in my mind what it means to love you after you are dead — but I still want to comfort and take care of you — and I want you to love me and care for me. I want to have problems to discuss with you — I want to do little projects with you. I never thought until just now that we can do that. What should we do. We started to learn to make clothes together — or learn Chinese — or getting a movie projector. Can’t I do something now? No. I am alone without you and you were the “idea-woman” and general instigator of all our wild adventures.

When you were sick you worried because you could not give me something that you wanted to and thought I needed. You needn’t have worried. Just as I told you then there was no real need because I loved you in so many ways so much. And now it is clearly even more true — you can give me nothing now yet I love you so that you stand in my way of loving anyone else — but I want you to stand there. You, dead, are so much better than anyone else alive.

I know you will assure me that I am foolish and that you want me to have full happiness and don’t want to be in my way. I’ll bet you are surprised that I don’t even have a girlfriend (except you, sweetheart) after two years. But you can’t help it, darling, nor can I — I don’t understand it, for I have met many girls and very nice ones and I don’t want to remain alone — but in two or three meetings they all seem ashes. You only are left to me. You are real.

My darling wife, I do adore you.

I love my wife. My wife is dead.


PS Please excuse my not mailing this — but I don’t know your new address.

Note: A response on Hacker News prompted me to find out the circumstances of this letter’s publication. It is part of a collection released by his daughter, Michelle Feynman, and published in a ~500-page volume, Perfectly Reasonable Deviations from the Beaten Track: The Letters of Richard P. Feynman. I can’t recommend it enough, especially for the < $10 price tag.

As mentioned above, Richard’s letter to Arlene was unsent and undiscovered until his death. But it was not unread. In the volume, Michelle writes this preface to it:

This letter is well worn – much more so than others – and it appears as though he reread it often.