Monthly Archives: January 2013

The Bastards Book of Regular Expressions

Well, I’m not quite done with my promised revision of the Bastards Book of Ruby. Or of Photography…but I’ve decided, oh what the hell, I should write something about regular expressions.

Actually, there is some method to this madness. As part of the process of updating the Ruby book, I realized I needed to spin off some of the larger, non-Ruby related topics. So, at some point, there will be mini-books about HTML and SQL. Regular expressions, as I keep telling people who want to deal with data, are incredibly important, even if you think you never want to learn programming. Hopefully this mini-book will make a strong case for learning regexes.

The second motive is I’ve been looking for a html/text-to-pdf workflow. So this is my experiment with Leanpub, which promises to turn a set of Markdown files into PDF/mobi/etc, while handling the selling process. I don’t expect to sell any copies of the BBoRegexes, but I hope to get a lot of insight about the mechanics behind Leanpub and if it presents a viable way for me to publish my other projects.

Check out the Leanpub homepage for my tentatively tiled book, The Bastards Book of Regular Expressions. Or, you could just read the mega-chapter on regexes in my Ruby book.

Edward Tufte’s defense of Aaron Swartz and the “marvelously different”

Edward Tufte speaks at Aaron Swartz's memorial on Jan. 19, 2013, in Cooper Union, NYC.

Edward Tufte speaks at Aaron Swartz’s memorial on Jan. 19, 2013, in Cooper Union, NYC.

There were many powerful speakers during yesterday’s memorial for Aaron Swartz in Cooper Union. Among them was data-visualization pioneer Edward Tufte, whom I hadn’t known had ever collaborated with Aaron. But given their similar orbits in democracy and data, it is not at all surprising that they were friends.

Tufte’s message included a surprising anecdote about his own hacking career. You can watch the video (parts 1, 2)

I’ve transcribed Tufte’s words here for posterity. It’s an interesting piece of hacker history and a touching defense of Aaron Swartz and others who are “marvelously and vigorously different.”

Note: A commenter has pointed me to a full version of Tufte’s talk. I’ll update with his full comments when I have time.

Parts of Democracy Now’s video feed appears to have cut out, as part of Tufte’s opening words are missing in the available feed, but here he is talking about how he and Aaron came to know each other:

[Aaron] tells the story of how he had a choice between taking a final exam or coming to hear me talk. As it turned out, he did both. We then would meet over the years for a long talk every now and then and my responsibility was to provide him with a reading list, a reading list for life

And then about two years ago, Quinn [Norton] and Aaron came to Connecticut, and he told me about the 4 and a half million downloads of scholarly articles. And my first question was:

“Why isn’t MIT celebrating this?”

The archival video feed cut off here and I can’t find the missing excerpt on Democracy Now’s website. From what I remember from being at the memorial, in this part, Tufte related how he told Aaron, half-seriously, that Aaron’s main fault was downloading millions of articles when only a few thousand were worth downloading.

Tufte then said that Aaron asked him if he knew Bill Bowen, the former president of Princeton University and the founder of JSTOR, the digital library that Aaron was accused of illegally accessing.

The second part of the video continues here:

…[Bowen] then became president of the Mellon Foundation and he had retired from the Mellon foundation. But he was asked by he foundation to handle the problem of JSTOR and Aaron.

So I wrote Bill Bowen an email about it. And I said first that Aaron is a treasure. And then I told a personal story about how I had done some illegal hacking as a student and had been caught at it and what happened.

In 1962, my housemate and I invented the first blue box. That’s a device that allows for free, undetectable, unbillable long-distance telephone calls.

And we got this up. And played around with it and at the end of our research came when we completed was what we thought was the longest long distance phone call ever made, which was from Palo Alto to New York time of day, via Hawaii.

Well, during our experimentation, AT&T, on the second day it turned out, had tapped our phone. But it wasn’t until about six months later when I got a call from a gentleman, A.J. Dodge, a senior security person at AT&T. And I said, “I know what you’re calling about”.

And so we met. And he said what we’re doing is a crime…But I knew it wasn’t serious because he actually cared about the kind of engineering stuff and complained that the tone signals we were generating were not up to standard. Because they recorded them and played them back into the network to see what numbers we were trying to reach and they couldn’t break through some of the noise of our signal.

He asked why we went off the air after about three months…And I said, well, we regarded it as an engineering problem and we made the longest long-distance telephone call…and that was it.

And so the the deal was, as I explained to my email to Bill Bowen, was that we wouldn’t try to sell this…we wouldn’t do any more of it, and that we would turn our equipment over to AT&T. And so they got a complete vacuum oscillator kit for making long distance phone calls.

But I was grateful for A.J. Dodge and, I must say, even AT&T, that they decided not to wreck my life.

And so I told Bill Bowen that he had a great opportunity here to not wreck somebody’s life. And of course he thankfully did the right thing.

Aaron’s unique quality was that he was marvelously and vigorously different.

There is a scarcity of that.

Perhaps we can be all a little more different too.

(Thanks to commenter Daniel for pointing out this Youtube full version)

This was the first that I had ever heard of Tufte being associated with phreaking. It’s not that he wouldn’t have the technical chops…it’s just that I couldn’t find any reference to it in the exhaustive articles that have been written about him. Eric Hellman, who also attended the memorial, said he chatted with Tufte afterwards who told him he had never talked of the incident in public.

After listening to Tufte talk, I Googled for any reference of him and Swartz. Here’s Swartz, “a loyal fan,” asking Tufte about the font he uses. And here’s Swartz describing a Tufte lecture.

Watch the rest of the recorded memorial at Democracy Now’s website. The message from Taren Stinebrickner-Kauffman was especially poignant and inspiring.

Watch live streaming video from democracynow at

The MIT Response to Aaron Swartz’s death

The past two days have been emotionally draining for me. It shows how much Aaron Swartz accomplished in his young life that even though I never met him in person, I was as crushed by the news of his suicide as if a close friend had died.

I had always thought that if anyone could fight to change a system for good, Aaron was one of the best equipped for our generation. His death sucked a lot of hope out, frankly. But the letter today from MIT president Rafael Reif restored a little of it again. MIT had taken a beating from Aaron’s family and never would I have expected an institution to be so contrite and so concerned about a non-student accused of committing a campus crime. The incident took place before Reif’s term but he shows no desire to wash his hands of it, appointing a strong defender of Internet rights to conduct a “thorough analysis” into how MIT contributed to this tragedy:

To the members of the MIT community:

Yesterday we received the shocking and terrible news that on Friday in New York, Aaron Swartz, a gifted young man well known and admired by many in the MIT community, took his own life. With this tragedy, his family and his friends suffered an inexpressible loss, and we offer our most profound condolences. Even for those of us who did not know Aaron, the trail of his brief life shines with his brilliant creativity and idealism.

Although Aaron had no formal affiliation with MIT, I am writing to you now because he was beloved by many members of our community and because MIT played a role in the legal struggles that began for him in 2011.

I want to express very clearly that I and all of us at MIT are extremely saddened by the death of this promising young man who touched the lives of so many. It pains me to think that MIT played any role in a series of events that have ended in tragedy.

I will not attempt to summarize here the complex events of the past two years. Now is a time for everyone involved to reflect on their actions, and that includes all of us at MIT. I have asked Professor Hal Abelson to lead a thorough analysis of MIT’s involvement from the time that we first perceived unusual activity on our network in fall 2010 up to the present. I have asked that this analysis describe the options MIT had and the decisions MIT made, in order to understand and to learn from the actions MIT took. I will share the report with the MIT community when I receive it.

I hope we will all reach out to those members of our community we know who may have been affected by Aaron’s death. As always, MIT Medical is available to provide expert counseling, but there is no substitute for personal understanding and support.

With sorrow and deep sympathy,

L. Rafael Reif

Why did infinite scroll fail at Etsy?

Descending down a Manhattan Subway

Descending down a Manhattan Subway

My point is not that infinite scroll is stupid. It may be great on your website. But we should have done a better job of understanding the people using our website” – Dan McKinley, Principal Engineer at Etsy

A few weeks ago, Etsy engineer Dan McKinley gave a talk on “Design for Continuous Experimentation.” It’s an interesting, humorous presentation on large scale A/B testing and I fully recommend you check it out (slides and video here).

The moral was that A/B testing – much like the code it tests – must be done in a modularized fashion. The “fail” case he gave was when Etsy spent months developing and testing infinite scroll to their search listings, only to find that it had a negative impact on engagement.

McKinley said Etsy’s main lesson from this was that their A/B testing strategy was too monolithic:

  1. Spend a ton of time building the infinite scroll feature and release it.
  2. Verify that people love it, then find a Brooklyn warehouse to throw a celebration party.

“Seeing more items faster is presumed to be a better experience”, McKinley said. But the A/B tests showed various negative effects of the feature, including fewer clicks on the results and fewer items “favorited” from the infinite results page. And curiously, while users didn’t buy fewer items overall, “they just stopped using search to find these items.”

So basically infinite scroll failed in every major way. But not only was Etsy’s team wrong in assuming that users would benefit from infinite sroll, McKinley said, they were wrong in automatically accepting the two underlying assumptions behind infinite scroll:

  1. Users want more results per page.
  2. Users want faster results.

(Assumption #2 was tested by artificially slowing down search for a cohort of users. These users did not necessarily react positively to slower results. But their engagement level was not statistically significantly lessened.)

The point of McKinley’s talk was that instead of having the goal of “test infinite scroll,” Etsy realized it needed to test each assumption separately, and this going forward is their game plan (the success case McKinley gives is the revamp of Etsy’s search box).

As I said, a great talk worth checking out. However, McKinley didn’t have an answer for this:

Even if user engagement isn’t positively influenced by more results per page or by faster results, why does the combination of both have a decidedly negative impact?

The decision to try infinite scroll was partially influenced by Google apparent success with “instant search results,” according to McKinley. And it’s a feature that is prominent among popular Tumblr themes, Pinterest, and of course, at Facebook and Twitter, so presumably their A/B testing has yielded good results.

But why not at Etsy? Or at Amazon, which sticks with 16 results per search page? Users are notoriously fickle about interface changes. But if the search algorithm still brings up the best results at the top, then a user who has only 16 options before clicking through has no better advantage than the user who has 16 plus an “infinite” number of lesser results, if the latter doesn’t have to do any additional work to get them.

McKinley said he didn’t know why infinite scroll didn’t succeed for Etsy. There wasn’t, as far as they could tell, a technical fault (i.e. infinite scroll breaking in a specific browser). It was just a bad thing and the reason why wasn’t clear.

Now the actual merits of infinite scroll itself is still a controversial feature – even if there aren’t technical issues, which there almost always are. But in Etsy’s use case, it seems that at worst there should have been no effect, not negative effect. The most jarring problem of infinite scroll is that there’s no footer, which presumably the average Etsy user doesn’t need when making viewing/purchasing decisions.

Maybe it’s not that users consciously dislike infinite scroll. But in practice, maybe they lose a sense of orientation? Some users may have the habit of going further and further into search results – “playing the field” so to speak – before they realize that pages 1 and 2 are the best options they have. With pagination, it’s fairly easy to get back to those pages.

But if these users don’t have the habit of bookmarking/favoriting/writing-down-the-names of items as they scroll through, perhaps the deluge of infinite scroll makes it more likely for users to forget where they once were and what once caught their eye? Or maybe it’s a lack of developing the habits needed to actually act: when users are forced to deal with the loading time of each page, they learn to do their favoriting/click-throughs/impulse-purchasing on those pages before moving on? These users, when allowed to scroll for more items than they were used to, might simply lose the habit of click-to-favorite/browse/buy – but never develop the habit of scrolling back up. Or they’re just overwhelmed from the information overload and don’t feel like taking action any longer.

Infinite scroll may be pleasant for browsing, but does it lead to inaction? It’s an interface issue that is likely less a technical question than a psychological. I wonder what other online retail interfaces have found?

Here’s a short Etsy forum discussion where users wonder where the infinite-scroll went. McKinley’s talk and slides are here.

On Hacker News, user oconnore makes a great observation here, in that least some users, when returning to the search page after backing up from a product page, would find that they had lost their place in the infinite search stream (probably the biggest problem with infinite scroll implementations). I didn’t mention this in the original version of this post, but McKinley opened his talk with an anecdote about another poor assumption: power users (who worked at Etsy and made the suggestion) often opened up search results in new windows because they wanted to do side-by-side comparisons. But when Etsy made that the default behavior, testing found that most users did not appreciate it.

So putting 2 and 2 together: Perhaps many Etsy users have the habit of clicking-through a search result and then backing up to the search page. When the infinite-scroll didn’t properly mark their place, they *really* got lost and the search experience would obviously not be a good one. Seems as good as explanation as any, and so maybe it really is a technical issue after all.

Another update: HN user gfodor, who worked on the project, said the back-button problem may have applied to some IE users. But as McKinley says in his talk, the negative search effect occurred across all browsers, including ones that did back-up correctly. McKinley does go into good detail about how Etsy determines control group composition (sellers vs. non-sellers, for example, are a much different user type), but as this blog post is near-infinitely long I suggest again you check out his excellent talk on his blog.

R.I.P. Sgt. Tyler Ziegel

Left: Sgt. Tyler Ziegel with his wife, Renee, in 2006. Photo by Nina Berman

Left: Sgt. Tyler Ziegel with his wife, Renee, in 2006. Photo by Nina Berman

A couple years ago, I randomly wrote a post because I was so affected by a photo exhibit at the Whitney Museum. The photos documented the life of Marine Tyler Ziegel, who suffered horrific burns during a suicide bomb attack in Iraq; the photographer, Nina Berman, was awarded the World Press Photo’s portraiture award in 2006 and Ziegel became one of the iconic images of the hell of war.

After writing that post, I might not have thought much more about Ziegel, who by all accounts lived a quiet life. But through some strange fluke with Google, that post ended up being one of the first search results for Ziegel’s name. The steady stream of visitors to it was a constant reminder of his sacrifice. He inspired such admiration and compassion that that random post is by far the most viewed page on this blog.

Sgt. Ziegel died last week on Dec. 26, 2012, after falling on ice. He was 30.

Berman’s image is unforgettable. But Ziegel’s story after the blast is also compelling. People Magazine did a profile of Ziegel shortly after his wedding in 2006:

For Ty and Renee the two years since have been a wrenching test of love and character. In a culture obsessed with physical perfection, Ty is now vulnerable to stares and whispers. Not naturally introspective and blunt to the point of gruffness, he says he doesn’t bother with what-ifs. Rather, he relies on a store of dark humor. “I’m thinking of writing a book, You Know You’ve Been Blown Up If…,” he says, in homage to redneck comic Jeff Foxworthy. “Like, ‘You know you’ve been blown up if a year later you bleed in the shower.'” One night, out to dinner with a Marine pal, Ty had a little fun with a man who was smoking. “My friend was like, ‘Hey buddy, do you mind? Do you see what happened to this smoker?'” Ty recalls. “The guy put his cigarette out and walked away.”

There’s also this 2007 interview with Berman in Salon (before Ziegel and his wife divorced):

Yes. I asked Ty, what do little kids say? Do little kids get scared? In my book, I’d photographed a really severely burned soldier. And when I was with him I’d see kids shy away and he would smile at them.

Ty would just laugh — he’s got a great sense of humor. Kids would say, “What happened to your ears?” and he’d say, “The bad guys took ‘em.” They’d say, “What happened to your nose?” and he’d say, “The bad guys took it.” I guess he tried to make some little game out of it to deal with it.

Rest in peace, Tyler. See the rest of Berman’s photos of Tyler’s life here.

Tyler Ziegel with his then fiancee, Renee.