On Thursday evening I went to an event at Twitter hosted by their Women in Engineering group featuring a series of lightning talks on the challenges of designing technologies that scale. As ever, I’ve put together a Storify of the event here.  Tom Woolway, one of Twitter’s engineering managers, also collated a tweet collection summarising the event.

In terms of technical topics, it was really exciting to get a peek under the hood at how Twitter approaches some of the biggest engineering challenges faced by any company.   The topics included scalability of TweetDeck, Twitter’s streaming API, scaling challenges in Manhattan (Twitter’s “eventually consistent” database), and the life of a tweet across the platform.  I’m hoping that the speakers will share their slides and I will update this post with a link if they do become available.

One of my key takeaways from the evening was that unlike the smaller tools I deal with on a daily basis, Twitter can’t afford to ignore edge cases or unexpected stresses on the system: with the kinds of volumes they’re dealing with, at any given time somewhere in the world an anomaly or an edge case WILL be happening and they need to know how or to very rapidly develop solutions for how to deal with that.

When asked what unanticipated challenges they experienced while scaling Twitter to deal with the massive amounts of interactions it currently has, Sharon Ly answered as follows:

There was a second component to the evening which may have surprised people considering that this was an evening of technical talks. During the Q&A panel, an attendee asked what approaches Twitter is taking to combat online harassment and abuse. Several people’s follow-up questions were devoted to these issues also, mainly because the audience appeared dissatisfied with the imprecise responses that were given to the initial question.

While I thought that the responses given to the question of how Twitter is handling online harassment were hesitant and addressed intentions more than strategies and solutions, I didn’t take this on its own as a sign that Twitter never thinks about these things: the people on the panel were there to answer questions about servers and redundancy measures and hashing tables. Whether or not that should be the case, the representatives on this panel were concerned with making sure the massive amounts of content that Twitter deals with gets delivered, not what the nature of that content is. I don’t think that means Twitter isn’t thinking about these questions.  There may be someone at Twitter who could provide really useful information or embark on a fruitful dialogue about what strategies Twitter is adopting in relation to online security, harassment, and safety. It was very unfortunate that such a person wasn’t in the room, though.

I found it notable that this is the first technical talk I’ve been to in London where this topic was even raised, let alone anyone attempting (however unsatisfactorily) to address it. I’m only one person so I’m hardly a representative sample, but I attend lots and lots and LOTS of technical talks and networking events around London, enough to feel like I now have a sense of the distinct cultures of each company that sponsors them. The only other time I recall hearing anyone speak about online harassment was when it was the specific subject of a lightning talk which was being hosted by Girl Geek Dinners a couple of years ago, and that was from the perspective of how people deal with online harassment as a cultural phenomenon, not what measures (technical or otherwise) companies should be putting in place to address it. Data privacy and cultural diversity, among company employees as well as the audience a product attracts, are both issues I’ve heard raised time and time again, but online harassment is not something in my experience that gets asked about nor voluntarily addressed–though I perceive the latter topic as very relevant and related to the first two. I think it’s no great exaggeration to assume that as more women and minorities are working in technical roles, there will consequently be an increase in the number of techies who have experienced online harassment firsthand. I’d like to think that they would be able to add to the discussion on this and to encourage the companies they work for to become advocates against harassment and abuse. Consequently I hope to see the issue addressed with more openness and a more firm focus on strategies in the future.

I think it says something particular–and positive–about Twitter’s culture that people in this forum felt comfortable to ask that question and to keep raising it when they felt the answers weren’t up to scratch.  Others noticed this also:

On the not-so-good side, Twitter’s summaries of the event in the form of tweet collections and blog posts about the event have completely glossed over that aspect of the evening’s discussion.

Like many others, I would really love to see a series of talks on this, particularly on the kinds of technical solutions that get proposed to address these issues. The time is clearly ripe for a brave company to step up to the plate and be frank about this.