All posts by user

The nature of “things”: Mukund Narasimhan’s talk at Facebook

Technology

Mukund Narasimhan, an alumnus of Microsoft and Amazon who is currently a software engineer with Facebook in Seattle, gave a very interesting presentation at Facebook’s Seattle offices this week. It was a right-sized crowd, in a right-sized office: just enough people, and the right sort of people to afford interesting conversations, and a very generous serving of snacks and beverages from Facebook in an open-plan office that had the look-and-feel of a well-funded startup.

Mukund isn’t posting his slides, and we didn’t take notes, so this isn’t an exhaustive report of the evening, but rather an overview of some aspects of the discussion.

Mukund’s talk was principally about data mining and the ways that Facebook is able to collate and curate vast amounts of data to create metadata about people, places, events and organizations. Facebook uses a variety of signals, the most important of which is user input, to deduce the “nature of things”: i.e. is this thing (entity) a place? Is this place a restaurant? Is this restaurant a vegetarian restaurant?

Some very powerful data mining techniques have been developed already, and this was illustrated most compelling by a slide showing a satellite image of a stadium: quite close to the stadium, on adjacent blocks, were two place markers. These marked the “official location” of the stadium, as told to Facebook by two separate data providers. The stadium itself was pin-pricked with a large number of dots, each marking a spot where a Facebook user had “checked in” to Facebook’s Places.

The visual contrast was dramatic: the official data providers had each estimated the position of the stadium to within a hundred yards, but the checkins of the Facebook users had perfectly marked the actual contours of the stadium.

Mukund’s talk was repeatedly interrupted by questions from the audience, since each slide offered a gateway to an entire topic, and eventually he ran out of time and we missed some of the material.

During the Q&A, Nikhil George from Mobisante asked a question that seemed highly relevant: was Facebook trying to create a “semantic Web“? Mukund sidestepped the question adroitly by pointing out there was no established definition of the term semantic Web, and that is certainly true – the Wikipedia article linked above is tagged as having “multiple issues”– but while Facebook may not be using the specifc protocols and data formats that some might argue are indispensible to semantic Webs, one could certainly make the case that deducing the nature of things, particularly the nature of things that exist on the Internet, is the main point of creating a semantic Web.

While much of the Q&A was about technical matters, a more fundamental question occupied our own minds: at the outset, Mukund asserted that the more people interact with (and, in particular, contribute to) Facebook, the better the Facebook experience is for them and all of their friends.

This, clearly, is the underpinning of Facebook’s business model: people must continue to believe, on a rational basis, that contributing data to Facebook – so that Facebook can continue to deduce the nature of things and offer these things back to their users to consume – offers some direct, reasonably tangible rewards for them and their friends.

Presumably, then, Facebook must be taking very concrete measures to measure this sense of reward that their users experience; we didn’t hear much about that last night since that wasn’t Mukund’s area of focus, but it must surely be well understood within Facebook that the promise of continued reward for continued user interaction – which is essentially their brand promise – must be kept at all times. Has a lot of research been done in this area? (There is, of course, the outstanding research done by dana boyd on social networks in general.)

At a more practical level, a question that bedevils us is how we can improve the signal:noise ration in our Facebook Wall! In our physical worlds, we all have some friends that are exceptionally chatty: some of them are also very witty, which makes their chatter enjoyable, but some are just chatty in a very mundane way. In our Facebook worlds, it is very easy for the chatty people (are they also exceptionally idle or under-employed?) to dominate the conversation.

In a physical world, if we find ourselves cornered by a boor at a party we would quickly, determinedly sidle away and find someone more interesting to talk to, but how does one do that in Facebook? One option, offered by Mukund, would be to turn off their posts, which seems rather like “unfriending” them altogether. But we don’t want to unfriend these people altogether, we just don’t want to hear every detail of every day.

Mukund suggested that by selecting hiding individual posts, as well as “liking” others more aggressively, we could send clearer indications of preferences to Facebook that would help the system improve the signal:noise ratio, and that’s what we have been trying over the past few days.

It is an intriguing topic to consider, and undoubtedly a difficult problem to solve, because you need to weed out individual messages rather than block entire users. For example, one Facebook friend used an intermission of the movie “Thor” to report that he was enjoying the movie. It’s great that he is enjoying the movie, but this low-value update spawned a low-value thread all of its own. We don’t want this person blocked altogether; we need some way of telling Facebook that updates sent during movie intermssions are not very valuable. If Facebook misinterprets our signals and relegates him to the dustbin, we might miss a more useful notification in the future, such as a major life event or career move.

The problem seems analogous to developing a version of Google’s PageRank, but at the message level. In the example above, if a post sent during a movie intermission is of low value, it would affect the ranking of everyone who piled on to create its low-value thread.

People like us who are more technical would probably prefer a more direct way of manipulating (i.e. correcting) the signal:noise ratio. Presumably someone at Facebook is working, even as we write this blog, on some visual tools that provide sliders or other ways for people to rank their friends and notifications. One idea that comes to mind might be a sort of interactive tag cloud that shows you who posts the most, but which also lets you promote or demote individual friends.

Some email clients and collaboration tools assume that people who email you the most are the ones that matter the most, but with a social network, wouldn’t this bias have the opposite effect? Wouldn’t the most chatty people be the ones who have the least to say that’s worth hearing?

One piece of good news from Mukund is that Facebook is working on a translator: one of our closest friends is a Swede, and his posts are all in Swedish which makes them incomprehensible. Having a built-in translator will certainly make Facebook more useful for people with international networks, although it will be very interesting indeed to see how Facebook’s translation deals with the idiosyncrasies of slang and idiom.

Update: we got it wrong. Facebook isn’t working on a translator; Mukund was referring to a third-party application.

One topic that particularly intrigues, but which we couldn’t raise with Mukund for lack of time, was Paul Adam’s monumental slideshare presentation on the importance of groups within social networks. Paul argues that people tend to have between 4-6 groups, each of which tend to have 2-10 people. (This is based upon the research behind Dunbar’s Number, which posits that there is a limit to the number of friends which whom one can form lasting relationships, and this number is around 150.)

Facebook still doesn’t have groups, which is surprisingly since Paul Adam decamped Google for Facebook soon after making his presentation available online. It is a massive presentation, but fascinating material and surprisingly light reading: just fast-forward to slide 64 and the picture there sums up the entire presentation rather well.

Update: we got that wrong, too. Faceb0ok does have a groups product, and it is becoming increasingly popular within the company itself

All in all, one of the most enjoyable presentations we have attended in recent days. Mukund needs special commendation for his fortitude and confident good humor in standing up before a savvy crowd and braving any and all questions about Facebook’s past, present and future.

Seattle’s “evening tech scene” is really getting interesting these days: perhaps we are seeing the working of a virtuous cycle where meetups and other events start to “up their game”!

A single-click if you are under 35, a double-click if you are over 35

Technology, Usability

When we first built Kerika, we deliberately modeled the user interface using the desktop application metaphor: projects were organized in folders, and mouse actions were as follows:

  • Select an item on the page with a single mouse click.
  • Open (or launch) an item on the page with a double mouse click.

It seemed the most natural thing in the world to us: everyone on the Kerika team liked it, and we assumed that our users would like just as much.

So it came as a very considerable surprise to us when we observed a generation gap among our beta users, in terms of what they considered to be the most natural way to open and launch items.

The breakpoint is roughly at the 35 years-old mark: people older than 35 had a strong preference for the double-click metaphor, and people under 35 had an equally strong preference for the single-click metaphor: where you select an item with one gesture, and then you select the action you wish to take from a menu that pops up.

The preference grew almost exponentially in intensity as you moved away from the 35-year breakpoint: people in their 50s, for example, had a very strong preference for double-clicking, while people in their early 20s were, for the most part, surprised by the notion that a double-click might do anything at all.

Our theory for this phenomenon is simple: roughly 15 years ago, the Netscape browser came into wide use. People who are older than 35 started their careers before the Web, and first learned to use desktop applications before learning to browse the Web. People under 35, on the other hand, probably first learned to use computers by surfing the Web at college.

(You might guess from all this that the Kerika team’s average age is well over 35, because it never occurred to us that the double-click metaphor might be strange or unappealing to anyone.)

At any rate, Kerika now supports the single-click metaphor exclusively – at this time. The initial feedback we got from our beta users was skewed by the younger demographic, and this caused us to reluctantly abandon the double-click in favor of the single-click. However, we are now hearing more from older users, and a future version – very soon! – will support both the single-click and double-click metaphors.

And while the Kerika application doesn’t run completely smoothly on iPads – for one thing, the Google Docs application doesn’t really run on iPads – supporting the single-click metaphor positions us to ensure that Kerika will run, intact, on tablet computers in the near future.

Use AT&T: it will boost your self-esteem

Technology

When you are in a startup, you always a sense of frustration that you haven’t been able to deliver all of your wonderful vision. Of course, it could happen in a big company as well, if you are really passionate about your job, but in a startup it happens all the time because entrepreneurs are wildly, dangerously passionate about what they do!

All the compromises that you had to make because of time and money constraints, the features you were forced to postpone, and this nagging sense of embarrassment that your 1.0 wasn’t all it could have been: “If only I had had more time and money, I could have built all these other cool features for our first release” you say to yourself.

If you ever feel a little depressed about the state of your version 1.0, there are two ways to boost your self-esteem and feel re-energized:

  • Remember the experienced entrepreneur’s mantra: “if you aren’t embarrassed by your first version, you have waited too long to ship“. We are not sure who coined this phrase – it’s been attributed to a number of people over the years – and there’s a great post called “1.0 is the loneliest number” that makes the case for shipping early, warts and all, and then iterating fast in response to customer feedback.

  • The other way to quickly start feeling better about your product is to visit the AT&T website to do something relative straightforward, like adding money to a pre-paid phone. The user experience that AT&T provides is guaranteed to make you feel better about your own product.

Why we chose Amazon’s EC2 over Google’s App Engine: the CTO’s Perspective

Kerika, Technology

A previous post discussed our decision to use Amazon’s EC2 service over Google’s App Engine and Microsoft’s Azure service primarily from a business perspective: the risks we weighed when we made our technology choices last August, and why the decision went in favor of EC2.

(We have also noted previously – lest anyone consider us uncritical fanboys! – that there are limitations with Amazon’s Web Services that have given us some heartburn.)

In today’s post, we present the CTO’s perspective: why EC2 was more attractive from a technology perspective:

The advantage of using Amazon’s EC2 over Google App Engine and Microsoft Azure is that for a highly interactive web applications such as Kerika, there needs to be multiple levels of caching of data.

Kerika maintains a cache within the browser so that it can avoid round trips to the server when the user moves objects around on the screen. Without that cache, the delays would be so long that users wouldn’t feel they were using a highly responsive desktop application – making it harder for us to .

There also needs to be a cache on the server side, and the most compelling reason for this is to reduce load on the database.

There are various ways to scale up a database, but none are as effective as simply minimizing the load on the database in the first place. A server-side cache is a simple way to do that. Because Kerika supports relationships between users, projects, idea pages, and so on, the server side cache has to maintain those relationships.

The Kerika server also must keep track of the state of the caches on its clients so that the it can maintain the clients in a current state, and thus avoid round trips to the server.

As a consequence of all this, Kerika uses servers with a large amount of RAM that needs to be quickly accessible for all of its users. Storing a large amount of data RAM is where EC2 becomes the only way to solve the problem. Because App Engine and Azure do not allow us to manage large amounts of RAM, they just weren’t good solutions for Kerika.

Another technical challenge is maintaining the long-lived browser connections that a Web application like Kerika depends upon. Google App Engine has the Channel API, but that doesn’t quite cut it for an application like Kerika.

Kerika needs to maintain several channels concurrently because users can be working with different combination of objects at the same time.

Kerika also makes use of broadcasts as a simple way to keep multiple users up to date. Because of EC2’s open architecture, we can use CometD as an off-the-shelf solution for client communication.

Finally, Google App Engine’s business model, which calls for charging two hours for each token doesn’t make economic sense in an environment where users are expected to navigate from page to page through the course of the day. EC2’s “pay-as-you-go” allows us to manage our traffic and keep operating costs down.

A surprisingly inelastic Elastic Load Balancer

Technology

Amazon’s Web Services are generally good, but they are not without flaws by any means, and in the past week we ran headlong into one of their limitations.

Some background, first: about a decade or two ago, people were very careful about how they typed in web addresses when using their browsers, because browsers weren’t very good at handling anything less than a perfectly formed URL. So, if you wanted to visit our website an eon ago, you probably would have typed in “http://www.kerika.com”. Ah, those were the days… You would direct people to “hot top dub dub dub kerika dot com”.

As lifeforms evolved, the “hot top” (did anyone really pronounce it as “hiptytipty colon whack whack dub dub dub dot domain dot com”?) got dropped, and we had the alliterative rhythm of “dub dub dub kerika dot com”.

From a branding perspective, however, having all that extra junk in front of your name isn’t very attractive, and we would rather be known as “Kerika.com” than “www.kerika.com”. After all, even in the crazy late 1990s when every company was guaranteed an immediate stock boost of 20% just by appending “.com” to their name, no one rushed to prefix their company names with “www.”

At Kerika, we decided that our canonical name of kerika.com was the way to go, which is why typing in “www.kerika.com” will land you at “kerika.com”. Setting that up is easy enough using the CNAME fields in DNS. So far, so hiptytipty…

The problem arose when we decided to use Amazon’s Elastic Load Balancer, so that we could have multiple servers (or, more precisely, multiple Amazon Machine Instances or AMIs) in the cloud, all hidden behind a single load balancer. It all sounded good in theory, but the problem came with routing all of our traffic to this ELB.

Getting www.kerika.com pointing to the ELB was easy enough by changing the CNAME in DNS, but in trying to kerika.com to also point to the ELB we fell into that gap that exists all too often between documentation and reality. But what really surprised us was finding out that the problem had been known to Amazon since May 2009, and it still hadn’t been fixed.

We can only guess why a known problem like this would remain unfixed for two years. The promised solution is supposed to be Route 53, Amazon’s new DNS for their cloud services. We heard an interesting presentation about Route 53 at Amazon’s premises a week ago; it was too bad we didn’t know then that our planned use of ELB would hit a brick wall or we could have pestered the Amazonians about this problem. According to the Route 53 website:

In the future, we plan to add additional integration features such as the ability to automatically tie your Amazon Elastic Load Balancer instances to a DNS name, and the ability to route your customers to the closest EC2 region.

We are waiting…

In the Spring, a young man’s thoughts turn seriously to marriage…

Technology

In the spring, a young man’s thoughts turn lightly to love…

(from “Locksley Hall,” by Alfred, Lord Tennyson.

Well, one young man’s thoughts turned to marriage. A self-professed “English code monkey” (when was the last time you saw that particular conjunction of words?) has been using Kerika to help plan out his wedding.

He lives locally, as does his fiance, but his parents live in the U.K. and his in-laws-to-be are in Texas, and so – in a very unexpected and utterly delightful use case for Kerika – this fellow has been using Kerika to plan his wedding.

We have been fortunate to share his experiences first hand, and since he is a very organized code monkey, he took a number of screenshots of his Kerika application that he has written about in his own blog. He calls it his “big fat geek wedding”!

Choosing between Google’s App Engine and Amazon’s EC2

Kerika, Technology,

Back in the summer, when we were making our technology choices, we had to decide which company was going to provide our “cloud”. It really boiled down to a decision between Google’s App Engine and Amazon’s EC2. Microsoft’s Azure service was still relatively new, and in talking to one of their product evangelists we got the distinct impression that Microsoft was targeting Azure for their biggest customers. (That was nice of their product evangelist: not taking us down the garden path if there was a mismatch between our needs and Microsoft’s goals!)

Here’s why we chose EC2 over App Engine:

  • Amazon is accessible: both Amazon and Google have engineering centers in our neighborhood, and Amazon’s is, of course, the much larger presence, but the real issue was which company was more accessible? Amazon does a lot of outreach to the startup community in the Seattle area – Jeff Barr is everywhere! – whereas Google is a lot more aloof. It’s much easier to find an engineer or product manager at Amazon who will talk to you, and that really makes a difference for a startup. Real people matter.
  • There was negative buzz about App Engine, at least back in August. People in the startup community were talking about suffering outages, and not getting very good explanations about what had gone wrong. Google’s aloofness didn’t help: there wasn’t anyone reaching out to the community to explain what was happening and there was little acknowledgment of problems coming through on Google’s official channels.
  • To beta or not to beta? Google’s rather lavish use of the “beta” label (remember the many years that Gmail was in beta?), and their recent decision to pull the plug on Wave added to our nervousness. Were they serious about App Engine, or was this another science experiment?
  • An infinite loop of web pages: in the absence of any physical contact with Google, we tried to learn about their service through their web pages – well, duh, of course, dude! The problem was that much of the information is presented in a series of rather small pages, with links to more small pages. As an approach to information architecture – particularly one that’s biased towards being “discoverable” by a search engine – this makes a lot of sense, and the Kerika website takes as similar approach. The risk with this approach is that you can very easily find that you are following links that take you right back to where you started. (And our website runs the same risk; we look forward to your comments detailing our failings in this regard!)

While we were pondering this choice, Google came out with a rather interesting offer: unlimited free storage for a select handful of developers. (How select? Probably as select as one of Toyota’s Limited Edition cars, production of which is strictly limited to the number they can sell.) The offer was presented in the usual mysterious way: you went to an unadvertised website and made an application. Someone would let you know “soon” whether you would get past the velvet rope.

We thought we had a good chance of getting included in the program: after all, Kerika was designed to add value to Google Docs, and we were using a number of Google’s services, including their Open ID for registration. And, indeed, we were selected!

The only problem was that the reply came about 4 weeks later: an eon in startup time. By that time, we had already decided to go with EC2…

(Is EC2 perfect? No, not by a long shot, and we were particularly annoyed to find a rather fatal flaw – from our perspective – in the way their Elastic Load Balancer can be configured with canonical web addresses: a problem that’s been noted for 2 years already. Not cool to sit on a problem for 2 years, guys. Not cool at all.)

Using Kerika to QA itself

Kerika, Technology

Since we are developing software for helping distributed teams collaborate more effectively, it was only natural that we would set up our own team as a distributed one.

Working remotely and relying principally on electronic communications, supplemented by some in-person meetings, has helped us understand better the challenges faced by distributed teams. As we believed from the outset, the biggest challenge has been keeping everyone on the same page in terms of our company strategy: business strategy, marketing strategy, and product roadmap.

Throughout our development process one of us would bemoan the lack of software like Kerika, that could help us build software like Kerika… Once the product became usable enough, we started “dogfooding” it –that elegant phrase invented by Microsoft that refers to a product team “eating it’s own dog food.”

One of the ways in which the Kerika team is using Kerika is for our QA: whenever we decide to work on a bug, we create a new project and name it after that bug number. Inside, we put our Build Verification Test (BVT) as well as the exit (success) criteria for that particular bug. It’s a neat trick: by going through the BVT, which we use as a sanity test before the developers hand off their code for QA, we end up creating a mini Kerika project for each bug.

For example, our BVT requires developers to upload a document to a Kerika page: well, for each bug, we upload a document that represents the exit criteria for that particular bug. The BVT requires users to go through various steps that provide a general coverage of Kerika’s main features. This means logging in using at least 3 different browsers (we usually test with Firefox, Safari and Chrome), and going through the process of adding Web links, videos, etc.

By using Kerika to test Kerika, at the end of each bug’s coding cycle we have a new project that we can look at and see whether it passed the BVT. It’s self-referential: the existence of a correctly set up project, with a particular team consisting of both Team Members and Visitors who perform certain actions, confirms that the BVT passed.

We combine the BVT with the exit criteria for each bug: these are derived from the reproduction steps of the bug report, plus the functional specifications. Going through the exit criteria for a particular bug, we end up with items in that bug’s project folder that confirm whether the bug was successfully fixed.

For example, if there was a bug about embedding YouTube videos on Kerika pages, the exit criteria would be such that at the end of the developer’s testing, the project would contain a YouTube video that could be examined to confirm that the bug was fixed.

So if the project for that bug is set up correctly at the end of the bug-specific repro steps and the BVT, then the developer knows he can check in his code for QA on our test environment. Of course, during our QA cycle we do more extensive testing, but this initial use of Kerika helps developers avoid breaking the build when they check in the code.

Pretty neat way of dogfooding, wouldn’t you agree?

Hello once again, world!

Kerika

After a rather long hiatus (over three years!), Kerika is back, and with a bang!

We owe the world some explanation of why Kerika disappeared, and why it is now reappearing…

Here’s what happened: when we launched Kerika back in 2006, it was as a desktop application, written entirely in Java so that it would run on Windows, Macs and Linux computers. People really liked the concept, particularly the innovative user interface and the ease with which one could do document management. But, there were two serious drawbacks with that first version:

  • The biggest problem was our reliance upon JXTA, an open-source peer-to-peer communication technology that had been hatched at Sun Microsystems (remember them?) by none other than the legendary Bill Joy. On paper, JXTA looked perfect: it’s theoretical model and architecture exactly matched our needs. In practice, however, it proved to be a disastrous technology choice.
  • The other big problem we had was that Kerika 1.0 was a desktop application, which meant that it needed to get downloaded, and users needed to configure their firewalls to let the JXTA traffic go through. This proved to be a huge hurdle for many people who were interested in the product, but couldn’t get it past their IT gatekeepers.

Eventually, we ran out of time and money, which is really the same thing from a startup’s perspective. Of the two flaws listed above, the dependency on JXTA was really the killer: it meant that we couldn’t reliably provide communication or transfer of files between users. (And the topic of JXTA really merits its own blog post.)

And, so, we had to pack up our tents and go get “regular” jobs.

That’s the story of why Kerika disappeared.

What’s more interesting, is the story of why Kerika is now reappearing:

A funny thing happened, in the 3 years that Kerika v1 was taken off the market: people kept writing in asking for the product. (We had never taken down the website, so the demos were still available; you just couldn’t download the product any more.)

This got us thinking that maybe Kerika was fundamentally a good idea, but we had screwed up the execution of that idea. And there was another thing that surprised us: in the intervening years, no one else released anything like Kerika –a flexible whiteboard on which you could sketch out your ideas and plans, and also embed your content.

Last Spring, we sent an email to our old user base, trying to understand better what it was they found attractive about Kerika, and, in the process, trying to gauge the interest in reviving the product. The replies we received convinced us that (a) Kerika was, fundamentally, a good idea, at heart, and (b) the needs it served were still not being met by anything else in the market.

In August we reconstituted the Kerika team: a different team than before, with the skills that we would need to rebuild Kerika, from scratch, as a Web application. We have been hard at work ever since, and have done a compete rebuild – not a single line of code was reused! – and now we are ready to present to you the fruits of our labors.

In the next few blog posts, we will talk about the new product, the challenges we faced, the choices we made, and the lessons we learned from Kerika v1 (or “K1” as we like to call it.)

Welcome back.