p'unk default blog
March 14th, 2011

Replacing Twitter: A Modest Proposal

Tom Boutell
Chief Software Architect

Everybody loved Twitter, then they started pushing the other birds out of the nest, now folks are mad. Yeah, well, that's what happens when you put all your eggs in one nest. That you do not own.

So let me ask you something. Why are microblogging and "regular" blogging different at all?

You can host a blog anywhere, right? So how many of you host your blog with wordpress.com? How many with tumblr? How many host your own (often via Wordpress software, since it is open source)?

It doesn't matter because they all interoperate via RSS feeds and you can shove those in your feed reader and you're good to go. You can use one of the big blog hosting companies, or switch at any time, knowing that they can't afford to get nasty with you because you have choices.

Why, exactly, does microblogging (what Twitter gives us) have to be any different?

If we all fired up instances of Wordpress or something even simpler tomorrow and started posting 140-character entries and added each other's microblog RSS feeds to our Google Reader or any feedreader of our choice, that could be iteration one of an open Twitter replacement.

Here are some objections to this, and ways to fix them.

  • No hashtags. If I can't write #monkeys and be instantly discovered by people who are also microblogging about monkeys, how will I discover new people to follow based on shared interests?

    Hashtags are not magic rainbow Twitter technology. They are a social convention invented recently by Chris Messina. Twitter does not own them, and if people start using them in their RSS feeds, Google, Bing and other freely competing search firms will look for them when indexing RSS feeds and help you search for them, browse them, etc. In fact, they are indexing your blog right now. Already you can be discovered in a search for a relevant topic. Firms that build niftier search implementations and mobile apps specifically for watching a hashtag mentioned in feeds will be rewarded for their efforts with more ad revenue. Ain't hard (OK, it's very hard, but it's a problem several companies are already solving for us).
  • RSS isn't designed for this. It's overdesigned for this in that it's ready for longer posts. But of course the brevity of Twitter is part of what we love about it. So microblog RSS feeds should declare themselves (see below in my suggested solution to the @names problem) and adhere to a 140 characters or less policy, with the post in both the title and the body for backwards compatibility with feed readers. Microblog clients (like Twitter clients) written specifically for microblogging will actively ignore anything after 140 characters, after first stripping any HTML markup provided to help vanilla feedreaders recognize links and follow @names.
  • No @names. How will I know who you are? On Twitter, if I want to recommend someone to you or retweet what they've said, I can say "RT @jane monkeys are badass." If everybody has a blog hosted heaven knows where, I can't succinctly tell you who they are and where they are in one pithy phrase.

    Unlike hashtags, this problem will not solve itself through the magic of the marketplace. I propose this simple solution: if I mention @monkeyman, that should reference the microblog RSS feed monkeyman.com/microfeeds/monkeyman.rss. If I mention tarzan@monkeyman, that should reference the RSS feed monkeyman.com/microfeeds/tarzan.rss. If I mention tarzan@monkeyman.ca, that should reference the RSS feed monkeyman.ca/microfeeds/tarzan.rss. This way it is still possible to have a short @name (get your own domain and set up the default feed), but cheapskates can avoid getting their own domain (they can be tarzan@monkeyman), and although .com is international and already has a distributed registry, you can also use a fully qualified name (tarzan@monkeyman.ca) if you don't want to use .com or just want to use a more natural address that looks like (and can be, if you wish) a valid email address, such as tarzan@monkeyman.com. See what I did there?
  • What about replies? We've already solved the @names problem. If a reply is from someone you follow, you're going to see it in the normal course of things, and a microblog client could choose to present them prominently. Replies from strangers are trickier. We could use trackback pings, with the knowledge on both sides that they wouldn't be visible to anyone but the original poster, so their effectiveness as a spamming technique is minimal (to be sure you'd have to block some people). We could also use the same super-simple "hey I posted" protocol I describe below for use in realtime aggregation, when replying to anyone whose feed specifies a place to announce replies.
  • How do I know who is following me? I'm tempted to be snarky here, but the truth is this is important. We discover many worthwhile people on Twitter when they choose to follow us. Counting ongoing followers isn't easy to do well, they can just stop hitting your feed. What's most important is to know about new followers. We could have a simple "hey I'm following you now" protocol, which makes you aware of new people after your site checks their feed to make sure it's really there. It couldn't be mandatory (after all people using plain feedreaders won't show up in this way), which is a departure from Twitter. I'm open to your thoughts on this and any part of this proposal.
  • What about verified Twitter accounts? I don't see a whole lot of regular people bothering with these, but you could use an SSL RSS feed with as hardcore a certificate as you like, including a Verisign cert where they require you to send documents to get it, etc. And clients could display this information easily enough, as browsers can now.
  • Spam! We'll drown in spam! No, we won't. Don't add spammy people, just as you don't add them to your Google Reader today. The only place we might drown in spam is in searches for hashtags,and notifications of new followers. Yes, Twitter is fighting spam in their content, but Google and Bing are also fighting spam in their search indexes. It is in their best interest to do at least as good a job as Twitter in this area. Verifying that new followers are not spammers might be an opportunity for someone like Akismet.
  • My friends will never find me again. Yes, they will. Your friends probably already use Google Reader or some other feedreader, because blogs already exist and they wanna read 'em. They can add your microblog directly to their feedreader. And as clients specifically focused on open microblogging appear, they can use those instead for a lighter, more Twitter-like feed of what's happenin'. Which they will do because they are awesome social media gurus. We're talking about replacing Twitter here, not Facebook.
  • No profiles! How will they see my beautiful face? RSS feeds already provide good ways to link to the original site the feed came from. This can point to your profile page. Which can use the rel="me" microformat on an image. It'll just work with both blog readers and clients that are specifically looking for it.
  • Will this work for realtime? Isn't this wildly inefficient? Twitter, LiveJournal, Facebook, etc. have optimized the way their data is provided to Google and Bing, on a "this is the same data you'd get by pounding our servers to death in one smooth continuous incoming stream" basis. Refetching millions of blog feeds over and over (getting a lot of posts you've already seen each time) can't be quite so realtime as that. We can help matters by supporting a simple "new-since" query string parameter to our feeds (virtually all existing blog hosting software would at least cheerfully ignore it without complaining), which would give us only posts since a previous post with the specified id. Couple this with using "if-modified-since" in the HTTP query and we're cooking with gas.

    But even that doesn't take us to a realtime level. So let's break a teensy bit of new ground here: we need a way of telling the aggregation services that our microblog feeds have been updated, which excuses them from pounding them more than once a day otherwise. Okay! Here's an insanely simple microprotocol for telling Google or Bing or the aggregation service that's driving the special microblog HDTV display at your conference that you posted an update:

    POST /some/aggregation/service/url author=jane@monkeys.co.uk

    This suggests that the aggregation service check http://monkeys.co.uk/microfeeds/jane.rss for new posts sooner rather than later. Note that you still have to have the actual post in your feed for this to work. That's a feature; it makes it harder to give the aggregator and real people different feeds (provided the aggregator has a few distinct IPs to surprise you with from time to time).

    All sensible people would add the big public aggregators to their microblog's list to notify; dedicated microblogging software would come with some usual suspects configured. And all clever people would add local aggregators that are watching for the hashtags of the conferences they attend.

  • What about private Twitter accounts? What about them? Private twitter accounts are a separate use case. RSS and traditional blogs aren't that well suited to addressing it. And Twitter is overwhelmingly about the public thing. If you're really interested, you can fork my Plog project on github and help move that ball down the field, or get involved in introducing Diaspora to the concept of security. But I think fixing public microblogging is a more interesting (and winnable) battle at the moment.

What about identi.ca? As Douglas Adams put it in The Hitchhiker's Guide to the Galaxy, "there are those who would say this has already happened." identi.ca is a Twitter alternative that runs on open source code that implements an open protocol stack that is built on RSS. And as a microblog host, they have a license that seems to obligate them to play nice with their users (though what happens if they get acquired is an interesting question). They seem to support interop of some kind with other implementations of the OStatus stack.

I haven't studied it in depth, but so far my impression is that it is not quite what we're after. identi.ca aims to be a more ethical monolithic host of microblogs that plays nicely with other big hosts. I'm interested in getting back to the completely distributed world of people hosting their own darn blog wherever they want and feedreaders doing the work of aggregating it all for you while competing search providers solve the findability problem. But I'm willing to be convinced that the OStatus protocols are part of the solution. For instance, PubSubHubbub might be useful in addressing the problem of fast realtime monitoring of hashtags.

I welcome your thoughts. And your tweets (@boutell). We're not there yet after all.

Tom Boutell
Chief Software Architect