Monday July 7, 2008

Building a Twitter Reader

One of the features of Twitter is that it runs 24/7/365. (Un)fortunately, I don't. So, I miss things. I especially miss tweets from people in the "other" hemispheres! I didn't want to miss things, so I looked for a solution.

Being a programmer myself, I wanted a solution I could control and tweak if necessary. However, I didn't want to write something from scratch if I didn't have to!

Twitter has a popular, published API, so I figured someone would have written what I wanted. Someone did. I found pyTwerp (written in Python).

From the pyTwerp documentation:

The whole concept of Twerp is to allow your Twitter data stream to be pulled from Twitter, formatted via a template you can control and then output to the console.

It also allows you to post a status message or send a direct message.

That's it in a nutshell.

pyTwerp looked perfect for the back end. What about the front end?

I could easily read the output files with any text editor, or as plain .txt files in my web browser. But I wanted something that looked a little nicer. Enter TWiki.

TWiki is a structured enterprise wiki with a lot of programmable features. By using TWiki, I don't need to convert the output to html in order to read it on the web and I get the following useful features:

  • Blank lines are retained as "paragraph" breaks.
  • Anything that begins with http:// is rendered as a clickable link
  • *word* is converted to bold; _word_ is converted to italic.
  • I can "script" the results, using TWiki to hide everything I'm not currently reading, or only show me this month's output.
  • I can easily adjust the look and feel with CSS.

How I Built My Twitter Reader

I started by downloading and installing pyTwerp and its required libraries. The installation of pyTwerp includes a utility script named twerp that lets you invoke pyTwerp as

   twerp <options>
Because I planned to use TWiki for reading, I configured the pyTwerp output format to add some TWiki markup code.

Twitter Reader

Then, I wrote a small shell script I named gotwerp. gotwerp performs various housekeeping tasks and runs twerp, creating files with date-stamped names, for example: TwitterLog2008Jun26.txt. Every day at midnight, a new file is created. I configured cron to run gotwerp every 10 minutes.

After testing, I realized I needed some postprocessing steps.

  1. The Twitter API returns entries in LIFO order. I prefer chronological order; otherwise, conversations get muddled. To handle this, I added some code to gotwerp to reverse the order of the output tweets.
  2. I also added some code to handle a few "gotchas" I found along the way. (See my complete writeup for details.)

Finally, I wrapped everything up in a small TWiki application which provides:

  • An easy way to choose which Twitter logs to read
  • A reasonable default choice of log (today)
  • Hiding of any logs I'm not currently reading
  • A table of contents with quick links into the list of choices.
  • A CSS class to make the results look nice on the page.
You can view my Twitter Reader in action in my TWiki. Read more about the gory implementation details (and see the code) at Building a Twitter Reader.
Dislaimer: If you want to implement my Twitter Reader, you will need a computer capable of running Python, Perl, and the Unix cron command. Capable systems include Linux, BSD, Mac OS X, and Windows (if properly configured, probably with cygwin installed). For any other system, your mileage will vary widely. I have no idea if Windows has something approximating cron.

I use Mac OS X, which is based on BSD Unix. I cannot help you with any other system!


Thank you to bear and decklin for pyTwerp; special thanks to decklin for answering bozo novice questions and pointing me toward seeing the template feature which I somehow had missed the first time out!

Building a Twitter Reader ( in category Computerware , Show & Tell ) - posted at Mon, 07 Jul, 18:34 Pacific | «e»


Post a comment

Any posted comments will be viewable by all visitors. Please try to stay relevant ;-) If you simply want to say something to me, please send me email.

All comments will be moderated. Thank you for your consideration.