# Evan A. Sultanik, Ph.D.

Evan's First Name @ Sultanik .com

Computer Security Researcher
Trail of Bits

Drexel University College of Computing & Informatics
Department of Computer Science

## Ballmer Peak

### A Generalization

Through my many years of coding, I have come to this realization:

The so-called "Ballmer Peak", as it is currently understood, is but a two dimensional projection of what in reality is a higher dimensional space, vi&.,

## AAMAS 2010

### The Ninth International Conference on Autonomous Agents and Multiagent Systems

If you've been keeping up, you know that I was attending the Ninth International Conference on Autonomous Agents and Multiagent Systems (AAMAS) last week in Toronto, Canada. The conference went really well, as did the two workshops I also attended:

1. The Third International Workshop on Optimisation in Multi-Agent Systems (OptMAS); and
2. The Twelfth International Workshop on Distributed Constraint Reasoning (DCR), which Rob and I co-chaired.

I presented my paper on distributedly solving art gallery and dominating set problems on Thursday. AAMAS also allows for full papers to additionally present a poster for the work. This was my first time making a poster purely in LaTeX, and it was a very smooth experience. I created a poster template for the Drexel CS department which can be downloaded here. You can view and download the presentation slides and poster for my paper here.

## Success in Linux

### Linux: Simultaneously the source of and solution to all of my computing problems.

Oh Linux, you’re simultaneously the source of and solution to all of my computing problems. I recently had my own version of Linux “success”. Read on to hear the whole story.

## Retirement Planning through Treasure Hunting

### A metaphor for the process of earning a Ph.D.

I've deliberated long and hard to decide whether or not to publish this blog entry.

I inevitably end up liking things that require extensive deliberation as to whether or not I like them.
@ESultanik
Evan Sultanik

I'm afraid that, given I've not yet completed my Ph.D., it will be misconstrued as boastful or, worse yet, exculpating. This is not my intention. If anything, I hope that my observations below will help others pursuing a Ph.D. to both rationalize their own situation and educate others on the process.

It's been a long while since I've posted to this blog, in large part due to the fact that I've been in the throes of wrapping up my degree and hunting for a job. As such, I get a lot of questions from non-academics as to when I will finish. It seems like many people are under the impression that a Ph.D. is "what you get if you tough it out and stay in school and study a few more years after undergrad." While this is technically true, many either don't realize or don't understand that a Ph.D.—at least in the sciences, engineering, and mathematics—requires independent research. A Ph.D. isn't earned through studying hard and passing tests. It's particularly frustrating when people see that I've been pursuing my Ph.D. for ~4 years (an average, if not short, duration for the degree) and immediately draw the conclusion that, "Since [he] hasn't graduated yet, [he] must not be studying hard enough!" Over the years, I've developed a metaphor that I give to people to explain the process:

Earning a Ph.D. is like trying to fund one's retirement through treasure hunting.

When I started my Ph.D., I had to choose a topic to study. That's like choosing a location in a vast field on which to begin digging for buried treasure. You know that the treasure is out there somewhere, but you're not sure where or how deep. This is not entirely up to chance: One's advisor(s)—experts in gold digging—do help in choosing the location, and there are ways to intelligently predict where the treasure might lie. The trouble is that one might dig for years and years and only be left with a huge pile of dirt. Worse yet, one might be only a few millimeters of soil away from the gold and not even know it. Every once in a while one might find a small nugget of gold in the dirt, egging him or her on, but there is no guarantee that one will eventually hit the mother lode. In other instances, one knows exactly where the treasure lies, but it is underneath an impenetrable rock that requires years and years of chipping away to exhume. In some cases one does find a huge treasure chest, but more often than not one's fortune is amassed from accumulation of the small nuggets. Defending one's dissertation, then, is like choosing to go into early retirement based off of one's fortune amassed from treasure hunting. Is it enough? Will I be able to support myself on what I've found so far? In order to complete one's degree, one has to defend his or her work to a committee of experts—all of whom are expert treasure hunters and, in a sense, one's competition. You have to convince them that your fortune will be enough to support yourself. And you have to do all of that without wasting too much time creating fanciful metaphors of debatable import.

## iPhone Toolchain on Linux

### A Tutorial

I have an iPhone. I also use Google Mail as my web-based mail client. Unfortunately, there is really no good way to get push Gmail on an iPhone. Even now, post firmware 3.0, these are the best ways:
1. Pay for a service like MobileMe.
Problem: service fees seem like overkill, and the push E-mail would be the only benefit I’d get from it.
2. Wrap Gmail’s IMAP service in an exchange server. There are some paid services that do this, however, Z-Push is free (if one can host it one’s self).
Problem: the iPhone only supports a single Exchange server at a time. Therefore, I’d have to choose between getting push E-mail versus over-the-air calendar/contacts synchronization that is currently provided through Google’s own “Sync” Exchange wrapper.
3. Write an app that uses the new Push Notification service in firmware 3.0 to remotely push mail to the phone.
Problem: this would probably be a very lucrative solution (i.e., I’ll bet lots of people would pay a nominal one-time fee for this app), but it would almost definitely be rejected from the App Store. Furthermore, it would require me to set up a back-end server running 24/7 to push the updates.
4. Jailbreak the phone and write a daemon that runs in the background, connects to Google’s IMAP service, and goes into IDLE mode.
Problem: the only Apple device I own is my iPhone; how might I compile my own apps for it? (Sure, my wife does have a PowerBook, but that would be cheating, right? Right‽)
Read on to discover how I was able to set up an iPhone development toolchain on Linux.

## File Drop

### Computer-to-Computer File Transfer for the Masses

Continuing the recent theme of posting-random-scripts-as-blog-entries…

I recently needed a quick and dirty way to send a really large (~1 gigabyte) file to someone. We were both on the same LAN, so it didn’t really make sense for me to upload it to my externally hosted web server. I do not have a web server installed on my laptop and, at the time, it seemed like overkill to install a web server just so I could send him my file. Using a thumb drive or scp would have been an option, but each would require the recipient to be physically at my computer (despite being on the same LAN, he was a 10 minute walk away). Therefore, I gave myself a 10 minute deadline to code my own solution (plus it would be a fun diversion from writing my journal paper due later that day).

Given that I had a whole 10 minutes (an eternity when it comes to Perl hacking), I figured I might as well make my method generalized (i.e., not only should my script be able to send files, but it should also be able to receive).

First, I had to decide on a method. FTP seemed like a logical choice, but, besides really tech savvy people, who has full-blown FTP clients installed these days? In keeping with my generality goal, my solution would ideally be usable by, say, my mom. And moms don’t know ‘bout my FTP. Everyone, my mom (and my mom’s mom) included, has a web browser and knows how to use it. Therefore, good ol’ HTTP it was. And I even had a bunch of old code to hack together!

I ended up with a script that I call filedrop. Here’s the usage:

$filedrop Version: filedrop 0.1 2009-07-01 http://www.sultanik.com/ Copyright (C) 2009 Evan A. Sultanik Usage: filedrop [OPTIONS] FILE_PATH Options: -s send a file by hosting it on a local web server (default) -r receive a file by accepting it from a local web server. FILE_PATH should be a directory to which the files should be saved. FILE_PATH will default to ‘.’ in this mode. -n, --num=N quit after sending/receiving N files. If N is less than zero the program will send/receive files until manually terminated. If N is zero then the program will immediately quit. Default is -1.  And here’s an example of how the file transfer went down: LeEtH4X0r: Y0 Home Skillet! Can you fry me up some juarez‽ Me: Indubitably! $ filedrop -s -n1 ./hugefile.tar.gz
Server running at: http://my_ip:47489/

Me: Go to http://my_ip:47489/
LeEtH4X0r: ZOMGKTHXBAI!!!!1ONE

Here’s the code:

#!/usr/bin/perl -w

use HTTP::Daemon;
use HTTP::Status;

my $version = “0.1”; my$date = “2009-07-01”;
my $copyright = “2009”; my$port = 80;

sub print_usage {
print “Version: filedrop $version$date http://www.sultanik.com/\n”;
print “Copyright (C) $copyright Evan A. Sultanik\n\n”; print “Usage: filedrop [OPTIONS] FILE_PATH\n\n”; print “Options:\n”; print “ -s send a file by hosting it on a local web server (default)\n”; print “ -r receive a file by accepting it from a local web server.\n”; print “ FILE_PATH should be a directory to which the files should be\n”; print “ saved. FILE_PATH will default to ‘.’ in this mode.\n”; print “ -n, --num=N quit after sending/receiving N files. If N is less than zero\n”; print “ the program will send/receive files until manually\n”; print “ terminated. If N is zero then the program will immediately\n”; print “ quit. Default is -1.\n”; print “\n”; } my$mode = “s”;
my $num = -1; my$last = “”;
my $nextIsN = 0; foreach my$arg (@ARGV) {
if($arg eq “-s”) {$mode = “s”;
} elsif($arg eq “-r”) {$mode = “r”;
} elsif($arg eq “-n”) {$nextIsN = 1;
next;
} elsif($arg =~ /-n(\d+)/) {$num = $1; } elsif($arg =~ m/--num=(\d+)/) {
$num =$1;
} elsif($nextIsN) {$num = $arg; } else { if(!($last eq “”)) {
print_usage() && die(”Invalid option: “ . $last . “\n”); }$last = $arg; }$nextIsN = 0;
}
if($last eq “” &&$mode eq “s”) {
print_usage() && die(”Path to a file to host expected!\n”);
} elsif($last eq “” &&$mode eq “r”) {
$last = “.”; } my$file = $last; exit(0) if($num == 0);

my $d = HTTP::Daemon->new(LocalPort =>$port) || HTTP::Daemon->new() || die;
print “Server running at: “, $d->url, “\n”; my$servings = 0;
while(my $c =$d->accept) {
while(my $r =$c->get_request) {
if($mode eq “s”) { if($r->method eq ‘GET’) {
$c->send_file_response($file);
$servings++; } else {$c->send_error(RC_FORBIDDEN)
}
} elsif($mode eq “r”) { if($r->method eq ‘POST’) {
$servings++; my$url = $r->content; while($url =~ m/.*?-+(\d+)\r\nContent-Disposition:.*? filename=”([^”]+)”.*?\r\n\r\n(.*?)\r\n-+\1-+(.*)$/ism){ my$id = $1; my$filename = $2; my$content = $3;$url = $4; my$newName = $filename; my$i = 0;
$newName =$filename . “.” . ++$i while(-e$file . “/” . $newName); if($i > 0) {
print “A file of named $filename already exists in$file!\n”;
print “Saving to “ . $file . “/” .$newName . “ instead.\n”;
$filename =$newName;
}
open(OUTFILE,”>” . $file . “/” .$filename) or die(”Error opening $file/$filename for writing!\n”);
binmode OUTFILE;
print OUTFILE $content; close(OUTFILE); print “Received$filename (ID: $id)\n”; }$h = HTTP::Headers->new;
$h->header(’Content-Type’ => ‘text/html’); my$msg = “UploadedUploaded!“;
$msg .= “Click here to upload another file.” if($num < 0 || $servings <$num);
$msg .= "“;$r = HTTP::Response->new( HTTP_OK, “”, $h,$msg);
$c->send_response($r);
} elsif($r->method eq ‘GET’) { print “Someone connected! Sending the upload form...\n”;$h = HTTP::Headers->new;
$h->header(’Content-Type’ => ‘text/html’);$r = HTTP::Response->new( HTTP_OK, “”, $h, “Upload Please specify a file, or a set of files: “);$c->send_response($r); print “Sent!\n”; } else {$c->send_error(RC_FORBIDDEN);
}
last if($num > 0 &&$servings >= $num); } last if($num > 0 && $servings >=$num);
}
$c->close; undef($c);
last if($num > 0 &&$servings >= $num); } close($d);


## Mail Notifier

An example of the notifier, well, notifying.

I recently caught a glimpse of how Gmail Notifier works on a friend’s Mac. It looked pretty cool. Unfortunately for me, though, there’s no reasonable facsimile in Linux. Sure, there are a couple options, but they aren’t available in Gentoo’s package management system. Given my recent experience dealing with E-mail from Perl, I figured it would be just as easy to write my own E-mail notifier as it would be to manually install these programs (along with their dependencies). I was right. I just spent the last ~20 minutes (while idling through a meeting) writing such an app. The code follows below. Its only dependency is XOSD.

Disclaimer: I blatantly cribbed some of my code from Flavio Poletti (for the MTA stuff) and Bill Luebkert (for the password input).

Future work: right now the code simply polls the mail server once every three minutes. In the future I’ll post an update that uses IMAP Idle to reduce bandwidth.

#!/usr/bin/perl -w

use Mail::IMAPClient;
use IO::Socket::SSL;
use File::HomeDir;

my $username = ‘youremail@domain.com’; my$sleeptime = 180; # Time between checks, in seconds.
my $conffile = File::HomeDir->my_home . “/.checkmail”; ######################################################$canceled = 0;
$inwhile = 0; sub get_passwd { # legal clear passwd chrs (26+26+10+24=86): “a-zA-Z0-9!#$%&()*+,-./:;<=> ?@[\]^“;
my @legal_clear = (’a’..’z’, ‘A’..’Z’, ‘0’..’9’, split //,
‘!#$%&()*+,-./:;<=> ?@[\]^’); my %legal_clear; foreach (@legal_clear) {$legal_clear{$_} = 1; }$| = 1;	# unbuffer stdout to force unterminated line out
my $ch = ‘’; while (defined ($ch = ReadKey ())) {
last if $ch eq “\x0D” or$ch eq “\x0A”;
if ($ch eq “\x08”) { # backspace print “\b \b” if$passwd;	# back up 1
chop $passwd; next; } if ($ch eq “\x15”) {	# ^U
print “\b \b” x length $passwd; # back 1 for each char$passwd = ‘’;
next;
}
if (not exists $legal_clear{$ch}) {
print “\n’$ch’ not a legal password character\n”; print ‘Password: ‘; next; }$passwd .= $ch; } print “\n”; ReadMode (’restore’); return$passwd;
}

$SIG{’INT’} = ‘INT_handler’; sub INT_handler { exit(0) if(!$inwhile);
$canceled = 1; print “\nCaught Signal; exiting gracefully!\n”; } print “Password: “; my$password = &get_passwd();

while(!$canceled) {$inwhile = 1;

my $socket = IO::Socket::SSL->new( PeerAddr => ‘imap.gmail.com’, PeerPort => 993, ) or (print STDERR “Warning: lost internet connection!\n” && next); # Perhaps we lost the internet connection? my$greeting = <$socket>; my ($id, $answer) = split /\s+/,$greeting;
die “problems logging in: $greeting” if$answer ne ‘OK’;

my $client = Mail::IMAPClient->new( Socket =>$socket,
User     => $username, Password =>$password,
Uid => 1,
)
or die “new(): $@”;$client->State(Mail::IMAPClient::Connected());
$client->login() or die ‘login(): ‘ .$client->LastError();

die(”Failed authentication!\n”) unless $client->IsAuthenticated();$client->examine(’INBOX’) or die “Could not examine: $@\n”; my @msgs =$client->unseen or die “Could not search the inbox! $@\n”; my$last_max = -2;
if(-e $conffile) { # Load the old largest open(CONFFILE, “<" .$conffile) or die("Error opening " . $conffile . "\n"); while() { my$line = $_;$last_max = $1 if($line =~ /^\s*last_max_uid\s*=\s*(\d+)\s*$/i); } close(CONFFILE); } my$max = -1;
my @over;
for my $msg (@msgs) {$max = $msg if$msg > $max; push(@over,$msg) if $msg >$last_max;
}

if($max >= 0) { open(CONFFILE, “>” .$conffile) or die(”Error opening $conffile for writing!\n”); print CONFFILE “last_max_uid = “ .$max . “\n”;
close(CONFFILE);
}

if($last_max >= 0) { open(OSDC, “| osd_cat -c green -p middle -A center -s 2 -l 5 -f \”-bitstream-bitstream vera serif-*-*-*-*-17-*-*-*-*-*-*-*\””); for my$m (@over) {
my $hashref =$client->parse_headers($m, “From”) or die “Could not parse_headers:$@\n”;
print OSDC “New mail from “ . $hashref->{”From”}->[0] . “!\n”; } close(OSDC); }$client->logout();
sleep $sleeptime; }  ## Awaiting Death ### In which I coerce processes to email me as they die. I’ve been running a number of experiments recently that require a lot of computing time. “A lot” in this case being on the order of days. It would therefore be nice to have a script that would automatically E-mail me when my experiments finish so I know to check the results. I fully expected there to be some magic shell script out there somewhere dedicated to this very purpose: sending out an E-mail when a specified process dies. Something like this: $ ./run_experiments&
[1] 1337
$emailwhendone 1337 Awaiting process 1337’s death...  As far as I can tell, however, there is no such script/program. So, as usual, I took it upon myself to write my own. The E-mailing part turned out to be a bit trickier than I had expected. I didn’t want my script to be dependent on the existence of a local mail server; therefore, I first tried using sSMTP. It turns out that sSMTP requires one to hard-code the remote SMTP server address in a .conf file, so that approach was out. Next I tried Mail::Sendmail, however, that module’s support for authentication is poor at best. That module also doesn’t support SSL, so emailing through servers like Google Mail is out. Therefore, I finally settled on using Net::SMTP::SSL, which unfortunately has four dependencies. Luckily for me, those dependencies are all easily installable on Gentoo: 1. dev-perl/Authen-SASL 2. dev-perl/IO-Socket-SSL 3. dev-perl/Net-SSLeay 4. dev-perl/Net-SMTP-SSL I call my script emailwhendone because, well, that’s exactly what it does. The code follows at the end of this post. Disclaimer:Robert Maldon (for the MTA stuff) and Bill Luebkert (for the password input). The script can be given one of two parameters: either the PID of the process for which to wait or the unique name of the process (if there are multiple processes with the same name you will need to use the PID). Right now I have the recipient E-mail address hard-coded; it should be fairly self evident from the code how to customize this. Here’s an example: $ ./run_experiments&
[1] 1337
$emailwhendone 1337 Password for youremail@domain.com: ******************* Waiting for process 1337 (run_experiments) to finish... The process finished! Sending an email to youremail@domain.com...$


Here’s the code:

#!/usr/bin/perl -w

use Net::SMTP::SSL;

my $destination = ‘youremail@domain.com’; my$server = ‘smtp.domain.com’;
my $port = 465; ##################################### sub usage { print “ Usage: emailwhendone [PID|PROCESS_NAME]\n”; } my$pid = $ARGV[0] or die &usage(); my$hostname = hostname;
my $pidmatch = -1; my$processmatch = “”;
my @pidmatches;

open PRO, “/bin/ps axo pid,comm |” or die ‘Failed to open pipe to ps’;

while() {
if($_ =~ m/^\s*(\d+)\s+(.+)$/) {
my $matchpid =$1;
my $matchprocess =$2;
if($matchpid eq$pid) {
$pidmatch =$matchpid;
$processmatch =$matchprocess;
@pidmatches = [$matchpid]; last; } elsif($pid =~ m/^\s*$matchprocess\s*$/) {
$pidmatch =$matchpid;
push(@pidmatches, $matchpid);$processmatch = $matchprocess; } } } close PRO; if(scalar(@pidmatches) <= 0) { if($pid =~ m/^\s*\d+\s*$/) { print "Error: no process with ID " .$pid . "!\n";
} else {
print "Error: no process named \"" . $pid . "\"!\n"; } exit(1); } elsif(scalar(@pidmatches) > 1) { print “There are multiple PIDs that match this process name!\n”; for my$match (@pidmatches) {
print $match . “\t” .$pid . “\n”;
}
exit(2);
}

sub get_passwd {
# legal clear passwd chrs (26+26+10+24=86): “a-zA-Z0-9!#$%&()*+,-./:;<=> ?@[\]^“; my @legal_clear = (’a’..’z’, ‘A’..’Z’, ‘0’..’9’, split //, ‘!#$%&()*+,-./:;<=> ?@[\]^’);
my %legal_clear; foreach (@legal_clear) { $legal_clear{$_} = 1; }
$| = 1; # unbuffer stdout to force unterminated line out ReadMode (’cbreak’); my$ch = ‘’;
while (defined ($ch = ReadKey ())) { last if$ch eq “\x0D” or $ch eq “\x0A”; if ($ch eq “\x08”) {	# backspace
print “\b \b” if $passwd; # back up 1 chop$passwd;
next;
}
if ($ch eq “\x15”) { # ^U print “\b \b” x length$passwd;	# back 1 for each char
$passwd = ‘’; next; } if (not exists$legal_clear{$ch}) { print “\n’$ch’ not a legal password character\n”;
print ‘Password: ‘, “*” x length $passwd; # retype *’s next; }$passwd .= $ch; print ‘*’; } print “\n”; ReadMode (’restore’); return$passwd;
}

print “Password for “ . $destination . “: “; my$password = get_passwd();

sub send_mail {
my $subject =$_[0];
my $body =$_[1];

my $smtp; if (not$smtp = Net::SMTP::SSL->new($server, Port =>$port,
Debug => 0)) {
die “Could not connect to server.\n”;
}

$smtp->auth($destination, $password) || die “Authentication failed!\n”;$smtp->mail($destination . “\n”);$smtp->to($destination . “\n”);$smtp->data();
$smtp->datasend(”From: “ .$destination . “\n”);
$smtp->datasend(”To: “ .$destination . “\n”);
$smtp->datasend(”Subject: “ .$subject . “\n”);
$smtp->datasend(”\n”);$smtp->datasend($body . “\n”);$smtp->dataend();
$smtp->quit; } print “Waiting for process “ .$pidmatch . “ (” . $processmatch . “) to finish...”; my$done = 0;
do {
$done = 1; open PRO, “/bin/ps axo pid |” or die ‘Failed to open pipe to ps’; while() { if($_ =~ m/^\s*$pidmatch\s*$/) {
$done = 0; last; } } close PRO; sleep(1); } while(!$done);

print “The process finished!\nSending an email to “ . $destination . “...”; &send_mail(’Process ‘ .$pidmatch . ‘ (’ . $processmatch . ‘) on ‘ .$hostname . ‘ finished!’, ‘It\’s done!’);

print “\n”;


### Journey to the Center of the Twitterverse

I’ve now been using Twitter for about six months. While Twitter’s minimalism is no doubt responsible for much of its success, I often pine for some additional social networking features. High up on that list is a simple way of representing my closest neighbors—perhaps through a visualization—without having to manually navigate individual users’ following/followers pages. A well designed representation could be useful in a number of ways:

1. It could expose previously unknown mutual relationships (i.e., “Wow, I didn’t know X and Y knew each other!);
2. It could reveal mutual acquaintances whom one did not know were on Twitter; and
3. Metrics on the social network could be aggregated (e.g., degrees of separation).
This afternoon I spent an hour or so hacking together a Python script, which I have dubbed TwitterGraph, to accomplish this. Here is an example of the ~100 people nearest to me in the network:

Usage: paste the code below into TwitterGraph.py and run the following:

$chmod 755 ./TwitterGraph.py$ ./TwitterGraph.py
You have 100 API calls remaining this hour; how many would you like to use now? 80
What is the twitter username for which you’d like to build a graph? ESultanik
Building the graph for ESultanik (output will be ESultanik.dot)...
.
.
.
$dot -Tps ESultanik.dot -o ESultanik.ps && epstopdf ESultanik.ps && acroread ESultanik.pdf$ dot -Tsvgz ESultanik.dot -o ESultanik.svgz


There are also (unnecessary) command line options, the usage for which should be evident from the sourcecode.

#!/usr/bin/python

import simplejson
import urllib2
import urllib
import getopt, sys
import re
import os

@property
def message(self):
return self.args[0]

if ‘error’ in data:

def fetch_url(url):
opener = urllib2.build_opener()
opener.close()
return url_data

def remaining_api_hits():
return data[’remaining_hits’]

def get_user_info(id):
global calls
json = None
calls += 1
else:
return data

def get_friends(id):
global calls
calls += 1
return data

def get_followers(id):
global calls
calls += 1
return data

last_status_msg = “”
def update_status(message):
global last_status_msg
# clear the last message
sys.stdout.write(“\r”)
p = re.compile(r”[^\s]”)
sys.stdout.write(p.sub(’ ‘, last_status_msg))
sys.stdout.write(”\r”)
sys.stdout.write(message)
last_status_msg = message
sys.stdout.flush()

def clear_status():
last_status_msg = “”

def save_state():
global history
global user_info
global friends
global followers
global queue
data = simplejson.dumps([history, user_info, friends, followers, queue])
bakfile = open(username + “.json”, “w”)
bakfile.write(data)
bakfile.close()

global friends
idxes = {}
idx = 0
for user in friends:
idxes[user] = idx
idx += 1
for user in friends:
if len(friends[user]) <= 0:
continue
amount_to_give = 1.0 / len(friends[user])
for f in friends[user]:
if str(f) in idxes:

try:
opts, args = getopt.getopt(sys.argv[1:], "hu:c:r", ["help", "user=", "calls=", "resume"])
except getopt.GetoptError, err:
print err
#usage()
sys.exit(2)

max_calls = -1

for o, a in opts:
if o in ("-h", "--help"):
#usage()
sys.exit()
elif o in ("-u", "--user"):
elif o in ("-c", "--calls"):
max_calls = int(a)
elif o in ("-r", "--resume"):
else:
assert False, "unhandled option"

if max_calls != 0:
# First, let's find out how many API calls we have left before we are rate limited:
update_status("Contacting Twitter to see how many API calls are left on your account...")
max_hits = remaining_api_hits()
if max_calls < 0 or max_hits < max_calls:
update_status("You have " + str(max_hits) + " API calls remaining this hour; how many would you like to use now? ")
max_calls = int(raw_input())
clear_status()
if max_calls > max_hits:
max_calls = max_hits
print “What is the twitter username for which you’d like to build a graph? “,

update_status(”Trying to open “ + username + “.dot for output...”)
dotfile = open(username + “.dot”, “w”)
update_status(””)
clear_status()
print “Building the graph for “ + username + “ (output will be “ + username + “.dot)...”

history = {}
calls = 0
user_info = {}
friends = {}
followers = {}

# Let’s see if there’s any partial data...
print “It appears as if you have some partial data for this user.”
resume = “”
print “Do you want to start off from where you last finished? (y/n) “,
resume = re.compile(r”\n”).sub(””, raw_input())
if load_prev == True or resume == “y” or resume == “Y” or resume == “yes” or resume == “Yes” or resume == “YES”:
bakfile = open(username + “.json”, “r”)
print str(len(friends)) + “ friends!”
bakfile.close()
print “The current queue size is “ + str(len(queue)) + “.”
else:
print “You are about to overwrite the partial data; are you sure? (y/n) “,
resume = re.compile(r”\n”).sub(””, raw_input())
if not (resume == “y” or resume == “Y” or resume == “yes” or resume == “Yes” or resume == “YES”):
exit

while len(queue) > 0 and calls + 3 <= max_calls:
next_user = queue.pop(0)
# Let's just double-check that we haven't already processed this user!
if str(next_user) in history:
continue
update_status(str(next_user) + "\t(? Followers,\t? Following)\tQueue Size: " + str(len(queue)))
if next_user in user_info:
info = user_info[next_user]
else:
try:
info = get_user_info(next_user)
except urllib2.HTTPError:
update_status("It appears as if user " + str(next_user) + "'s account has been suspended!")
print ""
clear_status()
continue
uid = next_user
uid = info['id']
history[uid] = True
user_info[uid] = info
update_status(info['screen_name'] + "\t(? Followers,\t? Following)\tQueue Size: " + str(len(queue)))
followers[uid] = get_followers(uid)
for i in followers[uid]:
if str(i) not in history:
history[i] = True
queue.append(i)
update_status(info['screen_name'] + "\t(" + str(len(followers[uid])) + " Followers,\t? Following)\tQueue Size: " + str(len(queue)))
friends[uid] = get_friends(uid)
for i in friends[uid]:
if str(i) not in history:
history[i] = True
queue.append(i)
update_status(info['screen_name'] + "\t(" + str(len(followers[uid])) + " Followers,\t" + str(len(friends[uid])) + " Following)")
clear_status()
sys.stdout.write("\n")
sys.stdout.flush()
save_state()

# Get some extra user info if we have any API calls remaining
# Find someone in the history for whom we haven't downloaded user info
for user in history:
if calls >= max_calls:
break
if not user in user_info:
try:
user_info[user] = get_user_info(user)
except urllib2.HTTPError:
# This almost always means the user’s account has been disabled!
continue

if calls > 0:
save_state()

# Now download any user profile pictures that we might be missing...
user_image_raw = {}
for u in friends:
_, _, filetype = user_info[u][’profile_image_url’].rpartition(”.”)
filename = username + “.images/” + str(u) + “.” + filetype
user_image_raw[u] = filename
if not os.path.isfile(filename):
urllib.urlretrieve(user_info[u][’profile_image_url’], filename)
update_status(”Profile pictures are up to date!”)
print “”
clear_status()

# Now scale the profile pictures
update_status(”Scaling profile pictures...”)
user_image = {}
for u in friends:
_, _, filetype = user_info[u][’profile_image_url’].rpartition(”.”)
filename = username + “.images/” + str(u) + “.scaled.” + filetype
user_image[u] = filename
if not os.path.isfile(filename):
# we need to scale the image!
update_status(”Scaling profile picture for “ + user_info[u][’screen_name’] + “...”)
os.system(”convert -resize 48x48 “ + user_image_raw[u] + “ “ + user_image[u])
update_status(”Profile pictures are all scaled!”)
print “”
clear_status()

print “”
clear_status()
update_status(”Calculating the stationary distribution...”)
iterations = 500
damping_factor = 0.25
st = [1.0]*len(friends)
last_percent = -1
for i in range(iterations):
users = 0
for u in friends:
users += 1
percent = round(float(i * len(friends) + users) / float(iterations * len(friends)) * 100.0, 1)
if percent > last_percent:
last_percent = percent
update_status(”Calculating the stationary distribution... “ + str(percent) + “%”)
idx = idxes[str(u)]
given_away = 0.0
give_away = st[idx] * (1.0 - damping_factor)
if give_away <= 0.0:
continue
for f in friends[u]:
if str(f) not in friends:
continue
fidx = idxes[str(f)]
given_away += ga
st[fidx] += ga
st[idx] -= given_away
print ""
clear_status()
# Now calculate the ranks of the users
deco = [ (st[idxes[u]], i, u) for i, u in enumerate(friends.keys()) ]
deco.sort()
deco.reverse()
rank = {}
last_st = None
last_rank = 1
for st, _, u in deco:
if last_st == None:
rank[u] = 1
elif st == last_st:
rank[u] = last_rank
else:
rank[u] = last_rank + 1
last_rank = rank[u]
last_st = st
print user_info[u]['screen_name'] + "\t" + str(rank[u])

update_status("Generating the .dot file...")

# Now generate the .dot file
dotfile.write("  /* A TwitterGraph automatically generated by Evan Sultanik's Python script! */\n")
dotfile.write("  /* http://www.sultanik.com/                                                 */\n")
for user in friends:
dotfile.write("  n" + str(user) + " [label=< ”)
dotfile.write(””)
dotfile.write(”“ + user_info[user][’name’])
if not (user_info[user][’name’] == user_info[user][’screen_name’]):
dotfile.write(”(” + user_info[user][’screen_name’] + “)”)
dotfile.write(“Rank: “ + str(rank[user]) + “ >”);
dotfile.write(” color=\”green\” shape=\”doubleoctagon\””)
dotfile.write(”];\n”)
dotfile.write(”\n”)
for user in friends:
for f in friends[user]:
if str(f) in friends:
dotfile.write(”  n“ + str(user) + “ -> “ + “ n” + str(f) + “;\n”)
dotfile.write(”}\n”)

dotfile.close()

print “”
clear_status()


## LaTeX

### In which Evan and Joe teach you how to make beautiful documents.

Earlier today, Joe Kopena and I once again presented our tag-team LATEX talk. Not familiar with LATEX? Why not read the Wikipedia article! It’s essentially a professional grade system for beautifully typesetting documents/books. There are various books and Internet tutorials that do a fairly good job of introducing the basics, so, in our talk, Joe and I cover some more advanced topics and also general typesetting snags that novices often encounter. We always get requests for our slides after each of our talks, so I figured I’d post them online (which is the purpose of this blog entry).

Note that the entire presentation was created in LATEX using Beamer. You may also want to read my notes on BIBTEX, which will eventually become a part of our talk. You can read some of Joe’s notes on LATEX on his personal wiki, here. Feel free to browse and/or post any of your general typesetting questions to this public mailing list.