More on Twitter!

Al3x wrote up a nice blog post talking about the future of twitter.

How work's going?

Since I started working at Twitter last month, I put up a standard work disclaimer along the side. It always applies.

Jack posted on the company blog: I have this graph up on my screen all the time. It should be flat. This week has been rough.

So we have open job postings for something called a Systems Engineer, which is what I do at Twitter. Systems Engineering means building systems where graphs like that stay flat and where downtime means it was either planned or making sure that particular problem won't happen again (if it can be avoided: typical engineering trade-offs apply).

Our problems are really interesting, I think. Lots of users, lots of connections, lots of messages flowing through the system, lots of endpoints, and lots of details to keep straight. All of this needs to be turned into a cohesive system that's simple to reason about and to run in order for me to consider my job a success. It's a tall order but it's what I signed up to do. I've been watching Twitter for a long time (I'm user #150) so I walked into things with my eyes wide open.

If you've been reading this blog for a while, you know that I'm more interested in engineering than hacking together a site. Thinking and then doing. Measuring and then reasoning. Making guesses and then testing them. There's a natural tension between cowboying around and Analysis Paralysis and you have to learn to walk that tightrope if you want to succeed and I think at Twitter, we work pretty hard to Do the Right Thing.

I'm writing this quick post because we're looking for great people who are interested in engineering big systems and in helping to make Twitter the utility-class company we see ourselves as needing to be. If you think you either have the skills or can learn them, please send us your resume to jobs@twitter.com.

Fun DTrace script

#!/usr/sbin/dtrace -s

syscall:::entry
/pid == $1/
{
 @sys[probefunc, ustack()] = count();
}

END {
    trunc(@sys, 2);
}

Tells you the 2 most often called system call/stack trace pair. Running it against firefox 3 beta while using Google Reader shows:

$ sudo ./syscalldist.d 240
dtrace: script './syscalldist.d' matched 428 probes
^C
CPU     ID                    FUNCTION:NAME
  1      2                             :END 

  munmap                                            
              libSystem.B.dylib`munmap$UNIX2003+0xa
              libSystem.B.dylib`free+0x6a
              CoreGraphics`CGEventCreateFromDataAndSource+0xbce
              CoreGraphics`CGSDecodeEventRecord+0x6a
              CoreGraphics`CGSDispatchDatagramsFromStream+0x28f
              CoreGraphics`snarfEvents+0x12a
              CoreGraphics`CGSGetNextEventRecordInternal+0x9f
              CoreGraphics`CGEventCreateNextEvent+0x2c
              HIToolbox`PullEventsFromWindowServerOnConnection(unsigned int, unsigned char)+0x58
              CoreFoundation`__CFMachPortPerform+0x75
              CoreFoundation`CFRunLoopRunSpecific+0xf51
              CoreFoundation`CFRunLoopRunInMode+0x58
              HIToolbox`RunCurrentEventLoopInMode+0x11b
              HIToolbox`ReceiveNextEventCommon+0x176
              HIToolbox`BlockUntilNextEventMatchingListInMode+0x6a
              AppKit`_DPSNextEvent+0x291
              AppKit`-[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:]+0x80
              AppKit`-[NSApplication run]+0x31b
              XUL`JSD_GetValueForObject+0xad6ce
              XUL`XRE_GetFileFromPath+0x61c563
              961
  mmap                                              
              libSystem.B.dylib`mmap+0xa
              libSystem.B.dylib`large_and_huge_malloc+0xcb
              libSystem.B.dylib`szone_malloc+0x1cf
              libSystem.B.dylib`malloc_zone_malloc+0x51
              libSystem.B.dylib`malloc+0x37
              CoreGraphics`CGEventCreateFromDataAndSource+0x15e
              CoreGraphics`CGSDecodeEventRecord+0x6a
              CoreGraphics`CGSDispatchDatagramsFromStream+0x28f
              CoreGraphics`snarfEvents+0x12a
              CoreGraphics`CGSGetNextEventRecordInternal+0x9f
              CoreGraphics`CGEventCreateNextEvent+0x2c
              HIToolbox`PullEventsFromWindowServerOnConnection(unsigned int, unsigned char)+0x58
              CoreFoundation`__CFMachPortPerform+0x75
              CoreFoundation`CFRunLoopRunSpecific+0xf51
              CoreFoundation`CFRunLoopRunInMode+0x58
              HIToolbox`RunCurrentEventLoopInMode+0x11b
              HIToolbox`ReceiveNextEventCommon+0x176
              HIToolbox`BlockUntilNextEventMatchingListInMode+0x6a
              AppKit`_DPSNextEvent+0x291
              AppKit`-[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:]+0x80
              997

Thrilling, I know!

DTrace for Java 6 on Leopard

When Java 6 for Leopard was released a few weeks ago, one thing that nobody seemed to notice was that Java now had DTrace probes on par with Java on Solaris.

What you expect is there:

• DTrace Topics: Java
• DTrace Probes in HotSpot VM

With one exception: jstack doesn't appear to work. ustack works fine.

$ sudo dtrace -x jstackstrsize=2048 -n 'syscall::read:entry /execname == "java"/ { jstack(); }' 

dtrace: description 'syscall::read:entry ' matched 1 probe
CPU     ID                    FUNCTION:NAME
  3  17600                       read:entry 

  2  17600                       read:entry 

  3  17600                       read:entry 

  3  17600                       read:entry 

  2  17600                       read:entry 

  2  17600                       read:entry 

  2  17600                       read:entry 

  2  17600                       read:entry

There should be java stack traces under each read:entry line. (This is true even with -XX:+ExtendedDTraceProbes enabled)

I used robey's scarling for my guinea pig and had a lot of fun poking around at it with dtrace.

John McCarthy has a good sense of humor

From an informal talk he gave at Stanford recently that was written up in Hacker News:

Q. Can computers know?

A. This is largely a question of definition. If a camera looked at a table, we could
say it "knows" that there are four containers of liquid on the table (which was true).

Q. Is there any definition of "know" in which computers cannot succeed?

A. Well, I suppose the biblical sense.

Q. Ha, well, what makes you think that?

A. They don't satisfy the necessary axioms (laughter)