Subscribe to Blog
Jun 19
2013

Dokku: The smallest PaaS implementation you've ever seen

Dokku is a mini-Heroku powered by Docker written in less than 100 lines of Bash. Once it’s set up on a host, you can push Heroku-compatible applications to it via Git. They’ll build using Heroku buildpacks and then run in isolated containers. The end result is your own, single-host version of Heroku.

Dokku is under 100 lines because it’s built out of several components that do most of the heavy lifting: Docker, Buildstep, and gitreceive.

  • Docker is a container runtime for Linux. This is a high-level container primitive that gives you a similar technology to what powers Heroku Dynos. It provides the heart of Dokku.
  • Buildstep uses Heroku’s open source buildpacks and is responsible for building the base images that applications are built on. You can think of it as producing the “stack” for Dokku, to borrow a concept from Heroku.
  • Gitreceive is a project that provides you with a git user that you can push repositories to. It also triggers a script to handle that push. This provides the push mechanism that you might be familiar with from Heroku.

There are a few other projects being developed to support Dokku and expand its functionality without inflating its line count. Each project is independently useful, but I’ll share more about these as they’re integrated into Dokku.

For now, here’s a screencast that shows how to set up Dokku, along with a quick walk-through of the code.

Comments
Jan 05
2013

Executable Tweets and Programs in Short URLs

A few weeks ago I was completely consumed for the better part of a day that I would have otherwise spent on more practical work.

Yeah, what? Weird, right? It started from a Twitter conversation earlier that day with my friend Joel:

This wishful brainstorming inspired me to start building exactly that. But first, a digression.

The idea reminded me of an idea I got from Adam Smith back when I was working on Scriptlets. If you can execute code from a URL, you could “store” a program in a shortened URL. I decided to combine this with the curl-pipe-bash technique that’s been starting to get popular to bootstrap installs. If you’re unfamiliar, take this Gist of a Bash script:

Given the “view raw” URL for that Gist, you can curl it and pipe it into Bash to execute it right there in your shell. It would look like this:

$ curl -s https://gist.github.com/raw/4464431/gistfile1.txt | bash
Hello world

Instead of having Gist store the program, how could we make it so the source would just live within the URL? Well in the case of curl-pipe-bash, we just need that source to be returned in the body of a URL. So I built a simple app to run on Heroku that takes the query string and outputs it in the body, a sort of echo service.

Letting you do this:

$ curl "http://queryecho.herokuapp.com?Hello+world"
Hello world

Which you could conceal and shorten with a URL shortener, like Bitly. I prefer the j.mp domain Bitly has. And since they’re just redirecting you to the long URL, you’d use the -L option in curl to make it follow redirects:

$ curl -L http://j.mp/RyUN03
Hello world

When you make a short URL from the bitly website, they conveniently make sure the query string is properly URL encoded. So if I just typed queryecho.herokuapp.com/?echo "Hello world" into bitly, it would give me a short URL with a properly URL encoded version of that URL that would return echo "Hello world". This URL we could then curl-pipe into Bash:

$ curl -Ls http://j.mp/VGgI3o | bash
Hello world

See what’s going on there? We wrote a simple Hello world program in Bash that effectively lives in that short URL. And we can run it with the curl-pipe-bash technique.

Later in our conversation, Joel suggests an example “app tweet” that if executed in Bash given a URL argument, it would tell you where it redirects. So if you gave it a short URL, it would tell you the long URL.

Just so you know what it would look like, if you put that program in a shell script and ran it against a short URL that redirected to www.google.com, this is what you would see:

$ ./unshortener.sh http://j.mp/www-google-com
http://j.mp/www-google-com
http://www.google.com/

It prints the URL you gave it and then resolves the URL and prints the long URL. Pretty simple.

So I decided to put this program in a short URL. Here we have j.mp/TaHyRh which will resolve to:

http://queryecho.herokuapp.com/?echo%20%22$url%22;%20curl%20-ILs%20%22$url%22%20|%20grep%20Location%20|%20grep%20-o%20'http.*'

Luckily I didn’t have to do all that URL encoding. I just pasted his code in after queryecho.herokuapp.com/? and bitly took care of it. What’s funny is that this example program is made to run on short URLs, so when I told him about it, my example ran on the short URL that contained the program itself:

$ curl -Ls http://j.mp/TaHyRh | url=http://j.mp/TaHyRh bash
http://j.mp/TaHyRh
http://queryecho.herokuapp.com/?echo "$url"; curl -ILs "$url" | grep Location | grep -o 'http.*'

You may have noticed my version of the program uses $url instead of $1 because we have to use environment variables to provide input to curl-pipe-bash scripts. For reference, to run my URL script against the google.com short URL we made before, it would look like this:

$ curl -Ls http://j.mp/TaHyRh | url=http://j.mp/www-google-com bash
http://j.mp/www-google-com
http://www.google.com/

Okay, so we can now put Bash scripts in short URLs. What happened to installing apps in Tweets? Building an apptweet program like Joel imagined would actually be pretty straightforward. But I wanted to build it in and install it with these weird programs-in-short-URLs.

The first obstacle was figuring out how to get it to modify your current environment. Normally curl-pipe-bash URLs install a downloaded program into your PATH. But I didn’t want to install a bunch of files on your computer. Instead I just wanted to install a temporary Bash function that would disappear when you leave your shell session. In order to do this, I had to do a variant of the curl-pipe-bash technique using eval:

$ eval $(curl -Ls http://j.mp/setup-fetchtweet)
$ fetchtweet 279072855206031360
@jf you asked for it... Jeff Lindsay (@progrium) December 13, 2012

As you can see by inspecting that URL, it just defines a Bash function that runs a Python script from a Gist. I cheated and used Gist for some reason. That Python script uses the Twitter embed endpoint (same one used for the embedded Tweets in this post) to get the contents of a Tweet without authentication.

The next thing I built installed and used fetchtweet to get a Tweet, parse it, put it in a Bash function named by the string after an #exectweet hashtag (which happens to also start a comment in Bash). So here we have a Tweet with a program in it:

To install it, we’d run this:

$ id=279087620145958912 eval $(curl -Ls http://j.mp/install-tweet)
Installed helloworld from Tweet 279087620145958912
$ helloworld
Hello world

We just installed a program from a Tweet and ran it! Then I wrapped this up into a command you could install. To install the installer. This time it would let you give it the URL to a Tweet:

$ eval $(curl -Ls http://j.mp/install-exectweet) 
Installed exectweet
$ exectweet https://twitter.com/progrium/status/279087620145958912
Installed helloworld from Tweet 279087620145958912
$ helloworld
Hello world

Where would I go from there? An app that calls itself into a loop, of course!

$ exectweet https://twitter.com/progrium/status/279123541054595074 && recursive-app
Installed recursive-app from Tweet 279123541054595074
Installed recursive-app from Tweet 279123541054595074
Installed recursive-app from Tweet 279123541054595074
Installed recursive-app from Tweet 279123541054595074
...

Obviously, this whole project was just a ridiculous, mind-bending exploration. I shared most of these examples on Twitter as I was making them. Here was my favorite response.

You may have noticed, it just happened to be 12/12/2012 that day.

Comments
Jan 01
2013

Where did Localtunnel come from?

Five years ago, async network programming scared me. I was a web developer. Working with the high level tools and frameworks of HTTP seemed much easier than any sort of serious low level networking. Especially since network programming would often also mean some kind of concurrent programming with threads or callbacks. I had mostly avoided multithreading and had no idea what an event loop was. I came from PHP.

Around 2007, I was starting to think about webhooks. One motivator was how you could use webhooks to let web developers, like me, build systems that used other protocols without them having to work with that protocol. For example, one of my first projects with webhooks was called Mailhooks. I wanted to accept email in my application, but I didn’t want to deal with email servers. I wanted to get an HTTP POST when an email came in with all the email fields nicely provided as POST parameters.

This is how I started working with Twisted. Twisted became my main tool to build webhook adapters for existing protocols. I even tried to generalize that idea in a project called Protocol Droid. Slowly I started to grok, and not fear, this kind of programming.

It’s funny how my desire to work with abstractions that didn’t exist yet to avoid a certain kind of programming was directly responsible for me eventually becoming an expert in that kind of programming.

Then in late 2009, I had another idea while thinking about webhooks. It would be great if I could expose a local web server to the Internet with a friendly URL. It should just be a simple command. There would have to be a server, but there could just be a public server that you didn’t even have to think about.

I committed the first prototype of Localtunnel to Github in January 2010. It was written entirely in Twisted. It also didn’t actually work. I really recommend taking a look because it was terrible. One of the challenges was multiplexing the HTTP requests into a single tunnel connection. My approach was so naive it just didn’t work. As soon as you made more than one request at a time, it broke.

A few months later, I decided to take a different approach. Instead of doing my own protocol, client, and server, I’d just make a wrapper around what I knew already worked: SSH tunneling. This was pretty quick to make happen, and that version is basically what’s been in production to this day.

This shortcut came with a lot of weird quirks. For example, the easiest way I found to implement an SSH tunnel client was a Ruby library, so I implemented the client in Ruby. The server, though, was in Python because I still only really knew Twisted for evented programming.

Actually, using SSH was the source of most of the quirks and annoyances. I was pretty bothered that it slowed down the initial user experience by requiring a public key to be uploaded. But most of the pain was operational. The server, sshd, would create a process for every tunnel. Localtunnel also needed its own user and to pretty much own the SSH configuration for that machine. Then, on occasion, something weird would happen where a tunnel would die and the process would go crazy eating up CPU. It would have to be manually killed or it would eventually bring the server to a halt. And, eventually, the authorized_keys file would become enormous from all the keys uploaded.

On top of all this, SSH is pretty opaque. It’s been around for so long and used so much that it certainly just works … you just don’t really know how. I still don’t know how SSH does tunneling or what the protocol looks like, even after trying to read the RFC for it.

By mid-2011, I was working at Twilio building distributed, real-time messaging systems at scale. I certainly came a long way from fearing async network programming. Localtunnel was still running the implementation based on SSH. By then it had quite a large user base and collected a number of bugs and feature requests. I also had my own operations and user experience wish list. With such a huge list of new requirements, so many problems with the current implementation, and a drastically different experience level and mindset, I decided to redesign Localtunnel from the ground up.

Since I was pretty consumed by Twilio, I didn’t have a lot of time to work on Localtunnel. I thought the biggest bang for buck in the long term would be to slowly work on the new version. They say software is never done, but I personally believe software can be finished. It just requires an aggressive drive for simplicity, and the only way you can make significant advances in simplicity is through redesign.

In the meantime, users continued to experience issues with the current implementation. These problems only got worse as it became more popular. For example, the biggest issue was that the namespace for tunnel names was too small. Users would get requests from old tunnels, and in rare cases tunnel names would get pulled out from under you while using them. This created confusion and a lot of emails and issue tickets, but it still worked with the occasional restart.

I’ve used this constant stream of complaints, which has been going on for almost two years, to make sure I keep making progress on the new version. In fact, I’m pretty sure I needed it because of my lifestyle of abundant projects.

Last week I finally released a beta of the new version. What’s interesting is that it’s a completely different architecture from what I started out with for the redesign. After the original unreleased prototype, there’s been 3 major approaches to implementation. In the coming weeks I’m going to share a more technical history of the architecture of Localtunnel, leading up to a deep exploration of what I hope will be its final form.

Comments
Dec 25
2012

Localtunnel v2 available in beta

A few years back, I released Localtunnel to make it super easy to expose a local web server to the Internet for demos and debugging. Since then, it’s gotten a ton of use. A few people even copied it and tried to make a paid service around the idea. Luckily, Localtunnel will always be free and open source.

With the release of Localtunnel v2, it will not only remain competitive with similar services, but continue to be the innovator of the group. I’ll post more on this later.

For now, let’s talk logistics. The current, soon-to-be-legacy Localtunnel stack includes the client that you install with Rubygems, and a server that runs on a host at Rackspace. These will continue to be available into 2013, but will be marked as deprecated. This means you should be making the switch to v2.

Besides the fact v1 will eventually be shutdown, there are a number of reasons to switch to v2. Here are some of the major ones:

  • It’s actively maintained. Bug reports, pull requests, and service interruptions are dealt with promptly.
  • No more mysterious requests from old tunnels. The subdomain namespace is much larger.
  • Custom subdomains. The new client lets you pick a tunnel name on a first come, first served basis.
  • Supports long-polling, HTTP streaming, and WebSocket upgrades. Soon general TCP tunneling.
  • No SSH key to start using it. A minor annoyance setting up v1, but it doesn’t exist in v2.

One implementation detail that affects users is that the client is now written in Python. This means you won’t use Rubygems to install it. Instead, you can use easy_install or pip.

$ easy_install localtunnel

On some systems, you may need to run this with sudo. If you don’t have easy_install, first make sure you have Python installed:

$ python --version

Localtunnel requires Python 2.6 or later, which comes standard on most systems. If you don’t have Python, you can install it for your platform. If easy_install isn’t available after you install Python, you can install it with this bootstrap script:

$ curl http://peak.telecommunity.com/dist/ez_setup.py | python

Once you’ve installed Localtunnel with easy_install, it will be available as localtunnel-beta. This lets you keep the old client to use in case anything goes wrong with v2 during the beta. Eventually, it will be installed as localtunnel, but only after v1 is shutdown.

Using localtunnel-beta is pretty much the same as before:

$ localtunnel-beta 8000
  Thanks for trying localtunnel v2 beta!

  Port 8000 is now accessible from http://fb0322605126.v2.localtunnel.com ...

Like I mentioned earlier, you can use a custom tunnel name if it’s not being used:

$ localtunnel-beta -n foobar 8000
  Thanks for trying localtunnel v2 beta!

  Port 8000 is now accessible from http://foobar.v2.localtunnel.com ...

Keep in mind v2 is in active development. There might be some downtime while I work out operational bugs, but you can always use the old version if you run into problems.

If you do run into any problems, you can ping me on Twitter. If you get traceback you can create an issue on Github. If you have more in-depth questions or want to get involved in development, check out the Localtunnel Google Group.

Comments
Dec 17
2012

HTTP Signatures with Content-HMAC

Today I wanted to propose another header. It would be used for signing HTTP content with HMAC, and is appropriately called Content-HMAC. In a previous post about the Callback header, I mentioned using an X-Signature header in callback requests to sign the payload of the callback. It looked like this:

X-Signature: sha1=<hexdigest of sha1 hmac>

The HMAC would be built with just the content of the request (i.e., no headers, no query params) and a secret key. This was borrowed directly from the PubSubHubbub spec, but the general idea of using HMAC to sign callback requests has become pretty standard in the world of webhooks. Here are details on how Google and Twilio use them.

Each of these providers is using their own header for basically the same use case. It would seem like there is an opportunity to standardize on a common header format for it. There’s been a number of proposals for a general Signature header to sign an entire request. There was a fairly comprehensive one proposed called Content-Signature. With signing, the difficulty is often getting the input string correct. Most signing mechanisms need to normalize their input. If you’ve ever had to deal with OAuth or AWS signatures, you’ll know what I’m talking about. With request signing, the headers pose a particularly tricky situation with signing since they often change as the request goes through proxies.

The idea of Content-HMAC is to focus on a simpler goal of signing just the content payload, since it’s normally treated as-is, and is not altered when going through proxies. The X-Signature proposal I had was a decent one, as is almost any cowpath-based proposal, but I realized it would probably be a good idea to limit the implied scope to what it’s really doing: providing an HMAC for request (or response) content.

It turns out there’s a similar header that’s not used that often anymore called Content-MD5. It was a simple mechanism to provide an MD5 digest of the content. My current proposal is to take this existing pattern and apply it to HMAC, giving us the Content-HMAC header:

Content-HMAC: <hash mechanism> <base64 encoded binary HMAC>

Here’s an example:

Content-HMAC: sha1 f1wOnLLwcTexwCSRCNXEAKPDm+U=

This proposal borrows its naming convention from Content-MD5, but the format is more similar to Authorization. The Authorization header allows multiple authorization schemes to be used. You define the scheme followed by a space and then the actual authorization data. Since HMAC allows different hashing techniques to be used, we use that pattern here to let you specify the hashing technique. We also take the existing pattern of base64 encoding used in several HTTP headers to make it conform even more to existing standards.

Content-HMAC was created for callback requests, but it’s a useful way to sign any HTTP request or response payload. For requests, it’s worth mentioning it only applies when there is a content payload, so for example it’s meaningless with GET requests.

It’s also very worth mentioning that the need for content signing is unnecessary when using HTTPS. It currently looks like the future will eventually be 100% SSL encrypted HTTP, but until then, there will always be situations where HTTPS is not available. Content-HMAC is perhaps a stop-gap until we reach that ideal. Until then, I think Content-HMAC is a good, standard way to add authorization to callback requests.

Let me know if you have any questions or feedback on this proposal. Further discussion is likely to happen on the Webhooks Google Group.

Comments