Web Served 8: Node.js, Redis, and real-time writing with Etherpad

Web Served is nearly over! This is the last regular part in the series, and so I want to kick it up a notch—several notches, actually!—and get away from installing tired old PHP applications. Instead, we’re going to go a little nutty and install Etherpad (formerly known, and still sometimes referred to as, “Etherpad Lite”). Some parts of this piece have previously appeared on my blog, but I’m aiming to expand a bit on that write-up and give you all something fun to play with.

Etherpad is a tool born out of the ashes of Google Wave, the abandoned and decommissioned real-time “e-mail replacement” that Google launched and then quickly killed. EPL is a real-time collaborative document editing application: it presents the user(s) with a blank canvas and lets them all make changes to it, and everyone’s changes are visible to everyone else in nearly real-time.

Stop. Collaborate and listen.

Credit: Lee Hutchinson

It’s a lot like Google Docs, except that the changes are generally shown a whole lot faster, and everyone’s changes are tracked and can be replayed like a VCR. Plus, it’s yours—that’s one of the main points of this entire series, after all! Rather than living in Google’s ephemeral application cloud, Etherpad lives on your server and you always have access to it (at least, assuming you have access to your server).

Why would I even want this?

Here at Ars, we use Etherpad with some regularity for collaborative editing. It’s really handy to be able to call up a coworker, create a new “pad,” paste in a document, and start talking through it and making changes as you talk. You can rewind and fast-forward back and forth across the document in various changed states to watch how the piece evolves through editing.

It’s also handy for quickly gathering a bunch of ideas from a bunch of different people. In this way, it fills some of the same functions as Google Drive applications do, except you’re hosting it yourself. As someone who regularly uses Google Drive for work (we keep our story dashboard in a Google Drive document and use it to coordinate who’s writing what throughout the day), it can occasionally be a frustrating tool to deal with—like when it loses the changes you just made and tells you that you must refresh the page to continue editing. Argh.

Another reason to get this set up is because it’s a neat application that relies on a number of different technologies to function, including Node.js (a server-side Javascript execution engine) and Redis (a fast in-memory NoSQL database), and setting up new, cool technologies is fun!

Going nuts with prerequisites

So, let’s get started on them prereqs, because there are a lot! Etherpad is a Node.js application—it’s written in Javascript and relies on Node.js to run it. Node.js is gaining a lot of traction in Web application developer circles because it’s fast, and it lets developers use their existing bits of Javascript knowledge to create “real” (or at least executed on the server instead of the client) Web-based applications. There’s a version available in Ubuntu’s repositories, but we’re going to add a custom PPA to ensure we’re running the latest version.

We’re also going to be installing Redis, which Etherpad uses as its database. Redis is a NoSQL database—specifically, it’s a key-value store (like memcached, which we installed back in part 3). Rather than operating like a traditional database with data arranged in a table and addressed through some form of structured query, Redis just stores a big list of keys and their associated values (like, “color” for a key and “red” for a value, for example). The big draw for Redis over other NoSQL databases is that Redis is entirely RAM-based. There are drawbacks to this (what if you lose power?), but the big advantage is that it’s blazing fast. Because Etherpad is pretty database-access-heavy, giving it a fast place to stash its data is a good idea. Plus, cool technology is cool!

As with Node.js, we need to add a PPA to ensure that we’re installing the latest version of Redis. Use the following commands to add the necessary PPAs to your server:

sudo add-apt-repository ppa:chris-lea/node.js
sudo add-apt-repository ppa:chris-lea/redis-server

Adding the PPAs looks like this.

Credit: Lee Hutchinson

Once those two PPAs are in, update your package listing and install both Node.js and Redis:

sudo aptitude update
sudo aptitude install nodejs redis-server

Etherpad download

Etherpad itself is available from Github, and we’re going to clone it down onto our local machine, just as we did with Vanilla in part 6.

Head to your Nginx directory at /usr/share/nginx, and use git pull to create a local copy of the Etherpad application from its git repository. To save time, we’re going to first elevate our shell to root privilege, since our regular user account should only have read access to the directory we’re in:

sudo /bin/bash
cd /usr/share/nginx/
git clone https://github.com/ether/etherpad-lite.git

That will kick off a lot of activity, and when it’s done, you’ll have an etherpad-lite subdirectory with the git repo’s contents.

Note that we’re not actually putting our Etherpad installation inside our Nginx Web root. This is because Nginx never actually needs to directly serve any files from inside the directory; the pages will all be generated by Node.js and proxied through Nginx…with one teeny, tiny exception, which we’ll get to at the very end.

Make Etherpad run as a service

We’ve got our local copy of Etherpad, but it’s not going to do us much good just sitting there—we need to set it up to run, by itself, as a service.

This is one of the advantages that prebuilt packages bring—when we installed Redis just now, for example, we didn’t have to think about what to do to make Redis run in the background by itself, because the Redis package took care of that for us. It created a local account for the Redis processes to run as, it set up a place for logs to be stored, and it took care of the init tasks for us so that Redis gets started when the computer boots up. We want the same kind of stuff for Etherpad, but we’re going to have to create it ourselves.

Create a user

At the root-elevated bash prompt, create an Etherpad user named “etherpadlite” by running the following command:

useradd -M -s /bin/false etherpadlite

This will create the user. The -M switch ensures that a home directory is not created (service accounts don’t need home directories), and -s /bin/false tells the system to use /bin/false as the account’s default shell—which is fine, because no one will ever log on as this particular user and no shell is needed.

Now that we have a user, there is one important step we need to make sure to do: we must change the ownership of the cloned Etherpad directory to that user. Otherwise, we won’t be able to actually get Etherpad up and running! To do that, use the chown command, like this:

chown -R etherpadlite:etherpadlite /usr/share/nginx/etherpad-lite/

Create logs for Etherpad

We also need to create a log file location for Etherpad to use and set up a log rotation policy for those logs so the operating system automatically tends to them and keeps them from growing too large.

Still as root, run these commands to create a directory for your Etherpad logs and to make sure that location is owned by your Etherpad account so that the application has the permissions to actually write things there:

mkdir /var/log/etherpad-lite
chown etherpadlite:adm /var/log/etherpad-lite

The chown command here sets ownership of the directory to the “etherpadlite” account, and also to the “adm” group (which needs to be there so Ubuntu can also change files in the directory).

Next, we need to tell the logrotate daemon about this location. This is a background application that keeps track of all the log files on the system; it can be configured to start a new log file for an application every day, to compress and archive older log files, to delete the oldest log files, and to do a whole bunch of other things.

To configure the logrotate daemon to also watch over our new Etherpad directory and to do its magic on files that show up there, we need to create an etherpadlite file inside of the /etc/logrotate.d/ directory, and add the following to that file:

/var/log/etherpad-lite/access.log /var/log/etherpad-lite/error.log {
  daily
  rotate 7
  missingok
  compress
  delaycompress
  missingok
}

This tells logrotate that there will be two files it should watch—one named access.log and the other named error.log—and then tells the daemon what to do with them. These options can be adjusted however you’d like; for a good explanation of what does what, try typing man logrotate at the bash prompt.

Setting up a service

We have a user and a log file location, and now we need to create the script to get Etherpad automatically started. Throughout this series, I’ve been using /etc/init.d/servicename start and stop when starting and stopping processes, but that’s really the old way to do things. Ubuntu supports (and prefers you to use!) the new, fancy Upstart method. So, rather than creating an old-school init script for Etherpad, we’ll make an Upstart job.

This is actually pretty easy: create a file named /etc/init/etherpad-lite.conf and paste in the following:

description "etherpad-lite"
start on started networking
stop on runlevel [!2345]
env EPHOME=/usr/share/nginx/etherpad-lite
env EPLOGS=/var/log/etherpad-lite
env EPUSER=etherpadlite
respawn
pre-start script
chdir $EPHOME
  mkdir $EPLOGS                       ||true
 chown $EPUSER:admin $EPLOGS          ||true
 chmod 0755 $EPLOGS                   ||true
 chown -R $EPUSER:admin $EPHOME/var   ||true
$EPHOME/bin/installDeps.sh >> $EPLOGS/error.log || { stop; exit 1; }
end script
script
  cd $EPHOME/
  exec su -s /bin/sh -c 'exec "$0" "$@"' $EPUSER -- node node_modules/ep_etherpad-lite/node/server.js \
  >> $EPLOGS/access.log \
  2>> $EPLOGS/error.log
end script

Then, test Etherpad by running the job:

start etherpad-lite

This might take a minute or so to complete, since it’s the first time Etherpad has been run. While this is going on, you can check the two Etherpad log files you’d previously created; if they’re filling up with stuff, that’s a good sign that the application is working.

If the job is able to start successfully, you’ll see an acknowledgement at the command line:

Looking good so far—our Upstart job starts and stops correctly.

Credit: Lee Hutchinson

If so, congratulations—Node.js is serving Etherpad!

But we’re only about halfway done. We still need to set up Nginx so that it proxies traffic to Node.js, so that you don’t have to have your bare instance of Node.js facing the Internet. We also need to actually do a bit of configuration on Etherpad, because right now it doesn’t know anything about how we want it to use Redis. In fact, if you look at the Etherpad error log right now, it’s probably throwing the same Error: EACCES, open 'var/dirty.db' error over and over again—don’t worry, we’ll fix it!

Telling Nginx what to do

Node.js is many things; it has the capability to function as a Web server by itself, with a bit of configuration. However, we haven’t put this much time and effort into building a robust Nginx set-up to not use it. Node.js isn’t running at the moment, but when it does, it listens for incoming connections on TCP port 9001. We’re going to configure Nginx to watch for HTTP requests to a path that doesn’t really exist and to reverse-proxy those requests to Node.js on port 9001.

Reverse proxying—that is, retrieving something from one or more servers on behalf of a requesting client—is something Nginx is very, very good at. In fact, that’s one of the predominant uses for Nginx—to sit in front of a pool of bigger, fatter application servers and act as a traffic cop between them and their users, quickly serving up static assets like images and text while the application servers generate the dynamic portions of a Web site.

We’re not going to do anything so complex here, but we are going to take advantage of Nginx’s presence on port 80. We can’t run both Nginx and Node.js on port 80 at the same time, and so by proxying our Etherpad traffic through Nginx, we don’t have to worry about telling users, “So, to use my cool Etherpad set-up, you’ll need to add ‘:9001′ at the end of the URL…” Nobody likes that.

Fortunately, we don’t need to do much to Nginx to tell it what to do. We simply need to make up a location for Etherpad, and tell Nginx to take traffic bound for that location and pass it on to the waiting Node.js process, listening on port 9001.

Add this to your active Nginx virtual host configuration file, which if you’ve been playing along at home is /etc/nginx/sites-available/www:

location /etherpad {
     rewrite /etherpad/(.*) /$1 break;
     rewrite ^/etherpad$ /etherpad/ permanent;
     proxy_pass http://localhost:9001/;
     proxy_redirect / /etherpad/;
     proxy_set_header Host $host;
     proxy_buffering off;
}

When I originally wrote this procedure up for my blog, I did it such that it used its own hostname instead of a subdirectory, and also such that all traffic was forced to run over HTTPS. If you’d prefer to set your Etherpad instance up this way (or as a blend—HTTPS only from a subdirectory, for example), you can alter the configuration to look more like the one I’m using on my blog.

However, since not everyone is going to want to use HTTPS all the time, I’ve borrowed the Nginx configuration above from the Etherpad Github pages. It uses rewrites to fiddle with the actual text of the URL that Nginx passes to Node.js. This is important because our Etherpad traffic is all going to be sent to and from a URL with a directory name at the end of it: http://yourhostname.com/etherpad. The Node.js process listening behind Nginx doesn’t know anything about any /etherpad subdirectory—it’s not a real subdirectory anyway, and certainly not one underneath the very real Etherpad directory we’ve created.

Using the rewrite directive lets Nginx take care of the confusion for us. Nginx will look for HTTP requests that match the regular expressions we entered above, and then alter the URLs on the fly before it proxies them to Node.js, and it will also translate back the replies it receives from Node.js. Neat!

Once that’s in, reload Nginx to make the change active:

/etc/init.d/nginx reload

Adding in Redis support

Time to turn our attention to the database-side of things. Rather than gunking up the Etherpad code with all the different connectors necessary to work with lots of different database packages, Etherpad actually uses a database abstraction layer called UeberDB. This way, it doesn’t particularly care what database you bring to the table: as long as UeberDB knows how to talk to it, we can use it.

It just so happens that there is an UeberDB connector for Redis, and we’re going to install it inside the Etherpad directory that we cloned down way back near the beginning of this tutorial. Fortunately, installing it is super-easy; we can use the Node Package Manager to automatically pull it down and slot it into the correct place.

Back at the bash shell, change your directory so that you’re at usr/share/nginx/etherpad-lite/src/node_modules/ueberDB, which is the location of the UeberDB package underneath the Etherpad directory, and install the Redis package:

cd usr/share/nginx/etherpad-lite/src/node_modules/ueberDB
npm install redis

Installing the Redis connector via Node Package Manager.

Credit: Lee Hutchinson

Configuring Etherpad

Getting closer! Now that we’ve given Etherpad the ability to store its stuff in our Redis database, we’re going to actually instruct Etherpad to do so.

To do this, we need to edit Etherpad main configuration file, which is at /usr/share/nginx/etherpad-lite/settings.json. Looking inside the file, there are lots of settings that you probably want to change (like "title", to give your Etherpad instance a more descriptive name), but for now we’re going to concern ourselves purely with the database ones.

Specifically, we’re going to modify "dbType" and "dbSettings" from their default values to instead look like this:

"dbType" : "redis",
"dbSettings" : {
      "host" : "localhost",
      "port" : 6379,
  "database" : 0
},

This tells Etherpad that it should use its Redis database driver and that the Redis server is localhost and is listening on TCP port 6379. I spent some amount of time trying to get Etherpad to communicate via Unix sockets to Redis instead of via TCP—Unix sockets are faster and more efficient—but I couldn’t make it work. I don’t think the UeberDB database abstraction layer used by Etherpad included support for it. However, if someone does figure it out, please do drop me a line!

The last setting, "database" : 0, tells Etherpad to use Redis’ db0. Redis can actually run multiple simultaneous independent key-value stores; the first one is db0 and is the one you’d most likely want to use first. If you end up setting up another Web application that uses Redis (like a second instance of Etherpad), you’d want to make sure that it’s using a database other than 0 so that the applications don’t eat each others’ data!

Is it working? Let’s take a peek!

We’ve got a couple of steps remaining, but it’s time to check our progress and make sure Etherpad is actually, you know, working.

Kick off that Upstart job you created to get Node.js activated and Etherpad up and running:

start etherpad-lite

Then, open up your Web browser and navigate to http://yourserver.com/etherpad. If you’ve done everything right, you should see the Etherpad front page:

IT WORKS

Credit: Lee Hutchinson

Woohoo!

But something’s missing…

If you’d like, you can stop now, because Etherpad is configured and everything’s working. However, there’s a little more we can do to add in some functionality that Etherpad is missing by default.

Right now, you can create a new “pad” to collaborate with simply by typing in a name. Try it out!

Creating a pad called “testpad” should yield a screen that looks like this. If this is what you’re seeing, then Etherpad is functioning correctly!

Credit: Lee Hutchinson

But there’s no easy way to see a quick list of all the existing pads on the server. This might be how you want it—for example, if your Etherpad instance is meant to be public, you might not want everyone who happens by to see that you’re working on a pad titled “MY SECRET FINANCIAL INFO AND PASSWORDS” or something like that. However, I’ve found it beneficial to see exactly what pads are available on the server to edit.

To do this requires a bit of extra work, but the results are worth it.

Showing a list of pads

This is perhaps a bit hacky, but it works well enough. We’re going to do a few different things to get a list of pads displayed on our Etherpad logon page. First, we need to figure out how to query our Redis database to actually produce the list. Then, we need to store that list in a text file. We need to be able to include that text file in our logon page. And, finally, we need to automate the Redis query so it’s running over and over again, to keep our list of pads current.

Making the list

The Etherpad Github repository has instructions on how to use the command line to ask various databases to produce a list of pads; using the syntax provided, we can quickly extrapolate how to do the same with the redis-cli command line tool. Create a couple of pads in Etherpad so we have some database content to work with (simply type a name for your pad into the “New Pad” page and click “OK,” then enter some content into the editing window). Then, from your root shell, run the following:

redis-cli keys 'pad:*' |grep -Eo '^pad:[^:]+' |sed -e 's/pad://'
|sort |uniq -c |sort -rn |awk '{if ($1!="2") {print $2 }}'

(If you’re cutting and pasting the above code, you may need to manually remove the linebreak in it.)

This set of commands is an excellent example of the power of the Unix command line and serves as a beautiful demonstration of one of the main focuses of Linux and all Unix-like operating systems: they are ultimately made up of a collection of very powerful tools that can manipulate each others’ textual output.

The commands are each separated by pipes; I suggest you try building the command line in sequence, pipe by pipe, to get a feel for what is happening. The pipes themselves tell bash to take the output of one command and use it for input into the next.

The first command, redis-cli keys 'pad:*', outputs a numbered list of all the Redis keys that contain the string “pad:”. Grep is then used to strip out some of the cruft with a regular expression, leaving behind only the string “pad:” and whatever follows it to the first colon. Then, sed replaces “pad:” with an empty string, deleting it and leaving us with a bare list of pads. Next we run the list through sort, which orders the results, and uniq is called with the -c flag, which counts each unique entry and displays the total. These results are then run through sort again, this time with -rn, which sorts them in descending order by number, using the counts at the front. Lastly, the entire mess is piped through awk, which looks at the two columns we’re left with, and if the first column is not “2” (which indicates that the pad exists but is empty), it displays the second column’s text.

The end result is a list of non-empty pads, though you can modify the final awk statement to awk '{if ($1>"1") {print $2 }}' if you want a list of all pads, empty or not.

So, we have our list of pads:

Building our way through that last command’s layers.

Credit: Lee Hutchinson

To get that list into a text file, you can simply append > output.txt onto the end of the command. Let’s go ahead and do that, and output the file to a location that Nginx can serve it from. Use this path and file name:

 /usr/share/nginx/html/padlist.txt

So the full command should now look like this:

redis-cli keys 'pad:*' |grep -Eo '^pad:[^:]+' |sed -e 's/pad://'
|sort |uniq -c |sort -rn |awk '{if ($1!="2") {print $2 }}'
> /usr/share/nginx/html/padlist.txt

(Again, remove the linebreaks if you’re copying and pasting.)

Displaying the list with SSI

Now that we have the command to produce our pad list, we need a way to actually display that list, and to do that, we’re going to take advantage of an Nginx feature called SSI, or server-side includes. This feature allows Nginx to display the contents of a file (like a text file containing a list of pads!) inside a Web page.

We’re going to sneakily modify Etherpad’s HTML so that it uses an SSI statement to grab our padlist.txt text file and display it. We also need to do a bit of CSS surgery so that the text is displayed in a manner that matches the existing formatting on the Etherpad logon page.

To make this happen, we need to edit the file /usr/share/nginx/etherpad-lite/src/templates/index.html, which is the file Node.js serves up via Nginx when you hit your Etherpad instance. We’re going to make two additions: first in the style section, and then in the body.

Open that file for editing, and add the following to the bottom of the style section:

.padlist {
  text-align: left;
}
.padlist ul {
  list-style-type: none;
  padding:0;
  margin-left:0;
  text-align: left;
  font-weight: bold;
}
.padlist li {
  padding-bottom: 5px;
}

Then, locate the wrapper div immediately following the style section, and insert the lines indicated below, just after the closing form tag (include the comments, so that if you want to change the section later, it’s obvious where your additions are):

<div id="wrapper">
  <div id="inner">
      <div id="button" onclick="go2Random()" class="translate">New Pad</div>
          <div id="label" class="translate">or create/open a Pad with the name:</div>
          <form action="#" onsubmit="go2Name();return false;">
          <input type="text" id="padname" autofocus x-webkit-speech>
          <button type="submit">OK</button>
          </form>
<!--Insert these lines-->
          <div class="padlist"><p>List of current pads:
              <ul><!--#include virtual="/padlist.txt" -->
              </p>
          </div>
<!--Done -->
      </div>
  </div>

The lines we’ve just inserted, specifically that include virtual line, are the magic bits. We’re creating a new div called “padlist,” and then we’re creating an unordered list made up of the contents of the padlist.txt file.

But, wait. How can we display padlist.txt as a ul, if its contents don’t have li tags around them? It’s easy—if you’ve got the right tools.

Making that pad list auto-refresh

We want our pad list to refresh itself every minute or so, and now we also need to make sure that it’s properly formatted with the right HTML tags. We’re going to do the generation and formatting with a script, and we’re going to use cron to auto-run the script every minute.

We’re going to park our script in /usr/local/bin, which is a good place to put scripts and binaries that you’ve created. Create a new file there named getpads.sh and stuff it full of the following:

#! /bin/bash
PADPATH="/usr/share/nginx/html/padlist.txt"
redis-cli keys 'pad:*' |grep -Eo '^pad:[^:]+' |sed -e 's/pad://'
|sort |uniq -c |sort -rn |awk '{if ($1!="2") {print $2 }}'
|sed -e 's/.*/<a href="\/p\/&">&<\/a>/' -e 's/^/<li>/' -e 's/$/<\/li>/'
> "$PADPATH"
echo "</ul>" >> "$PADPATH"
echo "<p>List updated at $(date "+%R %Z")</p>" >> "$PADPATH"

(Same deal as before—remove the linebreaks if you’re cutting and pasting this or it won’t work!)

You’ll recognize our redis-cli command up there for drawing forth the list of pads, but with a few additions on the end—specifically, a big long sed statement. Go ahead and run just the redis-cli command, stopping before the > "$PADPATH" portion.

Behold the power of text manipulation in the bash shell.

Credit: Lee Hutchinson

We’re using sed to not only insert a li tag at the beginning of each line, but also to create a hyperlink out of each line using that line’s name as the basis for the link. The hyperlinks will give single-click access to each pad. Finally, we’re closing the links and the list item tags, again with sed. This is the kind of shell magic that makes Unix folks cackle at Windows users and the poor stunted Windows command line (at least, prior to Powershell’s appearance).

The last two echo commands close the list and also add the date and time showing when the pad list was last updated. Old Unix hands will note that I’ve started the script off with a /bin/bash shebang; this is because the date command didn’t work properly when I tested with plain old sh. This isn’t really best practice, but since we’re using Ubuntu and we know /bin/bash is available, we can get away with it.

Save and exit the script, and then make it executable with the chmod command:

chmod +x /usr/local/bin/getpads.sh

Finally, to get that script executing automatically every minute, we need to edit the system crontab file, which is what the cron daemon uses to get its instructions. We want the script to fire every minute of every hour of every day, so open up /etc/crontab as root and add the following to it:

* *  * * *   root    /bin/bash /usr/local/bin/getpads.sh &> /dev/null

Give your server about a minute, and then check the modified date on /usr/share/nginx/html/padlist.txt. The timestamp should be updated to just a minute ago, indicating that cron is running the script and automatically updating the file.

Activating server side includes in Nginx

One final thing remains to be done: enabling SSI in Nginx. This is a simple change, and needs to be made in the www virtual host file, under /etc/nginx/sites-available. Add the following line to the server block near the top, right below the server_name directive:

ssi on;

Restart Nginx, and then, with fingers crossed, visit your Etherpad instance. If everything worked, you should see something like this:

Same as the first time, except now we have a list of our pads!

Credit: Lee Hutchinson

Create a new pad and add some text to it, then wait a minute or so and hit your Etherpad front page again. If your script is working and cron is automatically running it, you’ll see your new pad in the list:

And after waiting a minute to refresh, there’s our new pad, right where it should be.

Credit: Lee Hutchinson

Congrats, it works!

This has been one of the longest entries in the series, but we managed to cover a huge amount of ground—we’ve got Redis and Node.js running, we’ve got our real-time collaboration application available, and we managed to do some totally cool SSI hackery to extend its functionality a little bit!