Drupal vs Node.js or Drupal + Node.js

Submitted by tomo on December 25, 2012 - 4:01pm

What do Drupal and Node.js have in common? Besides the word "node" not much. They are both ways to do web development. Drupal is for building websites centered around managing, creating, and viewing mixed types of content. Node.js is optimized for near real-time updates in browsers connected for long periods of time to a website.

Drupal is software that runs PHP on top of a web server. It can run on Apache or nginx or other web servers. But it needs a web server to handle listening on a TCP connection and communicating with connected clients (browsers). Node.js, on the other hand, doesn't need an underlying web server because you can easily write a fast, basic web server using Node. Drupal is limited by the speed of PHP and the (most likely MySQL) database that is returning hundreds of queries per page load, and then limited further by the speed at which the web server can serve the data Drupal returns. Node.js can serve a page as quickly as it takes to run any JavaScript function.

Node.js is therefore much lower level than Drupal. Node.js can be the basis for writing scalable, responsive, real-time, single-page web apps. But it won't help you rapidly build any features used on the website. It won't serve as very expressive or patterned scaffolding for any advanced site structure, or help manage more (Drupal) nodes than you can count on your hands. Node doesn't have the breadth of third party web features that makes modern websites social, play nicely with thousands of APIs, or conduct e-commerce. Node.js isn't a web framework, nor a CMS. It's something very different.

Node.js is a framework for programming network servers. Node.js is also the only remaining mainstream way to run server-side JavaScript (others exist but never gained any following, perhaps because without V8 they were too slow or the idea was too ahead of its time). That may not be reason enough for most people to switch to Node, but for some people who perhaps want to share client-side and server-side code or really like JavaScript (because, well, PHP sucks!) then you might still want to use Node even without using it for its networking/events libraries. Those coming from a Python or Ruby on Rails background will see some similar ideas related to event-driven programming.

Why would you want to integrate some Node.js into your Drupal site? Here are some possible use cases

Dashboard: View realtime stats about your site, maybe from hits to your web server's access log. Think Google Analytics realtime view. The benefits increase the more people you have concurrently viewing the dashboard, and multiply when you allow changes to the dashboard that need to propagate out to all viewers with the dashboard open.

Chat: This is the most common example for Node.js. Chat servers need to support many open connections from different users and then update a lot of them when a message is received from one client. And these updates should happen quickly. Some chat protocols even show what the other person is typing as they're typing.

Multiplayer games: Think MMORPG. Games that are played on a server need to be as realtime as possible otherwise you get problems where different players see different versions of the game world. The server needs to be able to serve many simultaneous connections without taking up ever more server resources, CPU or RAM.

Group buying platforms where buyers need to be updated when the number of other buyers goes up. This also applies when resources are limited and shoppers need to know when inventory or availability decreases.

Why do those kinds of projects make sense with Node? They all benefit from high performance despite massive concurrency, realtime or really fast feedback, and the paradigm of event-driven programming on the backend with JavaScript just like in the browser. At least partly it has to do with Websocket in HTML5. Before Websocket, we had to use the hack-ish Comet or long-polling with ajax, but Websocket is the solution that was expressly designed for bidirectional network connections between browser and server. Of course it also has to do with the speed of the V8 JavaScript engine.

So how might we integrate our Node project into our Drupal site? Fortunately there's a module for that:

The module uses an independently installed Node.js server that you configure the module to use. Then you have some secret keys set up for the Node server and Drupal to securely communicate with each other. The module comes with some sub-modules with Node servers that can do things like Node-based notifications to Drupal sites and the browsers connected to them. Whatever message you want to broadcast to browsers can be done this way and the browsers can receive them instantly. This could be an actual text message or prices or scores to be updated or whatever data you might want to change in an existing session. And of course the Node server can listen for updates sent from browsers.

In conclusion, Node.js and Drupal are both web technologies with extremely different use cases but which can also work together for both content-ful and massively interactive websites/web apps. Drupal alone has no chance of doing realtime. By itself, Node.js for a blog or ecommerce site would be like programming games in assembly.

JavaScript as a serious programming language?

Unbeknownst to many folks, even those developing websites, JavaScript libraries are hot right now. One reason is that they're being used on both the frontend (browser) as well as the server via Node.js. And so Drupal 8 includes two new JavaScript libraries, beyond the mainstay of jQuery, which are popular because they rewrite how we use JavaScript, much like Drupal changes how we develop PHP software. They are the Backbone.js and Underscore.js JavaScript frameworks. Since they're in core, Drupal itself can use them in its own JavaScript code. However, as far as I can tell nothing in core, besides Backbone, is using Underscore yet.

Backbone.js ( gives structure to web applications by providing models with key-value binding and custom events, collections with a rich API of enumerable functions, views with declarative event handling, and connects it all to your existing API over a RESTful JSON interface.

The cool thing about Backbone is it splits the abstract representation from DOM events and HTML rendering in your JavaScript code. This is especially applicable to web apps which constantly get new data via AJAX from a REST server and then update the browser in-page rather than reloading a new URL and rendering the page's html on the server - for example via Drupal and it's normal theme/template system.

If you think of modern web applications like GMail or Trello (which is built on Backbone) they work mostly by not reloading the page but rather with each click of the mouse pulling in data from the server then rendering it somehow in the current page. Backbone is how Rdio, Hulu, and Pitchfork play music and videos continuously across "page loads". This can also be happening in the background independent of user activity to update content or display status messages or maybe show new email.


In Drupal, you'll see a lot of references to AHAH. AHAH is basically AJAX except that it's honest about the X, which in AJAX stands for XML, but in Drupal using AHAH it's actual HTML which can be inserted into a page. Or it could be JavaScriptON that's parsed and, using some custom JavaScript logic, turned into HTML by manipulating the DOM.

Without something like Backbone, this is how AJAXy applications are written in Drupal. We add in #ahah hooks that form elements and their widgets can use (when JavaScript is available.) But the login on the frontend is completely custom and usually is jQuery $.ajax callbacks that use jQuery selectors to find and replace DOM elements manually.

Backbone.js gives us a framework to represent things we want to update on the frontend in a structured way, with clear separation between the model and rendering. The model language directly maps to the MVC of Ruby on Rails. Then by updating the model, which is all the logic we really want, the visual representation is updated for us. Backbone can call Underscore templates, Mustache.js, or directly insert into the DOM server-side rendered (the old way we just mentioned) HTML.

So now our AHAH callbacks can return JSON that's pure data, and allow Backbone to handle rendering the HTML.

So what about Underscore.js?

Backbone itself depends on Underscore, but Underscore by itself is also useful. For example, see how Conway's Game of Life can be implemented in JavaScript without if-statements or loops (for, for-in, while) by using Underscore.js and by programming in a more functional style, with functionality similar to Ruby or Python. jQuery, long since included in Drupal core, also offers some functions like $.map or $.each but Underscore is much more complete. Underscore is a JavaScript utility library, not for writing any specific kinds of apps but for making all of your JavaScript code potentially cleaner, simpler, shorter.

Some _ functions:

Underscore also has some array and object helper functions, deep === comparison (isEqual), etc. See it all at

You can play around with Underscore.js right now by writing up some code at

Code Retreat 2012 - Ho Chi Minh City

Submitted by tomo on December 12, 2012 - 5:08pm

On Saturday December 8th, Ho Chi Minh City along with 200 other cities around the world celebrated a global Code Retreat.

What is a code retreat?

Like a meditation retreat, a code retreat is a time set aside to step out of the daily life and routine in order to get deep into practice. It's a full work day, and everyone attending is giving up their Saturday to do what they are probably paid to do from Monday to Friday. That's dedication to the craft. Not so many people came, partly due to poor advertising.

The code retreat was open to anyone but all attendees were fellow programmers, those with some experience already but who want to improve themselves. Besides myself, there was a motley crew of foreigners - an organizer from Japan, a local software tycoon originally from Spain, a Czech developer. Otherwise, all Vietnamese, including startup developers, university faculty, students, and corporate workers.

The venue was ERC Vietnam, the Singaporean MBA school who, along with two other Singaporean schools in Vietnam, is rumored to have lost its operating license from the Ministry of Education which would force it to close. However, the school was still open as usual. A Toastmasters club was also meeting in the room across the hall. There were scheduled outings for students posted in the elevators to be held at local bars and restaurants.

The coding

There were six sessions lasting about an hour each throughout the day and each session was the same both in format and content. The format was:

15 minutes - pair up with another programmer and discuss the strategy for writing the program - choose a language, maybe a framework, discuss data structures and algorithms
15 minutes - one person "drives" the keyboard, the other "navigates" - pair programming (an extreme programming methodology) where two developers use a single laptop to collaboratively develop software
15 minutes - switch up driver and navigator, although often times only a single person was familiar with the text editor or developing environment or even language and API (some languages heavily depend on libraries and APIs to do simple tasks which are difficult to do in the core language alone)
15 minutes - retrospective and break - we wrote down our learnings on sticky notes and collected them for each other to read and at the end of the day we had a whole day retrospective

What were we writing?

We were implementing Conway's Game of Life. Each pair (each session we picked new partners) wrote the game six times throughout the day, erasing the code they had written at the end of each session. The goal was not to write the game of life, but to get better at writing code by writing the same code over and over, by developing the same software over and over.

Halfway through the day some rules started to be added. First, we had to use TDD (test driven development) methodology (which some people didn't realize was feasible in JavaScript), then OOP (what if someone had chosen Scheme or some other functional language?). This was to encourage us to try new techniques and learn them from each other.

I was an outlier in that I wanted to use languages like Python, JavaScript, or even PHP instead of .Net framework languages. Instead of a massive IDE I was using Vim. In the end, in each session where it was up to me I chose Javascript. There was no need to build a new project from a template and then compile executables before running them. Just open a text file, save as html, and use Chrome as a development environment.

Later, more rules were imposed:
No if statements.
No loops (for, while, etc.).
No functions longer than a few lines (maybe 3 or 5 statements max).

We were encouraged to pick one rule and try to work within its constraints. I decided to apply all the rules!

These rules would be easy to abide by using a functional programming language but they're not impossible to follow in Python or Javascript either. But to replace loops using recursion, one thing I found was that nobody remembered how to write recursive functions and most people didn't even want to try as it was hurting their heads.

After the day was over, I took a look at underscore.js, a utility library for JavaScript which makes functional-like programming in JavaScript much easier.

Here's what I came up with.

<script src="underscore.js"></script>
// row and col are indices into 2d array 'grid', either 'today' or 'tomorrow'
function value(grid, row, col) { return (_.isUndefined(grid[row]) || _.isUndefined(grid[row][col])) ? 0 : grid[row][col]; }
function count_neighbors(grid, row, col) {
    return value(grid, row-1, col-1) + value(grid, row-1, col) + value(grid, row-1, col+1) +
           value(grid, row  , col-1) +                           value(grid, row  , col+1) +
           value(grid, row+1, col-1) + value(grid, row+1, col) + value(grid, row+1, col+1);
function next_state_if(neighbors, state) { return (neighbors == 3) ? 1 : ((neighbors == 2) ? state : 0); }
function next_state(grid, row, col) { return next_state_if(count_neighbors(grid, row, col), grid[row][col]); }
function calculate_tomorrow(today, tomorrow) {
    _.each(today, function (foo, row) {
        _.each([row], function (foo, col) { this.tomorrow[row][col] = next_state(, row, col); }, this);
    }, {today: today, tomorrow: tomorrow});
// helpers
function copy2d(ary2d) { return, function (list) { return list.slice(); }); }
function print_r(ary2d) { console.log(, function (list) { return list.join(''); }).join('\n')); }
function random_init() { return, function (list) { return, function (list) { return _.random(1); }); }); }
// demo main loop
var today = random_init();
var tomorrow = random_init();
    calculate_tomorrow(today, tomorrow);
    today = copy2d(tomorrow);
}, 400);

Archiving Twitter to Blog Post

Submitted by tomo on October 5, 2012 - 10:30am

Step 1: Install t (Ruby Twitter command line client). Run the following (substitute your own Twitter ID):

t timeline @tomosaigon --csv --number 3000 > tweets.csv

Step 2: Convert CSV to JSON using

Step pre-3:

cached=$(/usr/bin/grep $1 $CACHE)
if [[ $cached != "" ]]; then
        echo $cached|/usr/bin/cut -d' ' -f2 ; 
url=$(/usr/local/bin/curl -v $1 2>&1|/usr/bin/grep '^< Location: '|/usr/bin/sed -e 's/< Location: //')
echo $1 $url >> $CACHE
echo $url

Step 3: Open Chrome Developer Tools and in the command line type "var a = " then paste the copied JSON. Hit enter.

var out = '';
for (var i = 0; i < a.length; i++) {
    var t = a[i]; 
    var d = new Date(t['Posted at']); 
    if (d >= new Date('2012-09-01 00:00') && d <= new Date('2012-10-01 00:00')) {
        var tweet = t['Text'];
        var tcos = tweet.match(/http:\/\/\/\w+/g);
        if (tcos) {
            for (var m = 0; m < tcos.length; m++) {
                tco = tcos[m].replace(/\//g, '%252F');
                $.ajax({url:'' + tco, async:false, success:function(data){
                    tweet = tweet.replace(tcos[m], data.trim());
        tweet = tweet.replace(/("")/g, '"').replace(/(^|\s)@(\w+)/g, '\$1<a href="\$2" rel="nofollow">@\$2</a>').replace(/(^|\s)#(\w+)/g, '\$1<a href="!/search/%23\$2" rel="nofollow">\$2</a>')
        out += ((d.getMonth() + 1) + '/' + d.getDate() + ' ' + d.getHours() + ':' + d.getMinutes() + ' ' + tweet + ' ' + '<a href="' + t['Screen name'] + '/statuses/' + t['ID'] + '" rel="nofollow">#</a>');
        out += '\n';

How can we store tables or spreadsheets in Drupal content nodes?

Drupal is a Content Management System as well as a Content Management Framework. It's meant for facilitating the creation and editing of content, where content doesn't just mean simple text. CCK allows Drupal site managers to easily enable complex data types for their content. But sometimes it's not so easy to manage lots of simple data in the way that a spreadsheet (meaning Excel or Google Spreadsheet) easily manages many rows and columns of related data.

Sometimes you really want tabular data and you may not know for sure how many rows or columns you'll want beforehand, which is never a problem for Excel or Google Spreadsheet (which is also why sometimes people abuse Excel as a general purpose database). Maybe you are importing spreadsheets or are using spreadsheets with many rows and columns, and only want to use a section of the spreadsheet. Maybe you need to use spreadsheets because you are using formulas and want to do some graphing based on some numbers too. In fact, Excel can do much more. Rather than hoping all of those use cases will make it into a Drupal module anytime soon, wouldn't it be nice if you could just embed a spreadsheet in a node?

There are some modules for attaching tables or tabular data to nodes but in the end the interface can be a bit unwieldy. I discussed building something like a spreadsheet to quickly edit many values and below I present a demonstration.

What is already available that we could use to attach spreadsheets or something similar to nodes?

1. You could: Configure a Google spreadsheet that's either public or somehow accessed via a Google api, perhaps using OAuth2 or just assuming the viewer also has edit access to the spreadsheet. Google already lets you embed spreadsheets including the editable spreadsheet but they don't let you limit the view of the spreadsheet (limiting the view still lets everyone access the whole spreadsheet by changing the URL).

We would also need new nodes to automatically create sheets in the spreadsheet or new spreadsheets altogether and embed them in node edit. This would need to use their APIs since the embeddable Google Spreadsheets require an existing spreadsheet, otherwise this step needs to be done manually for each node - a new Google spreadsheet created per node.

This Redmine Google Docs plugin takes a similar approach and shows how the resulting data could be embedded.

(This might be worth investigating as well.)

2. Another approach would be to take an existing JavaScript or Java spreadsheet that could be embedded. This could be exactly what you want if you need the full functionality of Excel in each node. One such online spreadsheet is ZK Spreadsheet which is written in Java, and would let you have full formula and charting support.

3. Embed a widget like the form below as a CCK field in a node. It would be a new CCK field type where the editing widget is the below spreadsheet-like table and the display widget might be the same thing, an uneditable table, or the raw JSON string representation. In the database, the actual value of the field would be JSON (unless you wanted to use PHP's serialize() which would require POSTing values first instead of just posting the json value). This means you wouldn't relate or compare this field to anything else, as with any serialized data in a database. You could come up with some other schema involving columns and rows to store the table's values for each field in a node but I'm not sure it's worthwhile.

I created this demonstration but haven't turned it into a Drupal module. There is code to dump the edited values into a single JSON value to be stored in the database but the storage would be module-specific.

Get the JavaScript - then combine it with html and CSS, which you also see below.



About Saigonist Mutual Friends Facebook App

Submitted by tomo on August 20, 2012 - 9:22am

My friend Grover and I have a lot of mutual friends as I noticed on Facebook one day. And I know that I have a lot of friends with whom I share a large number of friends. I got curious and asked myself: "Who do I have the most shared friends with?" This question turned into a Facebook FQL query which gave me the data. Then I decided to turn it into a "product" and ended up learning stuff about Facebook's PHP and JavaScript SDKs.

The final result is this Facebook App: Saigonist Mutual Friends.

Just enable the app and you'll see a list of your top 10 friends with mutual friends. I got my answer: my friend Jodie and I share 158 friends.


Next steps? I'd like to visualize what cliques or subnetworks exist within my Facebook network...

Find Friends of a Friend on Facebook

Submitted by tomo on November 18, 2011 - 1:08am

Today I was playing around with Facebook's where you can test out their API.

First, you may want to logout of Facebook using the button there and log back in in order to get the "FB.login callback" Log message on the sidebar. Inside that, you'll see something like:

   "authResponse": {
      "accessToken": "AAAAAKvQdWksBAG8yLhVwqWyvgF2uu2eUahBQTZCPD5y2ilo2qZBbyjJ3DJRXDn4UONrVqAK28ZCSlouAtvdbxCc0ZAzQ0e8VbZBZCsdHmzUQZDZD",
      "userID": "12463924",
      "expiresIn": 5200,
      "signedRequest": "_ltfzUYHjPHyFZH6JnlbzVa-oejnPTud9aHK24eIWOc.eyJhbGdvcml0aG0iOiJITUFDLVNIQTI1NiIsImNvZGUiOiJBUUR2WU4tcTk4cW04X2xwbkg0LVU4Rk9YMUoxYVBmOUNMQkJPeEhlSGowQUZ3a2V6akRNWDlPNGExbkhkREZxT2laQVhWZW50aDRfd19EUk5uWm95NFlDYXFqWktweDZ1MDA5U2Fkc2hSUjQtWUdfUi05S21zY3F5RXlPTC1zbGRaVHBqZkRZS0lvdzFxYnJ2SjJteWxXYkd1dWFhY1h6U1FUWW8tc1V2Rm1oTU5fLV9xZzNxOXRadHJTSi1WdFRpUWciLCJpc3N1ZWRfYXQiOjEzMjE1NDc2MDAsInVzZXJfaWQiOiIxMjQ2MzkyNCJ9"
   "status": "connected"

You should copy the accessToken in order to do further testing which requires a token. Most of the new API calls can be tested right in your browser. For example:

That searches my account for users with the name 'Tomo', while using my token to authenticate. The token doesn't last very long and has definitely expired by the time you read this.

Now plug this code into the textarea.

<div id="profile_pics">
Profile pics here

// From an example

var profilePicsDiv = document.getElementById('profile_pics');
FB.getLoginStatus(function(response) {
  if (response.status != 'connected') {
    profilePicsDiv.innerHTML = '<em>You are not connected</em>';

// JSON output from
var people = {
   "data": [
         "name": "Tomo Coffee",
         "id": "100001851101897"
         "name": "Tomo Kiku",
         "id": "626405318"
         "name": "Tomomi Shiho",
         "id": "1755956208"
         "name": "Tomoman Bkk",
         "id": "1674446551"
         "name": "Tomori Moore",
         "id": "511540368"
         "name": "Tomomi Takihara",
         "id": "100001600551550"
         "name": "Tomohiro Morinaga",
         "id": "713242010"
         "name": "Onda Tomoyuki",
         "id": "100001455563427"
         "name": "Tomoko Okawara",
         "id": "100001514576923"
         "name": "Tomonari Kino",
         "id": "804330331"
   "paging": {
      "next": ""

var searchid = '1218273364';
var markup = '';

function genfn(i) {
    var fn = function(result) {
        //console.log(i); console.log(result); 
        for (var j = 0; j < result.length; j++) {
            if (result[j] == searchid) {
      'foaf search match',[i]);
                markup = (
                  '<fb:profile-pic size="square" ' +
                          'uid="' +[i].id + '" ' +
                          'facebook-logo="true"' +
                var profilePicsDiv = document.getElementById('profile_pics');
                profilePicsDiv.innerHTML += markup;
    return fn;

for (var i = 0; i <; i++) {
    person =[i];
    FB.api({method: 'friends.getMutualFriends', target_uid :}, genfn(i));



As you can see, I prefetched the results from searching "Tomo" and I set the id of a friend whose "Tomo" friends I wanted to find. This searches the results and filters for people who have the mutual friend 1218273364.

To go further, one should have controls for choosing the search term, automatically fetch search results and page through them to find more results, and have a control for choosing the mutual friend. This will allow you to search for mutual friends even if you don't have permission to view a friend's friends. Turning this into an app or Chrome extension is left as an exercise to the reader. :)

I've tracked down a source of the bug which breaks jquery (1.2.6) in FireFox (Chrome is fine) where you'll see a debug message of "z.indexOf is not a function". If you're running a minified jquery then the line number won't help locate the bug, but in this case it was around line 1715:

type == "^=" && z && !z.indexOf(m[5])

This code is triggered by jquery attribute filters like ^= (starts with) or *= or ~= and in this case I found that if z had been 0 then the code which checks "&& z" would short circuit and not try to reference indexOf of z.

Looking deeper, I found that z == -1 (not a string) and that this was because I was filtering on the 'value' attribute, and that in FireFox, the 'li' node was being given some value of -1. You can check this by running "$('li')" and checking out the returned values. In Chrome, there is no value. This difference causes a bug in FireFox.

One workaround is to use only use attribute filters when using selectors that select for specific tags which exclude 'li', at least for filtering on 'value'. For example, use 'input[value^=whatever]' instead of just '[value^=whatever]'.

Simple Way to Download YouTube HTML5 Videos

Submitted by tomo on November 24, 2010 - 3:52am

A lot of the existing YouTube video download scripts don't work today as YouTube have changed their "API" so those hacks no longer work. Programs that simple detect whatever media files get downloaded should still work.

Here's a dead simple way to download HTML5 videos from YouTube. Drag the following link to your bookmark bar. In Chrome, you have to set "Always show bookmark bar" for this to make sense.


Read the rest of this article...
Syndicate content
© 2010-2014 Saigonist.