Latest tinkerings – simply-json

Intro

Working with JSON data it is sometimes necessary to visualise it in a human readable way. Since we care about the number of bytes we send to the browser, JSON is usually stripped of any kind of unnecessary whitespace.

Good for size, bad for readability.

Doing a quick search reveals that there are already plenty of options for online formatters. Here’s just a few examples:

For their intended purpose, they all work – Trying them out though, I started to think of how it could be done in a simpler way and that it might be a fun little project, to try out on my own.

Application

For the lack of a better name, as the post title suggests, I decided to call it simply-json, feel free to suggest a more fitting name.

I wanted a minimal feature set, mostly what all the other formatters also provide:

  • Input a URL which points to some JSON data. Have it fetched and then formatted.
  • Input raw JSON and have it formatted.

I also included some design criteria, to formalise my idea of what “simpler” is:

  • No page noise. Content that isn’t relevant to solving the task at hand, should be kept to an absolute minimum.
  • No buttons. Why should I have to click a button, when the browser is perfectly capable of detecting when I’ve input some text in a textfield?

Supplemental features:

  • Highlighting of matching brackets. A feature I like, that some editors have – When moving the cursor over a bracket, the matching opening or closing bracket is indicated.
  • Collapsible regions. Clicking a bracket should collapse the content. Also an editor feature.
  • Loading indication when fetching data via URL. (http://www.ajaxload.info/)
  • Unobtrusive error indication. Borders around text boxes goes red on error.

Considerations

Due the to the same origin policy enforced by most browsers, it’s not immediately possible to request JSON from a different domain than the current one. To do it, some form of proxy is needed.

An easy, fire-and-forget solution would be to use Yahoo! Query Language. The problem with YQL though, is that it transforms the JSON into XML and, if requested as JSON, transforms it back again to JSON. This transformation is lossy, which means e.g. numbers are sent back as strings. {"number":42} => {"number":"42"}. According to the docs:

To prevent this “lossy” transformation, you append the query string parameter jsonCompat=new to the YQL Web Service URL that you are using.

At the time of writing, testing this does in fact reveal, that the loss in number transformation is fixed. What hasn’t been fixed yet though is null values, which is returned as "null" strings instead.

So much for YQL.

Custom proxy

Keeping things in the spirit of simplicity, a service that can proxy a GET request, shouldn’t be more than a few lines of code.

To that end, I chose to use Sinatra, which I’ve had good experiences with in the past. It really is an awesome lightweight web framework. Using Sinatra, this is all it takes:

require 'rubygems'
require 'sinatra'
require 'net/https'
 
get '/' do
  uri = URI(URI.encode(params[:uri]))
  https_session = Net::HTTP.new(uri.host, uri.port)
  https_session.use_ssl = true if uri.port == 443
  https_session.start
  response = https_session.get(uri.path + "?" + uri.query)
  response.body
end

N.B. Updated Feb 10, 2012. New version includes uri query part.

I specifically chose 'net/https' so https sources were also supported, it only adds one extra line and can handle plain http just as well.

The proxy should be very straightforward, assuming the endpoint is at thelabs/simply-json/proxy, making a request to "~thelabs/simply-json/proxy?uri=http://google.com" should be the equivalent of a request directly to "http://google.com".

The immediate problem though, is that the current web server, Apache, is already running on port 80 and the same origin policy even blocks requests to different ports on the same domain.

Fortunately, this can be solved with some config magic server-side.

Apache

Firing up the Sinatra service on localhost:6789, we need Apache to redirect traffic to the /proxy path, back to the service instead of trying to serve content from that directory. This is what is needed in the config, in my case in the virtual host for éncoder.dk:

<Proxy *>
Order deny,allow
Allow from all
</Proxy>
 
ProxyPass /thelabs/simply-json/proxy http://localhost:6789
ProxyPassReverse /thelabs/simply-json/proxy http://localhost:6789

In case Apache hasn’t loaded mod_proxy, enabling it can be done with this command:

$ a2enmod proxy proxy_http

That’s all

Working demo can be found here:

http://éncoder.dk/thelabs/simply-json/

Go check it out.

Launching éDoist – Todoist client for Symbian

Sometime last week I released my latest hobby project on the Ovi store. It is a QtQtuick application targeted at Symbian^3 based Nokia smart-phones.

For now it only features capabilites of a free account, since that is what I have myself.

Also worth noting, is that it is an online-only application at the moment. Adding offline capabilities to a future version is on my todo list however.

Support site with more screenshots: http://éncoder.dk/édoist
Ovi Store content page: http://store.ovi.com/content/183543

Update 09/08

The application is now open source under an MIT license. The source code is available at github: https://github.com/rhardih/edoist/

Using ncurses in C for Text User Interfaces, featuring Xcode

Premise

Being the *nix fanboy that I am, I love having terminal access to my system. Most *nix based OS’s have the same base set of awesome command-line tools. The majority of these are simply “set and run” programs, but some have Text User Interfaces, (TUI), as well. A few of my favourites include screen, bmon, htop and lynx. Be sure to check those out if you haven’t already.

If you’ve ever written a small command-line program that relied on any kind of user input, you’ve probably already coded your own rudimentary menu system at some point. It’s really not that difficult, but.. things can quickly get messy and you’d soon wish you had found an easier way of handling terminal control. Enter ncurses (new curses), a library for writing terminal-independent TUIs.

The project

There’s quite a few run-of-the-mill tutorials for curses out there, but doing a traditional “Hello World!” program just feels so uninspired. Instead we are going to do a simplified version of the classic game Snake, let’s call it PieceOfCakeSnake. In PieceOfCakeSnake you win simply by playing, there is no opponents, no consumables and no way of dying. Just a single, fixed-size, snake moving around in it’s little box world. The game starts right away upon launch and Snakey, our main character, is moving happily along from the get-go. The game ends when ‘x‘ is pressed.

Hammer and chisel

For no particular reason, I’m going to use Xcode for PieceOfCakeSnake and write it in C. If you want to use another editor be my guest, it’s much the same since it’s going to be run in a regular terminal anyway.

Now fire up Xcode and create a new project. Chose “Command Line Utility” > “Standard Tool”. Name the project and save it where you want.

Xcode have already created a main.c file for us that simply outputs “Hello World!” and exits. Click “Run” > “Console” and click “Build and Go”. If all is well, the program builds without error and you should see something like this:

Now, ncurses comes native with Mac OS X, but for other systems you might need to install it beforehand. E.g. for Ubuntu there is the libncurses5 package.

There is still a little more to be done before we start coding. To get access to all the ncurses functions we have to tell the linker, to include the library at compile time. This is done by adding the line #include <ncurses.h> at the top of main.c and by supplying the linker flag -lncurses to the compiler. If you are compiling this from the terminal with GCC, the command would be:

gcc main.c -lncurses -o pocs main.c

Where pocs is the resultant executable. In Xcode however, we rely on the provided build system and so, the linker flag is set in the project properties. Click “Project” > “Edit Project Settings”, chose the “Build” tab and find the field named “Other Linker Flags” and insert “-lncurses”.

That should do it, we are set to go.

Let’s see some code

In this first step, we’ll create a world for Snakey, a square box positioned in the middle of the terminal screen. Here’s the code:

#include 
 
#define WORLD_WIDTH 50
#define WORLD_HEIGHT 20
 
int main(int argc, char *argv[]) {
 
    WINDOW *snakeys_world;
    int offsetx, offsety;
 
    initscr();
    refresh();
 
    offsetx = (COLS - WORLD_WIDTH) / 2;
    offsety = (LINES - WORLD_HEIGHT) / 2;
 
    snakeys_world = newwin(WORLD_HEIGHT,
                           WORLD_WIDTH,
                           offsety,
                           offsetx);
 
    box(snakeys_world, 0 , 0);
 
    wrefresh(snakeys_world);
 
    getch();
 
    delwin(snakeys_world);
 
    endwin();
 
    return 0;
 
}

Running this example, you should see something like this:

Notice the WINDOW type. With ncurses everything is drawn on windows. By default, ncurses sets up a root window, stdscr, which backdrops the current terminal display.

To use it we call initscr(), which prepares the terminal for curses mode, allocates memory for stdscr and so forth.

The windows in ncurses are buffered, in the sense that you can do multiple drawing operations on a window, before making them show up on screen. To display the contents of a window in the actual terminal, the window needs to be refreshed.

For stdscr, this is done by calling refresh(), for child windows we use wrefresh(). This also shows the easy-to-remember naming convention used in the ncurses library – most functions that can be applied to stdscr, also has a counterpart, which applies to child windows, simply named by prepending a ‘w’ to the function name. E.g. refresh() and wrefresh(). We’ll se more of this in the finished version.

Instead of drawing the box manually, we take a shortcut by creating a new window and using the function box() to draw a border around the window. box() can use any displayable character to draw the borders. Using 0 defaults to a system specific line character.

Note: COLS and LINES are environment variables, that holds the current width and height of your terminal. That is the number of horizontal and vertical character positions available in the window.

The getch() function is simply there to pause program execution until some keyboard input is received. Thus a key press exits the program.

Functions delwin() and endwin() handles memory deallocation and returns the terminal to it’s former state. If these are omitted, the terminal will not behave as expected upon program termination and will probably need to be reset.

Time for some action

Now for the fun part – putting Snakey in his box and getting him to move about. Since this entry is about ncurses, I’m not going to go into the mechanics of the game itself. It’s a very simple implementation and the code should be rather self explanatory. You can download the source file here or just read from the following:

#include <ncurses.h>
 
#define TICKRATE 100
 
#define WORLD_WIDTH 50
#define WORLD_HEIGHT 20
 
#define SNAKEY_LENGTH 40
 
enum direction { UP, DOWN, RIGHT, LEFT };
 
typedef struct spart {
    int x;
    int y;
} snakeypart;
 
int move_snakey(WINDOW *win, int direction,
                snakeypart snakey[]);
 
int main(int argc, char *argv[]) {	
 
    WINDOW *snakeys_world;
    int offsetx, offsety, i, ch;
 
    initscr();
    noecho();
    cbreak();
    timeout(TICKRATE);
    keypad(stdscr, TRUE);
 
    printw("PieceOfCakeSnake v. 1.0  -  Press x to quit...");
 
    refresh();
 
    offsetx = (COLS - WORLD_WIDTH) / 2;
    offsety = (LINES - WORLD_HEIGHT) / 2;
 
    snakeys_world = newwin(WORLD_HEIGHT, 
                           WORLD_WIDTH, 
                           offsety, 
                           offsetx);
 
    snakeypart snakey[SNAKEY_LENGTH];
 
    int sbegx = (WORLD_WIDTH - SNAKEY_LENGTH) / 2;
    int sbegy = (WORLD_HEIGHT - 1) / 2;
 
    for (i = 0; i < SNAKEY_LENGTH; i++) {
        snakey[i].x = sbegx + i;
        snakey[i].y = sbegy;
    }
 
    int cur_dir = RIGHT;
 
    while ((ch = getch()) != 'x') {
        move_snakey(snakeys_world, cur_dir, snakey);
        if(ch != ERR) {
            switch(ch) {
                case KEY_UP:
                    cur_dir = UP;
                    break;
                case KEY_DOWN:
                    cur_dir = DOWN;
                    break;
                case KEY_RIGHT:
                    cur_dir = RIGHT;
                    break;
                case KEY_LEFT:
                    cur_dir = LEFT;
                    break;
                default:
                    break;
            }
 
        }
    }
 
    delwin(snakeys_world);
 
    endwin();
 
    return 0;
 
}
 
int move_snakey(WINDOW *win, int direction,
                snakeypart snakey[]) {
 
    wclear(win);
 
    for (int i = 0; i < SNAKEY_LENGTH - 1; i++) {
        snakey[i] = snakey[i + 1];
        mvwaddch(win, snakey[i].y, snakey[i].x, '#');
    }
 
    int x = snakey[SNAKEY_LENGTH - 1].x;
    int y = snakey[SNAKEY_LENGTH - 1].y;
    switch (direction) {
        case UP:
            y - 1 == 0 ? y = WORLD_HEIGHT - 2 : y--;
            break;
        case DOWN:
            y + 1 == WORLD_HEIGHT - 1 ? y = 1 : y++;
            break;
        case RIGHT:
            x + 1 == WORLD_WIDTH - 1 ? x = 1 : x++;
            break;
        case LEFT:
            x - 1 == 0 ? x = WORLD_WIDTH - 2 : x--;
            break;
        default:
            break;
    }
 
    snakey[SNAKEY_LENGTH - 1].x = x;
    snakey[SNAKEY_LENGTH - 1].y = y;
 
    mvwaddch(win, y, x, '#');
 
    box(win, 0 , 0);
 
    wrefresh(win);
 
    return 0;
}

And here is what the game should look like in the terminal:


There is a few new ncurses functions being used here. Let’s start at the top. In main I’ve added:

noecho();
cbreak();
timeout(TICKRATE);
keypad(stdscr, TRUE);
printw("PieceOfCakeSnake v. 1.0  -  Press x to quit...");

From top to bottom. noecho() subverts the terminal from printing back the users key presses. This is useful, since otherwise we would quickly have a lot of garbage on-screen from using the arrow keys to guide Snakey.

cbreak() disables line buffering and feeds input directly to the program. If this wasn’t called, character input would be delayed until a newline was entered. Since we would like immediate response from Snakey, this is needed.

timeout() sets an input delay, in milliseconds, for stdscr, which is applied during input with getch() and sibling functions. If the user doesn’t input anything within this given time period, getch() returns with value ERR. Useful in this part of the code, where we would like Snakey to move, even when we are not pressing any keys.:

while ((ch = getch()) != 'x') {
    move_snakey(snakeys_world, cur_dir, snakey);
    if(ch != ERR) {
        ...
    }
}

The keypad() function enables or disables special input characters for a given window. F keys and arrow keys for example.

printw() works like the standard library function printf. That is print a given string at the current cursor location.

To separate things a little, we have an auxiliary function move_snakey(), which handles movement and redrawing of Snakey within the box. There is a few ncurses specific functions in there as well:

You could chose to add and remove individual characters if you want to be explicit, but I’m lazy, so I clear the whole window and redraw it again every time Snakey has moved. Clearing is done with clear() for stdscr and wclear() for child windows.

The last function to mention is mvwaddch(), which moves, notice mv, to coordinate x, y, in a child window, notice w, and adds a character at that position.

The mv prepend, like w, is also a part of the naming convention of ncurses. Thus most drawing operations have an extended version, that besides the item to be drawn, takes a set of coordinates of where to move the cursor before drawing it. E.g. printw() and mvprintw.

Goodbye Snakey

PieceOfCakeSnake is a very simple demonstration and only shows a very small subset of the features available with ncurses. For the inspired reader however, it should be no problem extending the game with a menu system, a scoreboard and more, using only the small part demonstrated here.

References and source

Ruby on Rails: Sphinx, thinking-sphinx and PostgreSQL on Mac OS X

Premise

On a project of mine, I needed a full text search feature and after a bit of digging, decided to go with Sphinx. It seems like a very proven search engine and with Rails, it’s easy to use through the thinking-sphinx plugin. Normally I just go with the standard SQLite database, if the application doesn’t require a high powered database backend. Unfortunately Sphinx does not yet work with SQLite and as far as I know, needs to run against either MySQL or PostgreSQL. The choice of either MySQL or PostgreSQL is a bit religious I feel. They are both battle hardened DBMS’s and I won’t make compelling argument towards either one. This time however, PostgreSQL is the favored candidate.

Requirements

Before starting, make sure you can compile custom software on your system. For this it is assumed you have the following installed:

PostgreSQL

First things first, we need to install our database server and enable access from rails. PostgreSQL is readily available through MacPorts, so open up a terminal and enter the following command:

sudo port install postgresql84 postgresql84-server

When that is done we need to add the bin folder of the PostgreSQL installation to our PATH:

nano ~/.bash_profile

And then, depending on where your MacPorts installation puts your ports, mine is under /opt/local, add the following line to the file:

export PATH=/opt/local/lib/postgresql84/bin:$PATH

Now we can start up the server using the following command:

sudo launchctl load -w /Library/LaunchDaemons/org.macports.postgresql84-server.plist

Or the shortcut version:

sudo port load postgresql84-server

This goes to the background and makes sure the server is started up again after reboot. Next we want to create a default database and make the server listen for connections:

sudo mkdir -p /opt/local/var/db/postgresql84/defaultdb
sudo chown -R postgres:postgres /opt/local/var/db/postgresql84
sudo su postgres
initdb -D /opt/local/var/db/postgresql84/defaultdb
pg_ctl -D /opt/local/var/db/postgresql84/defaultdb -l ~/postgresql.log start

The logfile postgresql.log is placed in /opt/local/var/db/postgresql84/.

Although I’m a fan of using the command line, I recommend using a tool such as pgAdmin for everyday administration tasks, such as adding new users etc.

Per default, you can login with postgres as user, with a blank password. This is the default superuser account so I suggest changing it sooner rather than later.

Sphinx

Now Sphinx is available through MacPorts as well, even with a postgresql84 variant. I’ve tried installing this version, but couldn’t seem to get through without error. Somehow it still maintained some library dependencies for mysql5 and thus wouldn’t compile. So, instead we opt for a manual installation. First we need to install a couple of dependencies for Sphinx. Download these two archives and extract their content:

You should now have two folders named expat-2.0.1 and libiconv-1.13. From a terminal, navigate to each folder and type the following commands:

./configure --prefix=/usr/local
sudo make && sudo make install

Now it’s time to install Sphinx. Download it and extract as before:

Navigate to the sphinx-0.9.9 folder and enter:

export LDFLAGS="-L/usr/lib"
./configure --prefix=/usr/local --with-pgsql --without-mysql
sudo make && sudo make install

And that is it for Sphinx.

thinking-sphinx

Assuming you already have a Rails project, install the thinking-sphinx plugin like so:

script/plugin install git://github.com/freelancing-god/thinking-sphinx.git

This will add a bunch of new features and a couple of rake tasks. Now, let’s say we have a model Employee, with first names and last names and we would like to be able to search employees by either one. In our model, we can define what we want to have indexed, using the define_index method:

class Employee < ActiveRecord::Base
 
    define_index do
        indexes first_name
        indexes last_name
        set_property :enable_star => true
        set_property :min_infix_len => 4
    end
 
end

The set_property calls, are to enable wildcard searching with asterisks, *, and to index substrings. Eg. If we have an employee named “McLovin”, then we would get matches on “McLo”, “cLov”, “Lovi” and so forth.

Before we can utilize these indices, we need to tell Sphinx to do an index run. In your projects directory, issue these two rake tasks:

rake thinking_sphinx:index
rake thinking_sphinx:start

The Sphinx server should now be running in the backround and in your controller, you can now search for Employee names like so:

class SomeController < ApplicationController
 
    def index
        @employee = Employee.search params[:first_name]
    end
 
end

For more detailed information, see the documentation for Sphinx and thinking-sphinx:

References