One-liner for downloading music files from a Podcast feed, with grep, sed and wget

Premise

I like to listen to music while I code and usually it’s in the form of online radio or a podcast. I believe most portable music players have the ability to sync with podcast feeds nowadays, but I’m an old fashioned guy and sometimes I like to have the music files readily available on my own hard-drive, so I can move them about as I please.

Command

As an example, here is a feed from Tiësto’s club life, on podcast I listen to quite a lot:

http://www.radio538.nl/clublife/podcast.xml

Download it and the run this command:

  grep -E "http.*\.m4a" podcast.xml | sed "s/.*\(http.*\.m4a\).*/\1/" | xargs wget

This is a three step command, that does the following:

  • Extracts all lines, that contains the .m4a file urls using grep.
  • Strips all characters that are not part of the urls.
  • Feeds each line to wget one by one

Ofcourse this is very rudimentary and could easily be transformed into a more general purpose tool in a script. Setting for instance file types via parameter, or even avoiding having to download the feed file first. But this solved the job at hand for me.

Feel free to adapt in any way you please.

Installing Mac OS X from a USB Disk

I have a Macbook Pro which is a couple of years old and signs of age are starting to show. I wanted to upgrade to Snow Leopard recently, but the SuperDrive just isn’t working very well anymore. Whenever I tried to boot from the installation DVD, it just made a couple of disgruntled noises and then spat out the disc.

I’ve had this issue before with some mediums it couldn’t read. Sometimes it helps just reinserting it a couple of times and eventually it will start reading the disc. Not this time around though. After trying about 6-7 times, my patience had the better of me.

Luckily I’ve found a workaround and it is dead simple.

  • Grab one of those spare USB drives you have lying about anyway.
  • Create a new partition of say 8GB with Disk Utility.
  • Clone  the Mac OS X install DVD onto that partition.
  • Reboot with the drive attached to your Mac.
  • Install Mac OS X.

The details of how to do it can be found in this blogpost.

Using ncurses in C for Text User Interfaces, featuring Xcode

Premise

Being the *nix fanboy that I am, I love having terminal access to my system. Most *nix based OS’s have the same base set of awesome command-line tools. The majority of these are simply “set and run” programs, but some have Text User Interfaces, (TUI), as well. A few of my favourites include screen, bmon, htop and lynx. Be sure to check those out if you haven’t already.

If you’ve ever written a small command-line program that relied on any kind of user input, you’ve probably already coded your own rudimentary menu system at some point. It’s really not that difficult, but.. things can quickly get messy and you’d soon wish you had found an easier way of handling terminal control. Enter ncurses (new curses), a library for writing terminal-independent TUIs.

The project

There’s quite a few run-of-the-mill tutorials for curses out there, but doing a traditional “Hello World!” program just feels so uninspired. Instead we are going to do a simplified version of the classic game Snake, let’s call it PieceOfCakeSnake. In PieceOfCakeSnake you win simply by playing, there is no opponents, no consumables and no way of dying. Just a single, fixed-size, snake moving around in it’s little box world. The game starts right away upon launch and Snakey, our main character, is moving happily along from the get-go. The game ends when ‘x‘ is pressed.

Hammer and chisel

For no particular reason, I’m going to use Xcode for PieceOfCakeSnake and write it in C. If you want to use another editor be my guest, it’s much the same since it’s going to be run in a regular terminal anyway.

Now fire up Xcode and create a new project. Chose “Command Line Utility” > “Standard Tool”. Name the project and save it where you want.

Xcode have already created a main.c file for us that simply outputs “Hello World!” and exits. Click “Run” > “Console” and click “Build and Go”. If all is well, the program builds without error and you should see something like this:

Now, ncurses comes native with Mac OS X, but for other systems you might need to install it beforehand. E.g. for Ubuntu there is the libncurses5 package.

There is still a little more to be done before we start coding. To get access to all the ncurses functions we have to tell the linker, to include the library at compile time. This is done by adding the line #include <ncurses.h> at the top of main.c and by supplying the linker flag -lncurses to the compiler. If you are compiling this from the terminal with GCC, the command would be:

gcc main.c -lncurses -o pocs main.c

Where pocs is the resultant executable. In Xcode however, we rely on the provided build system and so, the linker flag is set in the project properties. Click “Project” > “Edit Project Settings”, chose the “Build” tab and find the field named “Other Linker Flags” and insert “-lncurses”.

That should do it, we are set to go.

Let’s see some code

In this first step, we’ll create a world for Snakey, a square box positioned in the middle of the terminal screen. Here’s the code:

#include 
 
#define WORLD_WIDTH 50
#define WORLD_HEIGHT 20
 
int main(int argc, char *argv[]) {
 
    WINDOW *snakeys_world;
    int offsetx, offsety;
 
    initscr();
    refresh();
 
    offsetx = (COLS - WORLD_WIDTH) / 2;
    offsety = (LINES - WORLD_HEIGHT) / 2;
 
    snakeys_world = newwin(WORLD_HEIGHT,
                           WORLD_WIDTH,
                           offsety,
                           offsetx);
 
    box(snakeys_world, 0 , 0);
 
    wrefresh(snakeys_world);
 
    getch();
 
    delwin(snakeys_world);
 
    endwin();
 
    return 0;
 
}

Running this example, you should see something like this:

Notice the WINDOW type. With ncurses everything is drawn on windows. By default, ncurses sets up a root window, stdscr, which backdrops the current terminal display.

To use it we call initscr(), which prepares the terminal for curses mode, allocates memory for stdscr and so forth.

The windows in ncurses are buffered, in the sense that you can do multiple drawing operations on a window, before making them show up on screen. To display the contents of a window in the actual terminal, the window needs to be refreshed.

For stdscr, this is done by calling refresh(), for child windows we use wrefresh(). This also shows the easy-to-remember naming convention used in the ncurses library – most functions that can be applied to stdscr, also has a counterpart, which applies to child windows, simply named by prepending a ‘w’ to the function name. E.g. refresh() and wrefresh(). We’ll se more of this in the finished version.

Instead of drawing the box manually, we take a shortcut by creating a new window and using the function box() to draw a border around the window. box() can use any displayable character to draw the borders. Using 0 defaults to a system specific line character.

Note: COLS and LINES are environment variables, that holds the current width and height of your terminal. That is the number of horizontal and vertical character positions available in the window.

The getch() function is simply there to pause program execution until some keyboard input is received. Thus a key press exits the program.

Functions delwin() and endwin() handles memory deallocation and returns the terminal to it’s former state. If these are omitted, the terminal will not behave as expected upon program termination and will probably need to be reset.

Time for some action

Now for the fun part – putting Snakey in his box and getting him to move about. Since this entry is about ncurses, I’m not going to go into the mechanics of the game itself. It’s a very simple implementation and the code should be rather self explanatory. You can download the source file here or just read from the following:

#include <ncurses.h>
 
#define TICKRATE 100
 
#define WORLD_WIDTH 50
#define WORLD_HEIGHT 20
 
#define SNAKEY_LENGTH 40
 
enum direction { UP, DOWN, RIGHT, LEFT };
 
typedef struct spart {
    int x;
    int y;
} snakeypart;
 
int move_snakey(WINDOW *win, int direction,
                snakeypart snakey[]);
 
int main(int argc, char *argv[]) {	
 
    WINDOW *snakeys_world;
    int offsetx, offsety, i, ch;
 
    initscr();
    noecho();
    cbreak();
    timeout(TICKRATE);
    keypad(stdscr, TRUE);
 
    printw("PieceOfCakeSnake v. 1.0  -  Press x to quit...");
 
    refresh();
 
    offsetx = (COLS - WORLD_WIDTH) / 2;
    offsety = (LINES - WORLD_HEIGHT) / 2;
 
    snakeys_world = newwin(WORLD_HEIGHT, 
                           WORLD_WIDTH, 
                           offsety, 
                           offsetx);
 
    snakeypart snakey[SNAKEY_LENGTH];
 
    int sbegx = (WORLD_WIDTH - SNAKEY_LENGTH) / 2;
    int sbegy = (WORLD_HEIGHT - 1) / 2;
 
    for (i = 0; i < SNAKEY_LENGTH; i++) {
        snakey[i].x = sbegx + i;
        snakey[i].y = sbegy;
    }
 
    int cur_dir = RIGHT;
 
    while ((ch = getch()) != 'x') {
        move_snakey(snakeys_world, cur_dir, snakey);
        if(ch != ERR) {
            switch(ch) {
                case KEY_UP:
                    cur_dir = UP;
                    break;
                case KEY_DOWN:
                    cur_dir = DOWN;
                    break;
                case KEY_RIGHT:
                    cur_dir = RIGHT;
                    break;
                case KEY_LEFT:
                    cur_dir = LEFT;
                    break;
                default:
                    break;
            }
 
        }
    }
 
    delwin(snakeys_world);
 
    endwin();
 
    return 0;
 
}
 
int move_snakey(WINDOW *win, int direction,
                snakeypart snakey[]) {
 
    wclear(win);
 
    for (int i = 0; i < SNAKEY_LENGTH - 1; i++) {
        snakey[i] = snakey[i + 1];
        mvwaddch(win, snakey[i].y, snakey[i].x, '#');
    }
 
    int x = snakey[SNAKEY_LENGTH - 1].x;
    int y = snakey[SNAKEY_LENGTH - 1].y;
    switch (direction) {
        case UP:
            y - 1 == 0 ? y = WORLD_HEIGHT - 2 : y--;
            break;
        case DOWN:
            y + 1 == WORLD_HEIGHT - 1 ? y = 1 : y++;
            break;
        case RIGHT:
            x + 1 == WORLD_WIDTH - 1 ? x = 1 : x++;
            break;
        case LEFT:
            x - 1 == 0 ? x = WORLD_WIDTH - 2 : x--;
            break;
        default:
            break;
    }
 
    snakey[SNAKEY_LENGTH - 1].x = x;
    snakey[SNAKEY_LENGTH - 1].y = y;
 
    mvwaddch(win, y, x, '#');
 
    box(win, 0 , 0);
 
    wrefresh(win);
 
    return 0;
}

And here is what the game should look like in the terminal:


There is a few new ncurses functions being used here. Let’s start at the top. In main I’ve added:

noecho();
cbreak();
timeout(TICKRATE);
keypad(stdscr, TRUE);
printw("PieceOfCakeSnake v. 1.0  -  Press x to quit...");

From top to bottom. noecho() subverts the terminal from printing back the users key presses. This is useful, since otherwise we would quickly have a lot of garbage on-screen from using the arrow keys to guide Snakey.

cbreak() disables line buffering and feeds input directly to the program. If this wasn’t called, character input would be delayed until a newline was entered. Since we would like immediate response from Snakey, this is needed.

timeout() sets an input delay, in milliseconds, for stdscr, which is applied during input with getch() and sibling functions. If the user doesn’t input anything within this given time period, getch() returns with value ERR. Useful in this part of the code, where we would like Snakey to move, even when we are not pressing any keys.:

while ((ch = getch()) != 'x') {
    move_snakey(snakeys_world, cur_dir, snakey);
    if(ch != ERR) {
        ...
    }
}

The keypad() function enables or disables special input characters for a given window. F keys and arrow keys for example.

printw() works like the standard library function printf. That is print a given string at the current cursor location.

To separate things a little, we have an auxiliary function move_snakey(), which handles movement and redrawing of Snakey within the box. There is a few ncurses specific functions in there as well:

You could chose to add and remove individual characters if you want to be explicit, but I’m lazy, so I clear the whole window and redraw it again every time Snakey has moved. Clearing is done with clear() for stdscr and wclear() for child windows.

The last function to mention is mvwaddch(), which moves, notice mv, to coordinate x, y, in a child window, notice w, and adds a character at that position.

The mv prepend, like w, is also a part of the naming convention of ncurses. Thus most drawing operations have an extended version, that besides the item to be drawn, takes a set of coordinates of where to move the cursor before drawing it. E.g. printw() and mvprintw.

Goodbye Snakey

PieceOfCakeSnake is a very simple demonstration and only shows a very small subset of the features available with ncurses. For the inspired reader however, it should be no problem extending the game with a menu system, a scoreboard and more, using only the small part demonstrated here.

References and source

Ruby on Rails: Sphinx, thinking-sphinx and PostgreSQL on Mac OS X

Premise

On a project of mine, I needed a full text search feature and after a bit of digging, decided to go with Sphinx. It seems like a very proven search engine and with Rails, it’s easy to use through the thinking-sphinx plugin. Normally I just go with the standard SQLite database, if the application doesn’t require a high powered database backend. Unfortunately Sphinx does not yet work with SQLite and as far as I know, needs to run against either MySQL or PostgreSQL. The choice of either MySQL or PostgreSQL is a bit religious I feel. They are both battle hardened DBMS’s and I won’t make compelling argument towards either one. This time however, PostgreSQL is the favored candidate.

Requirements

Before starting, make sure you can compile custom software on your system. For this it is assumed you have the following installed:

PostgreSQL

First things first, we need to install our database server and enable access from rails. PostgreSQL is readily available through MacPorts, so open up a terminal and enter the following command:

sudo port install postgresql84 postgresql84-server

When that is done we need to add the bin folder of the PostgreSQL installation to our PATH:

nano ~/.bash_profile

And then, depending on where your MacPorts installation puts your ports, mine is under /opt/local, add the following line to the file:

export PATH=/opt/local/lib/postgresql84/bin:$PATH

Now we can start up the server using the following command:

sudo launchctl load -w /Library/LaunchDaemons/org.macports.postgresql84-server.plist

Or the shortcut version:

sudo port load postgresql84-server

This goes to the background and makes sure the server is started up again after reboot. Next we want to create a default database and make the server listen for connections:

sudo mkdir -p /opt/local/var/db/postgresql84/defaultdb
sudo chown -R postgres:postgres /opt/local/var/db/postgresql84
sudo su postgres
initdb -D /opt/local/var/db/postgresql84/defaultdb
pg_ctl -D /opt/local/var/db/postgresql84/defaultdb -l ~/postgresql.log start

The logfile postgresql.log is placed in /opt/local/var/db/postgresql84/.

Although I’m a fan of using the command line, I recommend using a tool such as pgAdmin for everyday administration tasks, such as adding new users etc.

Per default, you can login with postgres as user, with a blank password. This is the default superuser account so I suggest changing it sooner rather than later.

Sphinx

Now Sphinx is available through MacPorts as well, even with a postgresql84 variant. I’ve tried installing this version, but couldn’t seem to get through without error. Somehow it still maintained some library dependencies for mysql5 and thus wouldn’t compile. So, instead we opt for a manual installation. First we need to install a couple of dependencies for Sphinx. Download these two archives and extract their content:

You should now have two folders named expat-2.0.1 and libiconv-1.13. From a terminal, navigate to each folder and type the following commands:

./configure --prefix=/usr/local
sudo make && sudo make install

Now it’s time to install Sphinx. Download it and extract as before:

Navigate to the sphinx-0.9.9 folder and enter:

export LDFLAGS="-L/usr/lib"
./configure --prefix=/usr/local --with-pgsql --without-mysql
sudo make && sudo make install

And that is it for Sphinx.

thinking-sphinx

Assuming you already have a Rails project, install the thinking-sphinx plugin like so:

script/plugin install git://github.com/freelancing-god/thinking-sphinx.git

This will add a bunch of new features and a couple of rake tasks. Now, let’s say we have a model Employee, with first names and last names and we would like to be able to search employees by either one. In our model, we can define what we want to have indexed, using the define_index method:

class Employee < ActiveRecord::Base
 
    define_index do
        indexes first_name
        indexes last_name
        set_property :enable_star => true
        set_property :min_infix_len => 4
    end
 
end

The set_property calls, are to enable wildcard searching with asterisks, *, and to index substrings. Eg. If we have an employee named “McLovin”, then we would get matches on “McLo”, “cLov”, “Lovi” and so forth.

Before we can utilize these indices, we need to tell Sphinx to do an index run. In your projects directory, issue these two rake tasks:

rake thinking_sphinx:index
rake thinking_sphinx:start

The Sphinx server should now be running in the backround and in your controller, you can now search for Employee names like so:

class SomeController < ApplicationController
 
    def index
        @employee = Employee.search params[:first_name]
    end
 
end

For more detailed information, see the documentation for Sphinx and thinking-sphinx:

References