Publish a perl application online in less than five minutes

This is one of my favourite pieces of knowledge I get to share today. Those clever people at heroku.com have made the business of publishing (modern) web applications exceedingly slick and easy. However, on the face of it (and I did hunt for a bit of time) they don’t support perl. Fret not; if you dig around under the bonnet it turns out they support any language/platform as long as someone provides a compatible ‘buildpack’. And it just so happens that Magnus Holm (a fine geek from Norway) has worked out the mechanics of getting your app publishable and provided a buildpack so it all happens behind the scenes.

(If you’re not writing your apps on Mojolicious your platform may not be modern enough — you’ll have to look at Magnus’s notes and code to see if you can become compatible. There are other buildpacks on the buildpack page.)

I will be deploying my web application from a debian server, so it made sense to create a chroot jail for it. Inside there I would be able to install third-party packages without worry of them invading my main diskspace. Wherever you deploy from, you’ll need to have root/superuser access so you can install the heroku tools.

mkdir -p /opt/jail/heroku
debootstrap squeeze /opt/jail/heroku
chroot /opt/jail/heroku
adduser --ingroup www-data --disabled-password --gecos 'app,,,' app
su - app

• First set up your account via the web page
• Sign up for free account
• Give email and password
• Acknowledge email
• Login and go to Dev Center (for reading later)

• Second install the heroku toolbelt
• Install ssh
• Install toolbelt

ssh-keygen
wget -qO- https://toolbelt.heroku.com/install.sh >/tmp/toolbelt.sh
view /tmp/toolbelt.sh
sh /tmp/toolbelt.sh
heroku login
heroku apps:create myapp --stack cedar \
  --buildpack http://github.com/judofyr/perloku.git

That works for debian and ubuntu — there are also installs for other platforms.
You can change the name later, but it’s slightly less trouble to get it right at the start.
(You can have multiple applications, just treat each one the same as the first.)
Application names need to be unique across heroku, so don’t be surprised if obvious names have already gone.
If you create an application without a buildpack (as I did) don’t worry, you can add a buildpack with

heroku config:add BUILDPACK_URL=http://github.com/judofyr/perloku.git

• Third set up your development directory
• Create a directory [in fact I created a chroot jail and a special user but that's only if you want complete isolation] and go into it

mkdir myapp
cd myapp
git init

• Fourth create your initial files based on Magnus’s files

vi Perloku ...
git add .
git commit -m 'Initial'
git push heroku master

• Fifth try out your new application

---

Referenced links

• Heroku
• Mojolicious
• Heroku Toolbelt
• Notes on buildpacks
• Magnus’s perl buildpack (Perloku)

 

Path copy

Frequently I want to copy files or directories from within one tree and inject them into another tree. In such cases I want to preserve timestamps, ownerships, and permissions, including those of all parent directories on its path, and I don’t want unspecified children of directories nor siblings of files. Let’s look at doing that with the various common tools available: tar, rsync, cp, install, cpio, afio.

We’ll have a list of paths in /tmp/paths and the objective is to copy them from /tmp/source to /tmp/target. Most of these methods face the issue that they create ancestor directories fresh rather than copy them. The solution is to ensure /tmp/paths contains every directory in every path, always listing parents before children. For example, if aa/bb/cc/d.txt is a path, then aa, aa/bb, aa/bb/cc are also paths and are included in the file in that same order.

tar

tar -C /tmp/source -cf - --files-from /tmp/paths --no-recursion \
| \
tar -C /tmp/target -xf - --same-owner --atime-preserve --same-permissions

(If your tar doesn’t have a -C option, you can use the classic workaround as follows.)

(cd /tmp/source && tar -cf - --files-from /tmp/paths --no-recursion) \
| \
(cd /tmp/target && tar tar -xf - --same-owner --atime-preserve --same-permissions)

rsync

rsync -lptgod --files-from=/tmp/paths /tmp/source /tmp/target

This does exactly what we want, but the nice people designing the interface provide an even easier to remember set of options which is equivalent.

rsync -a --no-r --files-from=/tmp/paths /tmp/source /tmp/target

One big advantage of rsync is that you don’t need to include the prefix paths (ancestor dirs) in the paths file. Less prep and an easy-to-remember invocation. Thank you rsync.

cp

while read path; do
  echo /tmp/source/$path
done </tmp/paths \
| \
xargs cp -pd --parents -t /tmp/target

While an interesting invocation of cp it doesn’t meet the requirements; it copies everything to /tmp/target/tmp/source instead. In fact I failed to find a successful invocation of cp, even putting it in a loop (and so losing efficiency), because it declines to copy a directory by itself.

Others

install is very similar to cp, but even less useful for this task (because it doesn’t preserve ownership). cpio has a terrible manpage and afio seems to have fallen out of favour (superseded by cpio). (To be fair, there’s no point them competing with tar and rsync in this market.)

See also: Using rsync to copy just a directory

 

Vim sessions and the missing first buffer

Recently I’ve found myself editing in excess of twenty source files concurrently; time to use a vim session so that I don’t need to specify the list of files again each time I start editing. Basic usage is simple.
vim firstfile then :e secondfile, etc. At any time after they’re all open, do :mksession! which will create a Session.vim file in your initial location. When you want to open up those same files again, use vim -S and you’re there. One great advantage of this is that :b8 for example, will always take you to the same file, even if you close and re-open that file — vim knows that you appreciate consistency and does its best to help.

However, there is one small mystery. Say you originally opened up a, b, c and then did :ls. You would see that a was buffer 1 and c was buffer 3. But when you re-open your session, you find that a is buffer 2 and c is buffer 4 — why no buffer 1? The answer is that vim kept buffer 1 for your headline file, but you specified none. I still find this a bit weird, but the working invocation is vim a -S. Now it has opened a as buffer 1 and all other files mentioned in Session.vim as the remaining buffers, so it looks again as it did originally.

Note that Session.vim contains a cd command (which you can edit of course) so you could keep several session files in your base location, relying on the individual session file to set you up in the appropriate destination. For example, have core.vim and plugins.vim in your home directory. Then let vim -S core.vim take you to the base directory for editing those files.

Note also that a huge degree of behaviour is configurable via the sessionoptions setting.

 

Time rounded to five minutes in MySQL

For reporting-style stats we often need to round down ‘now’ to the nearest 5 mins. For instance, my current project requires the code to identify the current ‘rolling twenty minutes’ with such rounding. An easy way to do this that works even in MySQL v4.0 is the following.

SELECT FROM_UNIXTIME(300 * FLOOR( UNIX_TIMESTAMP()/300 ))

which gives the most recently-passed 5-min datetime, taking 13:52:50 down to 13:50:00.
Of course, if you need rounding to the nearest 15 mins then the 300 would be changed to 900.

 

Transforming pictures for your Kindle

I love my kindle (3) — it has brought me back to reading fiction, reading more non-fiction than ever, being more productive in my work-related reading, and just appreciating quality texts. But to my surprise, I love its non-reading features almost as much. The ‘experimental’ browser is handy if you’re selective about which web pages you bookmark, and the couple of games might be handy while nervously passing time in a waiting room. Yesterday my daughter didn’t want to travel home in the car — offered her some music courtesy of the kindle and soon she was falling asleep to Rod Stewart. Best of all, when it’s asleep itself, it is my digital picture frame, showing some of my favourite family pictures without using any of its amazing battery life.

Here I’ll jot down my steps for quickly achieving top quality images tuned for the kindle while minimising strain on CPU and diskspace. (These notes are for B&W 6-inch (600 x 800) kindle screens, not DX nor Fire.)

First create a brand new folder to work in and put in it brand new copies of your favourite pictures. (In picassa the easiest way is to browse for a picture you like, then locate it on disk (<Ctrl>+<Rtn>) then copy & paste to your new folder.) For me that means high resolution colour images from my camera, which happen to be jpg. In picassa, define a ‘Custom format’ of 600 x 800 called “Kindle”. Point picassa at your new folder and carry out the following for each picture.

1. View and Edit (shortcut = <Rtn>)
2. Basic Fixes > Crop
3. Select Kindle, drag the rectangle around to suit
• Quickest is to choose one of the proposed crops, possible using ‘rotate’, then resize/move to suit
• Keep your produced images either all portrait or all landscape
4. Effects > B&W
5. Effects > Glow
• This is for cute pics of my daughter; might not be appropriate for your selection
6. Basic Fixes > Auto Contrast
7. Save

Your picture is now in the right ratio with pretty decent cropping and contrast, but it’s a fair bit bigger than it needs to be. If you don’t care about diskspace or how quickly pictures are rendered then you can skip the next section.

Now we open up gimp and optimise the file format.

1. File > Open
2. Image > Mode > Grayscale
3. Image > Scale Image
4. Set resolution to 166
5. Un-chain Width & Height so they can have unlinked values
6. Set them to be 600 and 800
7. You probably want ‘Cubic’ interpolation
8. File > Save As
9. Change suffix to “png”
10. Untick everything except ‘Save resolution’

Your picture is now in a smaller (png) file. (Might be worth comparing your ‘before’ and ‘after’ sizes, but with my pictures they became much smaller — less than 250 kB.)

If your kindle doesn’t have a ‘pictures’ folder, create one at its root and then create a subfolder for your pictures (eg /kindle/too_cute). Put your .png pictures into the sub-folder, go to ‘Home’ in the kindle, refresh via <Alt>+<z>, then you should be able to open the folder of pictures in the same way you would access a book.

To have these pictures be your new screensaver images, you need to follow the steps at Kindle Screensaver Hack, putting your pictures in the /linkss/screensavers folder.

There are a couple of improvements that could be made to this recipe. It’s ok for twenty pictures, but if you were doing two hundred, you would want to use gimp’s scripting approach. The generated picture files have meta-data relating to the edits, but I’d much rather copy the meta-data (eg creation date) from the original files. One day I’ll write a perl script to copy selected meta-data from each jpg to the corresponding png.

 

Converting HTML to eBook via Calibre

Having an ebook reader means you can take a library with you to remote locations, and in that library you might want to include tech docs that are available online as HTML. The ebook conversion tool calibre can convert HTML to many formats, but getting non-ugly results relies on a bit of fiddling. This is how I’m doing it today.

My ideal is to have a downloaded collection of web pages included as chapters in the ebook. This is done by downloading each file (via Firefox) with Save Page As... > Web Page, complete and putting each page into a shared folder. Then create an index page that links to each of those pages, in order. Test it by pointing your browser at your index page and checking the links work.

There is some configuration of calibre required.

Common Options. In ‘Look & Feel‘ leave ‘Input character encoding‘ blank.

Output Options. In ‘HTMLZ Output‘ leave ‘CSS‘ as ‘class’ and ‘class based CSS‘ as ‘external’.

Adding Books. Set ‘Automerge added books‘ to ‘Overwrite existing duplicate formats’.

Plugins. In ‘File type plugins‘ > ‘HTML to ZIP‘ set char encoding to ‘UTF-8′ and tick ‘Add linked files in breadth first order‘.

Then

1. add your book by adding your created index page,
2. convert, changing the output format to HTMLZ,
3. extract the resulting htmlz file (it’s really just a zip),
4. edit the generated index.html and style.css,
5. keep editing till your browser confirms you have it how you want it,
6. re-zip the files back into a file with .htmlz suffix,
7. use that file to create your real target format (eg mobi).

Without those steps you won’t just get ugly fonts and layout, you’ll also have unprintable chars thanks to faulty handling of encodings. For example, Firefox saves “&mdash;” as UTF-8 ‘em-dash’, and if that’s fed to calibre using its guessed default encoding it will come out as a garbled blob in your ebook. The above steps ensure calibre does the ‘add’ using UTF-8 (smartly converting it back to &mdash;) and then is allowed to make correct guesses from that point onwards.

 

Which features can I use in that version of Perl

I was just getting into the swing of using ‘defined-or’ (//) in my code to avoid some issues with evaluating my (tied) objects as booleans… and then I remembered that my code needs to work under Perl v5.8.3. Pooh! So this is my occasionally-updated list of features and their version of introduction.

  constant : v5.004_000
  //       : v5.010_000

There’s also a module that helps analyse your code to suggest a min version for it.

 

Run unit tests each time you save the source

Test driven development is nothing new, but recently I wanted to try out a stricter version of it in which the granularity is much finer — it became essential that I ran my unit tests for a file every time I hit <save>.

My current dev env is vim + bash + standard perl tests. For each area I’m working on, I define a brief script to declare the files of interest. In the below example I’m working on the Authen.pm plugin module and want the tests in cari_fn_authen.t to run each time the source is saved.

src/test/bin/run_authen:

#!/bin/bash
declare -x   TESTS="src/test/Cari/Fn/Authen/authen_unit.t"
declare -x   SOURCES="lib/Cari/Fn/Authen.pm"
${0/\/run*/\/run_tests.sh}

src/test/bin/run_tests.sh:

#!/bin/bash
declare -r   PROGNAME=${0##*/}
declare -x   PATH=/bin:/usr/bin
declare -i   DEBUG=0
declare      TESTS=${TESTS:?}
declare      SOURCES=${SOURCES:?}
declare      PERL="perl -Imod -Ilib -Isrc/test"

cache=$(mktemp -t ${PROGNAME}.XXXXXX)
function clean_up { rm -f ${cache:-/fictitious_path}; }
trap clean_up SIGHUP SIGINT SIGTERM

declare -i first_time=1
while [ $first_time -gt 0 ] \
        || inotifywait -qqe MOVE_SELF $TESTS $SOURCES; do
    echo -e "\n\n\n"$(date '+%F %R') $TESTS
    for t in "$TESTS"; do
        $PERL -T $t \
            | grep -vF ' skip ' \
            >$cache 2>&1
        result=$(tail -n 1 $cache)
        if [[ $result == 1..+([0-9]) ]]; then
            echo $result
        else
            cat $cache
        fi
    done

    echo -ne \\'033[0;43m'
    $PERL -wc $SOURCES 2>&1 \
        | grep -vF ' syntax OK'
    echo -ne \\'033[0m'
    first_time=0
done

At the beginning the script checks I've remembered to declare TESTS and SOURCES. Then it sets up a temporary file for cacheing test output. Once it's running, the only way to stop it is to interrupt it, so clean_up is required when interrupted.

Since we're monitoring vim saves, the loop guard has inotifywait look for the MOVE_SELF signal. Upon each iteration, it starts by spitting out a timestamp and the tests being invoked. (That's just to give me confidence the monitoring is active.) Then it runs each test, putting its output into the cache, checking the result (last line), and then spitting out the full output only if there was a problem.

The next part just checks the syntax of the source files. If a problem is detected is spits out the errors and warnings with a yucky muddy yellow background. [See here for other colours.]

The end result is I can do quite major refactoring and rewriting at a quicker pace, knowing that any problems I create will be spat onto the neighbouring screen.

 

Separating per-application init from per-request init in CGI::Application

One of the things that attracts (some) people to CGI::Application is the availability to have stripped down super efficient web applications. One of the strengths of it (CA) is that it is versatile enough to provide different strengths to different people, so in fact, a good proportion of CA users don’t actually care about what I call lean programming — as long as you’re not doing anything fancy you can get away with creating everything for every request that comes in. So that’s why some users advocate ignoring efficiency in the early stages; CGI::Application is already a pretty lean perl application framework, so any deployment that works has a good chance of being fairly efficient.

However, it turns out there are other users who require and cater for lean programming, including many of the plugin authors. The technique is very simple — you put your persistent objects/data in class storage and your per-request objects/data in your per-request CA object. (There are many tweaks to that model, including using traditional RDBMS, no-sql tuple pools, memcache variants, CPAN wrapper modules, etc, but usually the simple flavour is all you need and is easy to implement in CGI::Application.) Look at the CGI::Application::Plugin::Authentication source code if you want to see an example of the inner workings, but that’s not necessary (and the logic flow is not terribly transparent in that plug-in). The pattern goes:

1. Have a MyApp::Base which is a sub-class of CGI::Application and which all your MyApp (request handler) modules sub-class
2. In that class, define your cgiapp_init method
3. In cgiapp_init initialise all your application-wide configuration into class storage (eg %MyApp::Base::__Stash) using a guard to ensure you only do this once per perl interpreter:

   unless ($MyApp::Base::__Stash{config}) {
       $MyApp::Base::__Stash{config} = {};
       # Initialise application-wide values
       # Initialise plug-ins, using those values
       MyApp::Base->authen->config(
           # ...
       );
   }
   

4. For each plug-in you use, check whether it caters for class access in addition to object access. eg for CGI::Application::Plugin::Authentication, use MyApp::Base->authen->config rather than $myapp->authen->config and it will do the rest for you.
5. Then (still within your cgiapp_init) initialise any per-request data into your $myapp object:

       # Per-request initialisation
       $self->param(
           foo => 'bar',
           dbh => $MyApp::Base::__Stash{db_accessor}->connect,
           # ...
       );
   

You can take this further by ensuring as much as possible is done in the initial (parent) perl interpreter, for example doing initialisation before the first fork if using mod_perl2 with the ‘prefork’ MPM, but that’s the subject of a separate story.

 

Invoking a command each time a file is modified

There are many situations in which you would like something to run each time you save a file (or a process modifies a file). Since linux kernel 2.6 this has been possible via the inotify mechanism. All you need is a client package for whichever language (bash, perl, ruby, python, …) you want to use.
For perl we have liblinux-inotify2-perl and for bash we have inotify-tools which I’ll illustrate here.

In this example, each time one of the specified files is saved (in vim), the script of unit tests will be invoked. This particular line uses the fact that vim implements a ‘save’ by way of ‘move, move, delete’, so we detect the first ‘move’…

tests=test/cari_plugin_defaults.pl
sources="lib/Cari/Plugin/Defaults.pm lib/Cari/Plugin/Stash.pm"
while inotifywait -qqe MOVE_SELF $tests $sources; do ./$tests; done

This is useful in lots of examples where you want to quickly see the results of editing source file(s), for example editing LaTeX files.