Reusing Shotwell thumbnails in Nautilus

As I have lots of photos on my machine, thumbnails start to consume considerable amount of space on the disk.

Another problem, is that gnome-raw-thumbnailer isn't enabled in Ubuntu (Natty, Oneiric) by default anymore, so my raw photos don't get thumbnailed in Nautilus. And, if I enable it manually, thumbnails of vertical photos don't show with the correct orientation.

So, I have researched a bit the freedesktop thumbnail spec, gnome thumbnailer spec and how Shotwell stores its thumbnails and came up with a shell script that reuses Shotwell thumbnails for Nautilus.

Save the script below as /usr/bin/shotwell-raw-thumbnailer

#!/bin/bash
input=$1
output=$2

if [ -z $output ]; then
    echo "Usage: $0 input output"
    exit 1
fi

file=`echo -n ${input##file://} | perl -pe 's/%([0-9a-f]{2})/sprintf("%s", pack("H2",$1))/eig'`
md5=`echo -n $input | md5sum | awk '{print $1}'`

shotwell_id=`sqlite3 ~/.shotwell/data/photo.db "select id from PhotoTable where filename = '$file'"`
if [ -z $shotwell_id ]; then
    gnome-raw-thumbnailer $input $output
    exit
fi

thumb=`printf ~/.shotwell/thumbs/thumbs128/thumb%016x.jpg $shotwell_id`
if [ \! -e $thumb ]; then
    gnome-raw-thumbnailer $input $output
    exit
fi

replaceWithLink() {
    sleep 1
    ln -sf $thumb ~/.thumbnails/normal/$md5.png
}

# gnome-thumbnail-factory doesn't support links
cp $thumb $output

# however, linked thumbnails work, so replace them after a delay
replaceWithLink &

In order to make it work, you then need to register it as a thumbnailer in Gnome, put this to /usr/share/thumbnailers/shotwell.thumbnailer

[Thumbnailer Entry]
Exec=/usr/bin/shotwell-raw-thumbnailer %u %o
MimeType=image/x-3fr;image/x-adobe-dng;image/x-arw;image/x-bay;image/x-canon-cr2;image/x-canon-crw;image/x-cap;image/x-cr2;image/x-crw;image/x-dcr;image/x-dcraw;image/x-dcs;image/x-dng;image/x-drf;image/x-eip;image/x-erf;image/x-fff;image/x-fuji-raf;image/x-iiq;image/x-k25;image/x-kdc;image/x-mef;image/x-minolta-mrw;image/x-mos;image/x-mrw;image/x-nef;image/x-nikon-nef;image/x-nrw;image/x-olympus-orf;image/x-orf;image/x-panasonic-raw;image/x-pef;image/x-pentax-pef;image/x-ptx;image/x-pxn;image/x-r3d;image/x-raf;image/x-raw;image/x-rw2;image/x-rwl;image/x-rwz;image/x-sigma-x3f;image/x-sony-arw;image/x-sony-sr2;image/x-sony-srf;image/x-sr2;image/x-srf;image/x-x3f;

So, what does this script do?

• When Gnome (or Nautilus) needs a thumbnail, it runs this script
• The script checks if the image has an entry in the Shotwell database (~/.shotwell/data/photo.db)
• Then it checks if Shotwell has a thumbnail for it (in ~/.shotwell/thumbs)
• If yes, the script returns the already generated thumbnail to Gnome - no generation needed, so it works much faster
• If Shotwell doesn't have the thumbnail, the call is delegated to gnome-raw-thumbnailer that generates a new thumbnail, the old-fashioned way
• If Shotwell's thumbnail was used, the script will asynchronously replace the thumbnail in ~/.thumbnails with the link to Shotwell's file, avoiding a copy on the disk

The last step is the one that saves disk space. Unfortunately, it is not possible to return a link right away to Gnome - it can't read it for some reason. However, by putting a link directly under ~/.thumbnails later works perfectly, even if we put a .jpg file under the name of .png (as required by a spec). Png is actually a worse choice for thumbnailing of photos due to its lossless compression, so the disk savings are more than twofold with this script.

The next step would be to rewrite this in C or Vala to make even faster and maybe even make Shotwell create these links right away when it generates the thumbnails.

Deleting thumbnails for inexisting photos

Freedesktop for some years already has a spec on how applications should manage image thumbnails (use Next link there). The spec is now followed by majority of Gnome and KDE applications, including F-Spot, which is one of the very few applications that uses large 256x256 thumbnails under ~/.thumbails/large.

The spec specifies to store thumbnails in PNG format, naming the files after the MD5 sum of the original URLs of the original files, eg 81347ce6c37f75513c5e517e5b1895b8.png.

The problem with the spec is that if you delete or move image files, thumbnails stay there and take space (for my 20000+ photos I have 1.4Gb of large thumbails).

Fortunately, you can from time to time clean them by using simple command-line tricks, as the original URLs are stored inside of thumbnail files as Thumb:URI attributes. I don't recommend erasing all of your thumbnails, because regeneration will take time.

In order to create a list of matching thumbnail-original URL pairs, you can run the following in a terminal inside of either .thumbnails/large or .thumbnails/normal directories (it will take some time):

for i in `ls *.png`; do
identify -verbose "$i" | \
fgrep Thumb::URI | sed "s@.*Thumb::URI:@$i@" >> uris.txt;
done

This will get you a uris.txt file, where each line looks like the following:

f78c63184b17981fddce24741c7ebd06.png file:///home/user/Photos/2009/IMG_5887.CR2

Note that the provided thumbnail filenames (first tokens) can also be generated the following way from the URLs (second tokens) using MD5 hashes:

echo -n file:///home/user/Photos/2009/IMG_5887.CR2 | md5sum

After you have your uris.txt file, it can be easily processed with any familiar command-line tools, like grep, sed, awk, etc.

For example, in order to delete all thumbnails matching 'Africa', use the following:

for i in `cat uris.txt | fgrep Africa`; do rm $i >/dev/null; done

So, as you can see, it is pretty simple to free a few hundred megabytes (depending on the number of thumbnails you are deleting).

With this kind of trick you can even rename the thumbnails of moved files if you use md5sum to generate the new filenames from the URLs, as shown above. This will save you regeneration time.