Tech blog of Anton Keks

Deleting thumbnails for inexisting photos

2009-12-17T00:04:00.005+02:00

Freedesktop for some years already has a spec on how applications should manage image thumbnails (use Next link there). The spec is now followed by majority of Gnome and KDE applications, including F-Spot, which is one of the very few applications that uses large 256x256 thumbnails under ~/.thumbails/large.

The spec specifies to store thumbnails in PNG format, naming the files after the MD5 sum of the original URLs of the original files, eg 81347ce6c37f75513c5e517e5b1895b8.png.

The problem with the spec is that if you delete or move image files, thumbnails stay there and take space (for my 20000+ photos I have 1.4Gb of large thumbails).

Fortunately, you can from time to time clean them by using simple command-line tricks, as the original URLs are stored inside of thumbnail files as Thumb:URI attributes. I don't recommend erasing all of your thumbnails, because regeneration will take time.

In order to create a list of matching thumbnail-original URL pairs, you can run the following in a terminal inside of either .thumbnails/large or .thumbnails/normal directories (it will take some time):

for i in `ls *.png`; do
identify -verbose "$i" | \
fgrep Thumb::URI | sed "s@.*Thumb::URI:@$i@" >> uris.txt;
done

This will get you a uris.txt file, where each line looks like the following:

f78c63184b17981fddce24741c7ebd06.png file:///home/user/Photos/2009/IMG_5887.CR2

Note that the provided thumbnail filenames (first tokens) can also be generated the following way from the URLs (second tokens) using MD5 hashes:

echo -n file:///home/user/Photos/2009/IMG_5887.CR2 | md5sum

After you have your uris.txt file, it can be easily processed with any familiar command-line tools, like grep, sed, awk, etc.

For example, in order to delete all thumbnails matching 'Africa', use the following:

for i in `cat uris.txt | fgrep Africa`; do rm $i >/dev/null; done

So, as you can see, it is pretty simple to free a few hundred megabytes (depending on the number of thumbnails you are deleting).

With this kind of trick you can even rename the thumbnails of moved files if you use md5sum to generate the new filenames from the URLs, as shown above. This will save you regeneration time.

Announcing F-Spot Live Web Gallery extension

2009-08-26T22:55:00.007+03:00

I am happy to announce a new extension for F-Spot, the popular Linux photo management application - LiveWebGallery. Once installed, invoke it from the Tools menu in F-Spot's main window.

The extension contains a minimal web server implementation that serves the user's gallery over HTTP and can be viewed with any web browser, even on Mac and Windows. So now you are able to easily share your photos with family, friends, colleagues no matter what operating system and software they use by doing just a few mouse clicks in F-Spot. The only requirement is that they have to be on the same network, or be able to access your machine's IP address in some other way.

As you can see in the screenshot, you can choose whether to share photos with a particular tag, current view in F-Spot (allows you to create an arbitrary query) or the currently selected photos.

To activate the gallery (start the embedded lightweight web server), just click activate button in the top-right corner. On activation, the URL of the web gallery will appear, allowing you either to open it yourself or copy the link and provide to other viewers.

After that all the options can still be changed in the dialog and will affect all new viewers or those pressing the browser's reload button.

Most of us already know that many pictures are rarely viewed after they are made (Point, Shoot, Kiss It Goodbye). F-Spot tries to fix this with its very powerful tagging features - tags make it much easier to find photos made long ago. This, however, is no magic - the possibilities of finding the right photos when needed depends on how well you tag. Now, this extension allows to make tagging even more useful, because other people can help you with the most difficult part - properly tagging can sometimes be a lot of work. With this extension, you can delegate some it to other people! The gallery is not read-only - if you choose so, an editable tag can be selected and viewers can add/remove this tag from photos (currently only in the full photo view). This is especially useful to let other people to tag themselves in your library. For security reasons, editing is disabled by default.

As time goes by, a lot more features can be added to Live Web Gallery extension, especially related to editing photo metadata (tagging, editing descriptions, flagging for deletion).

As far as I know, being able to share your photos on the local network without any software or OS requirements is a unique feature of F-Spot now. No other photo management application can do this to date.

Downloading

The source code is on Gitorious, live_web_gallery branch (until is has been merged to the mainline).

To install, use the Edit->Manage Extensions menu in the F-Spot, click on Install Add-ins and then Refresh. After that LiveWebGallery should be available under the Tools category.

Or, alternatively, you can download the precompiled binary and put it to:
~/.config/f-spot/addins or /usr/lib/f-spot/extensions

Note: F-Spot 0.6 is required for it to work. You can already find deb/rpm packages for F-Spot 0.6 or 0.6.1 for most distributions and it will be included in the upcoming distro releases this autumn.

Hopefully, the extension will later be distributed with newer versions of F-Spot by default.

Enjoy! Comments are welcome!

Database Refactoring

2009-06-01T22:51:00.008+03:00

A couple of months ago I have made a short keynote titled Dinosaur Strategies: How Can Data Professionals Still Prosper in Modern Organisations, inspired by Scott Ambler's joke on the fictional Waterfall 2006 conference website.

(see the slides)

I primarily deal with 'application' aspects of software development using Agile practices, so I have a hard time understanding how some Data Professionals can be so behind in their evolution, and not doing some basic things like iterative development, unit tests, continuous integration, etc.

Last week I was asked to give a talk on Database Refactoring. The topic seemed challenging enough and as no Database Professionals cared to lead the topic, I decided to give it a try. The result is a motivational speech for both database developers as well as others in the software development process.

I have discussed the cultural conflict of database and OOP developers, the problem of refactoring tools available to relational database developers lagging behind, and some solutions to these problems that can help before these tools become available:

(1) Development Sandboxes
(2) Regression Testing
(3) Automatic Changelog, Delta scripts
(4) Proper Versioning
(5) Continuous integration
(6) Teamwork & Cultural Changes

Other discussed topics include Refactoring of Stored Code vs Database Schema, Agile Reality, Overspecialization (016n), Database not being under control, Database Smells, Fear of Change, Scenarios, Dealing with Coupling, Dealing with unknown applications, Proper versioning, Continuous Integration using sandboxes, and Delta Scripts (Migrations), which make evolutionary database schema possible.

The dinosaurs below are the reminder of my previous keynote available above. They come from the very nice Dinosaurs Song, available on YouTube, which I have actually played after the keynote itself.

Below are full slides of the Database Refactoring talk.

(click for PDF slides)

Versioning your home directory or documents with Git

2009-05-17T17:02:00.006+03:00

Git is a relatively new Version Control System, initially started by Linus Torvalds in order to manage the source code of Linux Kernel.

Although Randal Schwartz has stated that Git was not designed to version your home directory, it seems that many people are now trying to do so :-)

Some people have used CVS or Subversion for this purpose in the past, but to my mind, Git is suited better for this task for several reasons:

Git is grep-friendly (only stores it's metadata in a single .git directory at the root of working copy)
It is very easy to work with a local repository (just do git init and you're ready)
Git stores changes very efficiently (even binary files), so not much disk space is wasted, but don't forget to call git gc from time to time
Git repository is always available on your computer, even when you are offline, but on the other hand it is very easy to push your changes to a remote repository as well

All these things are much worse with CVS, which spams all versioned directories with CVS subdirs and stores each version of binary files fully. Subversion also requires more effort to setup, is less storage-efficient, and puts .svn subdirs everywhere.

Having said that, my setup is ultra-simple compared to others on the net!

To start versioning your home directory, just run this in the root of your home:

git init

This will initialize an empty local Git repository in ~/.git/ - this is the location that you can use when doing backups, but otherwise you shouldn't care about it anymore.

Then you need to tell Git to track your important files:

git add Documents
git add bin
git add whatever else you want to version
git commit -m "Adding initial files"

Then you can work normally with your tracked files and occasionally commit your changes to the repository with

git commit -a "description of changes you have done"

Note the "-a" above, that means to commit any changes made to any previously tracked files, so you don't have to use git add again. But don't forget to git add any new files you create before committing.

Use git status to show what files were changed since your last commit. Unfortunately, it will also list all untracked files in your home directory, so you may need to create a .gitignore file. You can get the initial version of this file using this command:

git status | awk '/#/ {sub("/$", ""); print $2}' > .gitignore

then, edit it and possible replace some full names partly with '*'. Don't forget to git add and git commit this file as well!

That's, basically, it! You may also try some GUI tools provided by git, eg gitk or git gui to browse your changes and do some changes if you can't remember the commands.

Moreover, I have some more ideas how to make all this more automatic that I am going to try laster:

Put git commit -a to user's crontab in order to commit changes automatically, eg daily
Create a couple of nautlus scripts (located in ~/.gnome2/nautilus-scripts) to make adding, comitting and other actions available directly from Nautlilus file manager in Gnome.

Happy versioning! And read the Git tutorial with either man gittutorial or on the official site.

Excessive memory usage by Oracle driver solved

2009-04-26T15:00:00.004+03:00

On my day job I deal with Internet banking. The Internet bank is a relatively large and high-load Java/Spring/Hibernate web application, which uses Oracle databases.

During our recent transition from a centralized data accessor (VJDBC) to local JDBC connection pools to reduce data routrip times, we have started having issues with memory usage in our application servers: some requests started to allocate tens to hundreds of megabytes of memory. While Garbage Collector was successfully reclaiming all this memory afterwards (no memory leaks), it still posed a problem of high peak memory usage as well as too frequent collections, also affecting the overall performance.

While profiling memory allocations with JProfiler, I have discovered that OracleStatement.prepareAccessors() is responsible for these monstrous allocations (up to 600 Mb at once, most in either char or byte giant arrays). Google has pointed to this nice article on reducing the default prefetch size, describing a very similar situation, however these guys have had problems with queries returning LOBs. We haven't used any LOBs in our problematic queries and haven't modified the defaultRowPrefetch connection property knowingly.

Further investigation led to the way we were using Hibernate: for some quesries that are expected to return large result sets, we were using the Query.setFetchSize() or Criteria.setFetchSize() methods with rather high values (eg 5000). This seemed reasonable, because we were also using the setMaxResults() method with the same value to reduce the maximum length of the returned ResultSet. However, after doing some upgrades of Java, Hibernate, and Oracle driver, this had started having these memory allocation side-effects. It seems that now Hibernate translates this fetchSize parameter directly to OracleStatement's rowPrefetch value, forcing it instantly allocate a rowPrefetch*expectedRowSize sized array even before it runs the actual query. This array can be ridicuosly large, even if the actual query returns only a few rows afterwards. Later investigation showed that also having the batch-size attribute in the Hibernate mapping files (hbm.xml) has exactly the same effect and also results in giant pre-allocations.

As a result, we had to review all batch-size and setFetchSize() values that we were using with our Hibernate queries and mappings, in most cases reducing them significantly. This would reduce the worst-case performance of some long queries (they would require more roundtrips to the database), but would also reduce the overall amount of garbage accumulating in the heap and thus reduce the frequency of garbage collections, having a positive impact on CPU load. Shorter results would run equally fast, so it makes sense actually to rely on average statictics of the actual responses when chosing optimal rowPrefetch values. The default value is 10, which is hardcoded in the Oracle driver.

For longer queries, the abovementioned article has proposed an idea of geometrically increasing the rowPrefetch (setting it twice as big for each subsequent fetch manually). This is a nice idea, but I wonder why Oracle driver can't do this automatically? This is how Java collections behave when they resize themselves. I haven't tried doing this with Hibernate yet, but I think it should be possible, especially if you use the Query.scroll() instead of Query.list().

Exchange calendar sync to iPhone through Evolution

2009-03-08T19:20:00.004+02:00

Finally, I have found an easy and transparent way of syncing my corporate calendar (on the evil MS Exchange server, of course) to my iPhone over-the-air, without involving any manual work. The same recipe can actually work for other mobile phones as well, read on!

Current syncing path is as follows:
Exchange → Evolution → (gcaldaemon) → Google Calendar → (Google Sync) → iPhone

If it looks long, don't worry - it is not so difficult in reality.

Exchange - this is where the calendar is stored, which very often is located behind a firewall, where it cannot be talked to directly from a mobile phone.
Exchange is accessed using Evolution Exchange connector (poor Outlook users may find this helpful).
gcaldaemon is an open-source tool for doing various interesting tricks with Google Calendar, including syncing it with Evolution, see below for details.
Google Calendar is a full-featured web-based calendar, where you can store several calendars. It is especially convenient together with GMail, but doesn't require it.
Google Sync is a new service from Google that can sync your Google Calendar (and contacts from GMail) to various programs and mobile devices.
iPhone is where you get both your personal and corporate calendars, always with you :-)

So, actually the keyword here is Google Calendar. As it is quite popular, it is already supported by a lot of software, so you can use it as a middle man in various synching situations, not only the one described here.

I have started testing the Google Sync from Google Calendar to my iPhone. Google had a nice idea to implement Exchange ActiveSync protocol, which is already supported by iPhone (probably there are lots of Google employees using iPhones). Now you just need to setup an Exchange account in your phone and configure it to talk to m.google.com instead of an Exchange server. This is another brilliant implementation after GMail started talking IMAP natively. Follow the instructions on the Google website linked above. As a bonus, you will get 2-way syncing of GMail contacts as well if you want. And everything will work via Push, so you will get almost instant updates when you change something on the either the phone or on the web.

Push syncing works on the iPhone by keeping HTTP connections open for as long as possible by sending a request and waiting for a response as long as your mobile operator's infrastructure permits (can get up to several hours). During this time no traffic is moving between iPhone and Google, so unless something is changed, no need to pay for any data, which is actually better than polling of server every 10 minutes or so. When changes are available, the server stops blocking the connection and immediately pushes data to the phone, hence the name.

Tip: to select which calendars to sync with your phone, navigate to m.google.com/sync with you phone and select what is needed. I have at least 2 calendars there: personal and corporate one - you can conveniently see appointments in different colors.

As soon as this works perfectly, all you need to do is get your Exchange (or any other) events to Google Calendar. This is very easy using the gcaldaemon. See their website for lots of usage scenarios. We are currently interested in file-based synchronization with Evolution. Note that Evolution now has native Google Calendar support as well, but this allows you viewing your exising Google Calendar in Evolution, but not syncing your corporate Exchange calendar to Google.

Evolution, while talking to Exchange, caches your calendar data in a file called cache.ics. You can find it in:

~/.evolution/exchange/exchange___username;auth=Basic@server_;personal_Calendar/cache.ics

substitute your own username name server there.

All you need is to configure gcaldaemon to monitor this file and send updates to Google Calendar, totally automatically. This way you will get one way sync from Exchange to Google, but this should be enough to not miss your all-important meetings at work, because then your phone will alert you whenever you are. I run gcaldaemon right after Evolution from the same launcher, so I don't have to worry about syncing anymore. For that, I have created the ~/bin/evolution file (local bin has priority in PATH, at least on Ubuntu), and this script on execution first runs /usr/bin/evolution, sleeps several seconds and then starts gcaldaemon.

Google sync actually supports syncing with many mobile devices, including the iPhone, Android phones, Nokia Series60 phones with Symbian (contacts only for now), Blackberry, and the awkward Windows Mobile. But even if you cannot directly sync calendar to you phone, you can ask Google to alert you with SMS before each meeting starts, which is almost as good, gcaldaemon will ask Google to do this by default for each event it syncs, provided that Google knows your mobile number. Give this a try - lots of operators are supported worldwide, it's not only US anymore.

The only thing that worries now that this just is another step towards Google taking over the World :-)

Human Interface Guidelines

2008-12-29T23:55:00.005+02:00

This is the recent talk I gave at an internal Training Day for developers of Swedbank.

Although Swedbank Estonia (formely Hansabank) has the best Internet bank in the region according to Global Finance Magazine, we still strive to develop our usability and user interface skills.

The talk was well received and was accompanied with some very nice slides, outlining the history of user interfaces, publicly available HIG documents, usability factors, comm0n design principles and other important points. Therefore it is worth reading for getting a compact introduction to the topic.

Human Interface Guidelines talk slides (PDF)

This time I tried using a minimalistic style for the slides without any (possibly) distracting backgrounds, inspired by Neal Ford's excellent talk "Ancient Philosophers & Blowhard Jamrobees" from the Agile 2008 conference. Neal's slides are mostly back with some nice stock pictures and maybe a few words; they help him talk and not just read what is written on the screen. This helps the listeners to concentrate on the performance of the speaker instead of being distracted by reading, and pictures help to visualise the concept, greatly increasing the influence on the audience. Of course, this kind of presentations need lots of skills and rehearsal from the speaker.

I haven't got as far with eliminating text (read: waste) from the slides, but to my mind this is anyway a great application of the simplicity principle I have talked about. Another principle I also tried to use in the slides (apart from the most important one - consistency) is the aesthetics - people like things that are visually appealing, so consistent style and reasonable animations can make a lot of sense. And please don't use these bundled presentation templates ever again :-)

Anyway, I hope that my slides will be useful and can at least spark some interest for researching this ultra important topic further. There are too many crappy user interfaces out there, so developers, keep this in mind!

Is the scanning of computer networks dangerous?

2008-12-29T22:49:00.008+02:00

Looking through some presentations I have done during the last year, I have come upon a talk I gave at the Baltic DB&IS 2008 Conference in Tallinn, Estonia.

This bi-annual academic conference takes places usually in either Estonia, Latvia or Lithuania, but often involves speakers from other countries as well. This year it was convenient for me to participate and talk a little about my open-source networking tool Angry IP Scanner.

The talk was quite successful, although not very academic, but it makes sense to post the slides to give some interesting information on Angry IP Scanner.

Baltic DB&IS 2008 - Is the scanning of computer networks dangerous? (PDF slides)

(a nice picture of Tallinn's silhouette is a bonus)

Use LightZone from F-Spot as external editor

2008-11-26T22:49:00.007+02:00

In a previous post I have shown how to teach LightZone, a non-destructive photo editor, to open files passed from the command-line (for some strange reason it doesn't do it out-of-the-box).

Now, I want to be able to use LightZone as an external editor from F-Spot, which I use for my photo workflow.

F-Spot has a convenient Open With menu when right-clicking on a photo, we just need to add LightZone to this menu. After some research, it appear that F-Spot uses the standard desktop entry specification files to populate this menu. These files can be located either in /usr/share/applications or in user's home, ~/.local/share/applications. You can use either location, but I prefer the latter one, because I have unpacked LightZone into my home as well.

Here is the working lightzone.desktop file:

[Desktop Entry]
Version=1.0
Type=Application
Name=LightZone Photo Editor
Exec=LightZone %u
TryExec=LightZone
Icon=/home/USERNAME/LightZone/LightZone_32.png
Terminal=false
Categories=Graphics;2DGraphics;Photography;RasterGraphics;GTK;
StartupNotify=true
MimeType=image/tiff;image/jpeg;image/x-canon-cr2;image/x-canon-crw;image/x-nikon-nef;image/x-pentax-pef;

This file expects that you have followed the previous post and already have LightZone in the PATH which accepts a filename to open on the command-line.

Exec - this line specifies what command to run, %u means to pass the selected file's URL on the command-line. For some reason with trial and error, I found that F-Spot is only able to pass URLs like this, specifying %f doesn't work with F-Spot. But if you look at the LightZoneOpener.java source, you will see that it supports URLs as well as filenames.
Icon - change this one to the full path of LightZone's icon, it supplied in the original archive.
Categories - this specifies where LightZone will appear in the ''Applications'' menu.
MimeType - here you must list all image mime types that you want LightZone to open. This is especially important for RAW files. As I own a Canon camera, I have most of my photos in CR2 format, so I need to be sure that image/x-canon-cr2 is in the list. I have also specified CRW, NEF and PEF mime types for older Canon, Nikon and Pentax cameras, respectively. These mime types are already registered in Ubuntu Intrepid (not sure about other distributions). Here is some info on how to register new mime types in Gnome, in case your camera's format is not registered yet.

After chosing the Open With->LightZone, F-Spot will ask whether to create a new version for the file. Select 'No' - this won't work with RAW files and LightZone anyway, because LightZone saves files automatically with the _lzn.jpg suffix and F-Spot doesn't know about it.

Getting this right requires patching F-Spot (I am going to do that later). For now, you will have to import the saved file manually, if you want it to appear in F-Spot.

Enable HTTP proxy in Gnome automatically

2008-11-24T19:56:00.011+02:00

I have a laptop with Linux (currently Ubuntu) which I use both at home and at work. The corporate security policy requires everyone to use the HTTP proxy server with authentication for web access, so when I come to work I had to manually enable it, and then disable again at home - not very convenient.

As a side note, Firefox 3+ is great in respecting the global or system-wide proxy configuration (System->Preferences->Network proxy or gnome-network-preferences) as well as gnome-terminal is very nice to set the http_proxy environment variable automatically when proxy is configured, making most command-line tools respect the global proxy setting as well, which is very cool.

So, before network profiles have arrived to Gnome or NetworkManager (I have seen some related commits in Gnome SVN), I still want to enable the proxy automatically depending on my location. Thankfully, NetworkManager supports execution of scripts when it brings interfaces up or down, so this is not difficult at all.

At least on Ubuntu, NetworkManager executes the scripts that are located in /etc/NetworkManager/dispatcher.d/ when it brings interfaces up. Inside of the script I can detect whether I am at work by checking the domain name in /etc/resolv.conf provided by the corporate DHCP server, or the beginning of the assigned IP address if domain can't be used for any reason.

OK, here is the working script for Ubuntu Karmic, Jaunty and Intrepid (Gnome 2.24+), see notes below for older versions. I have this script in /etc/NetworkManager/dispatcher.d/02proxy, because 01ifupdown already exists there.

It is an updated version, attempting to make the script suitable for more general use, eg in our company we now provide it in a .deb package for all Ubuntu-based laptops.

#!/bin/bash
# The script for automatically setting the proxy server depending on location.
# Put it under /etc/NetworkManager/dispatcher.d/02proxy
# Create also the /etc/NetworkManager/proxy_domains.conf, specifying the mapping of
# DHCP domains to proxy server addresses, eg "example.com proxy.example.com:3128"
# Written by Anton Keks

PROXY_DOMAINS="/etc/NetworkManager/proxy_domains.conf"

# provided by NetworkManager
INTERFACE=$1
COMMAND=$2

function gconf() {
 sudo -E -u $USER gconftool-2 "$@"
}

function saveUserConfFile() {
 echo "DOMAIN_USER=$DOMAIN_USER" > $CONF_FILE;
 echo "DOMAIN_PWD_BASE64="`echo $DOMAIN_PWD | base64` >> $CONF_FILE;
 echo "PROXY_HOST=$PROXY_HOST" >> $CONF_FILE;
 echo "PROXY_PORT=$PROXY_PORT" >> $CONF_FILE;
}

function enableProxy() {
 PROXY_HOST=`cat $PROXY_DOMAINS | grep $DOMAIN | sed 's/.* \+//' | sed 's/:.*//'`
 PROXY_PORT=`cat $PROXY_DOMAINS | grep $DOMAIN | sed 's/.*://'`

 # check if authentication is required
 http_proxy=http://$PROXY_HOST:$PROXY_PORT/ wget com 2>&1 | grep "ERROR 407"
 if [ $? -eq 0 ]; then
   AUTH_REQUIRED="true"
   CONF_FILE=$HOME/.proxy:$DOMAIN

   if [ ! -e $CONF_FILE ]; then
     DOMAIN_USER=`sudo -E -u $USER zenity --entry --text "Login name for domain $DOMAIN"`
     DOMAIN_PWD=`sudo -E -u $USER zenity --entry --text "Password for domain $DOMAIN" --hide-text`
     saveUserConfFile
   fi

   # load user proxy settings
   . $CONF_FILE
   # decode password
   DOMAIN_PWD=`echo $DOMAIN_PWD_BASE64 | base64 -d`
  
   # get Kerberos ticket (if it's configured)
   if echo $DOMAIN_PWD | sudo -E -u $USER kinit $DOMAIN_USER; then
     KLIST_INFO=`sudo -E -u $USER klist | fgrep Default`
     sudo -E -u $USER notify-send -i gtk-info "Domain login" "Kerberos ticket retrieved successfully: $KLIST_INFO"
   fi
 else    
   AUTH_REQUIRED="false"
 fi

 # setup proxy
 gconf --type string --set /system/proxy/mode "manual"
 gconf --type bool --set /system/http_proxy/use_http_proxy "true"
 gconf --type string --set /system/http_proxy/host $PROXY_HOST
 gconf --type int --set /system/http_proxy/port $PROXY_PORT
 gconf --type bool --set /system/http_proxy/use_same_proxy "true"  
 gconf --type bool --set /system/http_proxy/use_authentication $AUTH_REQUIRED
 gconf --type string --set /system/http_proxy/authentication_user $DOMAIN_USER
 gconf --type string --set /system/http_proxy/authentication_password $DOMAIN_PWD

 # notify
 sudo -E -u $USER notify-send -i gtk-info "Proxy configuration" "Your proxy settings have been set to: $DOMAIN_USER@$PROXY_HOST:$PROXY_PORT"
}

function disableProxy() {
 gconf --type string --set /system/proxy/mode "none"
 gconf --type bool --set /system/http_proxy/use_http_proxy "false"
 gconf --type string --set /system/http_proxy/host ""
 gconf --type bool --set /system/http_proxy/use_authentication "false"
 gconf --type string --set /system/http_proxy/authentication_user ""
 gconf --type string --set /system/http_proxy/authentication_password ""
}

# wait for gnome-settings-daemon to appear, ie until user logs in
for i in {1..100}; do
 if [ ! `pidof gnome-settings-daemon` ]; then
   sleep 5;
   echo "Waiting for gnome-settings-daemon to appear..."
 else
   break  
 fi
done
if [ ! `pidof gnome-settings-daemon` ]; then
 echo "gnome-settings-daemon is not running. exiting."
 exit 1
fi

# steal environment from the current non-root user
XENV=`xargs -n 1 -0 echo </proc/$(pidof gnome-settings-daemon)/environ`
# init DBUS connection string in order to reach gconfd
eval export `echo "$XENV" | fgrep DBUS_SESSION_BUS_ADDRESS=`
eval export `echo "$XENV" | fgrep USER=`
eval export `echo "$XENV" | fgrep HOME=`
eval export `echo "$XENV" | fgrep DISPLAY=`
eval export `echo "$XENV" | fgrep XAUTHORITY=`

if [ $COMMAND != 'up' ]; then
 disableProxy;
 exit
fi

DOMAIN=`cat /etc/resolv.conf | grep domain | sed 's/domain \+//'`
# check if we need to set proxy settings for this domain
if [[ -e $PROXY_DOMAINS && ! `cat $PROXY_DOMAINS | grep $DOMAIN` ]]; then
 echo "Proxy is not required for domain $DOMAIN"
 disableProxy
else
 echo "Setting proxy for domain $DOMAIN"
 enableProxy
fi

Don't forget to:

give this script execute permissions
have gconftool-2, zenity and kinit installed (gconf2, zenity, krb5-user packages in Ubuntu). Install gconf-editor as well for a graphical config editor.
create /etc/NetworkManager/proxy_domains.conf, specifying the mapping of DHCP domains to proxy server addresses, eg "example.com proxy.example.com:3128". Specify each domain on a new line.

The script doesn't need you to hardcode your username and the proxy password anymore - the script will ask you for these values on first run and then store them in $HOME/.proxy:$DOMAIN file, so the script is now perfectly usable on multiuser machines and doens't bug you in case of 'unknown' domains.

For more functionality, it even tries to retrieve the Kerberos ticket for you, if the kerberos is configured properly in /etc/krb5.conf. You can check if this is the case by running this on the command-line:

kinit your-user-name; klist

This works very well for me and saves several mouse clicks every morning :-)

Note to Gnome 2.22 and older users (Ubuntu Hardy, etc): I had this script initially done in Hardy, but after upgrading to Intrepid (Gnome 2.24) it stopped working. The reason was that starting from Gnome 2.24, the gconf setting of /system/http_proxy/use_http_proxy is not the primary one and has been replaced by /system/proxy/mode, which takes one of three values: 'auto', 'manual' and 'none'. In Intrepid, if you set only /system/http_proxy/use_http_proxy as before - it has no effect, you need to set /system/proxy/mode to manual, and this will set the value of the old setting to 'true' automatically.

Another thing introduced with Intrepid is the need to set the DBUS_SESSION_BUS_ADDRESS environment variable (the script steals it from the x-session-manager process) - this is because gconfd has switched to DBUS from CORBA for a communication protocol. If you have older Gnome, then you may omit these 2 lines involving DBUS.

Enjoy!

Opening files with LightZone from command-line

2008-11-12T23:29:00.005+02:00

LightZone is a very useful commercial photo editor with some unique features like non-destructive and layer-based editing. To my mind, the developers has taken a very clever approach to save resulting edits inside of smaller-size JPEG files (thumbnails), so that any program can be used for previewing the resulting image, but opening of the file in LightZone will load the original image and show all the edits again with the possibility to make any changes and export a full-resolution image. The cool thing here is that edited files are very small especially when compared to 10Mb+ source RAW photos, contain all the editing history and can be previewed quickly with any software. And all this runs on Linux (thanks to Java - write once, run almost anywhere).

The only problem with LightZone (at least the Linux version) is that it doesn't accept filenames from the command-line! You have to start the program and select the file manually using the embedded file browser. Of course, this is not an option if you want to run LightZone from another application as an external editor, eg from F-Spot (more on this in a later post).

To make a long story short, I have written a small Java program that takes a filename on the command-line and then modifies LightZone preferences, so when you run it next time it will directly open the specified image.

Here it is:

import java.io.*;
import java.net.*;
import java.util.prefs.*;

/**
 * LightZoneOpener - will modify LightZone preferences to open the 
 * specified file (image) on next startup.
 * This is useful to force LightZone to open a particular file from 
 * the command-line, just run this code before starting LightZone. 
 *
 * @author Anton Keks
 */
public class LightZoneOpener {

  public static void main(String[] args) throws Exception {
    if (args.length != 1) {
      System.err.println("Please specify filename to open in LightZone");
      System.exit(1);
    }
    String filename = args[0];
    if (filename.startsWith("file:"))
      filename = new URI(filename).getPath();
    File file = new File(filename).getCanonicalFile();
    if (!file.exists()) {
      System.err.println(file + " doesn't exist!");
      System.exit(2);
    }
    File fileDir = file.getParentFile();
    
    // set image folder as current one
    Preferences folderPrefs = Preferences.userRoot().node("com/lightcrafts/ui/browser/folders");
    int i = 0; File dir = file;
    while ((dir = dir.getParentFile()) != null) {
      folderPrefs.put("BrowserTreePath" + i++, dir.getName().isEmpty() ? "/" : dir.getName());
    }
    folderPrefs.remove("BrowserTreePath" + i);
    
    // set selected image in the current folder
    Preferences appPrefs = Preferences.userRoot().node("com/lightcrafts/app");
    appPrefs.put("BrowserSelectionMemory" + fileDir.getPath().hashCode(), file.getPath());
    // tell LightZone that last startup was OK just in case
    appPrefs.put("StartupSuccessful", "true");
    
    System.out.println("LightZone is now ready to open " + file + " on next start");
  }
}

Here is what to do:

save it to LightZoneOpener.java
compile with javac LightZoneOpener.java
run with java LightZoneOpener

Then you can create a small script that will automate the stuff for you (save it to ~/bin/LightZone):

#!/bin/bash
java -cp ~/bin LightZoneOpener "$@"
~/LightZone/LightZone

This assumes that you have extracted LightZone to ~/LightZone (in your home dir) and have the following two files in the ~/bin dir: the compiled LightZoneOpener.class and the script file LightZone (don't forget to set the execute permission with chmod a+x ~/bin/LightZone)

Here is all this pre-compiled. Just extract the file directly in your home and it will put all needed files into the bin directory. After next login your local bin will be in $PATH, so you will be able to use it.

Now you can run LightZone filename on the command-line in Linux (or using Alt+F2)! Have fun!

Anti-antivirus, someone?

2007-06-24T17:17:00.001+03:00

Many years ago, in good old DOS days, I thought that antivirus software was great. In these days it was common for antiviruses to start on system startup and scan drives (thankfully, the drives weren't that large).

In these days, most viruses could be considered a great work - most of them were technically difficult to write, were introducing very interesting tricks for infection and stealth techniques. However, after Windows OSs and the Internet became popular, a whole lot of new possibilities has appeared, how to break into computers. These times have brought trojans and worms to the attention of the public, which has mostly shaded viruses. Now, we receive a lot of email worms written by script-kiddies and call them 'viruses'.

Of course, as malware transformed, antivirus software
needed to follow up: they started to detect a lot more things that can be potentially harmful to poor computer users. Moreover, as these computer users don't usually understand what are they doing and what is dangerous and what is not, antivitus software vendors has taken a new mission: to protect from everything.

This is arguably a good mission, but IMHO it has started getting in the way too much. Nowadays, antivirus scanners have their hooks everywhere in the system: they monitor network traffic, they scan every opened and yet unopened file, they slow down the computer a whole lot. Sometimes it feels that we are back a few years in terms of computer performance - it is like Moore's law seems to be broken.

However, as everybody knows, nowadays the world is being controlled by money - so are the antivirus scanners. In order to make more profit, they tend to frighten the users by 'detecting' all kinds of stuff they may have on their computers and saying that it is dangerous. Moreover, this also has a side-effect of increasing the size of their 'virus' databases: "We detect 75,000 viruses, we are the greatest!". I wonder how many of these are REAL viruses! Even worse, now antivirus makers are close to controlling the world in some way - they tell users what they can use and what they cannot. They delete software without warnings.

Recently, McAfee and then Symantec started 'detecting' my open-source tool Angry IP Scanner. McAfee was the first, but they even didn't give any explanations. After long email discussions with them they told me that this is a 'potentially unwanted program' for their users and therefore it must be deleted. No matter that it is open-source, no matter that it has no installers and is never distributed automatically (the only way is to download it manually) nor it somehow abuses the system. It is just a tiny little exe file. If a user doesn't want - they can just hit the Delete button - and it's gone! Later, the trend was followed by Symantec. They at least have provided some information and classified the program as 'hacktool'. See their description here.

Actually, they both have hit a lot of their customers. There are thousands of thankful users of Angry IP Scanner around the world, especially among network administrators. I have got a lot of emails asking me to 'fix' this problem, but unfortunately I can't: antivirus makers, seeking for their profits, just don't listen to me.

So, if you are a user of antivirus software, please help to stop the evil: tell your vendors that they have gone too far. They are taking our freedom and ruining our computers.

I hope that antiviruses will never become popular on Linux. Almost complete lack of malware dangerous to regular users as well as freedom make it an unbeatable choice. Happy switching and fighting for the freedom in this imperfect cruel world! :-)