Lazy import of csv file in Clojure

I’ve been looking at Clojure now and then for a while… really wanna learn this stuff :) The other day I needed to import a very large CSV file… and it seemed like a good problem to try solve in Clojure.

I’m using Leiningen and started with the following project setup.

project.clj:

(defproject bbr "1.0.0"
  :description "Import BBR data"
  :dependencies [[org.clojure/clojure "1.3.0"]
                 [org.clojure/clojure-contrib "1.2.0"]
                 [org.clojure/java.jdbc "0.0.6"]
                 [mysql/mysql-connector-java "5.1.6"]
                 [clojure-csv "2.0.0-alpha2"]])

core.clj:

(ns bbr.core
  (:require [clojure-csv.core :as csv]
            [clojure.java.jdbc :as sql])
  (:use [clojure.java.io :only [reader]]))

(def db {:classname "com.mysql.jdbc.Driver"
         :subprotocol "mysql"
         :subname "//localhost:3306/mydb"
         :user "root"})

(defn parse-row [row]
  (let [v (first (csv/parse-csv row :delimiter \;))]
    (zipmap [:kvhx :road :no :floor :door :zip_code :city] v)))

(defn parse-file
  [filename]
  (with-open [file (reader filename :encoding "ISO-8859-1")]
    (sql/with-connection db
      (doseq [line (line-seq file)]
        (let [record (parse-row line)]
          (sql/update-or-insert-values "bbr" ["kvhx=?" (:kvhx record)] record))))))

The above should be able to import VERY large CSV files into the database because it reads one line from the file at the time.

Quick Launch Looking Glass

Today I took the plunge into something I’ve been wanting to do for a very long time: help developing Open Source Software. I’ve started off small :) I’ve been messing around with gnome-shell and looking into how extensions work. What better way to start than scratching and itch? I found it annoying to press Alt+F2 followed by “lg” and Enter to open Looking Glass (Gnome 3’s integrated debugger and inspector tool) all the time.

Now I’ve created a small extension which binds opening Looking Glass to a keyboard shortcut for Gnome 3.2.

To install: download Quick Launch Looking Glass and install it using gnome-tweak-tool or by extraction it in:

~/.local/share/gnome-shell/extensions/

Followed by a restart of gnome-shell.

I haven’t been able to figure out how to add a new custom key binding for a gnome-shell extension so I’ve used the reserved “run command 9” by Metacity until I figure out something better.

If you haven’t bound it already - here is an example on how to bind it to the key F12:

gconftool-2 -s --type string "/apps/metacity/global_keybindings/run_command_9" 'F12'

If the extension is enabled then Looking Glass will open when you press F12 (just close it by hitting Escape like normal).

Storing encrypted data in Amazon S3

I have a lot of digital photos which I would be very sad to loose. I was recommended Amazon S3 as a cheap storage solution and even though it is possible to use https to transfer your data to S3 in a secure way the data itself isn’t encryptet within Amazon. I’m looking for a solution where my photos of my son in the bathtub are secured from all prying eyes including the Amazon tech staff.

It is possible to implement encryption transparent on Linux with the following two modules:

  • s3fs - FUSE-based file system backed by Amazon S3
  • encfs - EncFS provides an encrypted filesystem in user-space

First you need to signup for Amazon S3 - remember you pay on for what you use.

Then create a bucket, I used the Getting Stated Guide that Amazon provides. The following example will use the bucket name: your.bucket

Get an “Access key” from the AWS Management Console, I just used the one which was created upon AWS account creation.

Then I installed s3fs following the instructions on the project website.

Put the “Access key” inside the file .passwd-s3fs in your home directory using the format accessKeyId:secretAccessKey. You can find more informaiton on the on s3fs wiki page.

Now mount your S3 storage using:

mkdir /media/s3
s3fs your.bucket /media/s3

Now try create a file within you storage in the cloud ie.:

touch /media/s3/test

After creation of the file check that you can see the file using the AWS Management Console. Permissions, timestamps etc. is stored in metadata in Amazon S3.

Now in order to apply a transparent encryption layer encfs needs to be installed. Using Ubuntu you do it like this:

sudo aptitude install encfs

Now apply transparent encryption by mounting /media/s3 thorugh encfs like this:

encfs /media/s3 /home/je/Pictures/Encrypted\ on\ S3

and follow the on screen setup. The setup process is only triggered the first time you mount a directory using encfs and it results in a XML filen with all your choices. Don’t delete the XML file (perhaps take a backup) and remember you encfs password.

Now try create a file within:

/home/je/Pictures/Encrypted on S3

And verify that is is unreadable through the AWS Management Console and the directory:

/media/s3

Thats it :)

To expose this to Windows clients then install Samba and share the directory:

/home/je/Pictures/Encrypted on S3

NO WARRENTY: I’m NOT an expert in encryption so I cannot guarantee that encfs is secure enough. Neither do I know how much overhead s3fs and encfs puts on top the actual data that you tranfer to S3.

jQuery Draggable/Droppable: Revert on drop

Background:

At work we are currently in the process of creating a general data hub which is easily expendable and configurable. It’s already in use exchanging EDI messages (EDIFACT) for a supplier on the danish electricity market.

For the configuration we settled on a GUI where you could drag different kinds of data manipulations into the data flow and this is all implemented in HTML with jQuery.

The problem:

Upon drop I had to use Ajax to ask the server if it was ok to drop the draggable. Since Ajax is asynchronous the drop event would return before the Ajax actually had finished. I decided that I didn’t want to force the request to the server to be synchronous instead I wanted the ajax success callback to be able to do the reverting.

Upon searching the net, the closest thing I came to a solution was defining a callback function for revert on the draggable, and this functionality isn’t even documented by the time of this writing.

The solution:

After playing a bit around I found an acceptable solution. I’ve created a small example reverting a drop using a confirm where you can test it (and see the code).

Screen real estate

Gnome global menu and Gnome Do in action
Gnome global menu and Gnome Do in action

My last laptop was the IBM Thinkpad x40 with a 12” screen and a resolution of 1024x768. I used it both private and for work for over five years several hours a day and wore down three batteries in that time. With such a little screen you find yourself exploring ways to get the most out of your screen. Since I prefer the keyboard over the mouse any day I’m not forced to have big icons and menus all over the place.

Here are the steps I’ve taken (things I’ve removed) and how get things done (without them).

Firefox

Press “View / Toolbars / Customize…” and remove anything except from the back, forward, location bar and search bar. Also select “Use small icons”. To stop loading a page just hit escape, to reload use F5 and home use Alt+Home. To get to the location bar hit Ctrl+L and to get to the search bar hit Ctrl+K. I also hid the Bookmarks toolbar (because the suggestions in the address bar usually gives me what I want… alternative you can give bookmarks tags (keywords) which you can type in the address bar to open the bookmark.

Bottom panel

I’ve removed the bottom panel on my Ubuntu installation since it had nothing I’d ever use. Hide all open windows or “Show desktop” is easily accessible using Ctrl+Alt+D. I never used the Trash icon i really rarely delete anything and when I do I usually hold shift while doing so (which skips the trash can) which leaves me with the “Window list”. The window list show the open windows on all workspaces and although I use tabs whenever I can get away with it (Gedit, firefox, terminal and sometimes Nautilus) I still usually have about eight open windows spread over my six workspaces. I’ve come into the habit of using the same workspaces for the same tasks (often one application). Example:

  • Workspace 1: Email
  • Workspace 2: Browsing
  • Workspace 3: Terminal
  • Workspace 4: Virtual machines
  • Workspace 5: Documents using OpenOffice or PDF using Evince
  • Workspace 6: Editor (mostly emacs but somtimes gedit).

This way I alwas know where the application I want for the task at hand is. I use the keyboard shortcuts (Ctrl+Alt+arrow keys) to get to it. I always know where to look for an application. Having all applications on the same workspace and use Alt+Tab or the window list I would have to use my eyes to locate and identify the application I want. So I find this method much faster. Just to be on the safe side I put an other applet called “Window Selector” in my panel in case I ever found myself in need of being able to select a window with the mouse. I sometimes use to show me a list of all the applications I’ve opened.

Gnome do

I can’t remember excatly how why or when I started using Gnome-Do but it is one of those things you didn’t know you needed until you started using it. It is one of the most powerful and versatile tools I’ve ever used on my desktop. Although its not there yet it certainly could be for the desktop whan a terminal is for a server. At the moment I’ve mostly use it to start applications, play, pause and skip music and start, stop, suspend and snapshot virtual machines in Virtualbox. Though I’m pretty sure the use of Gnome-Do will keep growing on me.

Global panel

After I found my self not using the application menu to start stuff anymore I decided to remove it from my top panel to get the extra space and test gnome-global-menu. I belive the gnome-global-menu project was inspired by Mac but whether Apple came up with the idea themselves or they got inspired somewhere else I don’t know. I the top of this post you can see how my desktop looks at the moment (just installed Ubuntu Lucid).