A blog in Clojure

I decided to write a blog in Clojure - my reasons:

  • It is alot more motivating working on something “real” - and I need practice to learn
  • Make markdown syntax for blog posts a first class citizen. This is is a tough fight with Wordpress (was? - haven’t poked at for some time now). Though wysiwyg (html) is just as important… now I just have to figure out a way to make both work well in parallel :D
  • Instead of just storing posts in the database and generating pages dynamic on request, I want the blog to generate static html pages whenever a post is created or updated. The same with comments.

I have put my code on Github and I’ll share my design ideas as it progresses.

Lazy import of csv file in Clojure

I’ve been looking at Clojure now and then for a while… really wanna learn this stuff :) The other day I needed to import a very large CSV file… and it seemed like a good problem to try solve in Clojure.

I’m using Leiningen and started with the following project setup.

project.clj:

(defproject bbr "1.0.0"
  :description "Import BBR data"
  :dependencies [[org.clojure/clojure "1.3.0"]
                 [org.clojure/clojure-contrib "1.2.0"]
                 [org.clojure/java.jdbc "0.0.6"]
                 [mysql/mysql-connector-java "5.1.6"]
                 [clojure-csv "2.0.0-alpha2"]])

core.clj:

(ns bbr.core
  (:require [clojure-csv.core :as csv]
            [clojure.java.jdbc :as sql])
  (:use [clojure.java.io :only [reader]]))

(def db {:classname "com.mysql.jdbc.Driver"
         :subprotocol "mysql"
         :subname "//localhost:3306/mydb"
         :user "root"})

(defn parse-row [row]
  (let [v (first (csv/parse-csv row :delimiter \;))]
    (zipmap [:kvhx :road :no :floor :door :zip_code :city] v)))

(defn parse-file
  [filename]
  (with-open [file (reader filename :encoding "ISO-8859-1")]
    (sql/with-connection db
      (doseq [line (line-seq file)]
        (let [record (parse-row line)]
          (sql/update-or-insert-values "bbr" ["kvhx=?" (:kvhx record)] record))))))

The above should be able to import VERY large CSV files into the database because it reads one line from the file at the time.

Quick Launch Looking Glass

Today I took the plunge into something I’ve been wanting to do for a very long time: help developing Open Source Software. I’ve started off small :) I’ve been messing around with gnome-shell and looking into how extensions work. What better way to start than scratching and itch? I found it annoying to press Alt+F2 followed by “lg” and Enter to open Looking Glass (Gnome 3’s integrated debugger and inspector tool) all the time.

Now I’ve created a small extension which binds opening Looking Glass to a keyboard shortcut for Gnome 3.2.

To install: download Quick Launch Looking Glass and install it using gnome-tweak-tool or by extraction it in:

~/.local/share/gnome-shell/extensions/

Followed by a restart of gnome-shell.

I haven’t been able to figure out how to add a new custom key binding for a gnome-shell extension so I’ve used the reserved “run command 9” by Metacity until I figure out something better.

If you haven’t bound it already - here is an example on how to bind it to the key F12:

gconftool-2 -s --type string "/apps/metacity/global_keybindings/run_command_9" 'F12'

If the extension is enabled then Looking Glass will open when you press F12 (just close it by hitting Escape like normal).

Storing encrypted data in Amazon S3

I have a lot of digital photos which I would be very sad to loose. I was recommended Amazon S3 as a cheap storage solution and even though it is possible to use https to transfer your data to S3 in a secure way the data itself isn’t encryptet within Amazon. I’m looking for a solution where my photos of my son in the bathtub are secured from all prying eyes including the Amazon tech staff.

It is possible to implement encryption transparent on Linux with the following two modules:

  • s3fs - FUSE-based file system backed by Amazon S3
  • encfs - EncFS provides an encrypted filesystem in user-space

First you need to signup for Amazon S3 - remember you pay on for what you use.

Then create a bucket, I used the Getting Stated Guide that Amazon provides. The following example will use the bucket name: your.bucket

Get an “Access key” from the AWS Management Console, I just used the one which was created upon AWS account creation.

Then I installed s3fs following the instructions on the project website.

Put the “Access key” inside the file .passwd-s3fs in your home directory using the format accessKeyId:secretAccessKey. You can find more informaiton on the on s3fs wiki page.

Now mount your S3 storage using:

mkdir /media/s3
s3fs your.bucket /media/s3

Now try create a file within you storage in the cloud ie.:

touch /media/s3/test

After creation of the file check that you can see the file using the AWS Management Console. Permissions, timestamps etc. is stored in metadata in Amazon S3.

Now in order to apply a transparent encryption layer encfs needs to be installed. Using Ubuntu you do it like this:

sudo aptitude install encfs

Now apply transparent encryption by mounting /media/s3 thorugh encfs like this:

encfs /media/s3 /home/je/Pictures/Encrypted\ on\ S3

and follow the on screen setup. The setup process is only triggered the first time you mount a directory using encfs and it results in a XML filen with all your choices. Don’t delete the XML file (perhaps take a backup) and remember you encfs password.

Now try create a file within:

/home/je/Pictures/Encrypted on S3

And verify that is is unreadable through the AWS Management Console and the directory:

/media/s3

Thats it :)

To expose this to Windows clients then install Samba and share the directory:

/home/je/Pictures/Encrypted on S3

NO WARRENTY: I’m NOT an expert in encryption so I cannot guarantee that encfs is secure enough. Neither do I know how much overhead s3fs and encfs puts on top the actual data that you tranfer to S3.

jQuery Draggable/Droppable: Revert on drop

Background:

At work we are currently in the process of creating a general data hub which is easily expendable and configurable. It’s already in use exchanging EDI messages (EDIFACT) for a supplier on the danish electricity market.

For the configuration we settled on a GUI where you could drag different kinds of data manipulations into the data flow and this is all implemented in HTML with jQuery.

The problem:

Upon drop I had to use Ajax to ask the server if it was ok to drop the draggable. Since Ajax is asynchronous the drop event would return before the Ajax actually had finished. I decided that I didn’t want to force the request to the server to be synchronous instead I wanted the ajax success callback to be able to do the reverting.

Upon searching the net, the closest thing I came to a solution was defining a callback function for revert on the draggable, and this functionality isn’t even documented by the time of this writing.

The solution:

After playing a bit around I found an acceptable solution. I’ve created a small example reverting a drop using a confirm where you can test it (and see the code).