A plugin system for clojure-blog

In order to support different syntax for posts, smileys and hook-ins in general I searched the web for inspiration and found the following link which have some cool info:

http://stackoverflow.com/questions/10272559/how-best-to-structure-and-build-clojure-apps-with-plugins

I’ve started small and decided to support changing the post content via a hook-in before handing it over to the html generator Enlive. The way it works is by registering any number of functions which can transform post content in a chain (vector). Zero functions means no transformation and the post content is serves as-is.

Here is some example code which can demo the thoughts behind the design so far:

(defonce changes (atom []))

(defn reg-change
  "Registers a fn (change) to a hook"
  [hook-name f]
  (swap! hook-name conj f))

(defn apply-changes
  "Applies all changes in the vector to the input"
  [change-vec input]
  (reduce #(%2 %1) input @change-vec))

(apply-changes changes "Test is good")

(defn good-is-bad [text] (clojure.string/replace text "good" "bad"))

(reg-change changes good-is-bad)

(apply-changes changes "Test is good")

A blog in Clojure

I decided to write a blog in Clojure - my reasons:

• It is alot more motivating working on something “real” - and I need practice to learn
• Make markdown syntax for blog posts a first class citizen. This is is a tough fight with Wordpress (was? - haven’t poked at for some time now). Though wysiwyg (html) is just as important… now I just have to figure out a way to make both work well in parallel :D
• Instead of just storing posts in the database and generating pages dynamic on request, I want the blog to generate static html pages whenever a post is created or updated. The same with comments.

I have put my code on Github and I’ll share my design ideas as it progresses.

Lazy import of csv file in Clojure

I’ve been looking at Clojure now and then for a while… really wanna learn this stuff :) The other day I needed to import a very large CSV file… and it seemed like a good problem to try solve in Clojure.

I’m using Leiningen and started with the following project setup.

project.clj:

(defproject bbr "1.0.0"
  :description "Import BBR data"
  :dependencies [[org.clojure/clojure "1.3.0"]
                 [org.clojure/clojure-contrib "1.2.0"]
                 [org.clojure/java.jdbc "0.0.6"]
                 [mysql/mysql-connector-java "5.1.6"]
                 [clojure-csv "2.0.0-alpha2"]])

core.clj:

(ns bbr.core
  (:require [clojure-csv.core :as csv]
            [clojure.java.jdbc :as sql])
  (:use [clojure.java.io :only [reader]]))

(def db {:classname "com.mysql.jdbc.Driver"
         :subprotocol "mysql"
         :subname "//localhost:3306/mydb"
         :user "root"})

(defn parse-row [row]
  (let [v (first (csv/parse-csv row :delimiter \;))]
    (zipmap [:kvhx :road :no :floor :door :zip_code :city] v)))

(defn parse-file
  [filename]
  (with-open [file (reader filename :encoding "ISO-8859-1")]
    (sql/with-connection db
      (doseq [line (line-seq file)]
        (let [record (parse-row line)]
          (sql/update-or-insert-values "bbr" ["kvhx=?" (:kvhx record)] record))))))

The above should be able to import VERY large CSV files into the database because it reads one line from the file at the time.

Quick Launch Looking Glass

Today I took the plunge into something I’ve been wanting to do for a very long time: help developing Open Source Software. I’ve started off small :) I’ve been messing around with gnome-shell and looking into how extensions work. What better way to start than scratching and itch? I found it annoying to press Alt+F2 followed by “lg” and Enter to open Looking Glass (Gnome 3’s integrated debugger and inspector tool) all the time.

Now I’ve created a small extension which binds opening Looking Glass to a keyboard shortcut for Gnome 3.2.

To install: download Quick Launch Looking Glass and install it using gnome-tweak-tool or by extraction it in:

~/.local/share/gnome-shell/extensions/

Followed by a restart of gnome-shell.

I haven’t been able to figure out how to add a new custom key binding for a gnome-shell extension so I’ve used the reserved “run command 9” by Metacity until I figure out something better.

If you haven’t bound it already - here is an example on how to bind it to the key F12:

gconftool-2 -s --type string "/apps/metacity/global_keybindings/run_command_9" 'F12'

If the extension is enabled then Looking Glass will open when you press F12 (just close it by hitting Escape like normal).

Storing encrypted data in Amazon S3

I have a lot of digital photos which I would be very sad to loose. I was recommended Amazon S3 as a cheap storage solution and even though it is possible to use https to transfer your data to S3 in a secure way the data itself isn’t encryptet within Amazon. I’m looking for a solution where my photos of my son in the bathtub are secured from all prying eyes including the Amazon tech staff.

It is possible to implement encryption transparent on Linux with the following two modules:

• s3fs - FUSE-based file system backed by Amazon S3
• encfs - EncFS provides an encrypted filesystem in user-space

First you need to signup for Amazon S3 - remember you pay on for what you use.

Then create a bucket, I used the Getting Stated Guide that Amazon provides. The following example will use the bucket name: your.bucket

Get an “Access key” from the AWS Management Console, I just used the one which was created upon AWS account creation.

Then I installed s3fs following the instructions on the project website.

Put the “Access key” inside the file .passwd-s3fs in your home directory using the format accessKeyId:secretAccessKey. You can find more informaiton on the on s3fs wiki page.

Now mount your S3 storage using:

mkdir /media/s3
s3fs your.bucket /media/s3

Now try create a file within you storage in the cloud ie.:

touch /media/s3/test

After creation of the file check that you can see the file using the AWS Management Console. Permissions, timestamps etc. is stored in metadata in Amazon S3.

Now in order to apply a transparent encryption layer encfs needs to be installed. Using Ubuntu you do it like this:

sudo aptitude install encfs

Now apply transparent encryption by mounting /media/s3 thorugh encfs like this:

encfs /media/s3 /home/je/Pictures/Encrypted\ on\ S3

and follow the on screen setup. The setup process is only triggered the first time you mount a directory using encfs and it results in a XML filen with all your choices. Don’t delete the XML file (perhaps take a backup) and remember you encfs password.

Now try create a file within:

/home/je/Pictures/Encrypted on S3

And verify that is is unreadable through the AWS Management Console and the directory:

/media/s3

Thats it :)

To expose this to Windows clients then install Samba and share the directory:

/home/je/Pictures/Encrypted on S3

NO WARRENTY: I’m NOT an expert in encryption so I cannot guarantee that encfs is secure enough. Neither do I know how much overhead s3fs and encfs puts on top the actual data that you tranfer to S3.