Roam notebook

Five minutes a day, two days per year

2010-07-25T15:20:30Z

People often hesitate to automate tasks. They reckon that that five-minute-once-a-day-thing doesn't really eat into their time and isn't worth the trouble of getting something else to do it for them instead.

Five minutes a day. It would mean not rushing to pick up the kids from school. Laying in bed five minutes longer each morning. Doesn't that sound good? It certainly does to me. Or look at it another way: five minutes per working day adds up to about 20 hours per year, or about two and a half days of working. Wouldn't it be great to have an extra two days to do some real work, or two more days off?

Decide what's important to you

Do yourself a favor and take some time to list your working day and look for the "low-hanging fruit". Write down all those dirty, boring little tasks and when you're done, get a developer you trust to have a look at it and let him pick out the stuff that can be automated with ease.

Before you know it you'll have something cheap that can perform those tasks by itself, leaving you your precious time with family and friends, doing more interesting things.

The value of one-man-shops

2010-07-24T18:20:30Z

Businesses often shy away from hiring freelancers for a job. They might have valid reasons to do so, but as a freelancer myself, I mostly see benefits to hiring people like me.

You see, a freelancer doesn't have to rely on someone else to tell him exactly what you, the customer, expects to get out of the project; he's right beside you every step of the way. He knows. And that's just the start. Contrary to big companies you know exactly who's working hard on delivering the goods. Got a question? Get an answer directly from the source. Got a request? Same thing. There's no need to do the hokey-pokey and wait for your contact to get the time to have a look at your e-mail, pass the question on and finally get back to you 2 days later with an answer.

It all boils down to this

You could write a book about the advantages and disadvantages of hiring a freelancer to build your next website or web application, but I feel the entire discussion can be reduced to the following images.

The freelancer

A freelancer is smaller, nimbler, more "agile". He can park in front of your door, load your stuff in the back and deliver the goods in a timely fashion. Like the driver of a van.

The company

The company more closely resembles a truck. Depending on the size of your business it's already a bit of a hassle to get them to come over and find a spot and you better have something large to put in the back or you're not even worth looking at. Of course, it is a bit pricey since there are more mouths to feed.

The choice

Got something big? Then only a truck will probably do. But for anything that fits the back of a delivery van, just get someone with a delivery van to help you out. It's faster, it's cheaper. If you're not sure about the size of your ~~load~~ project, ask a person who's judgement you trust to tell you what he thinks about it. Hell, ask me.

Django 1.2 upgrade: unique_error_message

2010-06-19T11:03:21Z

While upgrading a project from Django 1.1 to Django 1.2 everything went flawless. Well, expect this one thing...

In Django 1.1 you could customize the error messages for unique constraints over multiple fields by overriding unique_error_message in your form. In Django 1.2 much of the validation has been moved to the model, including the unique_error_message checks.

Using Django 1.1 you'd use following code in your form class:

def unique_error_message(self, field):  
    if field == ('field1', 'field2'):  
        return "You should not do that"  
    return super(MyForm, self).unique_error_message(field)

In Django 1.2 you'll have to change this to the following code in your model class:

def unique_error_message(self, model_class, unique_check):
    if unique_check == ('field1', 'field2'):
        return "You should not do that"
    return super(MyModel, self).unique_error_message(model_class, unique_check)

DRY KISS: django-stdfields

2011-11-13T15:30:45Z

Because Django doesn't really have a model field to track a duration and I always end up inventing something new to make working with choices on a field easier, I set out to finally build a solution that works for me: django-stdfields.

Duration in Django

When you need to record a duration in Django, you're out of luck using a standard Django model field. The DateField, DateTimeField and TimeField do exactly what they are supposed to do: record a date, date and time or only the time.

Instead, you could use the model field fields.MinutesField from django-stdfields. It's actually a simple extension of Django's models.PositiveIntegerField accompanied by a forms.MinutesField that will allow input in a format most users can understand. Instead of other existing third-party solutions, the MinutesField does exactly what its name suggests: track minutes, nothing more.

Users can input 8:30, resulting in 510 minutes. They can input 8.5 for the same result. Or even 8,5. Any integer value is interpreted as hours spent, any decimal value is interpreted as hours and part of an hour spent and anything of the form hh:mm is interpreted as the number of hours and exact number of minutes spent. It's basically the way you enter time spent on a task in Basecamp.

The tricky part is displaying those minutes back to your users. It's far easier to interpret 8:30 than 510. Simple solution: use the included minutes filter:

{% load stdfieldstags %}
...
{{ task.time_spent|minutes }}
...

When task.time_spent equals 510, this template will print 8:30.

Choices in Django

The choices argument for fields is great when you're dealing with a fixed number of possible values. The thing is... I always need to start messing with the constants further down the line. And then I need to get the label of the constant. I hate it because it gets gradually harder to find out what the hell you were doing two weeks later. The solution: enumerations.

Django-stdfields contains an easy way to create enums:

from django.db import models
from stdfields.models import Enum, EnumValue, EnumCharField

class CardinalDirection(Enum):
    NORTH = EnumValue('N', 'North')
    EAST = EnumValue('E', 'East')
    SOUTH = EnumValue('S', 'South')
    WEST = EnumValue('W', 'West')


class Wind(models.Model):
    direction = EnumCharField(enum=CardinalDirection, 
                            max_length=CardinalDirection.max_length())
    force = models.IntegerField()

You could still use a regular models.CharField and the choices argument. It's basically the same thing the EnumCharField and EnumIntegerField do behind the scenes.

But thanks to the Enumeration base class, things get easier to manage when you have to change stuff around (note: all fields can be used with South).

from django.db import models
from stdfields.models import Enum, EnumValue, EnumCharField

class CardinalDirection(Enum):
    NORTH = EnumValue('N', 'North')
    NORTHEAST = EnumValue('NE', 'Northeast')
    EAST = EnumValue('E', 'East')
    SOUTHEAST = EnumValue('SE', 'Southeast')
    SOUTH = EnumValue('S', 'South')
    SOUTHWEST = EnumValue('SW', 'Southwest')
    WEST = EnumValue('W', 'West')
    NORTHWEST = EnumValue('NW', 'Northwest')


class Wind(models.Model):
    direction = EnumCharField(enum=CardinalDirection, 
                            max_length=CardinalDirection.max_length())
    force = models.IntegerField()

We just added northeast, southeast, southwest and northwest to the possible values. Did you notice that this also changes the maximum length for the direction field of Wind?

Well, it doesn't matter: we're using the max_length method provided by Enumeration which will automatically determine the maximum possible length of the values — in this case 2. Just don't forget to create and run a schema migration with South.

But Enum might not work entirely as expected: when it's constructed, all contained EnumValue fields will be turned into regular fields:

>>> CardinalDirection.NORTH == 'N'
True
>>> CardinalDirection.NORTH_display == 'North'
True
>>> CardinalDirection.all()
[('N', 'North'), ('NE', 'Northeast'), ('E', 'East'), ('SE', 'Southeast'), 
('S', 'South'), ('SW', 'Southwest'), ('W', 'West'), ('NW', 'Northwest')]
>>> CardinalDirection.max_length()
2
>>> CardinalDirection.as_display(CardinalDirection.NORTH)
'North'

That's the gist of django-stdfields. It will save me a lot of time and I hope it does the same for you. Get it from Pypi or Bitbucket.

Converting a WordPress blog into a static site

2011-10-03T13:13:34Z

My personal blog at el73 hasn't seen an update in years. Constantly upgrading WordPress because of a newly discovered security hole on a website I no longer publish to feels a lot like wasting time.

That's why I decided to finally remove the WordPress parts from the equation and turn the blog into a static, archived site. Two tools made this conversion easier than I could have imagined from the start.

Wget

The first step was to archive the entire published site. With a little help from wget I got all of around 500 pages downloaded in a little less than an hour, thanks to Peter Upfold and this command: $ wget -mk -w 20 http://el73.be/

Cleaning up the HTML

So now I've got a local mirror of the site, but I need to change some things around. Since I'm not planning on keeping WordPress around, the form to add a comment will have to be removed. Plus there's a link to search the website's contents in the navigation. Since that's also powered by WordPress, it'll have to be removed as well.

My first attempt to clean up the HTML consisted of processing all the HTML files and removing those two parts using regular expressions. Now, I'm no command line guru chaining 5 separate commands into a single perfectly working combination of *nix glory. And as it turns out, OSX doesn't support the full range of arguments offered by grep on other systems such as Linux anyway. Sure, I could have delved in and wasted a lot of time on getting this to work, but it would be just that: wasting time. I'm not planning on doing this again anytime soon, so whatever works best and right now is going to get my vote.

As it turns out, PhantomJS is what worked best for me. PhantomJS allowed me to use a simple script that would walk the entire mirrored site and collect the HTML files. After that, it was a matter of loading each HTML file, removing the elements I wanted removed using plain DOM operations and saving the content back to file.

var cleanUpHtml = function(fp) {
    // Open the file at the given location
    console.log('Cleaning up HTML in ' + fp);
    var address = 'file://' + fp;
    console.log('Opening file at ' + address);
    var page = new WebPage();
    page.open(address, function(status) {
        if (status !== 'success') {
            console.log('Unable to load ' + fp);
        } else {
            var innerHtml = page.evaluate(function() {
                // Remove the comment form
                var el = document.getElementById('commentform');
                if (el) {
                    el.parentNode.removeChild(el);
                }
                // Remove the search link
                el = document.getElementById('nav');
                if (el && el.tagName == 'UL') {
                    var lis = el.getElementsByTagName('LI');
                    if (lis && lis.length == 4) {
                        el.removeChild(lis[3]);
                    }
                }
                // And return the inner HTML of the root
                return document.documentElement.innerHTML;
            });
            // Wrap the inner HTML
            fs.write(fp, '<!DOCTYPE HTML><html lang="nl">' + innerHtml + '</html>', 'w');
            console.log(fp + ' processed');
        }
    });
}

Upload... and profit!

So that's how I could turn an entire website backed by WordPress into a site consisting of nothing but static assets in about 2 hours of work, not counting the time needed to download and process the pages. Upgrading to the next two patched versions of WordPress would probably cost me the same or more.

Introducing Extractor

2011-07-30T19:58:30Z

Sometimes you're faced with an almost impossible decision, trying to make sure all your users are happy with what they're working with.

One such decision is where to place the comments and remarks on something crucial. Say, a chapter in a book review system. Place them on the bottom and you have to scroll back and forth between the comments and the text. Place them at the side and you're going to have to create a layout that makes this possible without hindering people working at lower screen resolutions. Thanks to Extractor there is another option: show the comments at the bottom, but let the user pull them out into a dialog, backed by jQuery UI. That way they can move the dialog around the page and resize it to their liking.

But...

Once you have loaded content into a dialog, you can't really get it back out. jQuery UI will move it to the bottom of your body element and often that's not what you'd like to see happening.

And that's where Extractor steps in: instead of simply letting jQuery's dialog do it's thing, it inserts a placeholder for your content and, when the dialog is closed, places the content right back where it belongs.

Have a look at the demo and grab a copy of Extractor.

Like jQuery UI? You'll probably love WidgetThemes, our new project that delivers high quality jQuery UI themes.

Writing tests is not a waste of time

2011-03-26T08:23:43Z

When providing estimates on a project I always include the time needed to write up the tests because they are often necessary to provide a quick turnaround on changing requirements and inevitable bugfixes.

Some managers and customers would prefer software to work as intended from the get-go without tests, but, unless you're developing the simplest of applications, tests are an absolute necessity. Simply telling your customer that "tests are necessary" probably won't do, so here's a list of reasons why they are important that I often employ.

Requirements change

Sure, the format of the files you need to process each night isn't going to change. Until the release is two days away and it turns out the format has changed from CSV to XML. With adequate tests in place, you can replace the file parser and know almost exactly where you stand without having to wait for the application to be tested by a human.

Of course, this does mean you'll have to change the tests for the file parser, but isn't that the easiest way to find out that your parser actually works?

During a project at the Belgian Post Group (now bpost), we used each new version, each change in requirements, of a pretty complex application to add more and more tests. The number of issues to pop up during user acceptance testing went down. Way down. So far down in fact that we had to find other things to do in the days leading up to the release, while a few months earlier we'd still be squashing bugs left and right.

Plus: the testing framework we developed during that project served as a baseline for Unitils, proving that not all testing code you write needs to be redone in your next project.

Code gets ugly

No matter how hard you try to think about the code you're going to write, at some point you'll start to notice duplicate code that can be merged into a single function and weird constructs that can be simplified from 200 lines to 10 lines. Tests have your back and make sure you can merge and simplify without breaking the app.

Imagine a piece of logic — let's say some kind of calculation based on government regulation — that's already been tested by the same person five different times. They will get sick of it and they will start to skip the testing. Why? Because it all worked up till now, so why would now be any different? And that's when you come along. You find out the code you've written three weeks ago in part of the calculation is actually the same as the code you've just finished writing. You merge both parts into a single function, but forget about that single tiny special case you needed to take into account.

Boom! Your new report works perfectly, but your calculation is broken. Chances are nobody will find out until it's too late.

Recently, I found out code I had been working on for days failed in some scenarios I hadn't even considered up till then. By the time the first fixes were applied, the code had grown out of control. And then another bug popped up.

I decided to add the newly discovered scenarios to the existing tests and rip out the entire part. I rewrote it to spec (i.e. the tests) in under 4 hours.

Tests are more valuable than comments

When you're the kind of developer who reads the comments detailing what a method does and how it will respond to certain input, you've probably seen your fair part of paradocs: the documentation accompanying the signature says "A" while your experience seems to indicate it's actually "X".

Given clear-cut test cases you'll soon find out whether the original developer (maybe it was you):

changed the code and forgot to update the docs; or
wrote buggy code and wrote the docs without verifying the code responded as expected

Tests clarify how you expect your code to be used.

Tests can speed up development

Don't believe me? In a current project we're using JBPM to embed workflows in a webapp. Using JUnit I can make sure that the implemented workflow works as expected. This means an email is sent out when expected to the people I expected. This means the Send out order step will always occur after the Verify payment step. This means I'll always have a quick comeback to an issue logged by a tester or customer titled "The workflow doesn't work as expected".

Besides, would you rather compile the entire application, start Tomcat, wait for the application to load and then start clicking away or would you prefer to hit the "Run test" button in your IDE and have the results displayed to you within seconds?

Here's another benefit: the next time your customer asks you whether it would be possible to add an extra step to the workflow and how much time and money it will cost, you'll open up your test cases and have a look. Maybe you'll add the step right away, to see how your tests will react. Imagine, just imagine, responding to the customer's request with a simple "Done!" instead of a long winding response detailing why you'll need 10 days to include a Review order step.

Tests are no magic bullet

Unfortunately, someone still has to make sure your web application works in Internet Explorer 7 and the "Log in" button is visible on an iPad. But that needs to happen less and less, while you add more and more tests.

Just make sure the things you're testing are worth it. When 2 + 2 no longer equals 4, you'll have bigger fish to fry.

jQuery dialog: keep your hands of my content!

2012-04-07T11:10:10Z

People who are new to jQuery UI often are surprised when they open a dialog using a piece of HTML that's on the page. Where's your HTML? What's happening? How do you prevent it from happening?

Poof! Gone

When you generate a dialog from part of your webpage, you'll notice that when you open the dialog that text disappears from the page and shows up in your dialog. Once you close that dialog, however, your text is gone.

jQuery UI actually takes the content you want to create a dialog from, wraps it in all kinds of divs to make sure it renders correctly and then places it at the bottom of your page. Hidden. Until you either open the dialog or the dialog auto-opens.

So how do you prevent that from happening? How do you keep the content on your page without jQuery UI moving it around?

Option 1: use jQuery Extractor

If you actually want to remove the content from the page, but you want jQuery UI to put it back when the dialog is closed, use jQuery Extractor.

This scenario is exactly why I created jQuery Extractor in the first place.

Option 2: copy before you pop

So you've got this HTML you want to load into a dialog.

<div id="mydialog" title="My dialog!">
    Oh, hi!
</div>

The easy way that does not work if you want to get your content back:

$('#mydialog').dialog();

So why don't we just make a copy of the HTML and load that? Now, be careful! Like we said before: the HTML is actually still part of the page. So when you're using an id attribute to identify the part you want to load, you'll get multiple elements with the same unique identifier. That's no good.

We've got to think a bit ahead. Let's adapt the HTML.

<div id="mydialog-wrapper">
    <div title="My dialog!">
        Oh, hi!
    </div>
</div>

We introduced a wrapper we can use to identify the HTML fragment we want to display in a jQuery UI dialog.

var dialogHtml = $('#mydialog-wrapper').html();
$(dialogHtml).dialog();

And that's the final part of the puzzle. We take the contents of our wrapper and wrap that HTML in a jQuery object which comes down to making a copy of that HTML. Then we call jQuery UI's dialog method on that instead of the original.

Like jQuery UI? You'll probably love WidgetThemes, our new project that delivers high quality jQuery UI themes.