A quick dive into Darktable

Full disclosure: I am not a professional photographer or graphic designer. 100% hobbyist.

That said, I do try a fair amount of applications for photo editing and use a fair amount of the tools they offer for image manipulation. Consider my thoughts as coming from the perspective of a hobbyist interested in getting great results (by my standards) with a relatively small investment in learning new software. I'm also use to working with Adobe Lightroom and have a Canon Digital Rebel XTi.

Over the past year I've switched from being a lifetime Windows user to an Ubuntu user. Since the switch I've been desperate for good photo editing/management software. Previously I'd used Lightroom. I love love love Lightroom. I had a few options to continue using Lightroom. I tried going the Windows virtual machine route but I found its responsiveness to be too slow. I felt like the dual-boot route was too much effort. File that complain under First-World Problems. I like my desktop setup and I don't want to reboot my computer just to use one app. I'm lazy.

Finally, I like to not spend money. Free software really appeals me. This lead me to look for Lightroom alternative that would work with Ubuntu and ideally would be free.

The post isn't about my application selection process but I did try Rawstudio and RawTherapee before choosing Darktable. I will say my preference for Darktable was immediate, though there's a few things to know to dive in and get to the "learning by using" part.

Keyboard shortcuts

I like shortcut keys. If you also like shortcut keys and are use to Lightroom you'll find that some shortcuts are the same but a lot aren't. If you want to see what the shortcut keys are and even change then to whatever you like here's how:

I also work on a laptop with a touchpad. You have a lot of zooming control with your scroll wheel, but I don't have one. Without a scroll wheel you need to rely on Alt+1. Note, if you hit it twice it zooms in twice. That is not obvious from just seeing the binding on the shortcuts tab.

Plugins

Each type of adjustment you can do to an image is broken into a plugin. Select a photo and go into Darkroom mode by hitting D. At the bottom right there is a "more plugins" button. Hover over the plugin to find out what the plugin means because the icon does not make it obvious at all. This process annoys me as it takes a fair amount of time just to know what your options are. Clicking on a plugin will turn the icon background grey indicating that the plugin has been added the the right panel. It will be under whatever the plugin's default category is. Clicking the icon twice will turn the icons background red. This means the plugin has also been added to the "favorite" category in the right panel. Click a third time to remove the plugin from the right panel.

Generally I use a few plugins and I use them a lot. My preference was to only add those plugins to the panel and make them all my favorites. This makes it a lot easier to use without having to keep switching between these kinda of poorly defined categories.

Time to play

With my shortcuts ready and plugins out where I can see this its easy to play around and make that transition from Lightroom to Darktable. Darktable doesn't have all the feautres and the easy of use that Lightoom has, but its still damn good and its free. I am extremely excited to see how this application will evolve over the next few years.

PCI DSS compliance and spaghetti code, Part 3

(Be sure to read Part 1 and Part 2 first!)

When writing new code I find its a good idea to start at your integration point and then write that client code, whether just by psuedo-coding or writing some test cases. For my payment gateway code I need code that takes user input, builds a request from that input, sends that request to the payment gateway, and gets a response back so we can act accordingly.

$factory = new RequestFactory();

// get sale builder
$builder = $factory->buildSaleRequest();

// get credit card, so we can set it
$builder->withCreditCard() 
    ->setCardType($_POST['cardType'])
    ->setCardNumber($_POST['cardNumber'])
    ->setExpirationDate($_POST['expirationDate']);

// get address, so we can set it
$builder->withBillingAddress()
    ->setCity($_POST['city'])
    ->setState($_POST['state']);

// send request to payment gateway
$response = $builder->execute();

This seems pretty simple. I have objects to accept my user input which execute the request and get the response. Marvellous. Under the hood its slightly less simple if you aren't particularly familiar with the Factory pattern, Builder pattern, Gateway pattern, Composite pattern, and Visitor pattern.

Today I'll start at the surface with our creational patterns being used here, the Factory and Builder pattern.

Factory pattern

Factories, like all other creational patterns, create new objects. At first they are often understood as a way to make its consumer get different types of object, which is correct, but people fail to see that they are places to return differently configured objects (potentially of the same type) as well. Factories are the place to stowaway configuration information. In Factories the configuration can be centralized, rather than forcing the consumer code to have to repeat configuration each time. For example:

class CarFactory {
    public function create() {
        $car = new Car();
        if($this->config->useElectricCars()) {
            $car->setEngine(new ElectricEngine());
        } else {
            $car->setEngine(new EnvironmentDestroyingBeastEngine());
        }
        return $car;
    }
}

We encapsulate the querying of the "config" object into this Factory method so we don't need to have other parts of the system be aware of this "config" or aware of any configuration of this object at all.

In the case of the payment gateway RequestFactory, we'll be grabbing the payment gateway credentials, which get used in each request, and passing them onto the Request object that gets returned for consumption. This way other code need not worry about where those come from. We've made that decision and we've encapsulated it in a Factory.

Builder pattern

Most people I've seen learning about the Builder pattern don't really get it for a while. This was true for myself as well. We know what it is basically - a class with a bunch of methods you can call to configure an object and then a method to return the newly created and configured object - but don't know when to use it. A quick look at the pattern summary talks about varying complex creation processes. Yea, it can do that, but you have good reason to create those Builders even if you need only one implementation now.

Here are a few indicators to help you know when its time to use a Builder.

Indicator #1: Wiring together collaborators in application code

Are you doing this in your application code? You've written a library to create cars and car parts and everywhere you use this library you're repeating this process...

$carPartFactory = new CarPartFactory();
$carFactory = new CarFactory();

$car = $carFactory->create();

$car->setEngine($carPartFactory->createElectricEngine())
    ->setWheels($carPartFactory->createGoodYearTires(4))
    ->setStereo($carPartFactory->createBadAssStereo());

You're creating more than one factory to wire together collaborators. Its Builder time, bitches.

$carBuilder = new CarBuilder();

$car = $carBuilder->addElectricEngine()
                  ->addGoodYearTires(4)
                  ->addBadAssStereo()
                  ->build();

Its easier to use, less duplication, easier to read, and if you ever need to vary that creation process its already in place.

Indicator #2: Factory method with optional parameters

Optional parameters are terrible. I think thats a whole post al by itself. For now, if you find yourself doing this...

public function create($first, $last, $middle = '', $prefix = '', $suffix = '')

...do this instead...

$builder->first($first)
        ->last($last)
        ->middle($middle);

Builders in Builders

Final note of the day on Builders. Object chaining can make usage pretty easy. We get use to returning $this in each method. Sometimes though we want the Builder to return another Builder which can configure its collaborator. In this case we return the Builder instead of $this, but its also good to have a method naming convention to let developers know instinctively which they'll get back.

Personally, and I think I picked this up from a book though I can't remember which one, I use "set" or "add", whichever reads best, when I'll be returning $this, and "with" when I'll be returning another Builder. Sometimes I might even have a "set/add" method and a "with" method for configuring the same option, where the "set/add" just uses default settings and the "with" method lets you dig in, but only if you want to.

// using "add"
$builder->addGoodYearTires();

// using "with"
$builder->withGoodYearTires() // returns TireBuilder
        ->setFancyRims()      // call TireBuilder methods
        ->setSnowTires();

class CarBuilder {
    public function withGoodYearTires() {
        // keep reference to builder so we can get its "product" 
        // and add it to the Car we're building
        $this->_tireBuilder = new TireBuilder();
        return $this->_tireBuilder;
    }
}

Stay tuned for Part 4...

Guitar Hero-like scrolling music on Ubuntu

This year I began to learn to play the drums. Its been fun but I am not the most dedicated student. Its a hobby and I slack off a lot, but I really would like to get better.

I first got interested in the drums when I tried them out on Guitar Hero. It was fun playing with friends and you could play for hours without thinking about it. You'd see yourself getting better and more comfortable. You'd notice what got tired or sore after playing for 3 hours so you'd adjust or do some Google-ing and find out how real drummers do it.

Guitar Hero certainly doesn't have the capacity to teach you like a human teacher could. It doesn't how the technique you play with, and that's a biggy. On the other hand, real teachers cost money and scheduled time, which at this point in my life is more than I want to give. Over the weekend I set out on a hunt for something like Guitar Hero with a few extra requirements.

• Work with a Alesis DM6 electric drum kit
• Runs on Ubuntu
• Can load songs of a format which can be easily found for free online (ex. MIDI)
• Scrolls music vertically (like Guitar Hero)
• Rates your ability to play with the song (like Guitar Hero)

(Note: I realize some electric kits can be connected to Guitar Hero directly but mine cannot. It has a USB-out, not MIDI-out.)

Synthesia via PlayOnLinux

My search brought me through a graveyard of abandoned projects but eventually I found Synthesia. Synthesia, as advertised, is almost what I want. Its actually made for learning/practicing piano, but it works off MIDI tracks, so theoretically it can work with other instruments as well.

Synthesia does not have an official Linux version. There is a forked Linux version, Linthesia, but it doesn't have a 64-bit version and Synthesia's wiki noted it was old and missing features. For a while I dug through discussion forums talking about how they managed to get Synthesia to work with Wine, but experiences seemed to vary and honestly I didn't want to dig in that deep.

Luckily I ended up stumbling upon the PlayOnLinux which, just by chance, added support for Synthesia two weeks ago. PlayOnLinux is a tool which quickly configures Wine for many Windows games and applications. I followed the Natty installation instructions and got PlayOnLinux installed without any issue. I then followed these instructions to install a PlayOnLinux application, in this case Synthesia. Lastly, I tweaked the Wine settings through PlayOnLinux to have Synthesia run in "windowed" mode by following these instructions.

From here Synthesia worked immediately with my wife's MIDI keyboard. Some articles suggest installing Timidity, which may be necessary in some use cases but not for mine. I simply had the keyboard input set to the MIDI keyboard, and the output also set to the MIDI keyboard. I had installed Timidity but it didn't seem to be necessary and it crashed once for me so I removed it.

Is that it for drumming with Synthesia?

No.

This is where I've left off. Synthesia registers my drum hits and can output to my drum kit. I tried downloading MIDI files with percussion tracks and tried playing a long. None of my hits seemed to match the drums Synthesia wanted me to hit. I believe this is a MIDI note mapping mismatch. For example, my snare drum produces note 38, says QMidiRoute. I suspect the MIDI track I was playing to did not have the snare drum sound as being note 38. Overall I'm not really familiar with MIDI at all so this could be a completely ignorant assessment but for now its what I'm working on. When/If I find a good solution I will post it.

PCI DSS compliance and spaghetti code, Part 2

(Be sure to read Part 1 first!)

The first iteration of my Command objects could continue to use our existing payment gateway code, but this would not be acceptable by the end of this project. This old payment gateway code relied heavily on global state to get configuration options and it executed database queries. If that were allowed the database servers which our CDE web server accessed would also be subject to audit. On top of that this code was poorly structured and lacked much of the functionality needed to complete the project.

Completely replacing a fairly large chunk of code is often a bad choice, but we had a mound of reasons for making the move.

Structuring payment gateway code

I worked on this project in parallel with one other teammate, Xiao. While I was tearing through legacy code and building command objects Xiao, wrote all of the new payment gateway code I'm about to describe. I helped him with up-front architecture consulting, code review throughout the development process, and some pieces of refactoring.

Each payment gateway we needed to support had a name-value-pair API (such as PayPal's) or an XML API (such as Authorize.net's). The data used to create each request was usually a combination of several complex objects: merchant account information, name information, addresses, "payment profile" information, etc. These objects created a Composite pattern. We used a Visitor pattern to descend into the Composite and build our final object in the format in which it would be send to the payment gateway.

This might seem counterintuitive because the structure we area creating didn't match their API structure at all. For example, an API that used name-value pairs might include an address. This address wouldn't be a separate structure that, as a whole, is part of our name-value pairs. The address parts (city, state, zip, etc) would be individually mixed up with name, credit card number, expiration date, and so on, because the name-value pair structure is completely flat.

The benefit to having a composite structure with strong typing is that it makes it impossible to make an invalid request. Each structure, like the address, has a set of fields you can define. Every place an address can be added, its specified by type. Passing an object of the wrong type causes an error instantly - do not pass GO, do not collect $200. This kind of defensiveness seems pretty important for a system as critical as this one (one that receives money).

Continue to Part 3!

PCI DSS compliance and spaghetti code, Part 1

Anyone taking a substantial amount of credit card payments will eventually stumble onto the Payment Card Industry Data Security Standard (PCI DSS). Meeting this standard likely adds significant complexity to your application, comes at fairly great expense, and will leave your developers and sysadmins with at least a few head-scratching moments where as they ask themselves "Why do I have to do this? It has nothing to do with security." The project I'm working on has been audited, but the final paperwork is not in yet. Things look good, but pass or fail, there's still plenty to learn from the experience already.

Minimizing the PCI DSS burden

My employer's web application has a fundraising module which needed to become PCI DSS compliant. There are many things we considered when deciding how to modify our fundraising module. Here are a few of the larger concerns:

• Hardware which handles credit card numbers is subject to audit (these systems combined are called the Cardholder Data Environment, or CDE)
• CDE hardware changes must be documented
• CDE code changes are subject to audit
• CDE code changes must go through a documented security-focused peer review
• Quality testing must be documented for all CDE code changes
• There are restrictions on who is permitted to deploy CDE code

As you can see, maintenance to a CDE has lots of extra overhead. It didn't make sense to add that to overhead anything other than our fundraising module so we decided to separate it from the rest of our application code as well as serving it from its own cluster of web servers. This solution also reduces the attack surface area of this sensitive part of the application.

We would obvious have to make some changes that would cost us this extra overhead but we realized we could keep that cost low by reducing the need for change within the CDE code. To do this we tried to keep the CDE code... well... as dumb as possible. Our application has many customizable types of fundraising forms used in various workflows, but we don't want the CDE code to know about that. That stuff is complicated. It changes, grows, and has a higher bug potential than something smaller and simpler. All that complexity should continue to live within our original application which the CDE code can communication with via a simple API. This sounds great, but how do we rip our application in two?

Plotting the course

Segmenting our application this way had huge implications for our existing fundraising code. The fundraising code had feature after feature piled on over 5 or 6 years. The user workflow stumbles through a 5000-line top-down PHP file of spaghetti code. Eventually other "helper" functions and classes might output to browser, redirect, or charge a credit card and then terminated the request with an exit() call. Eww! This code was so complex that it was hard to get a good idea of what all of it did. Without structure or comments it was unreadable and there was no test coverage.

When I first approached this problem I looked at in a very linear way. This is roughly how I saw the workflow for a basic fundraising page with all valid data submitted and no credit card processor errors:

1. User opens fundraising page on web server (not a CDE web server)
2. User submits form which posts data to CDE web server
3. CDE code pulls out credit card number and other sensitive data and validates it
4. CDE passes the validation results and non-sensitive data to a validate function on the web server API
5. Web server validates the rest of the posted data using some validation extracted from the spaghetti code
6. Validation passes so some other extracted code decides to return a "sale command" to the CDE (see Command Pattern)
7. CDE executes the command
8. CDE calls web server API telling it the transaction was successful
9. Web server uses some extracted code to determine if the user is to be shown a "thank you" message or redirected to a "thank you" page (each of which could be different depending on the workflow the user was executing)

All this code extraction was going to be a real pain. The validation and form building was tightly coupled using an old Pear library called QuickForm. The various user workflows were tangled together and extremely overcomplicated. Making a few huge tears in the middle of the application seemed high-risk given our relatively short timeline.

A-ha!

I soon realized this wasn't necessary. I didn't need to do all this extraction. I didn't need to rip this thing apart with a few huge, dangerous tears. What I needed to do was encapsulate those end points which triggered the output of data to the user's browser, a user redirect, or charging a user's credit card.

These encapsulated end points form Command classes which can be serialized and returned to the CDE in an API response. These commands are simple. Its not important what kind of page you are outputting, what kind of page you're redirecting to, or what kind of fundraising charge you're doing. The CDE code doesn't need to know. The CDE code just know how to handle certain sensitive fields, perform a credit card transaction, and execute these generic commands. All other decision making is delegated to the web server API.

For our first iteration I created these command classes and took all that structureless top-down spaghetti code and tossed it into a single class method (we'll call this class a Controller). I extracted global variable usage so their values could be passed in through the Controller's constructor. I also extracted the validation of the sensitive data and had it run outside the Controller with its results injected into the Controller (as would eventually happen with the validation running on the CDE web server). I used several legacy code refactoring methods described in one of my favorite programming books, Working Effectively with Legacy Code. These changes produced something I could release immediately.

Continue to Part 2!

Exceptions as part of "regular" control flow

I've heard this rule a lot: "Never use exception for control flow." Its an interesting statement to parse. We know what an exception is, but the definition of control flow is a little fuzzy, so lets clarify things a bit.

In computer science, control flow (or alternatively, flow of control) refers to the order in which the individual statements, instructions, or function calls of an imperative or a declarative program are executed or evaluated.

http://en.wikipedia.org/wiki/Control_flow

Immediately I'm confused. Is it possible to use an exception and not effect control flow? Nope. Exceptions are a means of controlling flow. Sometimes people making this point specify that its "normal" or "regular" control flow. "Normal" and "regular" are delightfully relative and unhelpful. Normal? Compared to what? What they're trying to get at is that they don't believe exceptions should be used unless there is an application error.

This seems odd considering that business layer classes aren't necessarily made for only one code path or even one application and therefore lack context to determine the exceptionality of the condition. I touch on this a bunch in my Exceptions vs Null article.

There are lots of "crazy" things you can do with exceptions and if you're interested in seeing more check out this article on c2.com. Here I will only be discussing three usages I found particularly interesting.

Exceptions as return values

This example comes from c2.com. I think even people who have never thought deeply about exception usage would never dream up this code. Still, I believe this might be the perfect example of what people are talking about when they say not to use exceptions for control flow.

void search( TreeNode node, Object data ) throws ResultException {
    if (node.data.equals( data ))
        throw new ResultException( node );
    else {
        search( node.leftChild, data );
        search( node.rightChild, data );
    }
}

Get it now? As a starting point I think we can all agree that this it batty. Its a search algorithm that throws an exception upon success. Really? The article correctly points out that this is a violation of the Principle of Least Astonishment. I know this code make me feel violated.

Exceptions as commands to the caller

} catch(MyService_Exception_CouldNotBeReached $e) {
    throw new MyOtherService_Exception_Retry("Couldn't reach my service, retry!");
}

This exception seems to be commanding the caller to retry something. Like the previous example, this also breaks the Principle of Least Astonishment. In order to benefit from the full functionality of the method you need to be setup to catch exception commands that it throws and follow out some other action to continue. This is a application-agnostic service making application-specific decisions (whether or not to retry). No thank you.

Exceptions as loop termination conditions

c2.com offers us another gem, and I don't mean that negatively.

try {
    for (int i = 0; /*wot no test?*/ ; i++)
        array[i]++;
} catch (ArrayIndexOutOfBoundsException e) {}

The first thing when I thought when I saw this was "What about things like Python's StopIteration?" One line later its mentioned. Yay! This article may be reading my mind. I do wonder if the fact that StopIteration exist in Python, Ruby, and Javascript now might start to carve away at exception/null failure/absence debate. I'd like to know more about the decision making that went into the the design of feature but so far have failed to find any discussions on the matter.

All that said, I'm not sure how I feel about the name - "stop" iteration. Sounds like the service commanding its caller. If I were to use a exception-terminated loop I'd prefer to explicitly specify the exception type rather than using a language generic exception with a name that is a command. Maybe something like this (which I would not be surprised to find already exists somewhere):

until(RecordNotFoundException $e) {
    print $recordSet->getNext()->getLabel() . "\n";
}

Categorizing exceptions: Subtypes vs Error codes

Today's problematic pattern: Catching a single, usually per-package, exception and then using an associated error code in a conditional statement to determine what else to do. I've seen several colleague programmers do it and they never seemed to think about doing it any other way. Its an oddly unconscious decision.

Consider the following code:

try {
    $http->get('/~bob');
    // do something
} catch (Http_Exception $e) {
    if($e->getCode() == 404) {
        // do something else
    }
}

When you use things like well-established HTTP status codes this almost seems reasonable. It gets a little weirder when you're talking about some internal code defined in some package no one knows anything about. Take this bank account withdrawal example:

try {
    $customerBankAccount->withdraw(100);
} catch (CustomerBankAccount_Exception $e) {
    if($e->getCode() == 123) {
        // 123 means they overcharged their account, SHIT!
    }
}

My problem with this is that we're using two tools for a single purpose. What is the purpose of an Exception subtype, Http_Exception or CustomerBankAccount_Exception, in these examples? Exception type categorization. What is the purpose of an exception code in these examples? Exception type categorization. Why are we categorizing the same exception using two different systems? Additionally, why would caller code be left to interpret magic numbers like 404 or some other constant value a programmer in your office came up with?

It can get worse. People create their own custom exception error codes with pre-specified ranges like code 100 to 200 are set aside for account balance errors and 200 to 300 are for "the bank spent all your money" -related errors. Once again, can't exception subtypes be used for this? The ranges are essentially more API for a developer to remember.

Better example time - GO!

This is more descriptive...

} catch (Http_Exception_NotFound $e) {

Sure, we all know what a 404 is anyway, but what about a 402 or a 203 or a 912 (Glenn Beck Logic Not Found)? If you're writing this code you're gonna need to figure out what you want to catch. The next guy reading through it probably doesn't want to figure it out and he'd appreciate it if was just looking at some plain English exception class names.

If you want to catch all those 4xx-class exceptions why not...

} catch (Http_Exception_ClientError $e) {

(The official category of 4xx errors is "Client Error")

I bet a lot of developers see this kind of problem and say "OH SHIT OH SHIT OH SHIT!!! I need to write a one line class with no body at all for like 20 exceptions, that's like 20 lines... OH SHIT OH SHIT OH SHIT!!!" I never quite get that. They would rather spend the time writing catch-blocks with if-blocks inside them that each duplicate some evaluation on a magic error code. It won't take very long for that try/catch/if/else boilerplate nonsense to make those 20 lines of exception definition code look mighty appealing.

Look what I found!

During my research for this post I stumbled upon discussions in many programming language communities about this idea that error codes can be used for exception message translations. "That sounds interesting," I thought. I've never considered that use before but as I read through a plan proposed to the Zend Framework I quickly began to dismiss the notion.

Thoroughly enjoy this proposal.

Currently, exceptions can only be handled on a per-class basis, or possibly by string comparison against the message.

Instead, we propose that exception codes be used throughout the framework, using a 4-byte hexadecimal format.

This has at least four obvious benefits.

First, it's now possible to distinguish exceptions based on the type of error instead of only by class. This allows users to handle them intelligently.

Second, codes let us do some interesting things with exception handling. Users would now be able to call a method such as:

if ($e->stringTooShort()) { 
    ... 
} 
if ($e->stringTooShort()) { ... }

thereby making exception special-casing easy.

Third, this gives us the ability to translate error messages using Zend_Translate (loaded only when an exception occurs) and using separate translation files (for example, .mo format). Not all developers speak English; those that do generally prefer to see exception messages in their native language.

http://framework.zend.com/wiki/pages/viewpage.action?pageId=22134

Who cares about point 4, I'm already gagging. Even the language irks me a bit. Just being able to do something doesn't make it "beneficial". A magical genie could come and grant me my one wish; the ability to shit dead baby seagulls. Its certainly something I couldn't do before, but is it beneficial? Maybe. I really hate baby seagulls.

I don't think I have to re-explain my feelings on Point 1 and 2, but whats the deal with lucky number 3? Oh, number 3. If you checked out the article you probably saw that each package in the whole framework has its own error code range preallocated. Its like preemptive namespacing for codes. No, it IS namespacing for codes. Yes, there's already a way to namespace exceptions; the same way we namespace any other class.

If the goal is to be able to translate any exception then we can't really do that with this approach. Other packages using the same error-code-to-string mapping scheme could overlap in code ranges, so we're at least going to have to catch each different base exception type for each package that has its own defined error code ranges. Then we're going to have to somehow map those to different translation files.

I'm not saying its necessarily easier to just map class names to exceptions, but I can't imagine it being harder. This new approach doesn't seem to be solving any problems and its creating a few new ones.

The proposal has tons of user comments. Some people pushed back. One user asked why PHP's Exception class took an error code in its constructor at all (a very good question). One of the proposal's authors answered:

The error code parameter is intended to be used exactly how we're using it: differentiating exceptions within exception "namespaces" (classes).

Sigh.

Finally, translations are for presentation. Are these presentation layer exceptions? No sir, sadly they are not. We're prepping our business layer exceptions with some data that is meaningless everywhere except the presentation layer assuming the presentation layer knows how to interpret the codes in the first place.

I don't know how to end this. I'd like to be shown something that throws my conclusions on its head, but from my reading thus far, I'm not seeing it. Feedback welcome.