19
Sep/11
0

Check out SQL::Abstract::Query

During the last few months I’ve been working on the CPAN distribution SQL::Abstract::Query. The drive for doing this is that DBIx::ResultSet (another module of mine) currently has lack-luster support for auto-incrementing IDs, has a custom and inflexible solution for GROUP BY, and has no support for JOINs.

To solve the first problem, with lack of good auto-incrementing ID support I wanted to make a generic module that would allow anyone to retrieve these IDs and could be used independently from DBIx::ResultSet. So, I set out to build DBIx::AutoID. I quickly found that there is a huge myriad of complexities to do this right for all the different databases out there. In the end I came to the conclusion that I need some way for an INSERT to be represented as an object so that DBIx::AutoID can modify it, if it needs to, before it is executed so that the ID may be returned.

The second two issues, bad support for GROUP BY and none for JOINS, really just required extending SQL::Abstract and overriding its behavior.

So, take a look at SQL::Abstract::Query on CPAN and let me know what you think. I’d really like to get some feedback. There is a large list of TODOs, but I’m confident that the interface as it is now won’t change much, just the internal guts need some work here and there.

Expect DBIx::AutoID (with inspiration from DBIx::Class) to be on CPAN in a few weeks as well. Then, I’ll get around to updating DBIx::ResultSet with all these new and more powerful features.

PS: The current development work is being done on the SQL-Abstract-Query github repository.

16
Aug/11
5

Using DBI Effectively: bind_columns()

One of the most under-utilized features of DBI is is the bind_columns() method. The majority (as in everyone) that I show bind_columns() to have never seen it before or used it. Here’s what it looks like:

my $sth = $dbh->prepare('SELECT name, email FROM users');
$sth->execute();
$sth->bind_columns( \my( $name, $email ) );

while ($sth->fetch()) {
    print "$name: $email\n";
}

bind_columns() takes a list of scalar references as its arguments. The form you see above with \my(…) is just a little known Perl-ism shortcut for:

my ($name, $email);
$sth->bind_columns( \$name, \$email );

There are two important reasons why bind_columns() is so awesome:

First, it greatly reduces the complexity of the code within the while() loop since you do not have to lookup in to an array ($sth->fetchrow_array()), de-reference an array ($sth->fetchrow_array()), or de-reference a hash-ref ($sth->fetchrow_hashref()). Instead the values themselves are available via appropriately named scalars.

Secondly, when using bind_columns() DBI is re-using the same scalars every time a fetch() is done which is much faster than creating an array or hash every fetch and typically causes the values to be copied one less time than normal. Benchmark it yourself – bind_columns() can make a huge difference when processing large sets of data.

2
Aug/11
3

Chained Git Hooks

So I’ve be wrestling with getting chained git hooks working. I’m surprised that Git doesn’t support this out-of-the-box as it seems very short-sighted. Here’s a Perl script I implemented to get this working:

#!/usr/bin/perl
use strict;
use warnings;

use autodie;
use File::Temp qw( tempfile );
use IPC::Cmd qw( run );

if (@ARGV and $ARGV[0] eq 'wrapper') {
    shift( @ARGV );

    my $hook_filename = shift( @ARGV );
    my $temp_filename = shift( @ARGV );

    open( STDIN, '<', $temp_filename );

    exec($hook_filename, @ARGV);
}

my $hook_type = $0;
$hook_type =~ s{^.+/}{};

my $git_dir = $ENV{GIT_DIR} || `git rev-parse --git-dir`;
chomp( $git_dir );
my $hook_dir = $git_dir . "/hooks";

opendir( my $dh, $hook_dir );
my @hooks = sort grep { /^${hook_type}_/ } readdir( $dh );
closedir( $dh );

my ($temp_fh, $temp_filename) = tempfile(UNLINK => 1);
while (my $line = <STDIN>) {
    print $temp_fh $line;
}
close( $temp_fh );

foreach my $hook (@hooks) {
    my( $success, $error_message, $full_buf, $stdout_buf, $stderr_buf ) = run(
        command => [$0, 'wrapper', "$hook_dir/$hook", $temp_filename, @ARGV],
    );
    print join('', @$full_buf);

    if (!$success) {
        die "Error running hook: " . $hook . ": $error_message\n";
    }
}

So, what you do is put this somewhere you can get at (perhaps just drop it in your hooks directory with the name “chained_hook”), chmod 755 it, and then symlink all your hooks to it, for example:

ln -s chained_hook applypatch-msg
ln -s chained_hook commit-msg
ln -s chained_hook post-commit
ln -s chained_hook post-receive
ln -s chained_hook post-update
ln -s chained_hook pre-applypatch
ln -s chained_hook pre-commit
ln -s chained_hook pre-rebase
ln -s chained_hook prepare-commit-msg
ln -s chained_hook update

Each of these chained hooks will now look for sub-hooks with the name “hookname_subhookname”. So, you could add a “post-receive_email” and “post-receive_foo” hooks and both will be called, in sorted order, by the post-receive hook.

So, I’m wondering – do people think this is a good solution? Are there better solutions? Is there a simpler way to write this script?

Tagged as: ,
21
Jul/11
0

Adding Files to Subversion With @ in Their Name

I just ran in to this today. If you try to do this:

svn add this\@that
svn: warning: 'this' not found

In order to ’svn add’ a file with a ‘@’ character in it you must end the add command with an ‘@’ character, as in:

svn add this\@that\@
A          this@that
3
Mar/11
1

Blue Box Maze

I needed to make sure my JS + jQuery skills were in good shape and created this maze app.

http://bluefeet.net/maze/

Use your arrow keys on your keyboard to move the blue box guy from the far left of the maze to the far right of the maze.

I opted to *not* use canvases as I wanted to play around with absolutely positioned divs, z-indexes, and PNG transparency. It worked out pretty good. If I was being practical I would have done this with canvases as it would be more efficient and much more flexible.

13
Aug/10
0

Presentation: Moose Best Practices

I wrote this for the TO.pm meeting on August 11th, 2010:

Tagged as: ,
10
Jun/10
0

Presentation: Tools of the CPAN Ninja

I wrote this for the TO.pm meeting on June 10th, 2010:

Tagged as: ,
3
Jun/10
0

BLOBs, Synonyms, and DBD::Oracle

INSERTing, UPDATEing, and SELECTing LOBs (CLOB/BLOB) in Oracle using Perl can be a PITA. Case in point: 2 years ago a fellow coworker ended up writing some extremely complex code using ora_lob_write(), etc, when all he really needed to do was use bind_param() and declare that the blob column is a blob so that DBD::Oracle would know to treat is specially. At the time I believe he did try something like:

my $sth = $dbh->prepare(q[
    INSERT INTO some_table (color, blob_data)
    VALUES (?, ?)
]);
$sth->bind_param( 1, 'red' );
$sth->bind_param( 2, $data_for_blob, {ora_type => ORA_BLOB } );
$sth->execute();

But then an error like “DBD::Oracle::st execute failed: ORA-04043: object some_table does not exist (DBD SUCCESS: OCIDescribeAny(view)/LOB refetch)” was raised. Which made no sense since some_table is very much an object and any number of SQL operations work on it… when BLOBs are not involved. In the end he ended up having to write code that was triple the size to get around this issue.

So, 2 years later and a little hair pulling, I come to figure out by luck that the problem is that the underlying DBD::Oracle code, that makes dealing with BLOBs simple, fails when you are using a synonym in your insert. All we had to do is fully spell out the table name, and it works!

my $sth = $dbh->prepare(q[
    INSERT INTO some_schema.some_table (color, blob_data)
    VALUES (?, ?)
]);
$sth->bind_param( 1, 'red' );
$sth->bind_param( 2, $data_for_blob, {ora_type => ORA_BLOB } );
$sth->execute();

Hopefully this helps others.

Tagged as: ,
21
May/10
1

Generating Constants in Perl

The other day @ $work I was throwing together a module that creates constants based on a fairly static table in the database. Of course I didn’t want to hard code the contants, I wanted them to be magically created by what was in the database. For the sake of this example I had a table called “toy_categories” where each record has a unique ID (toy_category_id) and a name that can only be letters and underscores. The data would look something like:

ID  NAME
--  --------------
12  dolls
 7  action_figures
92  education

And the resulting constants would look something like this:

TOY_CATEGORY_DOLLS = 12
TOY_CATEGORY_ACTION_FIGURES = 7
TOY_CATEGORY_EDUCATION = 92

If you read the core perl documents you might be lead to think that the constant pragma is the way to go. This is not the case – just because a particular library is distributed with Perl does not mean its a good tool, instead it usually means that the library cannot be removed or substantially modified for backwards-compatibility reasons. In my experience most core libraries are to be avoided – there are much better solutions on CPAN.

So, the CPAN solution to constants is Readonly. This module hooks in to Perl’s ability to mark a variable as read-only, much like how variables can be flagged as tainted or scalars as UTF. Make sure you grab Readonly::XS as well to get the full benefit of read-only variables without the performance hit.

So, if you were to create a module for these constants you might do something like this:

package MyApp::Constants::ToyCategories;
use strict;
use warnings;

use Readonly;
use Exporter qw( import );

our @EXPORT = qw(
    $TOY_CATEGORY_DOLLS
    $TOY_CATEGORY_ACTION_FIGURES
    $TOY_CATEGORY_EDUCATION
);

Readonly our $TOY_CATEGORY_DOLLS => 12;
Readonly our $TOY_CATEGORY_ACTION_FIGURES => 7;
Readonly our $TOY_CATEGORY_EDUCATION => 92;

1;

And then in some module you can access these constants:

use MyApp::Constants::ToyCategories;

if ($toy->category_id() == $TOY_CATEGORY_EDUCATION) {
    my $response = ask_question('Are you an educator?');
}

Now, like I said, I don’t want to hardcode the constants. I want them to be dynamically created by records in my toy_categories table which resides in my database. Its actually pretty simple to do this with some tricky evals:

package MyApp::Constants::ToyCategories;
use strict;
use warnings;

use Readonly;
use Exporter;
use Exporter qw( import );

our @EXPORT;

{
    my $dbh = code_that_returns_a_dbi_handle();
    my $sth = $dbh->prepare(q[
        SELECT toy_category_id, name
        FROM toy_categories
    ]);
    $sth->execute();
    $sth->bind_columns( \my( $id, $name ) );
    while ($sth->fetch()) {
        _export_variable( "toy_category_$name" => $id );
    }
}

sub _export_constant {
    my ($variable, $value) = @_;

    $variable = '$' . uc($variable);

    my ($error, $failed);
    {
        local $@;
        $failed = not eval("Readonly our $variable => \$value");
        $error = $@;
    }
    if ($failed) { die "Unable to create constant $variable: $error"; }
}

1;

What’s going on here? Let’s start at the top:

use Exporter qw( import );

This is the least intrusive way of using exporter and doesn’t pollute your namespace nearly as much as ‘use base qw( Exporter );’ does and is friendlies to other modules.

    my $dbh = code_that_returns_a_dbi_handle();

Replace this with whatever you use to get a DBI database handle. Take a look at DBIx::Connector for a great alternative to doing this directly with DBI.

    my $sth = $dbh->prepare(q[
        SELECT toy_category_id, name
        FROM toy_categories
    ]);
    $sth->execute();
    $sth->bind_columns( \my( $id, $name ) );
    while ($sth->fetch()) {
        ...
    }

I’m a big fan of writing my DBI selects in this fashion with bind_columns() because it tends to be the fastest way to get the data (versus fetchrow_hashref, etc) and tends to lead to the simplest code within the while loop since it just needs to use $id and $name versus $row->{$id} and $row->{name} (for example).

        _export_variable( "toy_category_$name" => $id );

While all of the code within _export_variable() could just be inlined right in the spot, that’s bad design. If you can get a piece of your code generalized and moved to a subroutine, do it.

    $variable = '$' . uc($variable);

Constants should always, due to convention, be uppercase. It is very important that you code to popular conventions because other people will most likely be working on your code later and if you come from a common expectation of how various things are done they will have an easier time ramping up to your code. I try to code to the Modern Perl / CPAN conventions.

    my ($error, $failed);
    {
        local $@;
        $failed = not eval ...;
        $error = $@;
    }
    if ($failed) { ... }

This bit of code was taken from nothingmuch’s blog entry where he announces Try::Tiny which provides a safe way to handle eval errors. Read up there if you want to understand why the code was written this way.

        $failed = not eval("Readonly our $variable => \$value");

While this code is several blocks deep in scope, this constant will exist in the package’s scope since it is being declared with ‘our’.

That’s it. After this was implemented I thought it was so useful, simple, but requiring the knowledge of a few tricks, that I’d share it with ya’all. Enjoy!

Tagged as: ,
17
Apr/10
1

YAPC::NA 2010 Talks are Being Approved

I just got my approval for the two proposals I had submitted for this coming YAPC::NA 2010! I’m very excited. This will be my first time doing any sort of talk at a YAPC. I’ve been wanting to do this for years and I am very grateful that the organizers of the conference considered me. Over the last few years I’ve been trying to hone my presentation skills by presenting various topics almost monthly at the Thousand Oaks Perl Mongers and Los Angeles Perl Mongers.

The first talk is a five-minute lighting talk about how to clone yourself in Perl. For those that are unaware: lighting talks are a great YAPC tradition where around a dozen speakers will present 5-minute or less talks. These talks range from very advanced computer science topics, down to the silly, and sometimes bizarre.

The other talk I’m doing is a 40-minute lecture/discussion showing examples of writing concise code. This will be an extended version of the similar presentation that I’ve given at mongers. I’ll be delving in to real-world cases where writing code in a more concise manner, reduces technical dept, leads to more robust code, and a whole lot more.

See you at YAPC!

Tagged as: