Hacking Thy Fearful Symmetry

Hacker, hacker coding bright
Powered by a Gamboling Beluga

Mass-Testing Dancer's Plugins

created: September 17, 2012

One of my highlights of YAPC::EU 2012 was to meet a few of the core Dancer dudes (and, yes, a blog entry about the conference is forthcoming, it only has been delayed by two weeks of utterly decadent vacations in the heart of the Rhine Valley). Amongst other things, dams filled me in on the progress of Dancer 2, and what tasks still lay ahead before this rewrite of our favorite micro-web framework can hit the streets. One of the items on the to-do list is to verify that most of the plugins already written for Dancer 1 will still work for Dancer 2. Well, I thought, that's just like doing smoke testing for a small subset of Perl modules. How hard can it be?

Yeah, I do tend to say silly things like that...

Corraling the Modules

First task: gather the modules to test. Finding all of Dancer's plugins is relatively simple. Mostly if your cpan client is configured to use the SQLite backend:


$ sqlite3 ~/.cpan/cpandb.sql 'select dist_name from dists \
    where dist_name like "Dancer-Plugin%"' \
    | perl -pe's/-/::/g' > plugins

If the output of that command is to be trusted, there are 76 Dancer plugins out there. Not a bad number. For keeping them around, I decided to have a go at Pinto, which turned out to be a joy.


perl -ne'chomp; next if /^\s*#/; `pinto -v pull $_`' plugins

Mind you, the command could have been as simple as


pinto -v pull < plugins

but for two things: I wanted to be able to comment out problematic plugins in the list and skip over them, and it seems that Pinto would abort commiting all of the pulled distributions if one of them goes wrong. And while pulling 76 distros and all their dependencies did generate a lot of wrongness, so calling pinto on each plugin independently was the fastest way out.

Testing the Herd

With all the modules nicely accessing, I thought I had it done.

AH!

It turns out that while Perl has an awesome ecosystem for testing, it's also quite befuddling at first, and most of the modules are almost but not quite what I was wanting. To wit: they are mostly aimed at smoking ALL THE THINGS on potentially many perls, and push the results to a smoking mothership rather than feed a local database. A specific module, SmokeRunner::Multi, seemed more promising for the task, and could even talk to a Smolder instance. ... well, to make a long story short, at the end I didn't use it (although I still keep it on the wing), but I ended up becoming its maintainer. Just don't ask me how that happened.

What I ended up doing, instead, is leverage the --test-only option of cpanm. I haven't found a way yet to use it in such a way that I would get the full TAP results for a distro (although I'm sure there's a way), but it reports the overall success or failure, which is already a start.

So, I have a way to test the modules. Now, I need to store the results. How about going easy and sleazy and using that DBIx::NoSQL::Store::Manager I hacked a few months back? (No, it's not on CPAN yet. Yes, I'm going to do something about that Real Soontm)

package DancerTest;

use strict;
use warnings;

use Moose;
 
extends 'DBIx::NoSQL::Store::Manager';
 
__PACKAGE__->meta->make_immutable(inline_constructor => 0);

1;

package DancerTest::Types;

use MooseX::Storage::Engine;
 
use MooseX::Types -declare => [qw/
    DateTimeClass
/];
 
use Moose::Util::TypeConstraints;
use DateTime;
use DateTime::Format::ISO8601;

class_type 'DateTimeClass' => { class => 'DateTime' };
 
MooseX::Storage::Engine->add_custom_type_handler(
    'DateTimeClass' => (
        expand   => sub {
            DateTime::Format::ISO8601->parse_datetime(shift)
        },
        collapse => sub { (shift)->iso8601 },
    ),
);
 
1;

package DancerTest::Model::Plugin;

use 5.10.0;

use strict;
use warnings;

use Method::Signatures;
use IPC::Cmd 'run';

use DancerTest;
use DancerTest::Types qw/ DateTimeClass /;

use Moose;
 
with 'DBIx::NoSQL::Store::Model::Role';

has '+store_key' => (
    default => method {
        join ' : ', $self->name, $self->timestamp;
    },
);
 
has name => (
    traits => [ 'StoreIndex' ],
    isa    => 'Str',
    is     => 'ro',
    required => 1,
);
 
has timestamp => (
    traits  => [ 'StoreIndex' ],
    isa     => 'DateTimeClass',
    is      => 'ro',
    lazy    => 1,
    default => sub { DateTime->now },
);

has version => (
    traits => [ 'StoreIndex' ],
    isa    => 'Str',
    is     => 'ro',
    lazy    => 1,
    default => method { 
        my $version;
        run( command => "pinto install --info " . $self->name, buffer => \$version );

        chomp $version;
        $version =~ s/^.*-//;
        $version =~ s/\.tar.gz$//;

        return $version;
    },
);

has dancer1_pass => (
    traits => [ 'StoreIndex' ],
    isa    => 'Bool',
    is     => 'ro',
    lazy    => 1,
    default => method { 
        $self->run_tests;
    },
);

has dancer2_pass => (
    traits => [ 'StoreIndex' ],
    isa    => 'Bool',
    is     => 'ro',
    lazy    => 1,
    default => method { 
        local $ENV{PERL5LIB} = '/home/yanick/work/perl-modules/Dancer2/lib';
        $self->run_tests;
    },
);

method run_tests {
    return scalar( run( command => "pinto install --test-only " . $self->name
    ) ) ? 1 : 0;
}

has verbose => (
    is => 'ro',
    isa => 'Bool',
    default => 1,
);

before 'store' => method {
    say "testing ", $self->name;
};

after 'store' => method {
    use Data::Printer;
    p $self->pack;
};
 
__PACKAGE__->meta->make_immutable;
 
1;

And with that in hand, we just need one more script to turn in the crank:

#!/usr/bin/perl 

use 5.14.0;

use DancerTest;
use DancerTest::Model::Plugin;

my $store = DancerTest->connect( 'plugins.db' );

$store->register;

while( my $module = <> ) {
    chomp $module;
    next if $module=~ /^\s*#/;
    $store->new_model_object('Plugin',
        name => $module,
        verbose => 1,
    )->store;
}

Aaaand:


$ ./test_plugins.pl plugins 
testing Dancer::Plugin::ExtDirect
\ {
    __CLASS__      "DancerTest::Model::Plugin",
    dancer1_pass   1,
    dancer2_pass   1,
    name           "Dancer::Plugin::ExtDirect",
    timestamp      "2012-09-18T01:41:52",
    verbose        1,
    version        1.03
}
testing Dancer::Plugin::TTHelpers
\ {
    __CLASS__      "DancerTest::Model::Plugin",
    dancer1_pass   1,
    dancer2_pass   1,
    name           "Dancer::Plugin::TTHelpers",
    timestamp      "2012-09-18T01:42:01",
    verbose        1,
    version        0.004
}

Final stroke, reporting the latest results:

package DancerReport;

use 5.10.0;

use base qw/DBIx::Class::Schema::Loader/;

package main;

my $schema = DancerReport->connect('dbi:SQLite:plugins.db');

my $plugins = $schema->resultset('Plugin');
my $rs = $plugins->search({
    'me.timestamp' => {
        '=' => $plugins->search(
            { 'x.name' => { '=' => { -ident => 'me.name' } } },
            { alias => 'x' }
        )->get_column('timestamp')->max_rs->as_query,
    },
});

say sprintf "%40s %10s %10s %10s", qw/ distro version dancer_1 dancer_2 /;

while( my $p = $rs->next ) {
    next if $p->version !~ /^\d/;
    say sprintf "%40s %10s %10s %10s", 
        map( { $p->$_ } qw/ name version / ),
        map { $_ ? 'passed' : 'failed' } map { $p->$_ } 
            qw/ dancer1_pass dancer2_pass /;
}

Which gives us:


$ ./report.pl                                            
                                  distro    version   dancer_1   dancer_2
               Dancer::Plugin::SQLSearch       0.04     passed     failed
                    Dancer::Plugin::I18N       0.40     passed     failed
                    Dancer::Plugin::Lucy      0.001     passed     passed
                Dancer::Plugin::EmptyGIF        0.3     passed     failed
                  Dancer::Plugin::Locale       0.01     passed     passed
        Dancer::Plugin::Locale::Wolowitz   0.122470     passed     failed
              Dancer::Plugin::RequireSSL   0.121370     passed     passed
                     Dancer::Plugin::Res     0.0003     passed     passed
                 Dancer::Plugin::Lexicon       0.05     passed     passed
                Dancer::Plugin::Cerberus       0.03     passed     passed
                Dancer::Plugin::Resource   1.122280     passed     failed
            Dancer::Plugin::ElasticModel       0.05     failed     failed
                Dancer::Plugin::Database       1.82     passed     failed
                   Dancer::Plugin::Email     0.1300     passed     passed
                  Dancer::Plugin::Scoped    0.02fix     passed     passed
                 Dancer::Plugin::SiteMap       0.11     passed     passed
                    Dancer::Plugin::DBIC     0.1506     passed     failed
               Dancer::Plugin::Authorize   1.110720     passed     passed
           Dancer::Plugin::FormValidator   1.122460     passed     failed
   Dancer::Plugin::Params::Normalization       0.51     passed     failed
                    Dancer::Plugin::REST       0.07     passed     failed
                   Dancer::Plugin::Mongo       0.03     passed     failed
                Dancer::Plugin::ORMesque   1.113100     passed     failed
                 Dancer::Plugin::Browser        0.4     passed     passed
               Dancer::Plugin::DebugDump       0.03     passed     passed
               Dancer::Plugin::Memcached       0.01     passed     failed
                     Dancer::Plugin::MPD       0.03     failed     failed
                    Dancer::Plugin::Feed        0.8     passed     failed
            Dancer::Plugin::MobileDevice       0.03     passed     failed
              Dancer::Plugin::Auth::RBAC   1.110720     passed     failed
           Dancer::Plugin::Auth::Twitter       0.02     passed     failed
              Dancer::Plugin::SimpleCRUD       0.61     passed     failed
  Dancer::Plugin::SporeDefinitionControl       0.11     passed     failed
               Dancer::Plugin::WebSocket     0.0100     passed     failed
                   Dancer::Plugin::Redis        0.2     passed     failed
         Dancer::Plugin::FormattedOutput       0.01     passed     passed
            Dancer::Plugin::FlashMessage      0.314     passed     failed
           Dancer::Plugin::MemcachedFast   0.110770     passed     failed
                Dancer::Plugin::Facebook      0.900     failed     failed
         Dancer::Plugin::ValidationClass   0.120490     passed     failed
               Dancer::Plugin::ProxyPath       0.03     passed     failed
               Dancer::Plugin::GearmanXS   0.110570     failed     failed
               Dancer::Plugin::FlashNote      1.0.3     passed     failed
                     Dancer::Plugin::SMS       0.01     passed     passed
          Dancer::Plugin::Fake::Response       0.03     passed     failed
              Dancer::Plugin::Cache::CHI      1.3.0     passed     failed
                Dancer::Plugin::Progress      0.001     passed     passed
            Dancer::Plugin::TimeRequests       0.03     passed     passed
                   Dancer::Plugin::Async      0.001     passed     passed
                Dancer::Plugin::XML::RSS       0.01     passed     passed
Dancer::Plugin::Auth::RBAC::Credentials::DBIC      0.003     passed     failed
              Dancer::Plugin::Passphrase      1.0.0     passed     failed
            Dancer::Plugin::ValidateTiny       0.05     passed     passed
                   Dancer::Plugin::Hosts       0.01     passed     passed
                  Dancer::Plugin::Bcrypt      0.4.1     passed     failed
                   Dancer::Plugin::Stomp     1.0101     passed     passed
            Dancer::Plugin::LibraryThing     0.0003     passed     passed
            Dancer::Plugin::DebugToolbar      0.016     passed     failed
              Dancer::Plugin::EscapeHTML       0.22     passed     failed
                Dancer::Plugin::EncodeID       0.02     passed     failed
           Dancer::Plugin::DirectoryView       0.02     passed     failed
                 Dancer::Plugin::NYTProf       0.21     passed     passed
             Dancer::Plugin::Auth::Basic       0.02     failed     failed
                Dancer::Plugin::Mongoose    0.00002     passed     failed
              Dancer::Plugin::Dispatcher       0.12     passed     failed
        Dancer::Plugin::Preprocess::Sass       0.01     passed     passed
          Dancer::Plugin::Auth::Htpasswd      0.014     passed     passed
           Dancer::Plugin::ElasticSearch      0.002     passed     passed
               Dancer::Plugin::ExtDirect       1.03     passed     passed
               Dancer::Plugin::TTHelpers      0.004     passed     passed

According to the report, 37 plugins pass their test under Dancer 1, but not Dancer2. Now we know exactly what work remains to do. So... Let's get crackin'.

comments powered by Disqus