Writing Pluggable Software
Tatsuhiko Miyagawa
miyagawa@gmail.com
Six Apart, Ltd. / Shibuya Perl Mongers
YAPC::Asia 2007 Tokyo
For non-JP attendees …
If you find \ in the code,
Replace that with backslash.
(This is MS' fault)
Tatsuhiko Miyagawa
Plaggable
Software
Tatsuhiko Miyagawa
Plaggable
Software
Tatsuhiko Miyagawa
Pluggable
Software
Tatsuhiko Miyagawa
Agenda
Tatsuhiko Miyagawa
#1
How to make
your app pluggable
Tatsuhiko Miyagawa
#2
TMTOWTDP
There's More Than One Way To Deploy Plugins
Pros/Cons by examples
Tatsuhiko Miyagawa
First-of-all:
Why pluggable?
Tatsuhiko Miyagawa
Benefits
Tatsuhiko Miyagawa
#1
Keep the app design
and code simple
Tatsuhiko Miyagawa
#2
Let the app users
customize the behavior
(without hacking the internals)
Tatsuhiko Miyagawa
#3
It's fun to write plugins
for most hackers
(see: Plagger and Kwiki)
Tatsuhiko Miyagawa
"Can your app do XXX?"
"Yes, by plugins."
Tatsuhiko Miyagawa
"Your app has a bug in YYY"
"No, it's the bug in plugin YYY,
Not my fault."
(Chain Of Responsibilities)
Tatsuhiko Miyagawa
Good Enough
Reasons, huh?
Tatsuhiko Miyagawa
#1
Make your app pluggable
Tatsuhiko Miyagawa
Example
Tatsuhiko Miyagawa
ack (App::Ack)
Tatsuhiko Miyagawa
grep –r for programmers
Tatsuhiko Miyagawa
Ack is a "full-stack"
software now.
Tatsuhiko Miyagawa
By "full-stack" I mean:
Easy install
No configuration
No way to extend
Tatsuhiko Miyagawa
Specifically:
These are hardcoded
Ignored directories
Filenames and types
Tatsuhiko Miyagawa
Ignored Directories
@ignore_dirs = qw( blib CVS RCS SCCS .svn
_darcs .git );
Tatsuhiko Miyagawa
Filenames and languages mapping
%mappings = (
asm
binary
cc
cpp
csharp
…
perl
…
);
Tatsuhiko Miyagawa
=>
=>
=>
=>
=>
[qw(
…,
[qw(
[qw(
[qw(
s S )],
c h xs )],
cpp m h C H )],
cs )],
=> [qw( pl pm pod tt ttml t )],
What if making
these pluggable?
Tatsuhiko Miyagawa
DISCLAIMER
Tatsuhiko Miyagawa
Don't get me wrong Andy,
I love ack the way it is…
Just thought it can be
a very good example for the tutorial.
Tatsuhiko Miyagawa
Quickstart:
Class::Trigger
Module::Pluggable
© Six Apart Ltd. Employees
Tatsuhiko Miyagawa
Class::Trigger SYNOPSIS
package Foo;
use Class::Trigger;
sub foo {
my $self = shift;
$self->call_trigger('before_foo');
# some code ...
$self->call_trigger('after_foo');
}
package main;
Foo->add_trigger(before_foo => \&sub1);
Foo->add_trigger(after_foo => \&sub2);
Tatsuhiko Miyagawa
Class::Trigger
Helps you to implement
Observer Pattern.
(Rails calls this Observer)
Tatsuhiko Miyagawa
Module::Pluggable SYNOPSIS
package MyClass;
use Module::Pluggable;
use MyClass;
my $mc = MyClass->new();
# returns the names of all plugins installed under
MyClass::Plugin::*
my @plugins = $mc->plugins();
package MyClass::Plugin::Foo;
sub new { … }
1;
Tatsuhiko Miyagawa
Setup plugins in App::Ack
package App::Ack;
use Class::Trigger;
use Module::Pluggable require => 1;
__PACKAGE__->plugins;
Tatsuhiko Miyagawa
Setup plugins in App::Ack
package App::Ack;
use Class::Trigger;
use Module::Pluggable require => 1;
__PACKAGE__->plugins; # "requires" modules
Tatsuhiko Miyagawa
Ignored Directories (Before)
@ignore_dirs = qw( blib CVS RCS SCCS .svn
_darcs .git );
Tatsuhiko Miyagawa
Ignored Directories (After)
# lib/App/Ack.pm
__PACKAGE__->call_trigger('ignore_dirs.add',
\@ignore_dirs);
Tatsuhiko Miyagawa
Ignored Directories (plugins)
# lib/App/Ack/Plugin/IgnorePerlBuildDir.pm
package App::Ack::Plugin::IgnorePerlBuildDir;
App::Ack->add_trigger(
"ignore_dirs.add" => sub {
my($class, $ignore_dirs) = @_;
push @$ignore_dirs, qw( blib );
},
);
1;
Tatsuhiko Miyagawa
Ignored Directories (plugins)
# lib/App/Ack/Plugin/IgnoreSourceControlDir.pm
package App::Ack::Plugin::IgnoreSourcdeControlDir;
App::Ack->add_trigger(
"ignore_dirs.add" => sub {
my($class, $ignore_dirs) = @_;
push @$ignore_dirs, qw( CVS RCS .svn
_darcs .git );
},
);
1;
Tatsuhiko Miyagawa
Filenames and languages (before)
%mappings = (
asm
binary
cc
cpp
csharp
…
perl
…
);
Tatsuhiko Miyagawa
=>
=>
=>
=>
=>
[qw(
…,
[qw(
[qw(
[qw(
s S )],
c h xs )],
cpp m h C H )],
cs )],
=> [qw( pl pm pod tt ttml t )],
Filenames and languages (after)
# lib/App/Ack.pm
__PACKAGE__->call_trigger('mappings.add',
\%mappings);
Tatsuhiko Miyagawa
Filenames and languages (plugins)
package App::Ack::Plugin::MappingCFamily;
use strict;
App::Ack->add_trigger(
"mappings.add" => sub {
my($class, $mappings) = @_;
$mappings->{asm} = [qw( s S )];
$mappings->{cc} = [qw( c h xs )];
$mappings->{cpp} = [qw( cpp m h C H )];
$mappings->{csharp} = [qw( cs )];
$mappings->{css} = [qw( css )];
},
);
1;
Tatsuhiko Miyagawa
Works great
with few lines of code!
Tatsuhiko Miyagawa
Now it's time to add
Some useful stuff.
Tatsuhiko Miyagawa
Example Plugin:
Content Filter
Tatsuhiko Miyagawa
sub _search {
my $fh = shift;
my $is_binary = shift;
my $filename = shift;
my $regex = shift;
my %opt = @_;
if ($is_binary) {
my $new_fh;
App::Ack->call_trigger('filter.binary', $filename, \$new_fh);
if ($new_fh) {
return _search($new_fh, 0, $filename, $regex, @_);
}
}
Tatsuhiko Miyagawa
Example:
Search PDF content
with ack
Tatsuhiko Miyagawa
PDF filter plugin
package App::Ack::Plugin::ExtractContentPDF;
use strict;
use CAM::PDF;
use File::Temp;
App::Ack->add_trigger(
'mappings.add' => sub {
my($class, $mappings) = @_;
$mappings->{pdf} = [qw(pdf)];
},
);
Tatsuhiko Miyagawa
PDF filter plugin (cont.)
App::Ack->add_trigger(
'filter.binary' => sub {
my($class, $filename, $fh_ref) = @_;
if ($filename =~ /\.pdf$/) {
my $fh = File::Temp::tempfile;
my $doc = CAM::PDF->new($file);
my $text;
for my $page (1..$doc->numPages){
$text .= $doc->getPageText($page);
}
print $fh $text;
seek $$fh, 0, 0;
$$fh_ref = $fh;
}
},
);
Tatsuhiko Miyagawa
PDF search
> ack --type=pdf Audrey
yapcasia2007-pugs.pdf:3:Audrey Tang
Tatsuhiko Miyagawa
Homework
Use File::Extract
To handle arbitrary media files
Tatsuhiko Miyagawa
Homework 2:
Search non UTF-8 files
(hint: use Encode::Guess)
You'll need another hook.
Tatsuhiko Miyagawa
Summary
Class::Trigger
+ Module::Pluggable
= Pluggable app easy
Tatsuhiko Miyagawa
#2
TMTOWTDP
There's More Than One Way To Deploy Plugins
Tatsuhiko Miyagawa
Module::Pluggable
+ Class::Trigger
= Simple and Nice
but has limitations
Tatsuhiko Miyagawa
In Reality,
we need more control
over how plugins behave
Tatsuhiko Miyagawa
1)
The order of
plugin executions
Tatsuhiko Miyagawa
2)
Per user configurations
for plugins
Tatsuhiko Miyagawa
3)
Temporarily
Disable plugins
Should be easy
Tatsuhiko Miyagawa
4)
How to install
& upgrade plugins
Tatsuhiko Miyagawa
5)
Let plugins
have storage area
Tatsuhiko Miyagawa
Etc, etc.
Tatsuhiko Miyagawa
Examples:
Kwiki
Plagger
qpsmtpd
Movable Type
Tatsuhiko Miyagawa
I won't talk about
Catalyst plugins
(and other framework thingy)
Tatsuhiko Miyagawa
Because they're
NOT
"plug-ins"
Tatsuhiko Miyagawa
Install plugins
And now you write
MORE CODE
Tatsuhiko Miyagawa
95% of Catalyst plugins
Are NOT "plugins"
But "components"
95% of these statistics is made up by the speakers.
Tatsuhiko Miyagawa
Kwiki 1.0
Tatsuhiko Miyagawa
Kwiki Plugin code
package Kwiki::URLBL;
use Kwiki::Plugin -Base;
use Kwiki::Installer -base;
const class_id
=> 'urlbl';
const class_title => 'URL Blacklist DNS';
const config_file => 'urlbl.yaml';
sub register {
require URI::Find;
my $registry = shift;
$registry->add(hook => 'edit:save', pre => 'urlbl_hook');
$registry->add(action => 'blacklisted_url');
}
Tatsuhiko Miyagawa
Kwiki Plugin (cont.)
sub urlbl_hook {
my $hook = pop;
my $old_page = $self->hub->pages->new_page($self->pages>current->id);
my $this
= $self->hub->urlbl;
my @old_urls = $this->get_urls($old_page->content);
my @urls
= $this->get_urls($self->cgi->page_content);
my @new_urls = $this->get_new_urls(\@old_urls, \@urls);
if (@new_urls && $this->is_blocked(\@new_urls)) {
$hook->cancel();
return $self->redirect("action=blacklisted_url");
}
}
Tatsuhiko Miyagawa
Magic implemented
in Spoon(::Hooks)
Tatsuhiko Miyagawa
"Install" Kwiki Plugins
# order doesn't matter here (according to Ingy)
Kwiki::Display
Kwiki::Edit
Kwiki::Theme::Basic
Kwiki::Toolbar
Kwiki::Status
Kwiki::Widgets
# Comment out (or entirely remove) to disable
# Kwiki::UnnecessaryStuff
Tatsuhiko Miyagawa
Kwiki plugin config
# in Kwiki::URLBL plugin
__config/urlbl.yaml__
urlbl_dns: sc.surbl.org, bsb.spamlookup.net, rbl.bulkfeeds.jp
# config.yaml
urlbl_dns: myowndns.example.org
Tatsuhiko Miyagawa
Kwiki plugins are CPAN modules
Tatsuhiko Miyagawa
Install and Upgrade plugins
cpan> install Kwiki::SomeStuff
Tatsuhiko Miyagawa
Using CPAN as a repository
Pros #1:
reuse most of current
CPAN infrastructure.
Tatsuhiko Miyagawa
Using CPAN as a repository
Pros #2:
Increasing # of modules
= good motivation
for Perl hackers
Tatsuhiko Miyagawa
Cons #1:
Installing CPAN deps
could be a mess
(especially for Win32)
Tatsuhiko Miyagawa
Cons #2:
Whenever Ingy releases
new Kwiki, lots of plugins
just break.
Tatsuhiko Miyagawa
Kwiki plugin storage
return if
grep {$page->id}
@{$self->config->cached_display_ignore};
my $html = io->catfile(
$self->plugin_directory,$page->id
)->utf8;
Tatsuhiko Miyagawa
Kwiki 2.0
Tatsuhiko Miyagawa
Same as Kwiki 1.0
Tatsuhiko Miyagawa
Except:
plugins are now
in SVN repository
Tatsuhiko Miyagawa
Tatsuhiko Miyagawa
Plagger plugin
package Plagger::Plugin::Publish::iCal;
use strict;
use base qw( Plagger::Plugin );
use
use
use
use
Data::ICal;
Data::ICal::Entry::Event;
DateTime::Duration;
DateTime::Format::ICal;
sub register {
my($self, $context) = @_;
$context->register_hook(
$self,
'publish.feed' => \&publish_feed,
'plugin.init ' => \&plugin_init,
);
}
Tatsuhiko Miyagawa
Plagger plugin (cont)
sub plugin_init {
my($self, $context) = @_;
my $dir = $self->conf->{dir};
unless (-e $dir && -d _) {
mkdir $dir, 0755
or $context->error("Failed to mkdir $dir: $!");
}
}
Tatsuhiko Miyagawa
Plagger plugin storage
$self->conf->{invindex} ||=
$self->cache->path_to('invindex');
Tatsuhiko Miyagawa
Plagger plugin config
# The order matters in config.yaml
# if they're in the same hooks
plugins:
- module: Subscription::Config
config:
feed:
- http://www.example.com/
- module: Filter::DegradeYouTube
config:
dev_id: XYZXYZ
- module: Publish::Gmail
disable: 1
Tatsuhiko Miyagawa
Plugins Install & Upgrade
>
#
>
>
notest cpan Plagger
or …
svn co http://…/plagger/trunk plagger
svn update
Tatsuhiko Miyagawa
Plagger impl.
ripped off by
many apps now
Tatsuhiko Miyagawa
qpsmtpd
Tatsuhiko Miyagawa
mod_perl for SMTP
Runs with tcpserver, forkserver
or Danga::Socket standalone
Tatsuhiko Miyagawa
Plugins: Flat files
rock:/home/miyagawa/svn/qpsmtpd> ls -F plugins
async/
greylisting
auth/
hosts_allow
check_badmailfrom
http_config
check_badmailfromto
ident/
check_badrcptto
logging/
check_badrcptto_patterns
milter
check_basicheaders
parse_addr_withhelo
check_earlytalker
queue/
check_loop
quit_fortune
check_norelay
rcpt_ok
check_relay
relay_only
check_spamhelo
require_resolvable_fromhost
content_log
rhsbl
count_unrecognized_commands sender_permitted_from
dns_whitelist_soft
spamassassin
dnsbl
tls
domainkeys
tls_cert*
dont_require_anglebrackets
virus/
Tatsuhiko Miyagawa
qpsmtpd plugin
sub hook_mail {
my ($self, $transaction, $sender, %param) = @_;
my @badmailfrom = $self->qp->config("badmailfrom")
or return (DECLINED);
for my $bad (@badmailfrom) {
my $reason = $bad;
$bad =~ s/^\s*(\S+).*/$1/;
next unless $bad;
$transaction->notes('badmailfrom', $reason) …
}
return (DECLINED);
}
Tatsuhiko Miyagawa
Actually qpsmtpd
Plugins are "compiled"
to modules
Tatsuhiko Miyagawa
my $eval = join("\n",
"package $package;",
'use Qpsmtpd::Constants;',
"require Qpsmtpd::Plugin;",
'use vars qw(@ISA);',
'use strict;',
'@ISA = qw(Qpsmtpd::Plugin);',
($test_mode ? 'use Test::More;' : ''),
"sub plugin_name { qq[$plugin] }",
$line,
$sub,
"\n", # last line comment without newline?
);
$eval =~ m/(.*)/s;
$eval = $1;
eval $eval;
die "eval $@" if $@;
Tatsuhiko Miyagawa
qpsmtpd plugin config
rock:/home/miyagawa/svn/qpsmtpd> ls config.sample/
config.sample:
IP
logging
require_resolvable_fromhost
badhelo
loglevel
rhsbl_zones
badrcptto_patterns
plugins
size_threshold
dnsbl_zones
rcpthosts
tls_before_auth
invalid_resolvable_fromhost relayclients
tls_ciphers
Tatsuhiko Miyagawa
config/plugins
# content filters
virus/klez_filter
# rejects mails with a SA score higher than 2
spamassassin reject_threshold 20
Tatsuhiko Miyagawa
config/badhelo
# these domains never uses their domain when greeting
us, so reject transactions
aol.com
yahoo.com
Tatsuhiko Miyagawa
Install & Upgrade plugins
Just use subversion
Tatsuhiko Miyagawa
Tatsuhiko Miyagawa
MT plugins are
flat-files
(or scripts that call
modules)
Tatsuhiko Miyagawa
MT plugin code
package MT::Plugin::BanASCII;
our $Method = "deny";
use MT;
use MT::Plugin;
my $plugin = MT::Plugin->new({
name => "BanASCII v$VERSION",
description => "Deny or moderate ASCII or Latin-1
comment",
});
MT->add_plugin($plugin);
MT->add_callback('CommentFilter', 2, $plugin, \&handler);
Tatsuhiko Miyagawa
MT plugin code (cont)
sub init_app {
my $plugin = shift;
$plugin->SUPER::init_app(@_);
my($app) = @_;
return unless $app->isa('MT::App::CMS');
$app->add_itemset_action({
type => 'comment',
key => 'spam_submission_comment',
label => 'Report SPAM Comment(s)',
code => sub {
$plugin->submit_spams_action('MT::Comment', @_) },
}
);
Tatsuhiko Miyagawa
Tatsuhiko Miyagawa
Tatsuhiko Miyagawa
MT plugin storage
require MT::PluginData;
my $data = MT::PluginData->load({
plugin => 'sidebar-manager',
key
=> $blog_id },
);
unless ($data) {
$data = MT::PluginData->new;
$data->plugin('sidebar-manager');
$data->key($blog_id);
}
$data->data( \$modulesets );
$data->save or die $data->errstr;
Tatsuhiko Miyagawa
Order control
MT->add_callback('CMSPostEntrySave', 9, $rightfields,
\&CMSPostEntrySave);
MT->add_callback('CMSPreSave_entry', 9, $rightfields,
\&CMSPreSave_entry);
MT::Entry->add_callback('pre_remove', 9, $rightfields,
\&entry_pre_remove);
Defined in plugins.
No Control on users end
Tatsuhiko Miyagawa
Conclusion
Flat-files
vs.
Modules
Tatsuhiko Miyagawa
Flat-files:
☺ Easy to install (Just grab it)
☻ Hard to upgrade
OK for simple plugins
Tatsuhiko Miyagawa
Modules:
☺ Full-access to Perl OO goodness
☺ Avoid duplicate efforts of CPAN
☻ Might be hard to resolve deps.
Subversion to the rescue
(could be a barrier for newbies)
Tatsuhiko Miyagawa
Nice-to-haves:
Order control
Temporarily disable plugins
Per plugin config
Per plugin storage
Tatsuhiko Miyagawa
Resources
Class::Trigger
http://search.cpan.org/dist/Class-Trigger/
Module::Pluggable
http://search.cpan.org/dist/Module-Pluggable/
Ask Bjorn Hansen: Build Easily Extensible Perl Programs
http://conferences.oreillynet.com/cs/os2005/view/e_sess/6806
qpsmtpd
http://smtpd.develooper.com/
MT plugins
http://www.sixapart.com/pronet/plugins/
Kwiki
http://www.kwiki.org/
Plagger
http://plagger.org/
Tatsuhiko Miyagawa