“Why Python and
other dynamic
languages are a new
way of thinking about
programming and why
it’s interesting to me
and possibly you if
you’re at all like me or
if you’re different than
me but just happen to
share that same specific
interest as me or if you
like progress and new
ideasbyand
stuff.”
Chris Gahan
First of all…


Python and other dynamic languages
(LIKE RUBY) are super-hot right now.
How hot?


IMAGINE A JULY DAY IN HELL AT HIGH
NOON AND THE WEATHER ALSO HAPPENS
TO BE ESPECIALLY HUMID THAT DAY FOR
SOME REASON
That’s hot.
You want some proof?
I’ll give ya proof!!
Cool Python Sh…stuff.













FuseFS
PyGame
PyOpenGL
PyMedia
PaiMei
Pyro (Python Robotics)
Matplotlib, SciPy
Fann
PIL (Python Imaging Library)
PyGTK, Tkinter, wxPython, wax
Google’s cluster
sCrAmBlEd?HaCkZ!
Orange (component-based data
mining software)












Scapy (packet masticator)
Twisted (everything, a kitchen
sink, and a side of fries)
CherryPy, SQLObject (web
applications)
BeautifulSoup
ElementTree
Psyco (optimizer)
PyPy
Pyrex
Maya, blender
Plone
Amarok
Subversion, Trac
And many, many, many more…
Why is it being used for so many
different applications?
It’s
AWESOME GLUE!
Mmm.. delicious glue…
What makes a good glue language?

High-level-ousity



You spend more time thinking about what you want to do than how you
want to do it
Python does the least to get in the way of you solving the problem
Excellent for text mastication

Text manipulation was well thought out in the language design



Strings have handy methods like “.startswith()” and “.endswith()” and
“.split()”
They can be sliced
List comprehensions


Dictionaries


doing transformations on large chunks of data is efficient
fast lookups of strings
Powerful regular expression library

No more hundreds-of-lines-of-IF-statements!
What makes a good glue language?

The built-in libraries let you do almost everything you want… easily!




Can call (and wrap) code written in other languages
Can combine disparate systems





Any computer that has Python installed contains almost all the modules you need
to do whatever you want!
No sysadmin dependency, no installing 9832749823742 packages
Connectors for every db format
Full access to Win32 API (COM, registry, all that…)
Excellent network protocol support
Easy to write C extensions
Maturity



Has been used in tons of industries so that there are a slew of libraries floating
around that let you do whatever you want
Stability (New versions don’t break all your code)
Speed! (Python’s built-in functions are heavily optimized in C)
What makes a good glue language?

Dynamicicity




You can load code modules at runtime
You can execute strings that came from
somewhere else as Python programs
You can do generative programming
You can introspect your code

This is a very cool feature. I think it needs its own
slide.
INTROSPECTION
You’d better have a good reason
for wasting a slide there, buddy!



Introspection is an incredibly powerful feature
when combined with a dynamic language
Introspection allows a program to look at itself
If the program can look at itself, and the
program can create itself dynamically, then you
can write a program that programs itself!
Why the hell would you want to
make a program that writes itself?

Because you’re lazy!
Isn’t laziness bad?


Short-term laziness is bad
Long-term laziness is good
Why?
“The reasonable man adapts himself to the world; the
unreasonable man persists in trying to adapt the world to
himself. Therefore, all progress depends on the unreasonable
man.”
-- George Bernard Shaw



Good-lazy programmers are lazy in that they
invent tools to do their work for them
Achieving laziness means improving your
technology, or tools
Improving your tools improves your
productivity
Variation in Productivity
“In a low-tech society you don't see much variation in
productivity. If you have a tribe of nomads collecting sticks for
a fire, how much more productive is the best stick gatherer
going to be than the worst? A factor of two? Whereas when you
hand people a complex tool like a computer, the variation in
what they can do with it is enormous. […]
(cont’d)
Variation in Productivity
[…] Productivity varies in any field, but there are few in which
it varies so much. The variation between programmers is so
great that it becomes a difference in kind. I don't think this is
something intrinsic to programming, though. In every field,
technology magnifies differences in productivity. I think
what's happening in programming is just that we have a lot of
technological leverage. But in every field the lever is getting
longer, so the variation we see is something that more and
more fields will see as time goes on. And the success of
companies, and countries, will depend increasingly on how
they deal with it.”
-- Paul Graham
Technology is what made this
country great!

If you’ve read the book Guns, Germs, and Steel,
you know that the only reason white people
conquered the world is because of all the
awesome technologies they stole

Gunpowder, writing, math, metallurgy, the centralized
state, horses, livestock, spaghetti…


basically, everything you need in the game Civilization to get
to the nuclear age.
The only thing they didn’t steal was germs

although, we did acquire them from livestock… so I
guess white people really are useless!
BUT ANYWAYS…
When do you need a program to
program itself?

A good example is when you do the same
thing over and over and over and over.
DRYness

A good rule of thumb is the DRY principle:
Don’t Repeat Yourself
Couldn’t you fix that without
introspection?



Okay, yes, you can fix the repetition by just
putting all the “things” in a list and iterating over
it like this:
But there an easier way than manually
specifying every element…
If you can see a pattern, then you can program
a computer to see it as well.
You could use locals()


locals() returns all of the variables in the
local scope as a dict
Instead of specifying each “thing”, you
could automatically grab all “things” like
this:
You could also improve debug
messages



Debugging by putting “print” statements everywhere is
tedious, and if you want to change how they’re all
displayed, you have to change every single one!
And what if you only want debug sometimes?
Why not make a debug function that figures out the
name of the function it was used in?
You could also improve debug
messages

Here’s what it looks like in the code:
(rest of the code omitted)
You could also improve debug
messages
And here it is in action
$ python snooch.py -v http://slashdot.org/
===========================================
snooch-o-matic 9000 pro elite pro v9.1 xp
------------------------------------------Getting all links from:
http://slashdot.org/
[snooch_urls_from_page] page url = http://slashdot.org/
[snooch_urls_from_page] url_is_url(found_url): False
[snooch_urls_from_page] Found: 'http://slashdot.org/http://www.ostg.com'
[snooch_urls_from_page] URL IS FILE!
[same_directory] base = ('http', 'slashdot.org', '/', '', ''), other = ('http',
'slashdot.org', '/http://www.ostg.com', '', '')
[snooch_urls_from_page] url_is_url(found_url): False
[snooch_urls_from_page] Found: 'http://slashdot.org/http://sf.net'
[snooch_urls_from_page] URL IS FILE!
[same_directory] base = ('http', 'slashdot.org', '/', '', ''), other = ('http',
'slashdot.org', '/http://sf.net', '', '')
etc..
How else can you DRY up your
code?
Mr. Bot




What if you wanted to write an IRC bot?
IRC bots have a bunch of actions that you
execute like this: !do_something
When you’re writing a program, you could have
a whole crapload of “if” statements that check
for the command, or a series of switch
statements.
But what if the program could figure out what
commands the bot supports by looking at the
names of the methods in the bot’s class?
Mr. Bot

Here’s a chunk of code from Mr. Bot’s
“logic core” (the class where all !actions
are coded):
Unittest

The unittest module also discovers methods by name and runs them
as tests with pretty output:
Creating new Classes automatically

Also known as “metaclasses”


A metaclass is a class that can change what happens
when you subclass another class
A good example is SQLObject



SQLObject creates a new classes that let you access your
database as if it’s a Python object
These classes are created on the fly, at runtime
This saves a lot of typing – you just supply the data needed
and no more.
Now that the computer is writing
most of the code, what do us
humans do?!

Now that you’ve become a metaprogramming ninja, you don’t have to
code as much. You’ve gotta spend all your
time THINKING!
You bastard! We had it so good!
Why’d you have to go and ruin it?


I know, I know, it seems unfair, but it’s a
cruel world out there.
If we don’t become more productive,
someone else is going to, like those
sneaky JAPANESE! Then we’re screwed.
So, how do we improve Thinking?

What about Lean Thinking!
See how the letters are leaning?
That’s called a “visual aid”.
What’s Lean Thinking?

The basic concept is this:



Do less activities that are a waste
Do more activities that are valuable
Examples of things that are a waste:

Paperwork


A bureaucratic organization produces mostly paperwork and
little value. Paperwork generally gets in the way of actually
doing things.
Surplus inventory

Produce products Just-in-Time™ instead of making tons and
tons of them, transporting them thousands of miles to a
warehouse, waiting 3 months, then transporting them all
back.
What’s Lean Thinking

Another thing that’s not valuable is overly
detailed program specifications



The only true representation of a program is
its source code.
Eventually, all specifications must be
transcribed as code, and that’s the thing that
gets run.
It’s easier to use the code as your One True
Reference instead of constantly converting
between code and specifications.
Wait, keeping it all in your head?

Well, okay, not ALL of it…

Don’t document everything – just the things that the
code is bad at describing.



Use cases
Keep only what you need on paper. Don’t overdo it.
Planning is good, but a lot of the time, actually
implementing something changes your plans.


Only plan things that you’re absolutely certain will not
change.
For everything else, learn as you go!
Cool! I like lean thinking!



Lean Thinking is just the beginning… Some
prolific software design process dudes got
together in 2001 and created the “Agile
Manifesto”.
The Agile Manifesto contains the current ideas
on the best way to produce software.
It’s goals are




maximal quality
maximal programmer satisfaction
minimal time
equal cost
Agile Development

Agile tries to keep focus on creating a process that
allows each individual in each team to find and use their
unique strengths





Flexibility
Dynamism
Learning
Trying new ideas
Adaptability and learning are essential

Continuous improvement of:



the people
the process
the source code
Agile Development

Agile has a lot to do with optimizing the
efficiency of people


Humans are the bottleneck, not computers!
Maximizing human efficiency is tricky


Humans are pattern recognizers, not state
machines
This means that people kinda suck at reading
and imagining code

We can understand small pieces at a time, but not
an entire system in all its gory detail.
Agile Development

Techniques to help improve the efficiency of programmers

Writing tests



Short release cycles



Get your program running and interfacing with real-world data as soon as possible
Helps keep you focussed on what’s important
Keep in constant communication with team members and customer



Tests let you use your module before you’ve written it
Keeps you secure in your code so you don’t have to worry about it
If the every member of the team doesn’t know the state of the program, then they’re
going to be afraid to change parts that they’re not responsible for, and they’ll also make
mistakes when integrating their code with the code of others
If the customer isn’t kept in the loop, then the project could start going in the wrong
direction. The customer is the end-user and ultimate judge of the finished product –
their feedback is the most important.
Pair programming

Two brains each doing different specialized tasks



brain #1: tactical [writing code]
brain #2: strategic and verification [thinking about architecture, verifying what brain #1 is
doing]
Keeps the programmers completely synchronized – two people will always be 100% up
to speed on a certain part of the program
PyPy has taken Agile to the
extreme: Sprints!




A PyPy Sprint is a two or three-day focused
development session in which developers pair
off together to focus on building a particular
subsystem
It ends up being incredibly productive, and is
really the best way to produce high-quality code
The alternative is to work slowly, unfocussedly,
individually.
Humans don’t work well like that! Our memories
fade over time.
Why does nCircle exist?

Because software has tons of flaws!

Thank the maker for bad development
practices and tools.
Why does software have tons of
flaws?

Because people are stupid!
Don’t be glib…

Well, okay, programming beyond a certain
level of complexity is too mentally taxing
for humans.
Fundamental Problems with
Software


With the advent of faster computers, and
with society pushing inexorably towards
the refinement and automation of all
things, software complexity has shot
through the roof
Programmers are digging themselves into
holes all over the place

Making their lives more complicated than it
needs to be
Basically…

We know how we want software to
behave, but we just don’t have the mental
hardware (or the software tools) to allow
us to continuously juggle a massive nest
of interconnected logical constraints and
remember every single one.

We get tired! We get hungry! We get
distracted by that videos of the monkey
peeing into his mouth!
The Future of Programming

Jonathan Edwards says:
“I discovered the creative exhilaration of programming at a formative age.
Yet all along I have also felt stifled by primitive, ill-suited languages and
tools. We are still in the Stone Age of programming. My quest is to better
understand the creative act of programming, and to help liberate it.
Reflection upon my own practice of programming has led me to three
general observations:



Programming is mentally overwhelming, even for the smartest of us. The more I
program, the harder I realize it is.
Much of this mental effort is laborious maintenance of the many mental models
we utilize while programming. We are constantly translating between different
representations in our head, mentally compiling them into code, and reverseengineering them back out of the code. This is wasted overhead.
We have no agreement on what the problems of programming are, much less
how to fix them. Our field has little accumulated wisdom, because we do not
invest enough in the critical self-examination needed to obtain it: we have a
reflective blind-spot.
(cont’d)
The Future of Programming

“From these observations I infer three corresponding
propositions:



Usability should be the central concern in the design of
programming languages and tools. We need to apply everything
we know about human perception and cognition. It is a matter
of “Cognitive Ergonomics”.
Notation matters. We need to stop wasting our effort juggling
unsuitable notations, and instead invent representations that
align with the mental models we naturally use.
The benefits of improved programming techniques can not be
easily promulgated, since there is no agreement on what the
problems are in the first place. The reward systems in business
and academics further discourage non-incremental change.”
Ruby + Domain Languages



Managing complex projects is about abstractions
Your program’s API is like its own domain-language
Good abstractions make writing programs easier because we can mentally manipulate them
better


“A change of perspective is worth 80 IQ points.” – Alan Kay
Intuitive paradigms are the most efficient way to allow a human to interface with a computer

The best programming paradigms are very intuitive:








Lists
Stacks
Dictionaries
Memory “space”
Objects with Actions
Threads
The implications of calling a function should be intuitive
Ruby allows you to make your API easier to understand by making it easy to create domain
languages

Ruby almost allows you to extend the language itself

A standard API will have a lot of boilerplate. In Ruby, it’s easy to make all that boilerplate disappear.

eg: ActiveRecord, Choice
Intentional Programming


This is probably going to be the first step
on a road towards an evolving system that
allows humans to represent programs on
their own terms.
Think of the Star Trek computer