Useful Python Techniques:
A brief introduction to List Comprehensions,
Functional Programming, and Generators
October 4, 2011
Joe Cross
Contents
1.
2.
3.
4.
Looping and List Comprehensions
Filter, Map, Reduce (and lambda!)
Functional Programming Basics + Decorators
Generators
1. Looping and List Comprehensions
for(int i = 0; i < 5; i++){
cout << witty_banter(i) << endl;
}
Looping
compared to Java/C++
for i in range(0, 10, 1):
a /= 2.0
for i in xrange(10):
a /= 2.0
Python
Java/C++
for(int i = 0; i < 10; i++){
a /= 2.0;
}
Looping
Looping
Range + Iter
aList = [0, 1, 2, 'hello', 2**-4]
for i in range(len(aList)):
print aList[i]
for item in aList:
print item
Looping
Looping
still not as fun as…
To double the values in a list and assign to a new variable:
winning_lottery_numbers = [0, 4, 3, 2, 3, 1]
fake_lottery_numbers = []
for i in range(len(winning_lottery_numbers)):
fake_lottery_numbers.append(2 * winning_lottery_numbers[i])
fake_lottery_numbers = []
for number in winning_lottery_numbers:
fake_lottery_numbers.append(2 * number)
Even though it’s an improvement over the tedium of c++ et. al, we can still do better.
Looping
List Comprehensions
Woohooo!
Syntax:
[<expression> for <value> in <collection> if <condition>]
winning_lottery_numbers = [0, 4, 3, 2, 3, 1]
fake_lottery_numbers = [2*n for n in winning_lottery_numbers]
List Comprehensions allow us to do all sorts of things:
• Single-function single-line code
• Apply a function to each item of an iterable
• Filter using conditionals
• Cleanly nest loops
List Comprehensions
List Comprehensions
Don’t nest too many!
Multi-variable functions in a single line using zip:
vec1 = [3, 10, 2]
vec2 = [-20, 5, 1]
dot_mul = [u*v for u, v in zip(vec1, vec2)]
dot_prod = sum(dot_mul)
Filtering:
readings = [-1.2, 0.5, 12, 1.8, -9.0, 5.3]
good_readings = [r for r in readings if r > 0]
Bad:
orig = [15, 30, 78, 91, 25]
finals = [min(s, 100) for s in [f+5 for f in orig]]
List Comprehensions
2. Filter, Map, Reduce
Life = map(evolution, irreducible complexity)
assert(sum(Life) == 42)
Filter
Syntax:
result = filter(aFunction, aSequence)
def isPos(number, lim = 1E-16):
return number > lim
>>> a = [-1,2,-3,4,-5,6,-7,8,-9,10]
>>> filter(isPos, a)
[2, 4, 6, 8, 10]
>>> filter(not isPos, a)
Traceback (most recent call last):
File "<pyshell#7>", line 1
filter(not isZero, a)
TypeError: 'bool' object is not callable
Filter, Map, Reduce
Filter + Lambda
Syntax:
[fnName] = lambda [args]: expression
def isPos(number, lim = 1E-16):
return number > lim
>>> filter(lambda n: not isPos(n), a)
[-1, -3, -5, -7, -9]
Filter, Map, Reduce
Lambda vs. def
def add(x, y):
return x + y
Ladd = lambda x, y: x+y
def printWords():
print "Words"
LprintWords = lambda: print "Words"
Filter, Map, Reduce
So… why use lambda?
When using verbose function declaration it is often the case that the function’s
verbose declaration can be verbose, even for functions that don’t require such verbosity.
Verbose
def ispos(n):
return n > 0
b = filter(ispos, aList)
b = []
for a in aList:
if a > 0:
b.append(a)
Vs.
b = filter(lambda n: n > 0, aList)
Not Verbose
Also, there are some valid concerns about namespace clutter and the like.
Verbose verbose verbose.
Filter, Map, Reduce
Map
Syntax:
result = map(aFunction, aSequence)
Compare to list comprehension:
winning_lottery_numbers = [0, 4, 3, 2, 3, 1]
1. fake_lottery_numbers = [2*n for n in winning_lottery_numbers]
2. fake_lottery_numbers = map(lambda n: 2*n, winning_lottery_numbers)
Filter, Map, Reduce
Reduce
Syntax:
result = reduce(aFunction, aSequence, [initial])
NOTE: results get accumulated on the left, and new values applied to the right.
so reduce(add, [1,2,3,4]) is processed as (((1+2)+3)4)
lambda factorial n: reduce(operator.mul, xrange(1, n))
Filter, Map, Reduce
3. Functional Programming +
Decorators
This isn’t your dad’s
Procedural (imperative) programming
A Simple Example
def add(x, y):
return x + y
def sub(x, y):
return x - y
def op(fn, x, y):
return fn(x, y)
def mul(x, y):
return x * y
def div(x, y):
return x / y
def mod(x, y):
return x % y
Functional Programming
Nested Functions
Speed vs. Obfuscation
def randNoGap(min_, max_):
#random.random() -> [0,1)
v = random.random()
return (max_ - min_) * v - min_
def randWithGap(min_, max_):
s = random.random()
v = randNoGap(min_, max_)
if s < 0.5:
return v
else:
return -v
#Same conditional using Python’s
#Ternary operator
#return v if s < 0.5 else -v
def rand(min_, max_, hasGap = False):
if hasGap:
return randWithGap(min_, max_)
else:
return randNoGap(min_, max_)
Functional Programming
Nested Functions
Speed vs. Obfuscation
(Continued)
def randomExplosion(minv, maxv, n):
particles = []
for _ in xrange(n):
vx = rand(minv, maxv, True)
vy = rand(minv, maxv, True)
vz = rand(minv, maxv, True)
vx2 = rand(minv, maxv, True)
vy2 = rand(minv, maxv, True)
vz2 = rand(minv, maxv, True)
r = rand(0,255,False)
g = rand(0,255,False)
b = rand(0,255,False)
mainParticle = [vx,vy,vz,r,g,b]
secondParticle = [vx2,vy2,vz2,r,g,b]
particleGroup = (mainParticle, secondParticle)
particles.append(particleGroup)
return particles
NO
Functional Programming
Nested Functions
Speed vs. Obfuscation
(Continued)
What we’d like to do:
velocities = [rndV() for _ in xrange(6)]
What it actually looks like:
velocities = [rand(minv,maxv,True) for i in xrange(6)]
With a functional wrapper, we re-map:
rndV -> make_rand_fnc(minv, maxv, True)
rndV() -> make_rand_fnc(minv, maxv, True)()
Functional Programming
Nested Functions
Speed vs. Obfuscation
def
(Continued)
rand(min_, max_, hasGap = False):
rand(minv, maxv, True)
rand(minv, maxv, True)
rand(minv, maxv, True)
def mkRand(min_, max_, hasGap = False):
def wrapper():
return rand(min_, max_, hasGap)
return wrapper
def randomExplosion(minv, maxv, n):
rVel = mkRand(minv, maxv, True)
rCol = mkRnd(0,255,False)
for _ in xrange(n):
vx = rVel()
vy = rVel()
vz = rVel()
vx2 = rVel()
vy2 = rVel()
vz2 = rVel()
r = rCol()
g = rCol()
b = rCol()
Functional Programming
Nested Functions
Speed vs. Obfuscation
(Continued)
def randomExplosion(minv, maxv, n):
particles = []
rndV = mkRand(minv, maxv, True)
rndC = mkRnd(0,255,False)
for _ in xrange(n):
velocities = [rndV() for i in xrange(6)]
r,g,b = [rndC() for i in xrange(3)]
mainParticle = velocities[:3] + [r,g,b]
secondParticle = velocities[3:] + [r,g,b]
particleGroup = (mainParticle, secondParticle)
particles.append(particleGroup)
return particles
Functional Programming
Decorators
Quickly apply common tasks to methods
Common pre + post function call tasks, such as:
• Caching
• Timing
• Counting function calls
• Access rights
@decorator
def myFunc(arg1):
print “arg1: “, arg1
@f1(arg)
@f2
def func(): pass
myFunc = decorator(myFunc)
def func(): pass
func = f1(arg)(f2(func))
Decorators
Decorators
Quickly apply common tasks to methods
def decorator(f):
print "This line is run once during func = decorator(func)"
def wrapper(*args, **kwargs):
print "This line is executed just before the function is called"
#Call the function
ret = f(*args, **kwargs)
print "This line is executed just after the function is called"
#Return the function's return
return ret
return wrapper
@decorator
def foo(bar):
print bar
On running, we get this output:
>>> ================================ RESTART ================================
>>>
This line is run once during func = decorator(func)
>>> foo(1)
This line is executed just before the function is called
1
This line is executed just after the function is called
Decorators
Decorators
Quickly apply common tasks to methods
Decorators using classes
class decorator(object):
def __init__(self, f):
print "This line is run once during func = decorator(func)"
self.f = f
def __call__(self, *args, **kwargs):
print "This line is executed just before the function is called"
#Call the function
ret = self.f(*args)
print "This line is executed just after the function is called"
#Return the function's return
return ret
Decorators
Decorators
Quickly apply common tasks to methods
(Rough) Timing
import time
class TIMED(object):
def __init__(self, f):
self.f = f
def __call__(self, *args):
start = time.clock()
ret = self.f(*args)
stop = time.clock()
print "{0}: {1} ms.".format(self.f.func_name, 1000*(stop-start))
return ret
Decorators
Decorators
Quickly apply common tasks to methods
@TIMED
def euler(f, t0, y0, h):
""" Euler's Method """
yn = y0 + h*f(t0,y0)
return yn
@TIMED
def RK2(f, t0, y0, h):
""" Heun's Method """
y_hat = y0 + h*f(t0,y0)
yn = y0 + h/2.0*(f(t0,y0)+f(t0+h, y_hat))
return yn
@TIMED
def RK4(f, t0, y0, h):
""" Standard RK4 """
k1 = f(t0, y0)
k2 = f(t0+h/2.0,y0 + h*k1/2.0)
k3 = f(t0+h/2.0,y0 + h*k2/2.0)
k4 = f(t0+h/2.0,y0 + h*k3)
yn = y0 + 1.0/6.0*h*(k1 + 2.0*k2 + 2.0*k3 + k4)
return yn
Decorators
Decorators
Quickly apply common tasks to methods
fns = [euler, RK2, RK3, RK4, jRK4]
t0 = scipy.linspace(-1,1)
y0 = scipy.ones(50)
h = 0.025
args = (f, t0, y0, h)
for fn in fns:
print fn(*args)
print
>>>
euler: 0.0181114469778 ms.
[ ... ]
RK2: 0.041656328049 ms.
[ ... ]
RK3: 0.0606733473757 ms.
[ ... ]
RK4: 0.0745587900587 ms.
[ ... ]
jRKN: 0.00150928724815 ms.
jRK4: 1.57358288492 ms.
[ ... ]
Decorators
4. Generators
Memory-conscious patterns
are kind of a big deal in scientific computing
Binary Tree from Array
(simplified interface)
class T(object):
def __init__(self, values = None, index = 0):
self.left = None
self.right = None
self.v = None
if values is not None:
self.loadValues(values, index)
def loadValues(self, values, index):
self.v = values[index]
n = len(values)
if index * 2 + 1 < n:
self.left = T(values, index * 2 + 1)
if index * 2 + 2 < n:
self.right = T(values, index * 2 + 2)
Generators
Guessing Game
def makeT(val, delta, levels, level = 0):
if level < levels:
t = T()
t.v = val
t.left = makeT(val-delta, delta/2, levels, level+1)
t.right = makeT(val+delta, delta/2, levels, level+1)
return t
Generators
Clean Code
def inorder(t):
if t:
for v in inorder(t.left):
yield v
yield t.v
for v in inorder(t.right):
yield v
Generators
Using our Generator
for v in inorder(a_tree):
print v
a = []
for v in inorder(a_tree):
a.append(v)
b = [v for v in inorder(a_tree)]
Generators
Questions?

“Yes, the slide with the code. What did that one do?”