LECTURE 10
Development Tools
DEVELOPMENT TOOLS
Since you are all in the early stages of your semester-long projects, today we will be
covering some useful modules and tools for developing larger python projects
(especially ones that involve multiple people).
• virtualenv
• logging
• testing
VIRTUALENV
virtualenv is a tool for creating isolated Python environments.
Let’s say you are working on two projects which require Twisted, a python-based
networking package. One of the projects requires Twisted 14.x, but the other
requires Twisted 13.x. Solve the problem by creating a custom Python environment for
each project.
To install virtualenv via pip:
$ pip install virtualenv
All of these examples are for a debian-based Linux distros.
VIRTUALENV
To create a virtual environment for a project:
$ cd my_project_folder
$ virtualenv venv
Essentially, we’re creating a copy of the Python interpreter (as well as a copy of the
pip and setuptools libraries) inside of a folder called venv. We can specify a
different Python version in the following way:
$ virtualenv -p /usr/bin/python2.7 venv
Use the –no-site-packages option to not include globally installed packages.
VIRTUALENV
After creating a new virtual environment, the next step is to activate it.
$ source venv/bin/activate
(venv) $
The name of the virtual environment will appear before your prompt after activation.
Anything you install at this point will install to your isolated environment, not to the
global site packages.
(venv) $ pip install twisted
VIRTUALENV
To deactivate a virtual environment:
(venv) $ deactivate
$
Now, we’re back to using the default Python interpreter and globally installed
packages. You can delete a virtual environment by simply deleting the folder
created, in this case called “venv”.
VIRTUALENV
For distributing your project and/or for easy set-up, freeze the current virtual
environment.
(venv) $ pip freeze > requirements.txt
This creates a list of the installed packages and versions inside of requirements.txt.
This can be used to rebuild the environment later. This is useful for allowing another
developer to run your project without having to figure out what packages and which
versions were used.
VIRTUALENV
Putting all of this together, the typical use of a virtual environment is as
follows:
$ virtualenv venv –no-site-packages
$ source venv/bin/activate
(venv) $ pip install -r requirements.txt
…
(venv) $ deactivate
$
LOGGING
The logging facility for Python is defined in the Standard Library as the module
logging.
You are encouraged to use the logging facility while developing your projects – it
aids with testing and debugging as well as allows your teammates to work with your
code without having to understand every line.
Logging can be used to do the following:
• Report events that occur during normal operation (e.g. for status monitoring, fault
investigation, or development).
• Issue a warning regarding a particular runtime event.
• Report suppression of an error without raising an exception (e.g. error handler in a
long-running server process).
LOGGING
There are a number of logging levels. Default level is WARNING, meaning no level
below WARNING will be logged.
Level
Use
DEBUG
Detailed info for development.
INFO
Logging of expected events.
WARNING
Application still behaving normally, but unexpected event happened or problem
is anticipated.
ERROR
Application is not behaving as it should due to some error.
CRITICAL
Critical error encountered that may have taken application down.
LOGGING
Most commonly, logging is performed by recording events to a log file.
import logging
logging.basicConfig(filename='example.log', level=logging.DEBUG)
logging.debug('This message should go to the log file')
logging.info('So should this')
logging.warning('And this, too')
The contents of example.log:
DEBUG:root:This message should go to the log file
INFO:root:So should this
WARNING:root:And this, too
LOGGING
You can log multiple modules in one central location. Simply initialize the logging in
your main module, and log as necessary from the others.
# myapp.py
import logging
import mylib
def main():
logging.basicConfig(filename='myapp.log', level=logging.INFO)
logging.info('Started')
mylib.do_something()
logging.info('Finished')
if __name__ == '__main__':
main()
# mylib.py
import logging
def do_something():
logging.info('Doing something')
LOGGING
You can log multiple modules in one central location. Simply initialize the logging in
your main module, and log as necessary from the others.
# myapp.py
import logging
import mylib
def main():
logging.basicConfig(filename='myapp.log', level=logging.INFO)
logging.info('Started')
mylib.do_something()
logging.info('Finished')
if __name__ == '__main__':
main()
# mylib.py
import logging
def do_something():
logging.info('Doing something')
myapp.log
INFO:root:Started
INFO:root:Doing something
INFO:root:Finished
LOGGING
You can change the format of the logged messages to be more useful.
import logging
logging.basicConfig(filename=“myapp.log”, format='%(asctime)s %(levelname)s %(message)s')
logging.warning(‘an event was logged.')
The contents of myapp.log:
2015-1-29 10:49:46,602 WARNING an event was logged.
LOGGING
A more complicated
example with rotating
log files. The logging
module defines many
components:
Loggers
Handlers
Filters
Formatters
import logging
def init_logging(logfile):
"""Logged messages are written to rotating logfiles of the form *logfile.x* where *x* is
an integer up to 5. The output *logfile* is the most recent output.""“
LOG_FILE = logfile
logger = logging.getLogger(__name__)
#initiate a logger object with the module name
logger.setLevel(logging.DEBUG)
fh = logging.handlers.RotatingFileHandler(LOG_FILE, maxBytes=10000, backupCount=5)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fh.setFormatter(formatter)
logger.addHandler(fh)
logger.info("*****RESTART*****")
if __name__ == “__main__”:
init_logging(‘path/to/log/file')
from twisted.internet import reactor
reactor.run()
logger.info(“Reactor started.”)
UNITTEST
The unittest module in the Standard Library is a framework for writing unit tests, which
specifically test a small piece of code in isolation from the rest of the codebase.
Test-driven development is advantageous for the following reasons:
• Encourages modular design.
• Easier to cover every code path.
• The actual process of testing is less time-consuming.
UNITTEST
Generally speaking, the process for creating unit tests is the following:
1. Define a class derived from unittest.TestCase.
2. Define functions within your class that start with ‘test_’.
3. You run the tests by placing unittest.main() in your file, usually at the bottom.
unittest defines a wide range of assert methods as well as set-up and cleanup
methods which you can use to define your test_ functions.
UNITTEST
Here’s an example of the simplest usage of unittest.
test_even.py
even.py
import unittest
import even
def even(num):
if num%2 == 0:
return True
return False
class EvenTest(unittest.TestCase):
def test_is_two_even(self):
self.assertTrue(even.even(4))
if __name__ == '__main__':
unittest.main()
carnahan@diablo:~>python2.7 test_even.py
.
---------------------------------------------------------------------Ran 1 test in 0.011s
OK
carnahan@diablo:~>
UNITTEST
test_even.py
even.py
import unittest
import even
def even(num):
if num%2 == 0:
return True
return False
class EvenTest(unittest.TestCase):
def test_is_two_even(self):
self.assertTrue(even.even(4))
def test_two_prime_even(self):
self.assertTrue(even.prime_even(2))
if __name__ == '__main__':
unittest.main()
def prime_even(num):
if even(num):
for i in range(num):
if num % i == 0:
return False
return True
return False
UNITTEST
carnahan@diablo:~>python2.7 test_even.py
.E
======================================================================
ERROR: test_two_prime_even (__main__.EvenTest)
---------------------------------------------------------------------Traceback (most recent call last):
File "test_even.py", line 10, in test_two_prime_even
self.assertTrue(even.prime_even(2))
File "/home/faculty/carnahan/CIS4930/demos/even.py", line 9, in prime_even
if num % i == 0:
ZeroDivisionError: integer division or modulo by zero
---------------------------------------------------------------------Ran 2 tests in 0.001s
FAILED (errors=1)
carnahan@diablo:~>
UNITTEST
test_even.py
even.py
import unittest
import even
def even(num):
if num%2 == 0:
return True
return False
class EvenTest(unittest.TestCase):
def test_is_two_even(self):
self.assertTrue(even.even(4))
def test_two_prime_even(self):
self.assertTrue(even.prime_even(2))
if __name__ == '__main__':
unittest.main()
def prime_even(num):
if even(num):
for i in range(2,num):
if num % i == 0:
return False
return True
return False
UNITTEST
carnahan@diablo:~>python2.7 test_even.py
..
---------------------------------------------------------------------Ran 2 tests in 0.000s
OK
carnahan@diablo:~>
We incrementally add unit test functions and run them – when they pass, we add more code and
develop the unit tests to assert correctness. Do not remove unit tests as you pass them.
It’s advisable to create pre-commit hooks on your repository that force the code to pass the unit
tests or else changes cannot be pushed.
SPHINX
Don’t have time to cover today, but check out the Sphinx package for your
documentation! It make it easy to create readable docs.
NEXT LECTURE
Starting Python applications.