Full Stack Python: 2020 Supporter's Edition
Table of Contents
1. Introduction........................................ 8
1.1 Learning Programming............................ 8
The Python Language............................ 12
Why Use Python?................................ 16
Python 2 or 3?................................. 19
Enterprise Python.............................. 23
1.2 Python Community............................... 27
Companies Using Python......................... 30
Best Python Resources.......................... 34
Must-watch Python Videos....................... 38
Podcasts....................................... 42
2. Development Environments........................... 47
2.1 Text Editors and IDEs.......................... 50
Vim............................................ 54
Emacs.......................................... 61
Sublime Text................................... 64
PyCharm........................................ 67
Jupyter Notebook............................... 69
2.2 Shells......................................... 74
Bourne-again shell (Bash)...................... 75
Zsh............................................ 77
PowerShell..................................... 78
2.3 Terminal multiplexers.......................... 80
tmux........................................... 82
Screen......................................... 83
2.4 Environment configuration...................... 83
Application dependencies....................... 85
virtual environments (virtualenvs)............. 91
Localhost tunnels.............................. 92
2.5 Source Control................................. 92
Git............................................ 98
Mercurial..................................... 104
3. Data.............................................. 107
3.1 Relational databases.......................... 111
2
Full Stack Python: 2020 Supporter's Edition
3.2
3.3
3.4
3.5
3.6
4. Web
4.1
4.2
PostgreSQL.................................... 116
MySQL......................................... 123
SQLite........................................ 127
Object-relational mappers..................... 131
SQLAlchemy.................................... 138
Peewee........................................ 143
Django ORM.................................... 146
Pony ORM...................................... 150
NoSQL......................................... 151
Redis......................................... 155
MongoDB....................................... 159
Apache Cassandra.............................. 162
Neo4j......................................... 165
Data analysis................................. 166
pandas........................................ 169
SciPy & NumPy................................. 172
Data visualization............................ 175
Bokeh......................................... 178
d3.js......................................... 181
Matplotlib.................................... 184
Markup Languages.............................. 185
Markdown...................................... 185
reStructuredText.............................. 188
Development................................... 190
Web Frameworks................................ 193
Django........................................ 198
Flask......................................... 206
Bottle........................................ 212
Pyramid....................................... 215
Falcon........................................ 221
Morepath...................................... 221
Sanic......................................... 223
Other web frameworks.......................... 225
Template Engines.............................. 227
Jinja2........................................ 231
Mako.......................................... 234
3
Full Stack Python: 2020 Supporter's Edition
Django Templates.............................. 235
4.3 Web design.................................... 236
HTML.......................................... 240
CSS........................................... 242
Responsive Design............................. 248
Minification.................................. 250
4.4 CSS Frameworks................................ 251
Bootstrap..................................... 251
Foundation.................................... 253
4.5 JavaScript.................................... 253
React......................................... 257
Vue.js........................................ 258
Angular....................................... 261
4.6 Task queues................................... 261
Celery........................................ 266
Redis Queue (RQ).............................. 270
Dramatiq...................................... 272
4.7 Static site generators........................ 272
Pelican....................................... 278
Lektor........................................ 280
MkDocs........................................ 282
4.8 Testing....................................... 283
Unit testing.................................. 287
Integration testing........................... 290
Debugging..................................... 291
Code Metrics.................................. 294
4.9 Networking.................................... 297
HTTPS......................................... 298
WebSockets.................................... 300
WebRTC........................................ 307
4.10 Web APIs..................................... 309
Microservices................................ 312
Webhooks..................................... 314
Bots......................................... 315
4.11 API creation................................. 318
API Frameworks............................... 323
4
Full Stack Python: 2020 Supporter's Edition
Django REST Framework........................ 324
4.12 API integration.............................. 325
Twilio....................................... 328
Stripe....................................... 330
Slack........................................ 331
Okta......................................... 333
4.13 Web application security..................... 334
SQL injection................................ 338
Cross Site Request Forgery................... 339
5. Web App Deployment................................ 341
5.1 Hosting....................................... 346
Servers....................................... 346
Static content................................ 350
Content Delivery Networks..................... 351
5.2 Virtual Private Servers (VPS)................. 352
Linode........................................ 353
DigitalOcean.................................. 353
Lightsail..................................... 354
5.3 Platform-as-a-Service......................... 354
Heroku........................................ 359
PythonAnywhere................................ 360
AWS Codestar.................................. 360
5.4 Operating systems............................. 361
Ubuntu Linux.................................. 365
macOS......................................... 367
FreeBSD....................................... 369
Windows....................................... 367
5.5 Web servers................................... 369
Apache HTTP Server............................ 373
Nginx......................................... 374
Caddy......................................... 378
5.6 WSGI servers.................................. 379
Green Unicorn................................. 384
uWSGI......................................... 387
mod_wsgi...................................... 387
5.7 Continuous integration........................ 388
5
Full Stack Python: 2020 Supporter's Edition
Jenkins....................................... 392
GoCD.......................................... 394
5.8 Configuration management...................... 395
Ansible....................................... 397
Salt.......................................... 399
5.9 Containers.................................... 400
Docker........................................ 403
Kubernetes.................................... 406
5.10 Serverless Architectures..................... 410
AWS Lambda................................... 414
Azure Functions.............................. 418
Google Cloud Functions....................... 419
6. DevOps............................................ 421
6.1 Monitoring.................................... 423
Datadog....................................... 427
Prometheus.................................... 428
Rollbar....................................... 429
Sentry........................................ 429
6.2 Web App Performance........................... 430
Logging....................................... 433
Caching....................................... 432
Web Analytics................................. 437
7. Meta
About the author.................................. 442
What "full stack" means........................... 442
6
Full Stack Python: 2020 Supporter's Edition
Introduction
You're knee deep in learning Python programming. The syntax is starting to
make sense. The first few "ahh-ha!" moments hit you as you learn conditional
statements, for loops and classes while playing around with the open source
libraries that make Python such an amazing programming ecosystem.
Now you want to take your initial Python knowledge and make something
real, like a web application to show off or sell as a service to customers. That's
where Full Stack Python comes in. You've come to the right place to learn
everything you need to know to create, deploy and operate your Pythonpowered projects.
Learning Programming
Learning to program is about understanding how to translate thoughts into
source code that can be executed on computers to achieve one or more goals.
There are many steps in learning how to program, including
1. setting up a development environment
2. selecting a programming language, of which Python is just one of many
amazing ecosystems that you can decide to use
3. understanding the syntax and commands for the language
4. writing code in the language, often using pre-existing code libraries
and frameworks
5. executing the program
6. debugging errors and testing for unexpected results
7. deploying an application so it can run for intended users
How should I learn programming?
There are several schools of thought on how a person should start learning to
program. One school of thought is that a lower-level programming language
such as Assembly or C are the most appropriate languages to start with
because they force new developers to write their own data structures, learn
about pointers and generally work their way through the hard problems in
computer science.
8
Full Stack Python: 2020 Supporter's Edition
There's certainly wisdom in this "low-level first" philosophy because it forces
a beginner to gain a strong foundation before moving on to higher level topics
such as web and mobile application development. This philosophy is the one
most commonly used in university computer science programs.
The atomic units of progress in the "low-level first" method of learning are
1. aspects of programming language understood (type systems, syntax)
2. number of data structures coded and able to be used (stacks, queues)
3. algorithms in a developer's toolbelt (quicksort, binary search)
Another school of thought is that new developers should bootstrap themselves
through working on projects in whatever programming language interests
them enough to keep working through the frustrations that will undoubtably
occur.
In this "project-based" line of thinking, the number of projects completed that
expand a programmer's abilities are the units of progress. Extra value is
placed on making the projects open source and working with experienced
mentors to learn what he or she can improve on in their programs.
Another way to learn that combines the project-based learning with defined
objectives is to play a computer game that will guide you through the learning
process. For example, TwilioQuest teaches the basics of Python in one of its
missions and then has a ton of free content for studying intermediate and
advanced topics.
Should I learn Python first?
Python is good choice in the project-based approach because of the extensive
availability of free and low cost introductory resources, many of which provide
example projects to build upon.
Note that this question of whether or not Python is a good first language for
an aspiring programmer is highly subjective and these approaches are not
mutually exclusive. Python is also widely taught in universities to explain the
fundamental concepts in computer science, which is in line with the "low-level
first" philosophy than the projects-first method.
In a nutshell, whether Python is the right first programming language to learn
is up to your own learning style and what feels right. If Ruby or Java seem like
9
Full Stack Python: 2020 Supporter's Edition
they are easier to learn than Python, go for those languages. Programming
languages, and the ecosystems around them, are human-made constructs.
Find one that appears to match your personal style and give it a try, knowing
that whatever you choose you'll need to put in many long days and nights to
really get comfortable as a software developer.
Practice problems
Working on practice programming challenges and studying their solutions in
Python or another language is a great way to learn whether you are just
starting or an experienced developer. Here are numerous open source
repositories and sites with practice problems and solutions:
• Pytudes are an awesome collection of Python programs to practice and
demonstrate skills. These problems go above and beyond the common
data structures and algorithm questions often found in other practice
problem sets.
• Interactive Python coding interview challenges is an awesome Jupyter
Notebook to learn and test your data structures and algorithms
knowledge in Python.
• Kindling project provides a wonderful list of resources that challenge
beginners with programming problems that beginners can solve to
grow their skills.
• Build your own "x" does not contain practice problems but instead
provides tutorials for how to build your own programming languages,
blockchain, bots, databases, frameworks and more awesome projects.
• Python Programming Exercises is a free short PDF book with exercises
across many standard Python language features such as dictionaries,
classes and functions.
• Code problems provides common algorithm and data structures
challenges with solutions in several programming languages including
Python.
• Python basics contains materials and exercises to learn basic Python 3
syntax such as variables, functions and lists.
• TeachCraft combines Minecraft with Python to learn coding.
10
Full Stack Python: 2020 Supporter's Edition
• 500 Data Structures and Algorithms practice problems and their
solutions covers a large swath of the computer science space. It is not
important to know all of these algorithms and data structures but
experience with many of them will be greatly beneficial in becoming a
better developer.
First-hand advice
These articles are written by programmers who explain how they learned to
code. They should not be taken as "this is how you must learn" but instead
give example paths you can think about taking as a beginner:
• Learning to program is a long read but goes through Dan's experience
in math and engineering before fully committing to software
development.
• Developing as a developer gives general advice on qualities necessary
to become a programmer, including persistence, respecting others and
considering ideas that are outside your comfort zone.
• Mastering programming by Kent Beck contains patterns and
observations for how experienced programmers he has worked with in
the past became great software developers.
• This Picture Will Change the Way You Learn to Code covers a well done
graphics of many up-to-date concepts and tools that developers use.
The post reminds you that you will not and should not learn everything
but that you should pick tools you want to gain experience in while
generally knowing what else is out there.
Teaching perspectives
Are you an experienced programmer working with new and junior
programmers? These articles give some insight into how you may want to
structure your teaching experience:
• Five Principles For Programming Languages For Learners is a
perspective on teaching children to program but is good advice for an
audience of any age.
11
Full Stack Python: 2020 Supporter's Edition
• Teach tech with cartoons is an awesome resource that explains how you
can use simple but fun drawings to teach otherwise difficult technical
concepts to students.
• Teaching programming to working professionals is a video podcast
with Trey Hunter about his experience teaching Python to experienced
professionals.
• Teach Yourself Computer Science is intended as a self-teaching tool
with many resources that are classic computer science textbooks. There
are also nice explanations for why each resource is useful in your
learning and teaching journey.
Python Programming Language
The Python programming language is an open source, widely-used tool for
creating software applications.
What is Python used for?
Python is often used to build and deploy web applications and web APIs.
Python can also analyze and visualize data and test software, even if the
software being tested was not written in Python.
Language concepts
Python has several useful programming language concepts that are less
frequently found in other languages. These concepts include:
• generators
• comprehensions
• application dependency handling via the built-in venv (as of Python
3.3) and pip (as of Python 3.4) commands
Generators
Generators are a Python core language construct that allow a function's return
value to behave as an iterator. A generator can allow more efficient memory
usage by allocating and deallocating memory during the context of a large
number of iterations. Generators are defined in PEP255 and included in the
language as of Python 2.2 in 2001.
12
Full Stack Python: 2020 Supporter's Edition
Comprehensions
Comprehensions are a Python language construct for concisely creating data
in lists, dictionaries and sets. List comprehensions are included in Python 2
while dictionary and set comprehensions were introduced to the language in
Python 3.
Why are comprehensions important?
Comprehensions are a more clear syntax for populating conditional data in
the core Python data structures. Creating data without comprehensions often
involves nested loops with conditionals that can be difficult for code readers
to properly evaluate.
Example comprehensions code
List comprehension:
>>> double_digit_evens = [e*2 for e in range(5, 50)]
>>> double_digit_evens
[10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 5
8, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 82, 84, 86, 88, 90, 92, 94, 96, 98]
Set comprehension:
>>> double_digit_odds = {e*2+1 for e in range(5, 50)}
{11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31, 33, 35, 37, 39, 41, 43, 45, 47, 49, 51, 53, 55, 57, 5
9, 61, 63, 65, 67, 69, 71, 73, 75, 77, 79, 81, 83, 85, 87, 89, 91, 93, 95, 97, 99}
Dictionary comprehension:
>>> {e: e*10 for e in range(1, 11)}
{1: 10, 2: 20, 3: 30, 4: 40, 5: 50, 6: 60, 7: 70, 8: 80, 9: 90, 10: 100}
General Python language resources
• The online Python tutor visually walks through code and shows how it
executes on the Python interpreter.
• Python Module of the Week is a tour through the Python standard
library.
• A Python interpreter written in Python is incredibly meta but really
useful for wrapping your head around some of the lower level stuff
going on in the language.
13
Full Stack Python: 2020 Supporter's Edition
• A few things to remember while coding in Python is a nice collection of
good practices to use while building programs with the language.
• Python internals: adding a new statement to Python
• Python tricks that you can't live without is a slideshow by Audrey Roy
that goes over code readability, linting, dependency isolation, and other
good Python practices.
• Python innards introduction explains how some of Python's internal
execution happens.
• What is a metaclass in Python is one of the best Stack Overflow
answers about Python.
• Armin Roacher presented things you didn't know about Python at
PyCon South Africa in 2012.
• Writing idiomatic Python is a guide for writing Pythonic code.
Python ecosystem resources
There's an entire page on best Python resources with links but the following
resources are a better fit for when you're past the very beginner topics.
• The Python Subreddit rolls up great Python links and has an active
community ready to answer questions from beginners and advanced
Python developers alike.
• The blog Free Python Tips provides posts on Python topics as well as
news for the Python ecosystem.
• Python Books is a collection of freely available books on Python,
Django, and data analysis.
• Python IAQ: Infrequently Asked Questions is a list of quirky queries on
rare Python features and why certain syntax was or was not built into
the language.
• A practical introduction to Functional Programming for Python coders
is a good starter for developers looking to learn the functional
programming paradigm side of the language.
14
Full Stack Python: 2020 Supporter's Edition
• Getting Started with the Python Internals takes a slice of the huge
CPython codebase and deconstructs some of it to see what we can learn
about how Python itself is built.
Comprehension resources
• Comprehending Python’s Comprehensions is an awesome post by Dan
Bader with a slew of examples that explain how list, dictionary and set
comprehensions should be used.
• Python List Comprehensions: Explained Visually explains how the
common idiom for iteration became syntactic sugar in the language
itself and how you can use it in your own programs.
• The Python 3 Patterns and Idioms site has an overview of
comprehensions including code examples and diagrams to explain how
they work.
• Comprehensions in Python the Jedi way shows off comprehensions
with a Star Wars theme to walk through the nuts and bolts. All
examples use Python 3.5.
• Idiomatic Python: Comprehensions explains how Python's
comprehensions were inspired by Haskell's list comprehensions. It also
provides clear examples that show how comprehensions are shorthand
for common iteration code, such as copying one list into another while
performing some operation on the contained elements.
• List comprehensions in Python covers what the code for list
comprehensions looks like and gives some example code to show how
they work.
• An Introduction to Python Lists is a solid overview of Python lists in
general and tangentially covers list comprehensions.
Python generator resources
• This blog post entitled Python Generators specifically focuses on
generating dictionaries. It provides a good introduction for those new
to Python.
15
Full Stack Python: 2020 Supporter's Edition
• Generator Expressions in Python: An Introduction is the best allaround introduction to how to use generators and provides numerous
code examples to learn from.
• Python 201: An Intro to Generators is another short but informative
read with example generators code.
• Iterators & Generators provides code examples for these two constructs
and some simple explanations for each one.
• Python: Generators - How to use them and the benefits you receive is a
screencast with code that walks through generators in Python.
• The question to Understanding Generators in Python? on Stack
Overflow has an impressive answer that clearly lays out the code and
concepts involved with Python generators.
• Generator Tricks for Systems Programmers provides code examples for
using generators. The material was originally presented in a PyCon
workshop for systems programmers but is relevant to all Python
developers working to understand appropriate ways to use generators.
Why Use Python?
Python's expansive library of open source data analysis tools, web
frameworks, and testing instruments make its ecosystem one of the largest
out of any programming community.
Python is an accessible language for new programmers because the
community provides many introductory resources. The language is also
widely taught in universities and used for working with beginner-friendly
devices such as the Raspberry Pi.
16
Full Stack Python: 2020 Supporter's Edition
Python's programming language popularity
Several programming language popularity rankings exist. While it's possible
to criticize that these guides are not exact, every ranking shows Python as a
top programming language within the top ten, if not the top five of all
languages.
The IEEE ranked Python as the #1 programming language in 2019, which
continued its hot streak after ranking it #1 in 2018, #1 in 2017 and #3 top
programming language in 2016. RedMonk's June 2019 ranking had Python at
#3, which held consistent from previous years' rankings in 2018 and 2017.
Stack Overflow's community-created question and answer data confirms the
incredible growth of the Python ecosystem and tries to determine why it
growing so quickly with their own analysis. In the 2020 Stack Overflow
developer survey the data indicated that Python was the fastest growing major
programming language and that there is a close alignment between the
languages and tools that developers choose to learn and the usage in
developers' professional work.
The TIOBE Index a long-running language ranking, has Python moving up
the charts to #3, climbing from #8 just a few years ago.
The PopularitY of Programming Language (PYPL), based on leading
indicators from Google Trends search keyword analysis, shows Python at #1.
GitHut, a visualization of GitHub language popularity, pegs Python at #3
overall.
These rankings provide a rough measure for language popularity. They are
not intended as a precise measurement tool to determine exactly how many
developers are using a language. However, the aggregate view shows that
Python remains a stable programming language with a growing ecosystem.
Why does the choice of programming language matter?
Programming languages have unique ecosystems, cultures and philosophies
built around them. You will find friction with a community and difficulty in
learning if your approach to programming varies from the philosophy of the
programming language you've selected.
17
Full Stack Python: 2020 Supporter's Edition
Python's culture values open source software, community involvement with
local, national and international events and teaching to new programmers. If
those values are also important to you and/or your organization then Python
may be a good fit.
The philosophy for Python is so strongly held that it's even embedded in the
language as shown when the interpreter executes "import this" and displays
The Zen of Python.
>>> import this
The Zen of Python, by Tim Peters
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
More perspectives on using Python
Programming language rankings and the philosophy behind a language
provide solid initial data points for why you should learn Python. These
resources also give perspectives on why people switched from other
programming communities and why they advocate for Python as a primary
language.
• How to argue for Python’s use explains that choosing a programming
language can be complicated but that Python is a very good option for
many use cases.
• Why I Push for Python gives one professor's rationale for promoting
Python to teach programming to undergraduates.
18
Full Stack Python: 2020 Supporter's Edition
• If you're wondering about the differences in Python's dynamically
typed system versus statically typed languages, be sure to read this
thorough explanation of the topic.
• Why I swapped C#.NET for Python as my default language and
platform (and won’t be going back) provides a viewpoint from someone
who is not a professional developer but uses coding to hack out some
projects. He found Microsoft's .NET ecosystem lacking when it came to
satisfying his needs and Python filled the gap for him with its wide
array of open source code libraries, package management and ability to
work well on platforms other than Windows.
• Python, Machine Learning, and Language Wars compares Python with
R, MATLAB and Julia for data science work. While Python is great for
deployment automation and web development, many non-developers
are first introduced to the language and ecosystem while getting data
analysis work done.
• Evangelizing Python for Business contains helpful hints if you are
trying to convince your company to use Python, particularly for web
development.
• Python: Beyond Just Web Apps supplies non-web development project
examples that use Python. The article also does a solid job comparing
and contrasting Python to other common programming languages such
as Java, Ruby and JavaScript.
• Python or Ruby for web development gives reasons for one language
compared to the other just focused on the web development space.
• If you are worried about the community split between Python 2 and
Python 3, you should go with Python 3. To read more about how
companies handled upgrading, check out Instagram's smooth move to
Python 3, practical steps for moving to Python 3 and lessons learned
from migrating to Python 3.
Python 2 or 3?
The Python programming language is almost finished with a long-term
transition from version 2 to version 3. New programmers often have
questions about which version they should learn. It can be confusing to hear
19
Full Stack Python: 2020 Supporter's Edition
that Python 3, which was originally released in 2008, is still not the default
installation on some operating systems such as macOS. However, that
situation is rapidly changing as the final version 2 release, Python 2.7, is
approaching its end-of-life that is definitively scheduled for January 1, 2020.
The simple answer right now is: learn Python 3, specifically the latest version
which as of October 2019 is Python 3.7. If for some reason you absolutely have
to learn Python 2, for example because your employer is working on a bunch
of legacy code, you will be able to transfer the majority of your knowledge
from Python 2 right into Python 3. Likewise, you will still be able to read and
write Python 2 code if you start with Python 3.
There are enough great resources out there that will teach you to code in
version 3 without any prior version 2 experience. Python 3 is the future and
you will not regret starting with the latest version of the programming
language.
There is one small caveat to the recommendation to go full-on Python 3. You
may infrequently come across lesser-used open source code libraries that
were originally written in Python 2 that do not completely support Python 3.
That was the case before 2019 with DevOps configuration management tools
such as Fabric or Ansible. However, those libraries now support Python 3 and
the usage problems that were frequent in years past are now typically not a
concern. Knowing how to upgrade Python 2 libraries to 3.x is still a useful skill
to apply at the edges of the Python open source community.
Visualizations and Projects
Since upgrading from Python 2 to 3 has been such a huge undertaking within
the community, many projects have sprung up to make the transition easier.
• six is a 2/3 compatibility library that is a dependency for many popular
Python projects to make it easier to support both Python 2 and 3 at the
same time.
• Python 3 Readiness is a visualization of which most popular 360
libraries (by downloads) are ready to be used with Python 3.
• The Python clock counts down the time until Python 2.x is no longer
maintained. While Python 2's retirement may still seem a long time
20
Full Stack Python: 2020 Supporter's Edition
away, it can take a lot of time and effort to migrate existing application
to the modified syntax in 3.x.
Porting to Python 3 resources
Moving an existing codebase to Python 3 from 2 can be a daunting task, These
resources were created by fellow developers who've previously gone through
the process and have advice for making it less painful.
• Python 3 Porting is an entire book with details for how to upgrade your
existing projects and libraries to Python 3.x.
• Moving from Python 2 to Python 3 is a PDF cheatsheet for porting your
Python code.
• The official porting code to Python 3 page links to resources on porting
Python code as well as underlying C implementations. There is also a
quick reference for writting code with Python 2 and 3 compatibility.
• Upgrading to Python 3 with Zero Downtime supplies advice on
transitioning a large existing Python 2 web application to Python 3.
Their process involved upgrading dependencies, testing and deploying
the new version before going back to clean up unnecessary code created
by the transition.
• Migrating to Python 3 with pleasure is a porting guide that focuses on
code that data scientists would typically use in their programs.
• Instagram Makes a Smooth Move to Python 3 explains their upgrade
process for getting all of their code over to Python 3 compatibility over
a period of about a year.
• Practical steps for moving to Python 3 is a podcast episode that goes
over migrating a large existing application's codebase to Python 3 from
Python 2.
• Lessons learned from migrating to Python 3 covers how a development
team with a large e-commerce site built on Django was able to upgrade
their project.
21
Full Stack Python: 2020 Supporter's Edition
Python 2 to 3 resources
The following resources will give you more context on how the community
feels the transition from Python 2 to 3 is going, as well as why you should
upgrade as soon as possible.
• Why should I use Python 3? is a detailed FAQ on important topics such
as unicode support, iteration improvements and async upgrades
provided by 3.x. There is also a great follow up post by the author titled
A Rebuttal For Python 3 that counters some arguments made by other
community members who are unhappy about various features in
Python 3.
• Want to know all of the advantages and what's changed in Python 3
compared to Python 2? There's an official guide to Python 3 changes
you'll want to read.
• Python 3 is winning presents data and graphs from PyPI to show that
at the current rate, by mid-2016 overall Python 3 library support will
overtake Python 2 support.
• Python 3 Retrospective from the Benevolent Dictator for Life is a talk
by Guido van Rossum on what is working, not working and still needs
to be done before the changover can be considered complete.
• The stages of the Python 3 transition provides perspective from a core
Python developer on how the transition from Python 2 to 3 is going as
of the end of 2015.
• How Dropbox rolled out one of the largest Python 3 migrations ever
explains how their transition began in 2015 and was successfully
completed in 2018. There is also a follow up post on incrementally
migrating over one million lines of code from Python 2 to Python 3 that
has more details on how hack weeks were able to help make enough
progress so the engineers could better estimate the scope of work when
the transition from 2 to 3 became critical to their development
toolchain.
• Zato: A successful Python 3 migration story examines the background,
preparation, execution and testing of moving an existing Python 2 code
base over to Python 3.
22
Full Stack Python: 2020 Supporter's Edition
• Why Python 3? randomly outputs valid reasons to use Python 3 over
2.x.
• Rules for Radicals: Changing the Culture of Python at Facebook is a
fascinating look at how Facebook moved from primarily Python 2 up to
Python 3 due to the efforts of a small passionate group of developers
within the company. Definitely worth watching to understand how to
shift a large organization with an established codebase.
• Porting to Python 3 is like eating your vegetables explains that there
are treats in Python 3 that are worth porting for and has some tips on
making the transition easier.
• Scrapy on the road to Python 3 support explains from the perspective
of a widely used Python project what their plan is for supporting
Python 3 and why it has taken so long to make it happen.
• All major scientific Python libraries have pledged to drop Python 2
support no later than 2020, when Python 2's maintenance life is over.
The pledge strongly encourages Python 3 adoption by publicly stating
their intentions.
• 10 awesome features of Python that you can't use because you refuse to
upgrade to Python 3 is a great slideshow with code snippets that show
useful new features of Python 3 that are not available in 2.x such as
keyword-only arguments, chained exceptions and the yield from
keyword.
Enterprise Python
One of the misconceptions around Python and other dynamically-typed
languages is that they cannot be reliably used to build enterprise-grade
software. However, almost all commercial and government enterprises
already use Python in some capacity, either as glue code between disparate
applications or to build the applications themselves.
What is enterprise software?
Enterprise software is built for the requirements of an organization rather
than the needs of an individual. Software written for enterprises often needs
to integrate with legacy systems, such as existing databases and non-web
23
Full Stack Python: 2020 Supporter's Edition
applications. There are often requirements to integrate with authentication
systems such as the Lightweight Directory Access Protocol (LDAP) and Active
Directory (AD).
Organizations develop enterprise software with numerous custom
requirements to fit the specific needs of their operating model. Therefore the
software development process often becomes far more complicated due to
disparate factions within an organization vying for the software to handle
their needs at the expense of other factions.
The complexity due to the many stakeholders involved in the building of
enterprise software leads to large budgets and extreme scrutiny by nontechnical members of an organization. Typically those non-technical people
place irrational emphasis on the choice of programming language and
frameworks when otherwise they should not make technical design decisions.
Why are there misconceptions about Python in enterprise
environments?
Traditionally large organizations building enterprise software have used
statically typed languages such as C++, .NET and Java. Throughout the 1980s
and 1990s large companies such as Microsoft, Sun Microsystems and Oracle
marketed these languages as "enterprise grade". The inherent snub to other
languages was that they were not appropriate for CIOs' difficult technical
environments. Languages other than Java, C++ and .NET were seen as risky
and therefore not worthy of investment.
In addition, "scripting languages" such as Python, Perl and Ruby were not yet
robust enough in the 1990s because their core standard libraries were still
being developed. Frameworks such as Django, Flask and Rails (for Ruby) did
not yet exist. The Web was just beginning and most enterprise applications
were desktop apps built for Windows. Python simply wasn't made for such
environments.
Why is Python now appropriate for building enterprise
software?
From the early 2000s through today the languages and ecosystems for many
dynamically typed languages have greatly improved and often surpassed some
24
Full Stack Python: 2020 Supporter's Edition
aspects of other ecosystems. Python, Ruby and other previously derided
languages now have vast, well-maintained open source ecosystems backed by
both independent developers and large companies including Microsoft, IBM,
Google, Facebook, Dropbox, Twilio and many, many others.
Python's open source libraries, especially for web development and data
analysis, are some of the best maintained and fully featured pieces of code for
any language.
Meanwhile, some of the traditional enterprise software development
languages such as Java have languished due to underinvestment by their
major corporate backers. When Oracle purchased Sun Microsystems in 2009
there was a long lag time before Java was enhanced with new language
features in Java 7. Oracle also bundles unwanted adware with the Java
installation, whereas the Python community would never put up with such a
situation because the language is open source and does not have a single
corporate controller.
Other ecosystems, such as the .NET platform by Microsoft have fared much
better. Microsoft continued to invest in moving the .NET platform along
throughout the early part of the new millennium.
However, Microsoft's enterprise products often have expensive licensing fees
for their application servers and associated software. In addition, Microsoft is
also a major backer of open source, especially Python, and their Python tools
for Visual Studio provide a top-notch development environment.
The end result is that enterprise software development has changed
dramatically over the past couple of decades. CIOs and technical executives
can no longer ignore the progress of Python and the great open source
community in the enterprise software development landscape if they want to
continue delivering business value to their business side customers.
Open source enterprise Python projects
Python is widely used across large enterprise organizations but the code is
often not put out as open source. If you come across projects that are
appropriate for this list, contact me to let me know:
• Collab by the U.S. government's Consumer Financial Protection Bureau
(CFPB) agency is a corporate intranet and collaboration platform for
25
Full Stack Python: 2020 Supporter's Edition
large organizations. The project is currently running and in-use by
thousands of CFPB employees.
• Pants is a build system for software projects with many distinct parts
and built with many different programming languages as is often the
case in large organizations.
Enterprise Python software development resources
The following articles cover topics in enterprise development that are often
not discussed when dealing with standard Python development.
• Talk Python to Me's fourth episode interviewed PayPal's lead developer
on Enterprise Python and Large-Scale Projects. They rebuke many of
the myths around Python for large scale projects including the variable
typing system and scalability.
• Building and deploying an Enterprise Django Web App in 16 hours
covers one developer's experience researching, building and deploying
an enterprise application in Python and Django.
• Mozilla's Enterprise InfoSec resources are programming language
agnostic but very useful to developers trying to understand all the
jargon that goes along with the enterprise security domain.
• The end of enterprise IT is a fascinating essay that actually does not
talk about Python in particular but shows how large enterprise IT
departments such as the one at ING Bank have to evolve their
structure, processes and tools to successfully ship software.
Programming languages such as Python are more likely to be used in
these updated polyglot and DevOps-driven environments.
• There are a couple of solid demystifying articles in CIO magazine
including this broad overview of Python in enterprises and this article
on why dynamic languages are gaining share for enterprise
development.
• JavaWorld wrote an interesting article about Python's inroads into
enterprise software development.
26
Full Stack Python: 2020 Supporter's Edition
• I gave a talk at DjangoCon 2014 on How to Solve the Top 5 Headaches
with Django in the Enterprise which covered both Python and Django
in large organizations.
• This StackExchange answer contains a solid explanation what
differentiates enterprise software from traditional software.
• There was a Python subreddit thread about Python in the enterprise
that's worth a look for broader opinions on Python compared to Java
and .NET in enterprise environments.
• Why are enterprises so slow? is not specific to Python but is a fantastic
article on the regulatory, cultural and financial reasons why large
companies often move so slowly.
Python Community
The Python programming language has a global community with millions of
software developers who interact online and offline in thousands of virtual
and physical locations.
Who drives the Python community?
There are tens of thousands of Python developers who help steer the
community with local, regional and global events. Most, if not all of them, are
members of the Python Software Foundation (PSF). The PSF is a 501(c)(3)
non-profit with the mission to "promote, protect, and advance the Python
programming language, and to support and facilitate the growth of a diverse
and international community of Python programmers." Anyone that uses
Python and has an active interest in the Python community can join the PSF
as a member. There are five classes of PSF members:
27
Full Stack Python: 2020 Supporter's Edition
1.
2.
3.
4.
5.
Basic Members
Supporting Members
Sponsor Members
Managing Members
Contributing Members and Fellows
Starting out by joining as a basic PSF member is a great way to show your
support and begin your own journey in working with the larger open source
ecosystem.
What is a PEP and why do they matter?
Python Enhancement Proposals (PEPs) are design documents that serve to
drive Python's continued evolution. There are three kinds of PEPs that serve
disinct purposes:
1. Standards track: enhances the Python language with new features
2. Informational: provides information to the Python community
3. Process: modifies or makes improvements on topics relevant to the
community but outside the Python language itself
PEP 1 defines what a PEP is and its purpose. In true computer science
fashion, PEP 0 also exists and it is an index of all the PEPs that have been
created.
The PEPs are important because they drive a transparent process to evolve
the language and broader ecosystems. Some other programming communities
are opaquely pushed by a single person or small cliques who refuse to
understand outside perspectives. The insular nature of some groups typically
causes decline over time as the original community members move on to new
projects and no new members take their place.
Conferences and events
The Python online community has fantastic resources for learning but talking
to your fellow developers in-person at conferences, meetups and hackathons
is a crucial way to discover new tools and coding approaches. The following
resources provide perspective on offline events like PyCon US.
• How to have a great first PyCon gives a slew of advice on how to get the
most out of your busy, and often overwhelming, PyCon experience.
28
Full Stack Python: 2020 Supporter's Edition
• Clinton Dreisbach wrote two fantastic retrospectives for PyCon US
2015 (his first PyCon) and PyCon US 2016. There are many other
retrospectives for other community-led conferences such as
EuroPython. These summaries can be a great way to get a slice of the
experience before purchasing a ticket and booking a trip.
• One reason among many that Python conferences are spectacular
experiences is the transparency about the how and why decisions are
made for event attendees. For example, PyCon is transparent about the
locations it selects for the conference as well as why the community has
a long-standing tradition of having speakers pay their own way just as
any conference attendee would. From the outside it may appear these
decisions are minor but they instill confidence in the stewardship of the
Python ecosystem by current community leadership.
• How I Learned to Stop Worrying and Love PyCon goes over one firsttime PyCon attendee's nervousness about attending the conference and
how she got over it while having a great time in the process.
• Don't Submit A Talk About Your Great Hack: Jesse's Seven Tips To Get
Into PyCon provides wonderful advice on how to approach the Call for
Proposals (CFP) when you want to speak at a Python conference. The
notes on how to balance specifics with vagarity in your title is
particularly useful.
Python community resources
The Python community exists both online and offline. The following resources
will help you get connected with other Python developers in both forms. They
also provide context on how some of the major decisions are made within the
community.
• The community page on Python.org provides a starter page with links
to community-run newsletters, resources and conferences.
• There are many large active online communities on Reddit and IRC
channels such as #python, #python-dev and #distutils.
• The Python community has a concept known as "Benevolent Dictator
For Life" that may appear odd to newcomers. Essentially, Guido Van
Rossum created the language and still has the ability to decide
29
Full Stack Python: 2020 Supporter's Edition
community arguments one way or the other. This post on the origin of
BDFL has more context about Guido's role.
• Python Community and Python at Dropbox is an interview with Jessica
McKellar, one of the most visible Python core committers and
organizers for her fantastic coding and community work. She explains
what it means to be a member and leader in the larger Python
community.
• The history behind the decision to move Python to GitHub is a
transparent and personal story by one of the Python Core Team
members, Brett Cannon, on why the main Python projects including
the language itself are now hosted for development on GitHub. The
post is a wonderful read about the history of where Python
development was centralized and how it moved from SourceForge to
svg.python.org and then over to GitHub.
• Python community trends in 2017/2018 by Ewa Jodlowska, who is
heavily involved in PyCon and the overall Python ecosystem, provides a
ton of data and metrics on areas of community interest such as Python
developers' locations, Python 3 adoption and years of experience as
professional developers.
• There are some amazing communities within the overall Python
ecosystem, such as PyLadies, which encourages women to be strong,
proactive community members.
Companies using Python
The Python programming language is widely used by companies around the
world to build web apps, analyze data, automate operations via DevOps and
create reliable, scalable enterprise applications.
There is also a fantastic list of organizations using Python on the Python.org
wiki as well as a detailed write-up of several top Python-powered companies
on Real Python's blog.
Many companies do not even realize they are using Python across their
organizations. For example, if a company is a "Java-only shop" but they use
IBM WebSphere as a web application server then they have to use Python to
30
Full Stack Python: 2020 Supporter's Edition
script the server's configuration! Python has a habit of getting in everywhere
regardless of whether the usage is intentional.
Financial institutions
Python is widely-used across financial institutions, whether they are hedge
funds, large banks or regulators (see "Government Agencies" section below).
• Goldman Sachs uses Python and often asks candidates about their
experience with the language during the interview process.
• You can see publicly what companies are using internally by looking at
job descriptions on sites like Glassdoor with "Python Goldman Sachs"
keywords and Indeed for JP Morgan Chase. Salaries and
responsibilities vary widely based on the role and whether Python is
used for data analysis, web application development or DevOps.
• PayPal uses Python across their entire infrastructure and often writes
great technical blog posts on packaging, optimization using C and
configuring DNS.
Large tech companies
Large technology companies tend to be polyglot (use many programming
languages rather than standardizing on one), with Python either as a primary
language or the "glue" that helps all the other languages fit together. The
following articles explain how these leading large companies like Uber, Twilio,
Netflix and Facebook uses Python in their development stacks.
• Uber's tech stack contains a significant amount of Python, which they
documented in a series of engineering posts. Part one describes the
lower backend levels, which are written in Python, with Node.js, Go
and Java mixed in. Part two explains the higher levels of the
marketplace and user interfaces.
• Twilio uses Python with Django and the Wagtail content management
system to power the amazing Twilio documentation as well as
TwilioQuest. They wrote a post about how TwilioQuest was built that
goes into detail on the code including the usage of the front-end Vue.js
framework. Twilio also uses Flask to run the REST API endpoints and
31
Full Stack Python: 2020 Supporter's Edition
open sourced the Flask-RESTful framework so other developers could
cut down the boilerplate in their web APIs.
• Netflix uses Python throughout their organization to run chaos
engineering tests and generally glue together the code from their highfunctioning polyglot teams. Netflix also wrote a 2019 update for PyCon
US to give more detail on what teams and projects work in Python.
• Python 3 at Mozilla explains how their "build system, CI configuration,
test harnesses, command line tooling and countless other scripts, tools
or Github projects are all handled by Python". So just about everything
a developer touches every day to build anything else needs Python to
hook into the larger organization!
• Google uses Python extensively and officially supports it internally as
one of their three core languages, the other two being Java and Golang.
While Google likely has every programming language running
somewhere in their infrastructure, Python receives priority support due
to its core language status.
• Dropbox is well-known for using Python across their application
development, infrastructure and operations. They also did a good job of
cornering the market on hiring well-known Python core contributors
for a period of time, such as Guido van Rossum and Jessica McKellar
(although Jessica is now at a new company that she co-founded).
• Facebook and Instagram use Python 3 at scale. They've been very vocal
about successfully making the migration from the Python 2 world into
Python 3.
• A significant portion of Reddit is built in Python and it is one of the
largest sites at scale to use the programming language.
• Increment covers usage of Python (and other programming languages)
at Lyft, Digital Ocean, Sauce Labs, Slack and Fastly in this awesome
overview post titled "What its like to be a developer at...".
Government agencies
Python usage in government agencies is widespread despite the reputation of
agencies as stodgy late technology adopters. Organizations range from
32
Full Stack Python: 2020 Supporter's Edition
financial industry regulators like the SEC and CFPB, to intelligence agencies
like the CIA, FBI and NSA.
• The Consumer Financial Protection Bureau (CFPB) not only uses
Python for running most of their applications but also open sources
many of those Python projects for other agencies (or any organization)
to use. For example, collab is a Django project that provides an
enterprise application for storing and looking up information on
employees and contractors.
• NASA uses Python extensively and open sources much of their
software.
• The United States Central Intelligence Agency (CIA) appears to be a
huge fan of using Python in its state sponsored hacking tools. They
even published their own Python code conventions documentation due
to how many developers at the agency are using the language.
• The SEC uses Python and proposes organizations use Python to comply
with regulations.
• A quick search for government jobs that require or recommend Python
via USAJobs turns up numerous listings at organizations such as the
Smithsonian Institution, Department of Education, Department of the
Navy and National Institute of Standards and Technology (NIST).
Industry-specific Python guides
Python is so widely used across various industries that developers have
written guides specific to their occupations for how to use Python. The
following resources are guides for using Python in astronomy, social sciences
and other fields rather than specific companies.
• Practical Business Python covers business-related topics such as how to
automate generating large Excel spreadsheets or perform analysis
when your data is locked in Microsoft Office files.
• Practical Python for Astronomers gives open source workshop
materials to teach astronomy students how to use Python for data
analysis.
33
Full Stack Python: 2020 Supporter's Edition
• Python for the Humanities is a textbook on the basics of text processing
in Python. The material ramps up quickly after the first chapter so you
will likely want to combine this walkthrough with other great Python
learning resources.
• Python for Social Scientists and Real Python's Python for Social
Scientists walkthroughs are specific to fields that work with a lot of
data gathered from studies such as psychology, sociology and
economics.
Best Python Resources
The Python community is amazing at sharing detailed resources and helping
beginners learn to program with the language. There are so many resources
out there though that it can be difficult to know how to find them.
This page aggregates the best general Python resources with descriptions of
what they provide to readers.
New to programming
If you're learning your first programming language these books were written
with you in mind. Developers learning Python as a second or later language
should skip down to the next section for "experienced developers".
• TwilioQuest is an free and incredible 16-bit adventure game that
teaches programming in the Python basics mission. It's an absolutely
way to stay engaged with the core concepts that can often be otherwise
dry to learn. There are also more advanced missions to learn about web
APIs, and new learning missions are added regularly on new
programming topics.
• Automate the Boring Stuff with Python is an incredible book for both
non-developers and professional developers alike. Each chapter walks
through a situation that can be automated using Python such as
manipulating images, organizing your files and programmatically
controlling your mouse and keyboard to handle any sort of tasks.
• CS for All is an open book by professors at Harvey Mudd College which
teaches the fundamentals of computer science using Python. It's an
accessible read and perfect for programming beginners.
34
Full Stack Python: 2020 Supporter's Edition
• This short 5 minute video explains why it's better to think of projects
you'd like to build and problems you want to solve with programming.
Start working on those projects and problems rather than jumping into
a specific language that's recommended to you by a friend.
• A Python Crash Course gives an awesome overview of the history of
Python, what drives the programming community and dives into
example code. You will likely need to read this in combination with
other resources to really let the syntax sink in, but it's a great article to
read several times over as you continue to learn.
• The Python projects tag on the Twilio blog is constantly updated with
fun tutorials you can build to learn Python, such as the International
Space Station Tracker with Flask and Redis-Queue, Choose Your Own
Adventures Presentations using Flask and WebSockets and Martianify
Photos with OpenCV.
• A Byte of Python is a beginner's tutorial for the Python language.
• Google put together a great compilation of materials and subjects you
should read and learn from if you want to be a professional
programmer. Those resources are useful not only for Python beginners
but any developer who wants to have a strong professional career in
software.
• The O'Reilly book Think Python: How to Think Like a Computer
Scientist is available in HTML form for free on the web.
• Looking for ideas about what projects to use to learn to code? Check
out this list of 5 programming projects for Python beginners.
• Python for you and me is an approachable book with sections for
Python syntax and the major language constructs. The book also
contains a short guide at the end to get programmers to write their first
Flask web application.
Python for specific occupations
Python is powerful for many professions. If you're seeking to use Python in a
specific field, one of these guides may be the most appropriate for you.
35
Full Stack Python: 2020 Supporter's Edition
• Python for Social Scientists contains a textbook, course outline and
slides for a college course that taught social scientists to use Python for
their profession.
• Practical Business Python is a blog that covers topics such as how to
automate generating large Excel spreadsheets or perform analysis
when your data is locked in Microsoft Office files.
• Python for the Humanities is a textbook and course on the basics of
Python and text processing. Note if you've never worked with Python
before the material ramps up quickly after the first chapter so you will
likely want to combine it with some other introduction to Python
resources.
• Practical Python for Astronomers provides open source workshop
materials for teaching students studying astronomy to use Python for
data analysis.
Experienced developers new to Python
If you can already program in another language, these resources are better for
getting up to speed because they are more concise when explaining
introductory topics.
• Learn Python in y minutes provides a whirlwind tour of the Python
language. The guide is especially useful if you're coming in with
previous software development experience and want to quickly grasp
how the language is structured.
• Developing a Real-Time Taxi App with Django Channels and Angular is
a great tutorial for jumping into a real project instead of a simple
starter application while learning common Python concepts and tools
such as Django, Angular, WebSockets and Redis.
• Developers familiar with other languages often have difficulty adapting
to accepted Python code style. Make sure to read the PEP8 code style
guidelines as well as The Elements of Python Style to know the Python
community standards.
36
Full Stack Python: 2020 Supporter's Edition
• Authentication with Flask, React, and Docker is another detailed
course that shows how to combine Flask, React, Docker and Heroku to
build a solid intermediate-to-advanced web application and deploy it.
• Essential Reads for Any Python Programmer is a great collection of
advice for developers coming to Python from another programming
language ecosystem such as Java.
• How to Develop Quality Python Code is a good read to begin learning
about development environments, application dependencies and
project structure.
• The Python module of the week chapters are a good way to get up to
speed with the standard library. Doug Hellmann is also now updating
the list for changes brought about from the upgrade to Python 3 from
2.x.
• Composing Programs shows how to build compilers with Python 3,
which is a good undertaking if you're looking to learn both more about
the Python language and how compiles work.
• free-for-dev is not specific to Python but it's a fantastic list of free tier
resources for experienced developers. The list is especially handy if you
want to try building a Python project and need new third party services
to kick around while experimenting.
Videos, screencasts and presentations
Videos from conferences and meetups along with screencasts are listed on the
best Python videos page.
Curated Python packages lists
• awesome-python is an incredible list of Python frameworks, libraries
and software. I wish I had this page when I was just getting started.
• easy-python is like awesome-python although instead of just a Git
repository this site is in the Read the Docs format.
• Hacker News Tools of the Trade is not specific to Python but almost all
of the tools and services are useful to building software projects.
37
Full Stack Python: 2020 Supporter's Edition
Podcasts
Take a look at the best Python podcasts section for a curated list of both
Python-specific and general software development podcasts.
Newsletters
Python's active community constantly publishes new tutorials and
walkthroughs. It is easier to keep up if you follow along by subscribing to
several email newsletters that round up and curate the best new resources. I
subscribe to all of the following newsletters and find that each one has its own
unique take on what resources are most important to send out to the
community.
• Python Weekly is a free weekly roundup of the latest Python articles,
videos, projects and upcoming events.
• PyCoder's Weekly is another great free weekly email newsletter similar
to Python Weekly. The best resources are generally covered in both
newsletters but they often cover different articles and projects from
around the web.
• Talk Python's Friends of the Show always highlights new episodes of
this wonderful podcast as well as new useful tools to add to your
developer toolbelt.
• Awesome Python Newsletter provides another solid selection of new
and existing tutorials along with an extensive issues archive with
previous links to resources.
Best Python Videos
If you prefer to learn Python programming by watching videos then this is the
resource for you. I've watched hundreds of live technical talks and combed
through videos to pick out the ones with great speakers who'll teach you the
most about the language and ecosystem.
This page links to the best free videos as well as other video lists so you can do
your own searching through the huge backlog of conference and meetup talks
from the past several years.
38
Full Stack Python: 2020 Supporter's Edition
Web development videos
The following web development videos cover the broad topics of using web
frameworks like Django, Flask, Pyramid and other frameworks, as well as web
design and deployments.
• My EuroPython 2014 "Full Stack Python" talk goes over each topic
from this guide and provides context for how the pieces fit together.
The talk slides are also available. Even though the talk is from 2014,
almost all of the general web development principles remain consistent
in 2018.
• Kate Heddleston gave a talk at PyCon 2014 called "Full-stack Python
Web Applications" with clear visuals for how numerous layers of the
Python web stack fit together. There are also slides available from the
talk with all the diagrams.
• Taking Django Async is a great overview by Andrew Godwin, who
created South (now Django Migrations as part of the core framework)
and Django Channels. He discusses the synchronous blocking worker
design of WSGI and why it is incompatible with asynchronous
protocols like WebSockets. A potential solution could be a new protocol
like Asynchronous Server Gateway Interface (ASGI), but how far would
the integration into Django need to go and would it be worth the pain?
Andrew does a great job of mixing the philosophical questions with
technical implementation details throughout the talk.
• Design 101 for Developers covers a difficult topic for many analyticminded developers to learn: design. The talk is not specific to web
development or web design but instead explains general design
principles such as spacing, consistency and making interactions
obvious for a user.
• How Netflix does failovers in 7 minutes flat by Amjith Ramanujam at
PyCon US was an excellent talk on handling network outages and
predicting the reliability of failover routes. This talk is worth watching
even for the data visualizations alone!
• Kate Heddleston and I gave a talk at DjangoCon 2014 called Choose
Your Own Django Deployment Adventure which walked through many
of the scenarios you'd face when deploying your first Django website.
39
Full Stack Python: 2020 Supporter's Edition
• The Discover Flask series is a detailed Flask tutorial on video with
corresponding code examples on GitHub.
• Designing Django's Migrations covers Django 1.7's new migrations
from the main programmer of South and now Django's built-in
migrations, Andrew Godwin.
• DjangoCon US videos from 2019, 2018, 2017, 2016, 2015, 2014, are all
available free of charge.
• DjangoCon EU videos are also available from 2017, 2016, and 2015.
• GoDjango screencasts and tutorials are free short videos for learning
how to build Django applications.
• The videos and slides from Django: Under the Hood 2015 are from
Django core committers and provide insight into the ORM,
internationalization, templates and other important web framework
topics.
Core Python language videos
The core Python programming language has many new features now that
almost all community resources are working on Python 3 instead of split
across legacy 2.x branches. The following videos cover topics within the core
Python language primarily relevant to Python 3 features although some can
be used with Python 2 as well.
• Jessica McKellar's Building and breaking a Python sandbox is a
fascinating walk through the lower layers of the Python interpreter.
• Brandon Rhodes' All Your Ducks In A Row: Data Structures in the Std
Lib and Beyond goes through how data structures are implemented,
how to select a data structure appropriate to your application and how
the list and dictionary can be used in many situations.
• Guido van Rossum's Python Language keynote talk from PyCon 2016
reinforced that there would be no Python version 2.8 and that
development on backported security releases into the Python 2 branch
would end by January 1, 2020. Guido also covered many topics
important to the Python language community like expanding the
number and backgrounds of core committers.
40
Full Stack Python: 2020 Supporter's Edition
• The talk Python Descriptors by Simeon Franklin explains the what and
why of this core Python language feature.
• David Beazley gives an amazing live coded performance to show
Python concurrency using threads, event loops and coroutines. David
makes the live coding look easy but a whole lot of work must've gone
into that talk.
• What is a Python Core Developer? explains the responsibilities,
projects, repositories and expectations of core Python committers as
well as how to become one.
• Google's Python Class contains lecture videos and exercises for
learning Python.
Video compilations
All major Python conferences, as well as most regional ones, release technical
talk videos for free. These sites either aggregate the thousands of videos that
have been released or are lists from specific conferences like PyCon US and
EuroPython.
• PyVideo organizes and indexes thousands of Python videos from both
major conferences and meetups.
• Incredible Technical Speakers is a repository I put together that
features software developer speakers talking about programming
language agnostic topics. The list is intended to emphasize professional
software developers who also have the ability to engage an audience of
peers with an exciting talk. These talks are relevant to all software
developers even though not every talk is specific to the Python
language.
• PyCon US videos from 2019, 2018, 2017, 2016, 2015 and 2014 are all
available online for free.
• All of the talk videos are available on YouTube for EuroPython 2019,
EuroPython 2018, EuroPython 2017, EuroPython 2016, EuroPython
2015, EuroPython 2014 and earlier years.
41
Full Stack Python: 2020 Supporter's Edition
Best Python Podcasts
The Python community has an embarrassment of riches when it comes to free
and low cost resources for both new and experienced software developers.
These great resources include several Python podcasts that are released on
regular schedules.
This page contains a list of active Python-specific and software engineering
high-quality podcasts.
Python-specific podcasts
These podcasts are run and recorded by Python developers who focus on
topics that are important to our field. Each podcast series has a long list of
previously-recorded episodes that are still relevant even if they are several
years old so you will always have some great material to listen and learn.
• Talk Python to Me focuses on the people and organizations coding on
Python. Each episode features a different guest interviewee to talk
about his or her work.
• Podcast.__init__ is another regular podcast that presents stories about
Python and interviews "with the people who make it great".
• Python Bytes is a new podcast from the creators of the above
mentioned "Talk Python to Me" and "Test and Code Podcast".
• Test and Code Podcast focuses on testing and related topics such as
mocking and code metrics.
• Django Riffs is a podcast for learning web application development in
Python using the Django web framework.
• Teaching Python is a podcast by two teachers about their adventures
teaching middle school computer science, problem solving, handling
failure, frustration, and success with teaching Python programming.
• Professor Philip Guo has a video podcast called PG Podcast, which
typically covers Python subjects.
• The Real Python Podcast is a weekly podcast with interviews, coding
tips, and conversation with guests from the Python community.
42
Full Stack Python: 2020 Supporter's Edition
Favorite podcast episodes
Here are a list of my favorite episodes from various Python podcasts before we
dive into entire podcast recommendations in the next section. Dig into these
episodes to get a feel for various personalities and styles in podcast
presentation.
• SQLAlchemy and data access in Python brought me some muchneeded context and understanding for how the object-relational
mapping library SQLAlchemy evolved. This episode interviews
SQLAlchemy's creator. The show host, Michael Kennedy, asks great
questions based on his in-depth research and prior usage of
SQLAlchemy.
• Python past, present, and future with Guido van Rossum covers the
history of Python, Guido's motivations in creating the language and
shepherding it through almost thirty years of releases. Fun fact: the
question about whether Python being open source contributed to its
success was my question when the podcast host Michael Kennedy
asked me what topics they should talk about.
• Deploying Python Web Applications. Spoiler alert: this is the episode I
was on Talk Python to Me explaining how Python web application
deployments work.
• Python Bytes talked extensively about object-relational mappers
(ORMs) on episode #39 where a lot of the discussion was based on the
Full Stack Python ORMs page. Thanks guys, it was great to hear the
feedback about what worked and what didn't work for you about the
explanations!
• The Python at Netflix episode of Talk Python to Me provided an
awesome view inside the largest internet site by bandwidth and how
Python fits into their polyglot organization.
• Another great Talk Python to Me episode, Python in Finance, explains
how Python is used in the broad finance industry for stock trading,
quantitative analysis and data analytics. Check this one out if you have
wondered how opaque private companies like hedge funds use Python
to make (a lot of) money.
43
Full Stack Python: 2020 Supporter's Edition
General software development podcasts
These podcasts are not specific to Python but cover relevant software
engineering topics and often have episodes on Python topics. At the very least
you will become a better general software developer by listening and learning
from them.
• Software Engineering Daily incredibly has a podcast every day with a
different developer on a huge range of relevant development subjects.
• All things Git talks to developers using, building and working with Git
with new episodes every two weeks.
• CodeNewbie interviews developers who are early in their software
development journey to find out their stories of why they are
programming and what they are working on. There are also interviews
with experienced developers building well-known projects.
• Developer on Fire interviews programmers, architects and testers with
their stories of success, failure and excellence.
• Command_line Heroes covers operating system-level topics as well as
DevOps.
• Embedded.fm goes into embedded systems and hardware hacking.
• The Changelog is a weekly podcast on general software development
matters.
• Full Stack Radio. No relation to Full Stack Python but another one to
check out!
• Exponent is not a software development podcast but it covers the
intersection of corporate strategy and technology in an in-depth way
that allows me to better understand the decisions businesses make
when building and releasing software. I listen to every episode (at 1.5x
speed) and it's well worth the 45 to 60 minutes spent listening to Ben
Thompson and James Allworth go deep on a weekly topic.
• Test Talks examines a software testing topic each week, usually with a
special guest focused on the area being examined.
• The Cloudcast focuses on cloud computing and DevOps-related topics.
44
Full Stack Python: 2020 Supporter's Edition
Data science and analysis podcasts
The data science community does not only use Python as its core
programming language but it plays a big role in almost every organization
performing data analysis. The following podcasts cover data science broadly
and often get specific into Python ecosystem tools.
• DataFramed is a data science podcast that often covers Python libraries
and other areas of interest to people using Python to analyze data.
• Data Skeptic covers data science, statistics, machine learning, artificial
intelligence and "scientific skepticism".
• Data stories is a podcast on data visualization.
• Partially Derivative was a podcast for machine learning, artificial
intelligence and the data community. They finished their last episode in
late 2017 but the episodes list contains a slew of existing content.
45
Full Stack Python: 2020 Supporter's Edition
Development Environments
A development environment is a combination of a text editor and a Python
runtime implementation. The text editor allows you to write code for your
applications. The runtime implementation, such as CPython or PyPy, provides
the method for executing your code.
A text editor can be as simple as Notepad running on Windows or a more
complicated integrated development environment (IDE) with syntax
checking, integrated test runner and code highlighting. A couple of common
IDEs for Python development are PyCharm and VSCode, both of which runs
on any major operating system.
Why is a development environment necessary?
Python code needs to be written, executed and tested to build applications.
The text editor provides a way to write the code. The interpreter allows it to be
executed. Testing to see if the code does what you want can either be done
manually or by unit and functional tests.
An example development environment
Here's what I (the author of Full Stack Python, Matt Makai) use to develop
most of my Python applications. I have a Macbook Pro with Mac OS X as its
base operating system. Ubuntu 18.04 LTS is virtualized on top with Parallels.
My code is written in vim and executed with the Python 3.6 release via the
command line. I use virtualenv to create separate Python interpreters with
47
Full Stack Python: 2020 Supporter's Edition
their own isolated application dependencies and virtualenvwrapper to quickly
switch between the interpreters created by virtualenv.
That's a common set up but you can certainly write great code with a much
less expensive set up or a cloud-based development environment.
Other developers' environments
Often the best way to figure out how to get comfortable in your own
development environment is to see examples of how other experienced
developers have set up their configurations. The following posts contain the
tools, editors and workflows that developers have taken the time to publicly
document.
• My Python development environment has a setup with Sublime Text,
Anaconda, PyCharm and the author's workflow for how to use the
different editors for different purposes.
• My Python Development Environment, 2018 Edition explains Jacob
Kaplan-Moss' (one of the original creators of the Django web
framework) local setup.
• The definitive guide to setup my Python workspace is geared towards
using Python for data science but the guide remains useful for
configuring your system for any type of Python work. There is some
solid advice in the post about not adulterating your global Python
installation as well as how to split out many virtual environments for
Python 2 & 3.
Cloud hosted dev environments
Several cloud-based development environments have popped up over the past
several years. These hosted environments can work well when you are
learning or stuck on a machine with a web browser but otherwise no
administrative privileges to install your own software. Most of these have free
tiers for getting started and then require payment as you scale up your
application.
• CodeAnywhere is a cloud IDE that can be used in the web browser or
on an iOS or Android device.
48
Full Stack Python: 2020 Supporter's Edition
• Cloud9 began as an independent company and is now owned by
Amazon as part of Amazon Web Services.
• code.xyz is an online text editor built by Stdlib that can integrate with
external web APIs.
• GitLab Web IDE is integrated into the GitLab web application for
modifying your Git repository files directly in your browser.
General dev environment resources
Development environments are unique to each programmer because Python
is used for many different purposes. The following guides range from web
development to DevOps and from getting started to data science. Even though
your environment requirements are unique, you should be able to find
someone who has set up something similar to what you need. Use that
configuration as a starting point and customize it from there.
• The Python subreddit had a nice thread with developers giving the
specifications to their Python development environments in this post
on What is in your Python Development Environment?.
• Real Python has an awesome, detailed post on setting up your Sublime
Text 3 environment as a full-fledged IDE.
• How to bootstrap a Python project covers using a virtualenv, where to
store your files, which version of Python to use and adding code
metrics libraries for checking syntax.
• Three Ways to Install Python on your Windows Computer provides
multiple avenues for Windows users to get Python on their machine
before setting up the rest of their development environment. Unlike
macOS and Linux, the Windows operating system does not include
Python with its default installation.
• PyCharm: The Good Parts shows you how to be more efficient and
productive with that IDE if it's your choice for writing Python code.
• JetBrains' PyCharm Blog is required reading if you're using the IDE or
considering trying it. One of the core developers also has an interview
on the Talk Python to Me podcast that's worth listening to.
49
Full Stack Python: 2020 Supporter's Edition
• The Joy of Linux Desktop Environments talks about desktop
environments, not specifically development environments, but
provides an explanation for why the core Linux operating system is
awesome for being unbundled from the desktop environment itself.
You can change your desktop environment from just a command line
without a windowing system to a full windowed system provided by
Gnome, KDE or Unity for using the system and getting your
programming work done.
• The Hitchhiker's Guide to Python has a page dedicated to development
environments.
• Setting up a Python Development Environment with and without
Docker explains the reasoning behind why and when to use various
tools in your local environment.
• Epic Development Environment using Windows Subsystem for Linux
is geared towards JavaScript developers but contains a slew of good
advice for developers trying to configure Windows.
Text Editors and IDEs
Text editors and integrated development environments (IDEs) are
applications for writing code. These applications are the primary user
interface for developers to create their own programs.
50
Full Stack Python: 2020 Supporter's Edition
Vim is an example of a text editor implementation that can be expanded into a
full Python IDE using configuration files and plugins.
Why is a text editor or IDE necessary?
Where will you write your code if you do not have a text editor? Your
development environment must include a text editor so you can enter, edit
and delete characters to create Python applications.
Preferrably your editor will have a monospace font. It will also get out of your
way, so no "smart" correction or automatic letter capitalization. The more
comfortable you become in your editor of choice the faster you can figure out
how to implement that next feature in your application or squash that pesky
bug that you just found.
What's the difference between editors and IDEs?
IDEs contain text editors but many text editors, for example Notepad
included with Windows, do not include IDE features. Many text editors such
as Vim or Emacs have IDE features by default but then can be further
customized to add file trees, syntax highlighting, line numbers and syntax
checking that is commonly found in full-featured IDEs.
Open source text editors
Open source provides an embarrassment of riches when its come to stable,
extendable text editors. Some version of these editors, such as the original vi
version of Vim, have been used for over 40 years! You can't go wrong with
using one of the editors as your development environment foundation.
The following text editor implementations can be upgraded with
configurations and plugins to become full-fledged IDEs when a developer
wants that kind of functionality.
• Vim is my editor of choice and installed by default on most *nix
systems.
• Emacs is another editor often used on *nix.
• Mu (source code) is an open source editor intended for Python
beginners.
51
Full Stack Python: 2020 Supporter's Edition
• Atom is an open source editor built by the GitHub team.
• Visual Studio Code by Microsoft provides spectacular Python support.
Python-specific IDEs
Editors built from the foundation up are not necessarily better than generalpurpose text editors and IDEs like Vim and Emacs but they are typically much
easier to configure for gathering code metrics, running unit tests and
debugging.
• PyCharm is a Python-specific IDE built on JetBrains' platform. There
are free editions for students and open source projects.
• Thonny is an open source Python IDE for new programmers. The tool
bakes in syntax highlighting, code completion, a simple debugger, a
beginner-friendly shell and in situ documentation to assist new
developers who are just starting to code.
• Wing IDE is a paid development environment with integrated
debugging and code completion.
• PyDev is a Python IDE plug in for Eclipse.
Proprietary (closed source) editors
There are some editors that are closed source that developers are very happy
using.
• Sublime Text versions 2 and 3 (currently in beta) are popular text
editors that can be extended with code completion, linting, syntax
highlighting and other features using plugins. If you are considering
using Sublime Text for Python development, check out this 2016 in
review - likes and dislikes about Sublime Text post that summarizes
many of the positives and negatives of using the editor.
• Komodo is a cross-platform text editor and IDE for major languages
including Python, Ruby, JavaScript, Go and more.
52
Full Stack Python: 2020 Supporter's Edition
Building your own text editor
One great way to learn more about how text editors work is by building your
own, even if it turns out to be a hacked-together proof-of-concept. These
resources give walkthroughs on building an editor, or explain how existing
editors work by digging into their source code.
• Build your own text editor provides an awesome tutorial for creating a
basic editor in the C programming language. This walkthrough is useful
to break open the black box of how a tool you use every day as a
programmer works under the covers.
• A brief glance at how various text editors manage their textual data is a
fun dive into the source code of vi, GNU Moe, Emacs, Scintilla and
GNOME GtkTextView/GtkTextBuffer.
• Xi: an editor for the next 20 years is an awesome technical talk about
designing a text editor with the current (2018) set of tools available to a
developer.
• Building a Text Editor for a Digital-First Newsroom gives some
wonderful insight into the New York Times' homegrown legacy text
editor and why they started building a new text editor named Oak that
is customized to the newsroom's workflow.
General text editor & IDE resources
These resources provide comparisons of various editors and give some deeper
insight into the IDE vs plain text editor debate.
• EditorConfig (source code) is an open source tool for keeping many
text editors and IDEs on the same code styles and configurations.
• PyCharm vs Sublime Text has a comparison of several features between
the two editors.
• What is the best IDE for Python tries to answer a loaded question and
gives some rationale behind choosing one application or another.
• Why I Deleted My IDE; and How It Changed My Life For the Better
contains some hyperbole but still has some solid reasoning why
integrated environments are not necessarily for everyone depending on
a developer's chosen workflow.
53
Full Stack Python: 2020 Supporter's Edition
• Text editing techniques every Front-End developer should know
contains tricks that experienced developers use in their editor of
choice. The author uses Sublime Text to demonstrate how the methods
work but they can be used in just about any editor.
Vim
Vim (source code), short for Vi IMproved, is a configurable text editor often
used as a Python development environment. Vim proponents commonly cite
the numerous plugins, Vimscript and logical command language as major
Vim strengths.
Why is Vim a good Python development environment?
Vim's philosophy is that developers are more productive when they avoid
taking their hands off the keyboard. Code should flow naturally from the
developer's thoughts through the keyboard and onto the screen. Using a
mouse or other peripheral is a detriment to the rate at which a developer's
thoughts become code. This "efficiency by keyboard" keeps Vim as one of the
most popular text editors despite having been around for decades. Few
programming tools have that kind of staying power.
Vim has a logical, structured command language. When a beginner is learning
the editor she may feel like it is impossible to understand all the key
commands. However, the commands stack together in a logical way so that
over time the editor becomes predictable.
Configuring Vim with a Vimrc file
The Vimrc file is used to configure the Vim editor. A Vimrc file can range from
nothing in it to very complicated with hundreds or thousands of lines of
configuration commands.
54
Full Stack Python: 2020 Supporter's Edition
Here's a short, commented example .vimrc file I use for Python development
to get a feel for some of the configuration statements:
" enable syntax highlighting
syntax enable
" show line numbers
set number
" set tabs to have 4 spaces
set ts=4
" indent when moving to the next line while writing code
set autoindent
" expand tabs into spaces
set expandtab
" when using the >> or << commands, shift lines by 4 spaces
set shiftwidth=4
" show a visual line under the cursor's current line
set cursorline
" show the matching part of the pair for [] {} and ()
set showmatch
" enable all Python syntax highlighting features
let python_highlight_all = 1
Here is how these configuration options look with a dark background on Mac
OS X while editing the markdown for this webpage (how meta!).
Take a look at another example using these configuration options, this time
with a light background and editing Python code from my Choose Your Own
Adventures Presentations project.
55
Full Stack Python: 2020 Supporter's Edition
The Vimrc file lives under the home directory of the user account running
Vim. For example, when my user account is 'matt', on Mac OS X my Vimrc file
is found at /Users/matt/.vimrc . On Ubuntu Linux my .vimrc file can be
found within the /home/matt/ directory.
If a Vimrc file does not already exist, just create it within the user's home
directory and it will be picked up by Vim the next time you open the editor.
Vim tutorials
Vim has a reputation for a difficult learning curve, but it's much easier to get
started with these tutorials.
• Learn Vim Progressively is a wonderful tutorial that follows the path I
took when learning Vim: learn just enough to survive with it as your
day-to-day editor then begin adding more advanced commands on top.
• A vim Tutorial and Primer is an incredibly deep study in how to go
from beginner to knowledgeable in Vim.
• Why Atom Can't Replace Vim discusses one of Vim's core principles:
command composability. Vim has a language where simple commands
are combined to execute more advanced operations. For example, in
command mode, $ moves to the end of a line. When $ is preceded by
d then everything to the end of the line is deleted. Over time the
simple commands become intuitive and the combinations become
56
Full Stack Python: 2020 Supporter's Edition
more powerful than having distinct commands such as a drop-down
menu with a specific option to delete all text until the end of the line.
• Vim as a Language explains the language syntax and how you can build
up over time to master the editor.
• How to install and use Vim on a cloud server along with How to use
Vim for advanced editing of code on a VPS are two detailed Digital
Ocean guides for getting up and running with Vim, regardless of
whether you're using it locally or on a cloud server.
• PacVim: a commandline game to learn Vim commands takes the
PacMan theme and teaches you how to use Vim by forcing you to move
around and use Vim commands while gaming.
• Ten years of Vim provides an insightful retrospective on one
experienced developer's journey with using Vim as a primary text
editor and development environment. I found the part about going
overboard with plugins before switching back to a simpler
configuration fascinating because it is the same path I've found myself
taking as I approach my own ten year mark with Vim.
• At least one Vim trick you might not know about is a collection of nonobvious keyboard shortcuts, many of which are infrequently used but
still useful.
• Vim Adventures is a cute, fun browser-based game that helps you learn
Vim commands by playing through the adventure.
• In Vim: revisited the author explains his on-again off-again
relationship with using Vim. He then shows how he configures and
uses the editor so it sticks as his primary code editing tool.
• Things About Vim I Wish I Knew Earlier explores the lessons one
developer learned while exclusively using Vim for several years. The
author includes using relative instead of absolute line numbering,
setting numerous configuration options and fuzzy finding to quickly
open files in other directories rather than expanding the whole path.
• Seven habits of effective text editing explains moving around
efficiently, fixing errors quickly and forming good habits.
57
Full Stack Python: 2020 Supporter's Edition
Vimrc resources
These are a few resources for learning how to structure a .vimrc file. I
recommend adding configuration options one at a time to test them
individually instead of going whole hog with a Vimrc you are unfamiliar with.
• A Good Vimrc is a fantastic, detailed overview and opinionated guide to
configuring Vim. Highly recommended for new and experienced Vim
users.
• Vim and Python shows and explains many Python-specific .vimrc
options.
• Vim as a Python IDE shows a slew of plugins and configuration options
for coding with Python in Vim.
• This repository's folder with Vimrc files has example configurations
that are well commented and easy to learn from.
• For people who are having trouble getting started with Vim, check out
this blog post on the two simple steps that helped this author learn
Vim.
Vim installation guides
These installation guides will help you get Vim up and running on Mac OS X,
Linux and Windows.
• Upgrading Vim on OS X explains why to upgrade from Vim 7.2 to 7.3+
and how to do it using Homebrew.
• The easiest way to install Vim on Windows 7+ is to download and run
the gvim74.exe file.
• On Linux make sure to install the vim package with
sudo apt-get install vim .
• If you're using PyCharm as your IDE you won't need to install Vim as a
separate text editor - instead use the IdeaVim PyCharm plugin to get
Vim keybindings, visual/insert mode, configuration with ~/.ideavimrc
and other Vim emulation features.
58
Full Stack Python: 2020 Supporter's Edition
Using Vim as a Python IDE
Once you get comfortable with Vim as an editor, there are several
configuration options and plugins you can use to enhance your Python
productivity. These are the resources and tutorials to read when you're ready
to take that step.
• VIM and Python - a Match Made in Heaven details how to set up a
powerful VIM environment geared towards wrangling Python day in
and day out.
• The python-mode project is a Vim plugin with syntax highlighting,
breakpoints, PEP8 linting, code completion and many other features
you'd expect from an integrated development environment.
• Vim as Your IDE discusses how to set up Vim for greater productivity
once you learn the initial Vim language for using the editor.
• Setting up Vim for Python has a well written answer on Stack Overflow
for getting started with Vim.
• If you're writing your documentation in Markdown using Vim, be sure
to read this insightful post on a Vim setup for Markdown.
Vim Plugin resources
• 5 Essential VIM Plugins That Greatly Increase my Productivity covers
the author's experience with the Vundle, NERDTree, ctrlp, Syntastic
and EasyMotion Vim plugins.
• Getting more from Vim with plugins provides a list of plugins with a
description for each one on its usefulness. The comments at the bottom
are also interesting as people have suggested alternatives to some of
the plugins mentioned in the post.
• Powerline is a popular statusline plugin for Vim that works with both
Python 2 and 3.
• VimAwesome is a directory of Vim plugins sourced from Vim.org,
GitHub and user submissions.
• Command-T is a Vim plugin for fast fuzzy searching files.
59
Full Stack Python: 2020 Supporter's Edition
• YouCompleteMe (source code) is a code-completion engine and plugin
that works for Python.
Vim Plugin Managers
If you use many Vim plugins together it is really handy to have a plugin
managers to sort out all of the dependencies. The following plugin managers
are the most commonly-used ones in the Vim ecosystem.
• Vundle comes highly recommended as a plugin manager for Vim.
• Pathogen is a widely used plugin manager.
• Vim-plug bills itself as a minimalistic Vim plugin manager.
Niche tutorials
After you have been using Vim for awhile there will be features you bump into
without realizing they were ever there. The following tutorials show how to
use some specific niche features. You may already know about these if you
have been using Vim for awhile but everyone's learning path is different so it's
useful to do a quick scan to make sure you are not missing anything.
• Vim’s absolute, relative and hybrid line numbers shows how to change
the line numbering scheme. There was a period of time I used relative
line numbers although I eventually switched back to absolute numbers.
The usefulness of these schemes is often dependent on what language
you are working in.
• A simpler Vim statusline explains how to customize your bottom screen
statusline without using plugins such as vim-powerline or vim-airline.
• The vim-clutch is a really cool project and walkthrough that shows how
you can create a foot pedal to switch between Normal and Insert modes
instead of using the typical ESC key (or a remapped key).
• How I'm able to take notes in mathematics lectures using LaTeX and
Vim explains how the author is able to keep up with mathematics
lectures by using Vim and LaTeX which produces gorgeous notes that
can be used to study.
60
Full Stack Python: 2020 Supporter's Edition
Emacs
Emacs (source code) is an extensible text editor that can be customized by
writing Emacs Lisp (Elisp) code.
Why is Emacs a good choice for coding Python?
Emacs is designed to be customized via the built-in Lisp interpreter and
package manager. The package manager, named package.el, has menus for
handling installation. The largest Lisp Package Archive is Melpa, which
provides automatic updates from upstream sources.
Macros are useful for performing repetitive actions in Emacs. A macro is just
a recording of a previous set of keystrokes that can be replayed to perform
future actions.
Hooks, which are Lisp variables that hold lists of functions to call, provide an
extension mechanism for Emacs. For example, kill-emacs-hook runs
before exiting Emacs so functions can be loaded into that hook to perform
necessary actions before the exiting completes.
Python plus Emacs resources
Emacs is programming language agnostic by design so it takes some effort to
customize the editor as a Python-specific development environment. The
following resources will walk you through the setups that other developers
have created for working with Python.
• Emacs - the Best Python Editor? continues the excellent Real Python
series showing how to get started with editors. In addition to this
61
Full Stack Python: 2020 Supporter's Edition
Emacs post, there are also posts on Vim and Sublime Text 3 specifically
for Python development.
• Python developers often use reStructuredText (RST) to document their
projects. This guide on Emacs Support for ReStructuredText is handy
for properly configuring your environment to work with RST files.
• Emacs as a Python IDE is a video of a technical talk where the speaker
sets up code completion, documentation lookup and the jedi-starter kit
on his Emacs environment.
• How do you create a robust Python IDE with Emacs (as the Text editor)
is a quality Stack Exchange thread that offers opinions about how to
best go about setting up Emacs for efficient Python development.
• Tricked out emacs for python coding is a short guide for handling
ReStructuredText documents and adding Python code metrics tools to
your Emacs environment.
General Emacs resources
Emacs, like any powerful tool, takes significant intentional practice to use
properly. These resources provide instructions for becoming comfortable with
the editor itself rather than specific Python environment configuration advice.
• GNU Emacs Manual provides an official in-depth review for how to use
Emacs.
• Emacs is sexy! provides a whole site with installation instructions,
cheat sheets and other materials for learning Emacs.
• Emacs Redux is a blog with tips and tricks for how to use Emacs
effectively.
• Emacs Rocks is a video tutorial series for Emacs.
• The Absolute Beginner's Guide to Emacs and Emacs as a Python IDE
are a couple of awesome detailed walkthroughs by Jessica Hamrick for
setting up Emacs for general development as well as Python coding.
• Compiling and running scripts in Emacs explains one workflow option
you can use with Emacs to run code from within the editor rather than
bouncing out to a shell.
62
Full Stack Python: 2020 Supporter's Edition
• What the .emacs.d?! provides a bunch of tiny optimizations for Emacs'
workflow.
• The Using Emacs Series is a set of videos along with an open source Git
repository of the configuration for using and gaining experience with
Emacs.
Programming Emacs with Elisp
Emacs can be completely customized and rewritten by using the Emacsspecific Lisp programming language named Emacs Lisp (Elisp). The ability to
completely modify the editor is part of what led to the old joke "a great
operating system, lacking only a decent editor". Nevertheless, Elisp is what
gives Emacs its text editing power despite the perception that the editor is
overkill for working with text.
These tutorials will help you learn the Elisp language and use it to modify
Emacs for your own purposes.
• An Introduction to Programming in Emacs Lisp is the "official"
introduction intended for a beginner programming audience.
• Emacs Lisp Guide is for developers that have been using Emacs for a
while and want to start making extensions to get more out of the editor.
• The Emacs Wiki contains advice and resources for getting oriented.
• Practical Emacs Lisp contains a bunch of useful code and focuses on
examples throughout the tutorial.
Notable Elisp Packages
Elisp is the LISP programming language dialect that Emacs using for adding
and customizing functionality in the editor. The following Elisp packages are
existing Elisp libraries that many developers using Emacs incorporate into
their environment.
• Magit allows the user to inspect and modify Git repositories from
within Emacs.
• company-mode creates a modular in-buffer completion framework.
• Flycheck provides syntax checking.
63
Full Stack Python: 2020 Supporter's Edition
• anaconda-mode is specific to Python development and allows code
navigation, documentation lookup and code completion.
• web-mode.el is a package for editing web templates like Jinja. Many
Python template engines are supported including Django templates,
Mako and Cheetah as well as JavaScript front-end frameworks.
Popular user configurations
Numerous custom Emacs user configurations exist that bundle together
custom Elisp packages and libraries to handle creating a powerful integrated
development environment. I recommend trying to configure Emacs yourself
before you dive into any of these configurations so it is easier to learn base
Emacs rather than get distracted by the customizations.
• Prelude is an enhanced Emacs version 24 distribution.
• A reasonable Emacs config shows a batteries-includes Emacs
configuration bundle.
• Emacs settings is a repository of configurations used in the Emacs
Rocks screencasts.
• Spacemacs mashes together Emacs' extensibility and Vim's ergonomic
text editing features.
Sublime Text
Sublime Text is a commonly-used text editor used to write Python code.
Sublime Text's slick user interface along with its numerous extensions for
syntax highlighting, source file finding and analyzing code metrics make the
editor more accessible to new programmers than some other applications like
Vim and Emacs.
64
Full Stack Python: 2020 Supporter's Edition
What makes Sublime Text awesome?
Sublime Text is often the first editor that newer programmers pick up because
it works on all operating systems and it is far more approachable than Emacs,
Vim or even PyCharm.
It is easy to get started in Sublime because the menus and options are
accessible by using a mouse. There are no different modes to learn like Vim's
normal and insert modes. The keyboard shortcuts can be learned over time
rather than all at once in the case of Vim or Emacs.
Sublime Text works well for beginners as soon as they install it and then can
be extended with many of the features provided by an IDE like PyCharm as a
developer's skill level ramps up.
An additional bonus of using Sublime Text as a Python developer is that
plugins are written in Python. Python developers can extend Sublime Text
with their own programming language rather than learn a new language like
Emacs' Elisp or Vim's Vimscript.
Why use any other editor if Sublime is so great?
Picking a text editor or IDE to use tends to be a weirdly personal decision for
each developer. Yet it makes sense when you realize that you are going to
spend hours upon hours every day in your chosen environment so why not
make sure it is one that is enjoyable and highly productive?
For some folks they prefer Vim's keyboard-driven style, PyCharm's Swiss
Army Knife set of Python tools or one of the many other editors with its own
strengths and weaknesses.
The only "best" editor choice is to pick one that works really well for you and
stick to it. Master your tool so it gets out of your way and enables as much
time in programming flow as possible.
Python-specific Sublime Text resources
There are many Python-specific Sublime Text tutorials and resources because
the editor is so frequently used to create Python applications. The following
links should get your editor customized with linters, code metrics, syntax
checking and many other integrated development environment features.
65
Full Stack Python: 2020 Supporter's Edition
• Setting Up Sublime Text 3 for Full Stack Python Development is a
spectacular tutorial that covers installing Sublime Text and configuring
a multitude of helpful Python programming plugins.
• Sublime Text 3 Heaven is a quick overview of the extensions, packages
and bonus toys that one developer uses for his own Sublime Text
development setup.
• Sublime Tutor is an interactive in-editor keyboard shortcuts tutorial
that plugs into Sublime so you can learn and become more productive
as you use the editor.
• Using Generators for Fun and Profit - Utility for developers is not
about setting up your Sublime Text environment but instead how to
create your own plugins using Python. The tutorial is written by the
author of a Sublime Text plugin who uses generators to implement
features with Sublime's API.
• Turning Sublime Text Into a Lightweight Python IDE shows the basic
settings and configuration specific to using Sublime with Python as
more than just a text editor.
• Setting up Sublime Text 3 for Python Type Checking shows one way of
setting up support for Python 3.6 static type checking in Sublime.
• Three steps to lint Python 3.6 in Sublime Text walks through setting up
Flake8 to enforce code style guidelines and show you the errors and
warnings in Sublime as you are working.
• Text editing techniques every front-end developer should know gives
examples in Sublime Text of time-saving text manipulation you may
not have known existed such as line bubbling, ragged line selection,
AceJump and transpose. While the techniques can be used in most
editors the provided video clips show how to perform each of these
shortcuts in Sublime.
General Sublime Text resources
Sublime Text can be used for much more than Python development and there
are many useful tutorials that are not targeted at a specific programming
language which are still useful.
66
Full Stack Python: 2020 Supporter's Edition
• Super charge your Sublime Text 3 to increase your productivity
provides many shortcuts and tricks for using the editor.
• Disassembling Sublime Text uses a binary disassembler to dive into the
reverse engineered source code of Sublime Text because it is not open
source software.
• Sync your sublime text 3 configurations safely and easy explains how to
mitigate configuration conflicts that can arise when trying to use copied
files from one computer to another.
• 7 shortcuts of a highly effective Sublime Text user shows keyboard
shortcuts for opening any file, going to any specific block of text,
handling multiple cursors and more.
Sublime Plugin resources
Sublime Text plugins are written in Python which makes it convenient for our
ecosystem to customize the editor. The following resources provide
information on writing your own plugins as well as great community plugins
you will want to take a look at adding to your installation.
• Sublime's documentation covers plugin basics, the API for plugins and
gives a "Hello, world!"-level example that you can extend.
• Sublime Text plugin development basics has some good advice and
further resources.
• The 25 Best Sublime Text Plugins for Front End Developers is not
specific to Python development but there is a bunch of overlap between
plugins useful for general front-end development and any Python web
development project.
• 5 Awesome Sublime Plugins you Won’t Find in Top Plugin Posts covers
some lesser-known plugins and how you can find your own via Package
Control's trending plugins section.
PyCharm
PyCharm is a text editor and integrated development environment specifically
designed for writing Python code.
67
Full Stack Python: 2020 Supporter's Edition
PyCharm resources
• JetBrains provides courses for the PyCharm Educational Edition that
can be used to learn any edition of the IDE.
• The Mastering PyCharm Talk Python to Me course is awesome when
you want to invest in your skills for using the IDE well.
• Worth the switch to Pycharm? is a solid discussion thread with
different developers' perspectives on using PyCharm for coding their
applications.
• How to Get Started with PyCharm and Have a Productive Python IDE
covers the basics of configuring PyCharm for running code within
virtualenvs, macros, using the console and code completion.
• Using PyCharm with Pyramid is specific to developing and debugging
with the Pyramid web framework.
• Just switched from Atom to PyCharm is a Reddit /r/learnpython
thread with a positive experience of switching to PyCharm along with
some comments and feedback from other developers.
• PyCharm Vs Visual Studio Code For Python Development compares
the editors on performance, extensions and resource consumption for
Python development.
• PyCharm has excellent first-party official documentation for getting
started and configuring the IDE. The advantagee to using the official
docs is that they tend to be more up-to-date than community blog posts
that were not published recently because PyCharm has new major
releases twice per year.
• Getting Started with PyCharm is another JetBrains-produced tutorial
that is exceptionally in-depth and just right for beginners who need
68
Full Stack Python: 2020 Supporter's Edition
help with every step of setting up and creating their first Python
project.
• Setting Up a Python Development Environment with PyCharm is
focused on the EV3 Lego Mindstorm Linux distribution but has some
good information and steps for anyone setting up PyCharm for the first
time.
• flask-pycharm-templates are snippets that are commonly used when
building Python, Jinja2 and Flask applications that you can import into
your environment to use during development.
• Unit Tests and Doctests in PyCharm is a beginner's guide on why you
would want to use tests and doctests in your Python code and run them
with PyCharm.
Jupyter Notebook
Jupyter Notebook (open source code), which began as the iPython Notebook
project, is a development environment for writing and executing Python code.
Jupyter Notebook is often used for exploratory data analysis and
visualization.
Project Jupyter is the top-level project name for all of the subprojects under
development, which includes Jupyter Notebook. Jupyter Notebooks can also
run code for other programming languages such as Julia and R.
How does Jupyter Notebook work?
The key piece of Jupyter Notebook infrastructure is a web application that
runs locally for creating and sharing documents that contain embedded code
and execution results.
69
Full Stack Python: 2020 Supporter's Edition
How are IPython Notebook and Jupyter Notebook related?
IPython Notebook was the original project that proved that there was great
demand among data scientists and programmers for an interactive,
repeatable development environment. Jupyter Notebook became the new
official name for the overall project during The Big Split after the IPython
Notebook project matured into distinct submodules such as the interactive
shell, notebook document format and user interface widgets tools. However,
the IPython Notebook name sticks around as the Python backend for Jupyter
Notebook which is seriously confusing if you are searching the internet and
come across both current and old articles that use all of these names
interchangeably.
Jupyter Notebook beginner tutorials
Jupyter Notebook's powerful analysis and visualization environment can be
intimidating even for experienced developers that are new to the tool. The
following tutorials will explain the basics so you can quickly figure out your
own productive workflow.
• Jupyter Notebook for Beginners: A Tutorial is a great place to start if
you have never before used the tool. The guide covers installation,
terminology, the user interface and how to publish your notebooks to
the web. Screenshots walk you through some of the more confusing bits
as you are getting up and running.
70
Full Stack Python: 2020 Supporter's Edition
• First Python Notebook is a free guide on analyzing data with Python
and Jupyter Notebook. It covers many "Hello, World!"-style examples
in both data analysis topics and more general software development
areas like Git, GitHub and Markdown.
• IPython Or Jupyter? covers the evolution of the Notebook concept
from its origins in the IPython Notebook implementation through the
IPython and Jupyter split that happened in 2015 that separated
IPython Notebook into logical subprojects. The post kicks off with
some fun lesser-known historical context on other data science
notebook projects such as MATLAB and Mathematica to set the stage
for IPython and Jupyter's creation.
• How to use Jupyter Notebooks in 2020 (Part 1: The data science
landscape) is a high-level overview post that starts a series on Jupyter
Notebooks. This first post covers why a tool like Jupyter Notebook is
needed in the broader landscape of data science. Part 2 examines the
growth of the Jupyter ecosystem and the jump from exploratory
analysis notebooks to production notebooks.
• Jupyter Notebook Best Practices contains tips for beginners such as
learning shortcuts and properly documenting the analysis you work on.
• How to Version Control Jupyter Notebooks explains how Jupyter
Notebooks are stored in JSON, the issues with that format for source
control and how to get around the problem.
Example Notebooks
Example Notebooks are easy to fire up and see how other people are working.
These resources are highly recommended after you read a couple of tutorials
and play around with the tool.
• Peter Norvig's collection of Jupyter Notebooks is a an incredible
resource for example projects.
• Building and Exploring a Map of Reddit with Python is a detailed
notebook that digs into public Reddit data while explaining the "what"
and "why" along the way.
71
Full Stack Python: 2020 Supporter's Edition
• This gallery of interesting Jupyter Notebooks provides many great
examples across numerous programming languages.
• jupyter-samples contains an extensive set of notebooks along with
public data sets that can be used for analysis.
• learn-python3 is geared towards Python beginners who want to learn
basic syntax and standard library features such as about string
manipulation, functions and iteration.
Intermediate to advanced Jupyter Notebook tutorials
Once you get the hang of the basics there are a slew of ways to connect your
notebooks to third party APIs and use more advanced Python libraries with
your code. These walkthroughs cover a range of topics from niche tricks to
common but advanced situations like advanced interactive visualizations.
• Advanced Jupyter Notebook Tricks — Part I and Building Interactive
Dashboards with Jupyter (Part 2) have a ton more details on ways to
set up Jupyter Notebooks as dashboards and export results to other
formats.
• Creating Interactive Dashboards from Jupyter Notebooks shows how
to use public Reddit data for a data analysis project as an example to
display in dashboards running in a Jupyter Notebook.
• mapboxgl-jupyter library along with the quickstart show you how to
visualize geospatial data within your notebooks.
• Reproducible Data Analysis in Jupyter is a fantastic series of videos by
Jake Vanderplas that shows how to move your code from the
interactive Jupyter environment into packaged, tested Python code
that is suitable for deployment to a production environment.
• 28 Jupyter Notebook tips, tricks and shortcuts explains many of the
lesser-known keyboard shortcuts and mechanisms to output settings.
• Boost Your Jupyter Notebook Productivity covers hotkeys, data
plotting, shell commands, timing and other topics you will eventually
want to handle within your notebooks as you get comfortable in the
environment.
72
Full Stack Python: 2020 Supporter's Edition
• Making Publication Ready Python Notebooks explores the plugins that
the author uses when creating and exporting reports from Jupyter.
• PyData has an extensive list of Jupyter Notebook talks from past
events. JupyterCon has a similarly extensive talks list that is also worth
watching.
• Integrate Google Sheets and Jupyter Notebooks answers the common
question of how to extract data directly from a Google Sheet and start
working with it in your Jupyter Notebook. The screenshots help a lot to
make sure you avoid getting lost in the sea of menus along the way.
• Analyzing Cryptocurrency Markets using Python uses a freely-available
Bitcoin API as a source data set for a data analysis and visualization
project in Jupyter.
• nbdev: use Jupyter Notebooks for everything shows how to use the
nbdev tool to create a literate programming environment within a
Jupyter Notebook so that you can do all of your debugging and
refactoring there rather than switching between a more traditional IDE
and Jupyter.
• Running Jupyter Notebooks on GPU on AWS: a starter guide explains
how to run notebooks on Amazon Web Services using a graphicsprocessing unit (video card), which for some machine learning
situations can result in significantly faster execution times.
• Python & Big Data: Airflow & Jupyter Notebook with Hadoop 3, Spark
& Presto walks through a data pipeline that combines several
commonly-used data analysis tools with a Jupyter Notebook.
• Ansible-jupyter-kernel is a kernel that allows you to run Ansible tasks
and playbooks from within your Jupyter environment.
• Jupyter Notebooks in the IDE explains how to use Jupyter files in
Visual Studio Code or PyCharm with Jupytext, which defines the
pairing information and notebook kernel.
• The Notebook Wars is not a tutorial but instead points to the
weaknesses that become apparent when using Jupyter and the current
generation of notebook projects. The article raises many good points
about barriers to entry although you could also argue some of these
73
Full Stack Python: 2020 Supporter's Edition
issues have been mitigated by Jupyter, just not as much as some people
would like to see. Overall there is a lot to enjoy reading here and reflect
on so that the community can continue making Jupyter a fantastic
environment for development.
• Creating Presentations with Jupyter Notebook shows how to make
slides out of your cells so you can present your work like a traditional
presentation.
• The number of open source Jupyter Notebooks on GitHub is exploding
and this post attempts to estimate the growth using Python, pandas
and a scraped data set.
• Reproducible Jupyter Notebooks with Docker explains when to use
Docker in combination with Jupyter Notebooks as well as the
instructions for creating a dockerfile to build your images.
• JupyterLab GPU Dashboards contains a Bokeh server and TypeScript
code for displaying GPU utilization charts.
Shells
Shells are computer user interfaces that typically refer to a text-only or
primarily text-based command prompt.
The above screenshot shows the bash shell with an active Python virtual
environment named fullstackpython within the macOS Terminal
application.
Shell resources
• cmd is the Pythonic standard library module that can be used for
building your own shells. The Python CmdModule wiki page has a great
overview of the module and its capabilities.
74
Full Stack Python: 2020 Supporter's Edition
• Give your Python program a shell with the cmd module shows a short
code example of how to use cmd to build a simple shell.
• Super Charge Your Shell For Python Development covers aliases,
environment variables via Autoenv and some basic shell commands
often used during development.
• Terminal latency quantifies the impact of lag in your keystrokes
appearing on the screen. It's a fascinating look at how a small
difference of tens of milliseconds causes some shells and editors to feel
slow while others are snappy.
• Why Create a New Unix Shell? is a post by the creator of Oil shell that
goes into the rationale for building a new shell even though so many
others such as Bash, zsh, PowerShell and KornShell already exist.
• explainshell (source code) is a wonderful little tool that shows how
input and arguments in the shell break down and are interpreted by
commands. The data is pulled from the Ubuntu man pages.
• Shell productivity tips and tricks covers how to increase your
effectiveness on the shell across topics such as navigating history,
autocompletion, and pattern matching.
Bash shell
The Bourne-Again SHell (source code), almost always referred to simply as
"Bash", interprets and executes input entered from a source such as the user
or a program. Bash is an implementation of the shell concept and is often
used during Python software development as part of a programmer's
development environment.
75
Full Stack Python: 2020 Supporter's Edition
Bash resources
• Bash Guide for beginners is an entire book for those new to working
with commandlines. It covers commands, paths, Bash shell scripting,
variables and many other critical topics that are necessary to move
from beginner to advanced Bash user.
• Advancing in the Bash shell covers important concepts such as bang
syntax, movement commands, tab completion and aliases.
• Mastering Bash and Terminal shows methods for repeating commands,
changing directories and handling background processes.
• Ten Things I Wish I’d Known About Bash covers some edge cases that
are very useful to know about such as proper exit code usage and
configuration options through the set command. There is also a great
follow up post called Ten MORE Things I Wish I'd Known About Bash
that covers new topics such as on-the-fly command re-execution using
the carrot character. The Seven Surprising Bash Variables post
continues the series by examining built-in variables such as
PROMPT_COMMAND , CDPATH and REPLY which can simplify your scripts
by using values that Bash already has stored for you.
• Google's Shell Style Guide covers how to write consistent, maintainable
shell scripts, which is particularly important if you have ever tried to
debug a hacky shell script that was never meant to be used by anyone
other than the original author.
• 101 Bash Commands and Tips for Beginners to Experts is a well-done
laundry list of tricks to explore.
• Bash scripting quirks & safety tips explains Bash basic programming
constructs like for loops and variable assignment then goes into ways
to avoid weird issues in your code.
• Safe ways to do things in bash shows you how to not shoot yourself in
the foot by using safe coding practices with your shell scripts.
• The Bash Infinity Framework source code provides boilerplate and a
standard library for Bash projects so they are easier to read and
maintain. If you have ever tried to read someone else's Bash scripts or
even your own after setting them aside for a couple of months, you
76
Full Stack Python: 2020 Supporter's Edition
know that anything which makes readability better is a major step up
from vanilla Bash.
• Static status is a Bash application that generates a hostable,
customizable status page for your services.
• Replacing Bash scripts with Python is a guide on using using Python
for administrative scripting, including what to do about replacing
invaluable command line tools such as awk , sed and grep .
• Using Aliases to Speed Up Your Git Workflow has a bunch of shell
aliases that make it easier for you to execute complicated or uncommon
Git commands.
• Creating a bash completion script is a great tutorial that walks you
through a reasonably complex Bash script for completing syntax in
other Bash shell scripts.
• 6 Tips Before You Write Your Next Bash Cronjob covers starting your
scripts with shebang, redirecting output, timeouts and sudo privileges.
• Better Bash history shows how to make your Bash history more useful
by having it store more previous commands (which takes up more
persistent storage but is not a huge deal in 2019) and add timestamps
to the history command.
• 9 Evil Bash Commands Explained presents a list of commands you
should never run, but can learn about their destructive abilities by
reading through the descriptions provided by the author.
• Bash Quick References is a cheat sheet for common operators and
signals that come up when working with scripts.
• Anybody can write good bash (with a little effort) covers the basics of
shell scripting and provides some recommendations for creating more
maintainable scripts such as using linters and formatters.
Zsh shell
Zsh interprets and executes input entered from text sources such as user input
or from another application. Zsh is an implementation of the shell concept
77
Full Stack Python: 2020 Supporter's Edition
that is frequently used during Python software development as part of a
programmer's development environment.
Zsh beginner's resources
• Getting Started with Zshell has a short video showing off Zsh features
and then shows how to handle aliases, globbing and parameter
expansion.
• Switching to Zsh goes on a bit about how Zsh is better than Bash in his
opinion, then shows how to use it as your default shell with a custom
theme.
• Become A Command-Line Power User With Oh My ZSH And Z
provides a long in-depth tutorial on a slew of Zsh features and how to
use them if you have never previously used Zsh.
• No, Really. Use Zsh. goes through the laundry list of advantages
provided by Zsh compared to other shells.
Zsh configuration
• Oh My Zsh is a configuration manager for Zsh.
• Zsh Configuration From the Ground Up is a wonderfully-written
detailed post on the setup that the author uses along with why he
enjoys using Zsh for development.
PowerShell
PowerShell is a commandline user interface for Windows that is often used as
part of a Python programmer's development environment.
78
Full Stack Python: 2020 Supporter's Edition
PowerShell resources
• A Python Developer's Guide to Powershell explains the PowerShell
scripting language then shows how to combine a Python script and a
PowerShell script to automate web scrapining downloads.
• ChatOps with PowerShell covers how to use the Python-based chatbot
named ErrBot. It also presents example code to connect ErrBot to
applications you are running.
• PowerShell in Azure Functions shows how to use PowerShell code in
Azure serverless Functions.
• Getting Started with Windows PowerShell is a guide for your first steps
with PowerShell.
• Windows Command-Line: Backgrounder and Windows CommandLine: The Evolution of the Windows Command-Line give historical
perspective on how the Windows shell has evolved from MS-DOS days
into the current Windows 10 world.
• Learning a New REST API with PowerShell covers how to create the
common GET, POST, PUT and DELETE requests that are used to work
with RESTful APIs. The the tutorial also contains some tips and tricks
for reading API documentation and how to use PowerShell more
effectively in these situations.
• PowerShell in Azure Functions shows you how to use PowerShell
scripting language code in your Azure Functions. The language is only
in experimental mode on Azure Functions but could be useful if you
have a bunch of existing scripts that you want to use on the serverless
platform.
79
Full Stack Python: 2020 Supporter's Edition
• PowerShell Core support in AWS Lambda is an announcement post
that PowerShell can be used in AWS Lambda Functions along with how
to get started.
• PowerShellBuild provides common reusable tasks for building
PowerShell modules.
• Invoke-Build is a build automation tool. There is also extensive
documentation on their wiki.
• Two months with Powershell on a UNIX examines one developer's
experience using PowerShell not on Windows but on a *nix operating
system. The author covers the advantages and disadvantages he found
during his experience and some of the bugs he hopes are fixed in
PowerShell 7.
Terminal Multiplexers
A terminal multiplexer provides separation between where a shell is running
and where the shell is accessed. Each shell can be running on a different
computer, but to the developer it does not matter where each shell is being
executed.
For example, a developer could have many shells running within a terminal,
like the following screenshot.
The above terminal window is using the tmux terminal multiplexer
implementation with two windows and three panes.
80
Full Stack Python: 2020 Supporter's Edition
Why are terminal multiplexers awesome?
Developers gain greater control over the usage of their shells by working with
a terminal multiplexer.
Shells are typically executed locally on a computer but terminal multiplexers
allow one or more virtual shells to be run within a single terminal. Shells can
also be left running within the multiplexer and attached to again from a
different machine.
Terminal multiplexers are used by developers to run many virtual shells
within a single terminal. These shells can be run via a mix of local, remote,
containerized and virtualized resources. The shells can also be persisted and
moved while running from one computer to another.
Terminal multiplexer implementations
Many terminal multiplexer implementations exist, including:
• tmux
• screen
• byobu
• Pymux (source code) is a terminal multiplexer implementation written
in Python that clones the functionality of tmux. Like tmux and Screen,
Pymux makes it easier for programmers to use many shells within a
single terminal window during development.
Terminal multiplexer resources
• Terminal multiplexers provides a wonderful overview of the subject,
including the history of various implementations and why you would
want to use one for development.
• Terminal multiplexer commands is a comparison of equivalent key
command in the two most popular implementations, tmux and screen.
• Byobu vs. GNU Screen vs. tmux — usefulness and transferability of
skills gives solid answers on this (now closed) question of the
usefulness of the major terminal multiplexer implementations.
• Pymux discussion on Hacker News
81
Full Stack Python: 2020 Supporter's Edition
tmux
tmux (source code) is a terminal multiplexer implementation often used
during Python development on Linux and macOS. tmux grants greater control
over a programmers's development environment by making it easier to use
many shells at once and attaching to both local and remote shell sessions.
tmux resources
• Tmux and Vim - even better together explains how using tmux with the
Vim editor provide better multi-window navigation using only
keystrokes and other benefits. As an avid tmux+Vim user myself, I can
attest to how great these two tools complement each other.
• Making tmux Pretty and Usable - A Guide to Customizing your
tmux.conf
• Tmux Pairing Anywhere: On Your Box
• Using tmux Properly
• Differences between tmux vs screen
• The Power Of tmux Hooks
• There are a slew of "cheat sheets" for tmux out there, here are a few
good ones:
◦
◦
◦
◦
tmux cheat sheet
shortcuts & cheatsheet
tmux & screen cheat-sheet
Multiplexers: tmux and screen
82
Full Stack Python: 2020 Supporter's Edition
Screen
Screen (source code) is a terminal multiplexer implementation often used
during Python development on Linux and macOS operating systems. Screen
makes it easier for programmers to use many shells within a single terminal
window while developing their applications.
Screen resources
• Learn to use screen, a terminal multiplexer
• SCREEN - the terminal multiplexer
• Using GNU Screen to Manage Persistent Terminal Sessions
Environment configuration
Properly configuring a development, test or production environment is
important for the successful execution of your Python application.
There are many de facto names for environments:
1.
2.
3.
4.
5.
Local development
Development / integration
Test
Staging
Production
83
Full Stack Python: 2020 Supporter's Edition
The above list provides the common environment names but there can be a
limitless number and each organization has their own configuration.
Environment variables
Environment variables are modifiable system values that can be read by
Python applications to affect a program's execution.
One answer I found very useful when learning about getting environment
variables in Python code is knowing the difference between os.getenv and
os.environ.get. Either one can be used in your applications but there are slight
differences that can make one better than the other in various situations.
Other useful environment variables resources: * The Twelve-Factor App
describes a method for securing environment data for your applications. The
twelve factors are commonly referenced across many programming
ecosystems, not just Python, so it's worthwhile to familiarize yourself with
how to use this method to configure your applications.
• Everything You Need to Know About the Twelve-Factor App breaks
down the original twelve-factor app source material and provides solid
additional advice and context.
• Environment variables on Ubuntu Linux
• Why you shouldn't use ENV variables for secret data
• Environment variables in Windows
• Security of infrastructure secrets elaborates on techniques to protect
your secret tokens such as API keys as well as the threats that are out
there which put your secrets at risk.
Environment configuration resources
• Staging Servers, Source Control & Deploy Workflows, And Other Stuff
Nobody Teaches You
• Deployments best practices explains the differences between various
environments and why you need each one.
• Best practices for staging environments
• Staging environment vs Production environment
84
Full Stack Python: 2020 Supporter's Edition
• A good QA team needs a proper software staging environment for
testing
Application Dependencies
Application dependencies are the libraries other than your project code that
are required to create and run your application.
Why are application dependencies important?
Python web applications are built upon the work done by thousands of open
source programmers. Application dependencies include not only web
frameworks but also libraries for scraping, parsing, processing, analyzing,
visualizing, and many other tasks. Python's ecosystem facilitates discovery,
retrieval and installation so applications are easier for developers to create.
Finding libraries
Python libraries are stored in a central location known as the Python Package
Index. PyPi contains search functionality with results weighted by usage and
relevance based on keyword terms.
Besides PyPi there are numerous resources that list common or "must-have"
libraries. Ultimately the decision for which application dependencies are
necessary for your project is up to you and the functionality you're looking to
build. However, it's useful to browse through these lists in case you come
across a library to solve a problem by reusing the code instead of writing it all
yourself. A few of the best collections of Python libraries are
• Python.org's useful modules which groups modules into categories.
• GitHub Explore Trending repositories shows the open source Python
projects trending today, this week, and this month.
• This list of 20 Python libraries you can’t live without is a wide-ranging
collection from data analysis to testing tools.
• Wikipedia actually has an extensive page dedicated to Python libraries
grouped by categories.
85
Full Stack Python: 2020 Supporter's Edition
Isolating dependencies
Dependencies are installed separately from system-level packages to prevent
library version conflicts. The most common isolation method is virtualenv.
Each virtualenv is its own copy of the Python interpreter and dependencies in
the site-packages directory. To use a virtualenv it must first be created with
the virtualenv command and then activated.
The virtualenv stores dependencies in an isolated environment. The web
application then relies only on that virtualenv instance which has a separate
copy of the Python interpreter and site-packages directory. A high level of how
a server configured with virtualenv can look is shown in the picture below.
Installing Python dependencies
The recommended way to install Python library dependencies is with the pip
command when a virtualenv is activated.
86
Full Stack Python: 2020 Supporter's Edition
Pip and virtualenv work together and have complementary responsibilities.
Pip downloads and installs application dependencies from the central PyPi
repository.
requirements.txt
The pip convention for specifying application dependencies is with a
requirements.txt file. When you build a Python web application you
should include requirements.txt in the base directory of your project.
Python projects' dependencies for a web application should be specified with
pegged dependencies like the following:
django==1.11.0
bpython==0.12
django-braces==0.2.1
django-model-utils==1.1.0
logutils==0.3.3
South==0.7.6
requests==1.2.0
stripe==1.9.1
dj-database-url==0.2.1
django-oauth2-provider==0.2.4
djangorestframework==2.3.1
Pegged dependencies with precise version numbers or Git tags are important
because otherwise the latest version of a dependency will be used. While it
may sound good to stay up to date, there's no telling if your application
actually works with the latest versions of all dependencies. Developers should
deliberately upgrade and test to make sure there were no backwardsincompatible modifications in newer dependency library versions.
setup.py
There is another type of dependency specification for Python libraries known
as setup.py. Setup.py is a standard for distributing and installing Python
libraries. If you're building a Python library, such as twilio or underwear you
must include setup.py so a dependency manager can correctly install both the
library as well as additional dependencies for the library. There's still quite a
bit of confusion in the Python community over the difference between
requirements.txt and setup.py, so read this well written post for further
clarification.
87
Full Stack Python: 2020 Supporter's Edition
Open source app dependency projects
pip and venv are part of Python 3's standard library as of version 3.3.
However, there are numerous other open source libraries that can be helpful
when managing application dependencies in your projects, as listed below.
• Autoenv is a tool for activating environment variables stored in a .env
file in your projects' home directories. Environment variables aren't
managed by virtualenv and although virtualenvwrapper has some
hooks for handling them, it's often easiest to use a shell script or .env
file to set them in a development environment.
• Pipenv is a newer Python packaging and dependency management
library that has seen some adoption in place of the standard pip
library.
• Pipreqs searches through a project for dependencies based on imports.
It then generates a requirements.txt file based on the libraries
necessary to run those dependencies. Note though that while this could
come in handy with a legacy project, the version numbers for those
libraries will not be generated with the output.
• pip-check presents a nicely-formatted list of all your installed
dependencies and the status of whether or not updates are available for
each of them.
• pip-name is a straightforward library that looks up package names on
PyPI and tells you whether or not the library name is already taken.
Code library packaging guides
There are many steps in creating and distributing packages on PyPI and your
own hosted application dependency servers. Many of these steps involve
writing configuration files that are not as well documented as some other
areas of Python development. These resources are the best ones I have found
so far to get up to speed on building and releasing your own packages.
• Python Packaging User Guide provides a collection of resources to
understand how to package and distribute Python code libraries.
• Alice in Python projectland is an amazing post that takes the reader
from simple Python script into a complete Python package.
88
Full Stack Python: 2020 Supporter's Edition
• Perils of packaging covers several edge cases that come up when trying
to put together pieces like Travis CI, PyPI and conda. The post walks
through the errors and how to get around them until they are smoothed
out by updates to the tools.
• How to Publish Your Package on PyPI is for developers who have
created a code library they would like to share and make installable for
other developers.
• How to Submit a Package to PyPI presents the basic steps like signing
up for a PyPI account and other accounts that go along with the
tutorial. It then walks through the configuration code for setting up
continuous integration and deploying your package.
Application dependency resources
The following links provide advice on how to use Python packages as well as
package your own dependencies for projects or consumption by other
developers.
• Python's New Package Landscape covers the history of Python
packaging tools and examines the problems with dependency isolation
and the dependency graphs that newer tools such as Pipenv, Poetry,
Hatch and pipsi aim to solve.
• Python Packaging Is Good Now is a wonderfully written blog post. It
provides historical context on why Python's code library packaging was
painful for a long time, and what's been fixed to make building and
installing application dependencies so much better.
• A non-magical introduction to virtualenv and pip breaks down what
problems these tools solve and how to use them.
• There’s no magic: virtualenv edition breaks open the virtual
environment "black box" to show you what the tool is doing when you
use its commands.
• Using pyenv to manage your Python interpreters explains how the
pyenv tool can make it easier to switch between different versions of
Python for each project and gives a brief review of the important things
to know when using the tool such as local versus global scope.
89
Full Stack Python: 2020 Supporter's Edition
• Testing & Packaging examines a configuration for ensuring tests run
against your code and how to properly package your project.
• 12 Alternatives for Distributing Python Applications in 2020 covers
packaging with Docker, Vagrant, PyInstaller, Briefcase, virtual
environments, Pipx and several more options for bundling and running
Python code.
• Python Application Dependency Management in 2018 presents some
critical analysis and critique oof the existing Python dependency
management tools including newer ones such as pipenv and Poetry.
• Occasionally arguments about using Python's dependency manager
versus one of Linux's dependency managers comes up. This provides
one perspective on that debate.
• Open source trust scaling is a good piece for the Python community
(and other programming communities) that is based on the left-pad
NPM situation that broke many dependent packages in the Node.js
community.
• Major speed improvements were made in pip 7 over previous versions.
Read this article about the differences and be sure to upgrade.
• Typosquatting programming language package managers shows how
many packages on centralized dependency servers for Python, Node.js
and Ruby can be vulnerable to "typosquatting" where a developer
either confuses a fake package for the correct one or simply makes a
typo when specifying her dependency list.
• The Many Layers of Packaging goes up and down the packaging stack
and even covers bits about virtual environments and security. It's well
worth investing some time to read this post to get an overview of the
many layers involved in dependency packaging.
Application dependencies learning checklist
1. Ensure the libraries your web application depends on are all captured
in a requirement.txt file with pegged versions.
2. An easy way to capture currently installed dependencies is with the
pip freeze command.
90
Full Stack Python: 2020 Supporter's Edition
3. Create a fresh virtualenv and install the dependencies from your
requirements.txt file by using the
pip install -r requirements.txt command.
4. Check that your application runs properly with the fresh virtualenv and
only the installed dependencies from the requirements.txt file.
Virtual environments (virtualenvs)
Virtual environments, implemented by the library virtualenv and venv (added
to Python standard library in Python 3.3 via PEP 405), separate project
dependencies, such as the Django library code, from your code projects. For
example, if you have three projects, one that uses Django 1.7, another that
uses Django 2.0 and another project that does not use Django at all, you will
have three virtualenvs that each contain those dependencies separated from
each other.
How do virtualenvs work?
Virtualenv provides dependency isolation for Python projects. A virtualenv
creates a separate copy of the Python installation that is clean of existing code
libraries and provides a directory for new application dependencies on a perproject basis. A programmer can technically use a virtualenv for many
projects at once but that is not consider to be a good practice.
Virtual environment resources
• Package management in Python 2 or 3 (Linux and Mac) with virtualenv
or venv
• There’s no magic: virtualenv edition
• Virtual environments dymystified
• What is the relationship between virtualenv and pyenv?
• Setting up Python on a Unix machine (with pyenv and direnv)
• venv — Create Virtual Environments
• Python development environment, 2018 edition
91
Full Stack Python: 2020 Supporter's Edition
Localhost tunnels
A localhost tunnel establishes a connection between your local machine and a
remote connection. The connection is intended to proxy traffic from a
publicly-addressable IP address and URL to your local machine. Localhost
tunnels are most useful for allowing a tester to connect to a server running on
your local development system so they can try out an in-development
application you are building but have not yet deployed.
Localhost tunnel services
There are numerous localhost tunnel services that have similar features. The
following services are listed in order from ones I have had the most
experience with to the ones I have not used.
• ngrok is the service I use most often. It is easy and worth the small fee
to upgrade your account with a few extra features such as fixed,
customizable subdomains. There is also a Python wrapper for ngrok
called pyngrok that makes it easy to programmatically access the ngrok
client from Python applications.
• Localtunnel is a localhost tunnel written in Node.js.
• Burrow provides another service, albeit one that I have not used
myself.
Source Control
Source control, also known as version control, stores software code files with a
detailed history of every modification made to those files.
Why is source control necessary?
Version control systems allow developers to modify code without worrying
about permanently screwing something up. Unwanted changes can be easily
rolled back to previous working versions of the code.
Source control also makes team software development easier. One developer
can combine her code modifications with other developers' code through diff
views that show line-by-line changes then merge the appropriate code into the
main code branch.
92
Full Stack Python: 2020 Supporter's Edition
Version control is a necessity on all software projects regardless of
development time, codebase size or the programming language used. Every
project should immediately begin by using a version control system such as
Git or Mercurial.
Monorepo vs Multirepo
There is a spectrum of philosophies for how to store projects within source
code repositories.
On one extreme end of the spectrum, every line of code for every project
within an organization is stored in a single repository
repository. That approach is called
monorepo and it is used by companies like Google. On the other end of the
spectrum, there are potentially tens of thousands or more repositories that
store parts of projects. That approach is known as multirepo or manyrepo.
For example, in a microservices architecture, there could be thousands of
microservices and each one is stored within its own repository. No one
repository contains the code for the entire application created by the
interaction of the microservices.
There are many hybrid strategies for how to store source code that fall
between these opposite approaches. What to choose will depend on your
organization's needs, resources and culture.
Source control during deployment
Pulling code during a deployment is a potential way source control systems fit
into the deployment process.
93
Full Stack Python: 2020 Supporter's Edition
Note that some developers recommend deployment pipelines package the
source code to deploy it and never have a production environment touch a
source control system directly. However, for small scale deployments it's often
easiest to pull from source code when you're getting started instead of figuring
out how to wrap the Python code in a system installation package.
Source control projects
Numerous source control systems have been created over the past several
decades. In the past, proprietary source control software offered features
tailored to large development teams and specific project workflows. However,
open source systems are now used for version control on the largest and most
complicated software projects in existence. There's no reason why your
project should use anything other than an open source version control system
in today's Python development world. The two primary choices are:
• Git is a free and open source distributed version control system.
• Mercurial is similar to Git, also a free and open source distributed
version control system.
• Subversion is a centralized system where developers must check files in
and out of the hosted repository to minimize merge conflicts.
94
Full Stack Python: 2020 Supporter's Edition
Hosted version control services
Git and Mercurial can be downloaded and run on your own server. However,
it's easy and cheap to get started with a hosted version control service. You
can transition away from the service at a later time by moving your
repositories if your needs change. A couple of recommended hosted version
control services are:
• GitLab has both a self-hosted version of its open source software as
well as their hosted version with pricing for businesses that need
additional hosting support.
• GitHub is a software-as-a-service platform that provides a user
interface, tools and backup for developers to use with their Git
repositories. Accounts are free for public open source development and
private Git repositories can also be hosted for $7 per month.
• BitBucket is Atlassian's software-as-a-service tool that with a user
interface, comparison tools and backup for Git projects. There are
many features in BitBucket focused on making it easier for groups of
developers to work on projects together. BitBucket also has private
repositories for up to five users. Users pay for hosting private
repositories with more than five users.
General source control resources
• Staging Servers, Source Control & Deploy Workflows, And Other Stuff
Nobody Teaches You is a comprehensive overview by Patrick McKenzie
of why you need source control.
• Version control best practices is a good write up of how to work with
version control systems. The post is part of an ongoing deployment
guide written by the folks at Rainforest.
• A visual guide to version control is a detailed article with real-life
examples for why version control is necessary in software development.
• An introduction to version control shows the basic concepts behind
version control systems.
• What Is Version Control? Why Is It Important For Due Diligence?
explains the benefits and necessity of version control systems.
95
Full Stack Python: 2020 Supporter's Edition
• Version control before Git with CVS goes into the history of version
control systems and defines three generations, of which CVS and SVN
were part of the second generation while Git and Mercurial are thirdgeneration version control systems.
• About version control reviews the basics of distributed version control
systems.
• Why not Git? covers SQLite's development workflow and why they do
not use Git as their version control system.
Monorepo vs multirepo resources
Monorepo versus multirepo version control strategies are a weirdly
contentious topic in software development, likely because once a policy is set
for an organization it is exceptionally difficult to change your approach. The
following resources give more insight into the debate on how to structure your
repositories.
• Monorepo, Manyrepo, Metarepo is an awesome guide to varying ways
of structuring your source repositories that contain more than one
project. The guide covers advantages and disadvantages of common
approaches used in both small and large organizations.
• Repo Style Wars: Mono vs Multi goes into the implications of using one
side or the other and why it is unlikely you can create a combination
solution that will give you the advantages of both without the
disadvantages.
• Why Google Stores Billions of Lines of Code in a Single Repository
covers the history and background of Google's source control
monorepo, which is one of if not the largest monorepo for an
organization in the world.
• Advantages of monorepos goes into the advantages of using a
monorepo and does not discuss the downsides but admits there are
many so the decision is not clear-cut on using either strategy.
• Monorepos and the Fallacy of Scale argues that having all of an
organization's code in a single repository encourages code sharing. The
author considers the concerns often raised about tight coupling
96
Full Stack Python: 2020 Supporter's Edition
between components in a monorepo code base but says that the
advantages outweigh the disadvantages overall.
Git distributed source control system
Git is the most widely-used source control system currently in use. Its
distributed design eliminates the need to check files in and out of a
centralized repository, which is a problem when using Subversion without a
network connection. There is a full page on Git with further details and
resources.
Subversion resources
Apache Subversion (source code), often just called "Subversion" or "SVN", is a
source control system implementation.
• The SVN book is the free online version of the O'Reilly Version Control
with Subversion book.
• How to use Subversion (SVN) lays out the basic concepts and provides
the first few steps for getting started tracking files.
• 10 Most Used SVN Commands with Examples is a good refresher list if
you've used SVN in the past but it has been awhile since you worked
with all the commands.
Source control learning checklist
1. Pick a version control system. Git is recommended because on the web
there are a significant number of tutorials to help both new and
advanced users.
2. Learn basic use cases for version control such as committing changes,
rolling back to earlier file versions and searching for when lines of code
were modified during development history.
3. Ensure your source code is backed up in a central repository. A central
repository is critical not only if your local development version is
corrupted but also for the deployment process.
4. Integrate source control into your deployment process in three ways.
First, pull the project source code from version control during
97
Full Stack Python: 2020 Supporter's Edition
deployments. Second, kick off deployments when code is modified by
using webhooks or polling on the repository. Third, ensure you can roll
back to a previous version if a code deployment goes wrong.
Git
Git is a distributed open source source control (also referred to as "version
control") system commonly used to track and manage file changes. Git is
frequently used as the version control system for Python projects.
Why is Git widely-used by developers?
Git is a distributed version control system (DVCS) compared to the
centralized models previously provided by Subversion and CVS. Files would
need to be "checked out" over the network by a single person at a time while
she was working. The network transfer speed as well as the blocking check out
model became a significant bottleneck, especially for large development
teams.
Git clones a full repository and its entire history instead of just the current
state of a file. Developers only require a network connection when pulling
updates and pushing changes to a backup repository. The commit log and file
histories are stored and transmitted far more efficiently than prior version
control systems to maximize the effectiveness of the distributed version
control design.
Another issue with traditional VCS was that it was difficult to create branches.
Take a look at this tutorial on managing a CVS repository as an example of the
98
Full Stack Python: 2020 Supporter's Edition
confusion the existing non-distributed models could cause. Git simplified the
branching process with simplified commands such as git checkout -b and
faster branch merging and clean up. In contrast to earlier version control
systems, Git encourages developers to create local branches and experiment
in them without impacting a stable master branch.
GitHub also helped to drive Git as the overwhelming version control favorite
by providing the open source community with free open remote Git
repositories. GitHub's web application user interface, issue tracking and pull
request features for maintainers and consumers also encouraged more
collaboration than Git alone. Recently, GitHub's third-party marketplace has
begun to add more features by integrating continuous integration servers like
as Jenkins and code metrics services.
Beginner Git tutorials
Git can take awhile to wrap your head around, even for experienced software
developers. The following tutorials can quickly get you up to speed.
• The official Pro Git book is available online for free. It is awesome both
as a step-by-step walkthrough and as a bookmarked reference on
specific topics.
• Git from the inside out provides a spectacular walkthrough for
developers who have used Git before but want to go deeper in
understanding what each command does under the covers instead of
simply using the tool as a black box.
• Think like a Git is another introduction that focuses more on the graph
theory and conceptual ideas behind Git to help the reader understand
what's happening as they use Git commands.
• Git and GitHub in plain English is a high-level overview of both Git and
GitHub. This guide is intended for both non-programmers and junior
developers who want to learn everything from terminology to
workflow.
• A Hacker's Guide to Git is a free ebook written for experienced
developers that contains both the syntax and the conceptual ideas
behind how Git works.
99
Full Stack Python: 2020 Supporter's Edition
• A Designer's Guide to Git gives a beginner's Git overview for nonprogrammers. The tutorial also covers using Git clients such as the
GitHub desktop application.
• Git in Six Hundred Words is a concise essay explaining what happens
when you add and commit files in a Git repository.
• 19 Tips For Everyday Git Use is a laundry list of helpful Git tips on
commands such as git bisect , git stash and git difftool .
• git ready presents beginner, intermediate and advanced tips for how to
use Git. The example commands and their results are great for learning
Git piece-by-piece.
Advanced Git tutorials and resources
You won't learn Git in an afternoon or even a few months of usage. After sixplus years of working with Git I still get tripped up and have a lot to learn.
These tutorials have taught me some of the beyond-the-basics edge cases.
• Flight rules for git contains common commands that answer specific
desired tasks such as "I want to discard specific unstaged files"
( git checkout filename ) and "I want to rename a branch"
( git branch -m newname ).
• Shadows Of The Past: Analysis Of Git Repositories explains how you
can extract some surprising data from Git repositories' commit history,
such as which developers are domain experts in certain tools, potential
hot spots in the code and coupling between source code files. This is a
great read once you get past the basics of using Git.
• Write yourself a Git! is a tutorial for building your own version of Git
from scratch with 503 lines of Python code. The result is obviously not
as full-featured as the real Git implementation but this program is
awesome for understanding how Git's internals work.
• Phil Nash shows how to use the git reflog command in Git back to
the future.
• On undoing, fixing, or removing commits in git is a fantastic overview
of how to unscrew a whole slew of bad situations you may find yourself
in if you use Git for long enough.
100
Full Stack Python: 2020 Supporter's Edition
• High-level Problems with Git and How to Fix Them is a long-form
article on how to fork properly (and how not to use them) and how to
not go crazy using branches and remote repositories.
Specific Git resources
Large tutorials are great for getting started with Git. However, sometimes you
need tactical support or want to learn new tricks to add to your workflow.
These resources will come in handy for specific Git subjects.
• How to Write a Git Commit Message provides strong advice that will
help you write consistent, concise and contextual messages on your
commits. Commit messages are especially important when working
with others on a long-lasting project where you dive through the
commit history via git log and related commands.
• How to squash Git commits explains how to use the git rebase
command in interactive mode to consolidated the number of commits
in your history. This technique is useful when a group of commits are
related and it's easier to understand them as a single commit rather
than a collection of smaller commits.
• Oh shit, Git! is a profanity-filled description of tips to get you out of
binds you may find yourself in when you get too tricky with Git
commands.
• Tips for a disciplined git workflow is less about workflow and more
about how to write self-explaining commit messages, self-containing
each commit and modifying branch history when you muff up before it
is merged into master.
• Another Git catastrophe cleaned up goes through a difficult merge
scenario that required deep Git understanding to properly fix.
• Erlang's source code provides a concise explanation on writing good
commit messages that any programming ecosystem can learn from.
• Git internals is a presentation that covers how Git stores data, how to
work with the Git history, and good practices for using Git based on the
knowledge of how it works internally.
101
Full Stack Python: 2020 Supporter's Edition
• Chasing a bad commit examines the git bisect command and how
it can be used in either interactive mode or on its own with
git bisect run to find the problematic code commit that needs to
be fixed.
• How Microsoft uses Git gives a high-level overview of their repository
structure and hosting at the extremely large scale organization.
• GitTips is a list of pro tips to clean up common issues and how to dive
through Git history to find specific text.
• Git allows command aliasing, which allowed one developer to create
his own list of lesser known Git commands that alias more complicated
Git lines.
• Little things I like to do with Git has some nice tips such as easily
viewing branches you recently worked on and generating a changelog
from your commits.
• Git from the inside out demonstrates how Git's graph-based data
structure produces certain behavior through example Git commands.
This is a highly recommended read after you've grasped the basics and
are looking to go deeper with Git.
• How I configure my git in a new computer shows how to handle a
.gitconfig file, with an example Gist that the author uses for his own
environment.
• How to Quickly and Correctly Generate a Git Log in HTML is an
interesting look at how string processing on *nix systems works by
generating an HTML page from a Git log. If you need to output your Git
commits somewhere and are having trouble writing your own script
you should check out some of the interesting solutions the author
presents.
• Better Git configuration explains global config options, revisions and
merging along with several other commands that can be customized to
your taste.
• Why does Git use a cryptographic hash function? explains that the
SHA-1 hash isn't used for security on Git, it's a consistency check.
SHA-1 has been broken in practice so Git needs to transition to a
102
Full Stack Python: 2020 Supporter's Edition
stronger hash without proven collisions but it's not quite as big of a
concern compared to security-related projects that use SHA-1.
• The anatomy of a git commit digs into the tree and commit objects that
underpin the Git source control system. This is an awesome read to get
a view on how Git works under the commands you're using to
manipulate these objects.
Git Workflows
Teams of developers can use Git in varying workflows because of Git's
distributed model and lightweight branching. There is no "right way" to use
Git, especially because development teams can range in size from a single
developer up to entire companies with thousands of developers in a
repository. The only correct answer is to let the developers decide on a
workflow that maximizes their ability to frequently commit code and
minimize merge conflicts.
• git-flow shows one possible way for small teams to use Git branches.
GitHub Flow explains why at GitHub they do not use the git-flow
model and provides an alternative that solves some of the issues they
found with git-flow.
• Git Workflows That Work is a helpful post with diagrams to show how
teams can create a Git workflow that will help their development
process.
• Comparing workflows provides a slew of examples for how developers
on a team can handle merge conflicts and other situations that
commonly arise when using Git.
GitHub resources
GitHub is a software-as-a-service application owned by Microsoft that makes
it easier to collaborate with other developers on centralized Git repositories.
The site also provides a remote backup location for repositories as well as
secure, private repository storage. The following tutorials show how to get
started using Git on GitHub.
103
Full Stack Python: 2020 Supporter's Edition
• Introduction to Git and GitHub for Python Developers covers basic
usage for Git and working with repositories locally and on the GitHub
service.
• Hello World: GitHub edition and Git and GitHub learning resources
are GitHub's official guide and learning resources.
• A Beginner’s Git and GitHub Tutorial shows how to perform your first
commit and back it up on GitHub.
Mercurial
Mercurial is a distributed open source source control (also known as "version
control") system written in Python for tracking and handling file
modifications. Mercurial can be used as the version control system for Python
projects.
Mercurial tutorials
• The official Mercurial tutorial goes through the basics. It has great
examples for syntax and expected output.
• Getting started with Mercurial for version control seems mistitled
because it is actually a short tutorial on how to build Mercurial
extensions in Python. With a few lines of Python code it shows how to
create your first extension and test it.
• Mercurial: The Definitive Guide is a free online version of the O'Reilly
book.
104
Full Stack Python: 2020 Supporter's Edition
• Monoroke is a Mercurial server written in Rust designed to be used for
very large monorepos that have thousands of commits affecting
millions of files per hour.
105
Full Stack Python: 2020 Supporter's Edition
Data
Data is an incredibly broad topic but it can be broken down into many
subsections, including (in no particular order):
•
•
•
•
•
•
•
•
•
data processing / wrangling
machine learning
data analysis
visualization
geospatial mapping
persistence via relational databases and NoSQL data stores
object-relational mappers
natural language processing (NLP)
indexing, search and retrieval
The Python community has built and continues to create open source libraries
and tutorials for all of the above topics.
Why is Python a great language choice for data tasks?
Python has a wide array of open source code libraries available and a diverse
community of people with different backgrounds who contribute to make
those libraries better each day.
In addition, Python data manipulation code can be combined with web
frameworks and web APIs to build software that would be difficult to create
with a single other language. For example, Ruby is a fantastic language for
building web applications but its data analysis and visualization libraries are
very limited compared to what is currently available in the Python ecosystem.
How did Python become so widely used for working with
data?
Python is a general purpose programming language and can be applied to
many problem areas. Over the past couple of decades, Python has become
increasingly popular in the scientific and financial communities. Projects such
as pandas grew out of a hedge-fund while NumPy and SciPy were created in
academic environments then improved by the broader open source
community.
107
Full Stack Python: 2020 Supporter's Edition
The question is: why Python was used to created these projects? The answer is
a mix of luck, the growth of the open source community as Python was
maturing and wide adoption by people not formally trained as computer
scientists. The pragmatic syntax and explicit style helped very intelligent
people without programming backgrounds to pick up the language and get
their work done with less fuss than other programming languages. Over time
the code used in the financial world and scientific community was shared at
the same time global open source communities were developing, further
spreading their usage among a broader base of software developers.
There's no doubt some of the momentum behind Python's wide adoption for
all types of data manipulation was that it happened to be the right language in
the right place at the right time. Nevertheless, it was ultimately the hard work
of a massive number of engineers and scientists around the world who created
the incredible mix of data code libraries available today.
Data inspiration
Sometimes you just need to see it to understand how data analysis,
visualization and storytelling can intersect in a meaningful way. The following
resources do a great job of telling stories with data. There are more links to
stories listed on the data analysis and data visualization pages.
• Data — from objects to assets covers the history of data collection and
usage, from 150 years ago to today. The article covers how initial steps
by individual scientists sponsored by wealthy patrons in the 1800s gave
way to systematic collection by governments and businesses in the
20th century. A significant amount of personal data is now held by a
few dozen large corporations worldwide such as Google, Amazon and
Facebook. The article covers some of the implications of data as a
valuable asset and in general is a great read as a high-level overview of
on this topic.
• Metadata Investigation : Inside Hacking Team presents what metadata
is and how it can be used to track people even though it is often
thought of as less of a problem than typical stored data.
• A visual introduction to machine learning is a spectacular example of
data visualization to explain what a machine learning model does on a
San Francisco and New York housing data set.
108
Full Stack Python: 2020 Supporter's Edition
• Earthquake recurrence and survival analysis: How long should we wait
for an overdue earthquake? combines earthquake data with questions
around earthquake recurrence probabilities to tell its story.
• Data Science Project: Profitable App Profiles for App Store and Google
Play is a tutorial that shows you how to use iOS and Android app store
data for business analysis. This post is part of a larger series on how to
get your first job as a data scientist which is all worth your time reading
to understand the intersection of working with data to figure out its
value to companies sand organizations.
Example data sets
Looking for freely-available data to use in your projects but aren't sure where
to get it? The following links have large free, open data sets.
• Check out the awesome public datasets project repository for data in
many different categories ranging from finance to museums.
• Kickstarter datasets are scraped JSON and CSV structured monthly
data from Kickstarter projects.
• Data is Plural is a weekly newsletter that highlights open data that you
can use for your projects. I have been a subscriber to the newsletter for
a couple of years now and love seeing the wide variety of data sources
that are freely available.
• Data analysis and machine learning projects provides more than just
the data, it also includes instructions and code for working with the
data in your own development environment.
• Discovering millions of datasets on the web introduces Google's dataset
search and explains what they learned from iterating on earlier
versions of it before they released this one.
General Python data resources
• PyData is a community for developer and users of Python data tools.
They put on fantastic conferences around the world and fund the
continued development of open source data-related libraries.
109
Full Stack Python: 2020 Supporter's Edition
• Anaconda is one of the leading Python companies that pours a
tremendous amount of time and funding into the data community.
• A crash course in Python for scientists provides an overview of the
Python language with iPython Notebook for those in scientific fields.
• The videos of Travis Oliphant on Python's Role in Big Data Analytics:
Past, Present, and Future and Building the PyData Community give
historical perspective on how the Python data tools have evolved over
the past 20ish years based on his first-hand experience as a leader and
member in that community.
• The Open Source Data Science Masters is a well-crafted free
curriculum and set of resources for students who want to learn both the
theory and technologies for working with data.
• Reproducible research: Stripe’s approach to data science goes through
the workflow and tools such as Jupyter Notebook that Stripe for their
data analysis across the company.
• The Definitive Data Scientist Environment Setup explains how to set
up both a hardware and software configuration that is conducive to
data science research and analysis.
Databases
A database is an abstraction over an operating system's file system that makes
it easier for developers to build applications that create, read, update and
delete persistent data.
Why are databases necessary?
At a high level web applications store data and present it to users in a useful
way. For example, Google stores data about roads and provides directions to
110
Full Stack Python: 2020 Supporter's Edition
get from one location to another by driving through the Maps application.
Driving directions are possible because the data is stored in a structured
format.
Databases make structured storage reliable and fast. They also give you a
mental framework for how the data should be saved and retrieved instead of
having to figure out what to do with the data every time you build a new
application.
Relational databases
The database storage abstraction most commonly used in Python web
development is sets of relational tables. Alternative storage abstractions are
explained on the NoSQL page.
Relational databases store data in a series of tables. Interconnections between
the tables are specified as foreign keys. A foreign key is a unique reference
from one row in a relational table to another row in a table, which can be the
same table but is most commonly a different table.
Databases storage implementations vary in complexity. SQLite, a database
included with Python, creates a single file for all data per database. Other
databases such as PostgreSQL, MySQL, Oracle and Microsoft SQL Server have
more complicated persistence schemes while offering additional advanced
features that are useful for web application data storage. These advanced
features include but are not limited to:
1. data replication between a master database and one or more read-only
slave instances
2. advanced column types that can efficiently store semi-structured data
such as JavaScript Object Notation (JSON)
3. sharding, which allows horizontal scaling of multiple databases that
each serve as read-write instances at the cost of latency in data
consistency
4. monitoring, statistics and other useful runtime information for
database schemas and tables
Typically web applications start with a single database instance such as
PostgreSQL with a straightforward schema. Over time the database schema
evolves to a more complex structure using schema migrations and advanced
111
Full Stack Python: 2020 Supporter's Edition
features such as replication, sharding and monitoring become more useful as
database utilization increases based on the application users' needs.
Most common databases for Python web apps
PostgreSQL and MySQL are two of the most common open source databases
for storing Python web applications' data.
SQLite is a database that is stored in a single file on disk. SQLite is built into
Python but is only built for access by a single connection at a time. Therefore
is highly recommended to not run a production web application with SQLite.
PostgreSQL database
PostgreSQL is the recommended relational database for working with Python
web applications. PostgreSQL's feature set, active development and stability
contribute to its usage as the backend for millions of applications live on the
Web today.
Learn more about using PostgreSQL with Python on the PostgreSQL page.
MySQL database
MySQL is another viable open source database implementation for Python
applications. MySQL has a slightly easier initial learning curve than
PostgreSQL but is not as feature rich.
Find out about Python applications with a MySQL backed on the dedicated
MySQL page.
Connecting to a database with Python
To work with a relational database using Python, you need to use a code
library. The most common libraries for relational databases are:
• psycopg2 (source code) for PostgreSQL.
• MySQLdb (source code) for MySQL. Note that this driver's
development is mostly frozen so evaluating alternative drivers is wise if
you are using MySQL as a backend.
112
Full Stack Python: 2020 Supporter's Edition
• cx_Oracle for Oracle Database (source code). Oracle moved their open
source driver code from SourceForge to GitHub in 2017.
SQLite support is built into Python 2.7+ and therefore a separate library is not
necessary. Simply "import sqlite3" to begin interfacing with the single filebased database.
Object-relational Mapping
Object-relational mappers (ORMs) allow developers to access data from a
backend by writing Python code instead of SQL queries. Each web application
framework handles integrating ORMs differently. There's an entire page on
object-relational mapping (ORMs) that you should read to get a handle on
this subject.
Database third-party services
Numerous companies run scalable database servers as a hosted service.
Hosted databases can often provide automated backups and recovery,
tightened security configurations and easy vertical scaling, depending on the
provider.
• Amazon Relational Database Service (RDS) provides pre-configured
MySQL and PostgreSQL instances. The instances can be scaled to
larger or smaller configurations based on storage and performance
needs.
• Google Cloud SQL is a service with managed, backed up, replicated,
and auto-patched MySQL instances. Cloud SQL integrates with Google
App Engine but can be used independently as well.
• BitCan provides both MySQL and MongoDB hosted databases with
extensive backup services.
• ElephantSQL is a software-as-a-service company that hosts
PostgreSQL databases and handles the server configuration, backups
and data connections on top of Amazon Web Services instances.
113
Full Stack Python: 2020 Supporter's Edition
SQL resources
You may plan to use an object-relational mapper (ORM) as your main way of
interacting with a database, but you should still learn the basics of SQL to
create schemas and understand the SQL code generated by the ORM. The
following resources can help you get up to speed on SQL if you have never
previously used it.
• Select Star SQL is an interactive book for learning SQL. Highly
recommended even if you feel you will only be working with an objectrelational mapper on your projects because you never know when you
will need to drop into SQL to improve a generated query's slow
performance.
• A beginners guide to SQL does a good job explaining the main
keywords used in SQL statements such as SELECT , WHERE , FROM ,
UPDATE and DELETE .
• SQL Tutorial teaches the SQL basics that can be used in all major
relational database implementations.
• Life of a SQL query explains what happens both conceptually and
technically within a database when a SQL query is run. The author uses
PostgreSQL as the example database and SQL syntax throughout the
post.
• A Probably Incomplete, Comprehensive Guide to the Many Different
Ways to JOIN Tables in SQL elaborates on one of the trickiest parts of
writing SQL statements that bridge one or more tables: the JOIN .
• Writing better SQL is a short code styling guide to make your queries
easier to read.
• SQL Intermediate is a beyond-the-basics tutorial that uses open data
from the US Consumer Financial Protection Bureau as examples for
counting, querying and using views in PostgreSQL.
General database resources
• How does a relational database work? is a detailed longform post on
the sorting, searching, merging and other operations we often take for
114
Full Stack Python: 2020 Supporter's Edition
granted when using an established relational database such as
PostgreSQL.
• Databases 101 gives a great overview of the main relational database
concepts that is relevant to even non-developers as an introduction.
• Five Mistakes Beginners Make When Working With Databases explains
why you should not store images in databases as well as why to be
cautious with how you normalize your schema.
• DB-Engines ranks the most popular database management systems.
• DB Weekly is a weekly roundup of general database articles and
resources.
• Designing Highly Scalable Database Architectures covers horizontal
and vertical scaling, replication and caching in relational database
architectures.
• Online migrations at scale is a great read on breaking down the
complexity of a database schema migration for an operational
database. The approach the author's team used was a 4-step dual
writing pattern to carefully evolved the way data for subscriptions were
stored so they could move to a new, more efficient storage model.
• A one size fits all database doesn't fit anyone explains Amazon Web
Services' specific rationale for having so many types of relational and
non-relational databases on its platform but the article is also a good
overview of various database models and their use cases.
• SQL is 43 years old - here’s 8 reasons we still use it today lists why SQL
is commonly used by almost all developers even as the language
approaches its fiftieth anniversary.
• SQL keys in depth provides a great explanation for what primary keys
are and how you should use them.
• Exploring a data set in SQL is a good example of how SQL alone can be
used for data analysis. This tutorial uses Spotify data to show how to
extract what you are looking to learn from a data set.
• Databases integration testing strategies covers a difficult topic that
comes up on every real world project.
115
Full Stack Python: 2020 Supporter's Edition
• GitLab provided their postmortem of a database outage on January 31
as a way to be transparent to customers and help other development
teams learn how they screwed up their database systems then found a
way to recover.
• Asynchronous Python and Databases is an in-depth article covering
why many Python database drivers cannot be used without
modification due to the differences in blocking versus asychronous
event models. Definitely worth a read if you are using WebSockets via
Tornado or gevent.
• PostgreSQL vs. MS SQL Server is one perspective on the differences
between the two database servers from a data analyst.
Databases learning checklist
1. Install PostgreSQL on your server. Assuming you went with Ubuntu
run sudo apt-get install postgresql .
2. Make sure the psycopg2 library is in your application's dependencies.
3. Configure your web application to connect to the PostgreSQL instance.
4. Create models in your ORM, either with Django's built-in ORM or
SQLAlchemy with Flask.
5. Build your database tables or sync the ORM models with the
PostgreSQL instance, if you're using an ORM.
6. Start creating, reading, updating and deleting data in the database
from your web application.
PostgreSQL
PostgreSQL, often written as "Postgres" and pronounced "Poss-gres", is an
open source relational database implementation frequently used by Python
applications as a backend for data storage and retrieval.
116
Full Stack Python: 2020 Supporter's Edition
How does PostgreSQL fit within the Python stack?
PostgreSQL is the default database choice for many Python developers,
including the Django team when testing the Django ORM. PostgreSQL is
often viewed as more feature robust and stable when compared to MySQL,
SQLServer and Oracle. All of those databases are reasonable choices.
However, because PostgreSQL tends to be used by Python developers the
drivers and example code for using the database tend to be better documented
and contain fewer bugs for typical usage scenarios. If you try to use an Oracle
database with Django, you'll see there is far less example code for that setup
compared to PostgreSQL backend setups.
Why is PostgreSQL a good database choice?
PostgreSQL's open source license allows developers to operate one or more
databases without licensing cost in their applications. The open source license
operating model is much less expensive compared to Oracle or other
proprietary databases, especially as replication and sharding become
necessary at large scale. In addition, because so many people ranging from
independent developers to multinational organizations use PostgreSQL, it's
often easier to find developers with PostgreSQL experience than other
relational databases. There is also ancedotal evidence that PostgreSQL fixes
bugs faster than MySQL, although to be fair there has not been a
comprehensive study comparing how the two projects handle defect
resolution.
The PostgreSQL core team also releases frequent updates that greatly enhance
the database's capabilities. For example, in the PostgreSQL 9.4 release the
117
Full Stack Python: 2020 Supporter's Edition
jsonb type was added to enhance JavaScript Object Notation (JSON) storage
capabilities so that in many cases a separate NoSQL database is not required
in an application's architecture.
Connecting to PostgreSQL with Python
To work with relational databases in Python you need to use a database
driver, which is also referred to as a database connector. The most common
driver library for working with PostgreSQL is psycopg2. There is a list of all
drivers on the PostgreSQL wiki, including several libraries that are no longer
maintained. If you're working with the asyncio Python stdlib module you
should also take a look at the aiopg library which wraps psycopg2's
asychronouos features together.
To abstract the connection between tables and objects, many Python
developers use an object-relational mapper (ORM) with to turn relational
data from PostgreSQL into objects that can be used in their Python
application. For example, while PostgreSQL provides a relational database
and psycopg is the common database connector, there are many ORMs that
can be used with varying web frameworks, as shown in the table below.
Learn more about Python ORMs on that dedicated topic page.
PostgreSQL data safety
If you're on Linux it's easy to get PostgreSQL installed using a package
manager. However, once the database is installed and running your
118
Full Stack Python: 2020 Supporter's Edition
responsibility is just beginning. Before you go live with a production
application, make sure to:
1. Lock down access with a whitelist in the pg_hba.conf file
2. Enable replication to another database that's preferrably on different
infrastructure in a separate location
3. Perform regular backups and test the restoration process
4. Ensure your application prevents SQL injection attacks
When possible have someone qualified do a PostgreSQL security audit to
identify the biggest risks to your database. Small applications and
bootstrapped companies often cannot afford a full audit in the beginning but
as an application grows over time it becomes a bigger target.
The data stored in your database is the lifeblood of your application. If you
have ever accidentally dropped a production database or been the victim of
malicious activity such as SQL injection attacks, you'll know it's far easier to
recover when a bit of work has been performed beforehand on backups,
replication and security measures.
Python-specific PostgreSQL resources
Many quickstarts and tutorials exist specifically for Django, Flask and other
web application frameworks. The ones below are some of the best
walkthroughs I've read.
• Setting up PostgreSQL with Python 3 and psycopg on Ubuntu 16.04
provides instructions for getting a fresh Ubuntu install working with
PostgreSQL and Python 3.
• This post on using PostgreSQL with Django or Flask is a great
quickstart guide for either framework.
• This article explains how and why PostgreSQL can handle full text
searching for many use cases. If you're going down this route, read this
blog post that explains how one developer implemented PostgreSQL
full text search with SQLAlchemy.
• django-postgres-copy is a tool for bulk loading data into a PostgreSQL
database based on Django models. Say hello to our new open-source
119
Full Stack Python: 2020 Supporter's Edition
software for loading bulk data into PostgreSQL is an introduction to
using the tool in your own projects.
• How to speed up tests in Django and PostgreSQL explains some hacks
for making your schema migration-backed run quicker.
• Thinking psycopg3 is written by a developer who has worked on this
critical Python library for interacting with PostgreSQL since 2005. The
author writes up thoughts on what should change if backwardsincompatible changes are ever introduced in a new hypothetical future
version.
• Records is a wrapper around the psycopg2 driver that allows easy
access to direct SQL access. It's worth a look if you prefer writing SQL
over using an ORM like SQLAlchemy.
o Postgres Joins and Django Querysets is a well done post with a specific
example of how a standard Django ORM query can lead to degraded
performance due when obtaining data from many related tables. The
prefetch_related command and database performance monitoring tools
can help analyze and alleviate some of the issues in these unoptimized
queries.
• Loading Google Analytics data to PostgreSQL using Python is a quality
tutorial that combines API calls with psycopg and PostgreSQL to take
data from Google Analytics and save it in a PostgreSQL database.
• 1M rows/s from Postgres to Python shows some benchmarks for the
performance of the asyncpg Python database client and why you may
want to consider using it for data transfers.
General PostgreSQL resources
PostgreSQL tutorials not specific to Python are also really helpful for properly
handling your data.
• Why PostgreSQL? (5 years later) covers the improvements that have
been made to PostgreSQL over the past five years. It's amazing to see
how far this project has come and how it continues to evolve.
120
Full Stack Python: 2020 Supporter's Edition
• The Internals of PostgreSQL is a book that goes into how PostgreSQL
works, including core topics such as query processing, concurrency
control and the layout of heap table files.
• PostgreSQL Weekly is a weekly newsletter of PostgreSQL content from
around the web.
• My Favorite PostgreSQL Extensions - Part One and part two are
roundups of useful PostgreSQL extensions that augment the standard
PostgreSQL functionality.
• An introduction to PostgreSQL physical storage provides a solid
walkthrough of where PostgreSQL files are located on disk, how the
files store your data and what mappings are important for the
underlying database structure. This post is an easy read and well worth
your time.
• Braintree wrote about their experiences scaling PostgreSQL. The post
is an inside look at the evolution of Braintree's usage of the database.
• There is no such thing as total security but this IBM article covers
hardening a PostgreSQL database.
• Handling growth with Postgres provides 5 specific tips from
Instagram's engineering team on how to scale the design of your
PostgreSQL database.
• Inserting And Using A New Record In Postgres shows some SQL
equivalents to what many developers just do in their ORM of choice.
• Following a Select Statement Through Postgres Internals provides a
fascinating look into the internal workings of PostgreSQL during a
query.
• Locating the recovery point just before a dropped table and logging
transactions that dropped tables are two posts that show you how to
recover from an accidentally dropped table. In the first post the author
shows how recovery is possible with recovery points while the second
post shows how to put logging in place to assist in future recoveries.
• awesome-postgres is a list of code libraries, tutorials and newsletters
focused specifically on PostgreSQL.
121
Full Stack Python: 2020 Supporter's Edition
• While you can use a graphical interface for working with PostgreSQL,
it's best to spend some time getting comfortable with the commandline interface.
• Backing up databases is important because data loss can and does
happen. This article explains how to back up a PostgreSQL database
hosted on an Amazon Web Services EC2 instance if managing your own
database on a cloud server is your preferred setup.
• Is bi-directional replication (BDR) in PostgreSQL transactional?
explores a relatively obscure topic with the final result that BDR is
similar to data stores with eventual consistency rather than consistency
as a requirement.
• PostgreSQL-metrics is a tool built by Spotify's engineers that extracts
and outputs metrics from an existing PostgreSQL database. There's
also a way to extend the tools to pull custom metrics as well.
• Creating a Document-Store Hybrid in Postgres 9.5 explains how to
store and query JSON data, similar to how NoSQL data stores operate.
• This slideshow on high availability for web applications has a good
overview of various database setups common in production web
applications.
• The JSONB data type was introduced in PostgreSQL 9.4 to make it
easier to store semi-structured data that previously NoSQL databases
such as MongoDB covered. However, there are times when using
JSONB isn't a good idea and this blog post covers when to avoid the
column type.
PostgreSQL monitoring and performance
Monitoring one or more PostgreSQL instances and trying to performance
tune them is a rare skillset. Here are some resources to get you started if you
have to handle these issues in your applications.
• This guide to PostgreSQL monitoring is handy for knowing what to
measure and how to do it.
• Craig Kerstiens wrote a detailed post about understanding PostgreSQL
performance.
122
Full Stack Python: 2020 Supporter's Edition
• The Practical Guide to PostgreSQL Optimizations covers using cache
sizes, restore configurations and shared buffers to improve database
performance.
• This article on performance tuning PostgreSQL shows how to find slow
queries, tune indexes and modify your queries to run faster.
• What PostgreSQL tells you about its performance explains how to
gather general performance metrics and provides the exact queries you
should run to get them. The article also covers performance monitoring
and how to analyze trigger functions.
• PostgreSQL monitoring queries is a simple GitHub repository of SQL
queries that can be run against a PostgreSQL instance to determine
usage, caching and bloat.
• PgSQL Indexes and "LIKE" examines why LIKE queries do not take
advantage of PostgreSQL indexes when the locale is set to something
other than the default "C", which is for the North American UNIX
default. The gist is that you need to build a special index to support
LIKE whenever you use a locale other than "C".
• The PostgreSQL page on PopSQL has a ton of useful syntax snippets
categorized by type of action you want to perform using SQL.
MySQL
MySQL is an open source relational database implementation for storing and
retrieving data.
123
Full Stack Python: 2020 Supporter's Edition
MySQL or PostgreSQL?
MySQL is a viable open source database implementation for Python web
applications. MySQL has a slightly easier initial learning curve than
PostgreSQL. However, PostgreSQL's design is often preferred by Python web
developers, especially when data migrations are run as an application evolves.
Python Drivers for MySQL
Accessing MySQL from a Python application requires a database driver (also
called a "connector"). While it is possible to write a driver as part of your
application, in practice most developers use an existing open source driver.
There was a major issue with MySQL drivers since the introduction of Python
3. One of the most popular libraries called MySQLdb did not work in its
existing form with Python 3 and there were no plans to update it. Therefore a
fork of MySQLdb named mysqlclient added Python 3 compatibility.
The mysqlclient fork was good in that existing MySQLdb users could drop
mysqlclient into existing projects that were upgrading to Python 3. However,
the fork often causes confusion when searching for which Python driver to use
with MySQL. Many developer simply decide to use PostgreSQL because there
is better support for Python drivers in the PostgreSQL community.
124
Full Stack Python: 2020 Supporter's Edition
With that driver support context in mind, it's absolutely possible to build a
Python 3 web application with MySQL as a backend. Here is a list of drivers
along with whether it supports Python 2, 3 or both.
• mysqlclient is a fork of MySQLdb that supports Python 2 and 3.
• MySQL Connector is Oracle's "official" (Oracle currently owns MySQL)
Python connector. The driver supports Python 2 and 3, just make sure
to check the version guide for what releases work with which Python
versions.
• MySQLdb supports Python 2 and was frequently used by Python web
applications before the mass migration to Python 3 began.
• PyMySQL is a pure Python (no C low-level code) implementation that
attempts to be a drop-in replacement for MySQLdb. However, some
MySQL APIs are not supported by the driver so whether or not your
application can use this connector will depend on what you're building.
What organizations use MySQL?
The database is deployed in production at some of the highest trafficked sites
such as Uber, Twitter, Facebook and many others major organizations.
However, since MySQL AB, the company that developed MySQL, was
purchased by Sun Microsystems (which was in turn purchased by Oracle),
there have been major defections away from the database by Wikipedia and
Google. MySQL remains a viable database option but I always recommend
new Python developers learn PostgreSQL if they do not already know MySQL.
Python-specific MySQL resources
The following resources show you how to work with MySQL in your Python
code either directly through SQL queries or less directly with an objectrelational mapper (ORM) like SQLAlchemy or the Django ORM.
• Python MySQL tutorial uses the MySQL Connector Python library to
demonstrate how to run queries and stored procedures in your Python
applications.
125
Full Stack Python: 2020 Supporter's Edition
• Python 3.4.0 with MySQL database and Python 3 and MySQL provide
context for the commonly asked question about which database
MySQL driver to use with Python 3.
• Terrible Choices: MySQL is a blog post about specific deficiencies in
MySQL's implementation that hinder its usage with Django's ORM.
• MySQL Python tutorial uses the MySQLdb driver to connect to a
MySQL server instance and shows some examples for inserting and
querying data.
General MySQL resources
There are many programming language agnostic tutorials for MySQL. A
handful of the best of these tutorials are listed below.
• How to Install and Use MySQL on Ubuntu 16.04 is a quick tutorial for
getting up and running on Ubuntu Linux.
• 28 Beginner's Tutorials for Learning about MySQL Databases is a
curated collection on various introductory MySQL topics.
• A Basic MySQL Tutorial doesn't have the most original title but it's a
good walkthrough of your first few steps in MySQL for creating users
and working with tables.
• mycli is a command line interface for MySQL that includes command
completion and other super handy features.
• Bye Bye MySQL & MongoDB, Guten Tag PostgreSQL goes into details
for why the company Userlike migrated from their MySQL database
setup to PostgreSQL.
• MySQL sharding at Quora provides details behind Quora's at-scale
infrastructure and how their MySQL sharding evolved over time.
• Growing up with MySQL is a story about how one company went
through dramatic growth and had to keep up with it by quickly scaling
their MySQL database.
• Monitoring MySQL metrics is the first of a three part series, with the
other parts on collecting metrics and monitoring & collecting
specifically with the DataDog tool. The series explains what metrics you
126
Full Stack Python: 2020 Supporter's Edition
should be collecting and monitoring in your production database along
with the purpose for why those metrics are important.
• gh-ost (source code) is a schema migration tool built by GitHub and
open sourced to the development community. The advantages of gh-ost
are sustainable workloads on the master node to allow it to keep
serving inbound query requests and the ability to pause the migration.
The post on how to use gh-ost pairs nicely with GitHub's detailed writeup on how they perform backups, failover and schema migrations in
MySQL infrastructure testing automation at GitHub.
• The unofficial MySQL optimizers guide is intended for experienced
developers who need to get better performance out of MySQL for their
specific use cases.
• The Ultimate Postgres vs MySQL Blog Post provides comparisons of
data types, default values, arrays, joins and many other differences
between MySQL and PostgreSQL.
SQLite
SQLite is an open source relational database included with the Python
standard library as of Python 2.5. The pysqlite database driver is also included
with the standard library so that no further external dependencies are
required to access a SQLite database from within Python applications.
127
Full Stack Python: 2020 Supporter's Edition
Useful SQLite tools and code
SQLite is used in such a wide variety of industries that there are open source
tools and example code for all kinds of edge case uses. Here are several tools
and bits of code I have found useful while coding my applications:
• sqlitebiter (source code) is a command-line tool for converting various
data formats such as comma-separated values (CSV), HTML,
Markdown and JSON (among others) into a SQLite database file.
• Scout (source code) is a Flask-powered search server for SQLite
backends. The introductory post is really handy for getting started with
Scout.
• Datasette makes it easy to expose JSON APIs from your SQLite
database without coding up a custom web application. Make sure to
check out the Datasette getting started guide as well.
• SQLite Browser is an open source graphical user interface for working
with SQLite.
• The Membership SQLite SQL scripts provide example code for storing
user accounts, roles and authentication tokens in web applications.
• ExtendsClass is an online SQLite browser.
SQLite tutorials
It's a good idea to brush up on the basics for using SQLite before you use the
database in your project through SQL scripts or via an object-relational
mapper. These tutorials will help you get started.
• A simple step-by-step SQLite tutorial walks through creating databases
as well as inserting, updating, querying and deleting data.
• A Minimalist Guide to SQLite shows how to install SQLite, load data
and work with the data stored in a new SQLite database.
• Python SQLite3 Basics covers how to connect to a SQLite database in
Python, executing statements, committing and retrieving saved values.
• sqlite3 - embedded relational database is an extensive tutorial showing
many of the common create, read, update and delete operations a
developer would want to do with SQLite.
128
Full Stack Python: 2020 Supporter's Edition
• The official sqlite3 module in the Python stdlib docs contains a bunch
of scenarios with code for how to use the database from a Python
application.
• Finding bugs in SQLite, the easy way explains how a bug was found and quickly fixed - in the SQLite codebase. It's a great short read which
shows that the code is well-tested and maintained.
• Data Analysis of 8.2 Million Rows with Python and SQLite explains
how you can load a large dataset in to SQLite and visualize it using the
Plotly service.
• SQLite: The art of keep it simple uses C code examples from SQLite's
codebase to show how its design has been kept consistent and tight
throughout 15+ years of active development. There's also a great design
document on the SQLite site that covers many of these principles.
• My list of SQLite resources is a nice roundup of useful tools to use with
SQLite and tutorials for learning more about the database.
• Python SQLite3 tutorial provides another beginner's tutorial using the
built-in sqlite3 Python standard library module.
• A SQLite tutorial with Python covers both SQL and Python code to
interact with SQLite.
Specific SQLite scenarios
These are solid resources if you are looking to solve a particular problem you
are having with SQLite rather than going through a general tutorial.
• Let's Build a Simple Database is an awesome read where the author recreates a SQLite-type database for learning purposes.
• We are pretty happy with SQLite & not urgently interested in a fancier
DBMS gives the rationale behind one development teams' decision to
stick to SQLite instead of porting to another relational database such as
MySQL or PostgreSQL.
• This overview of SQLite as part of the Databaseology Lectures is
amazing because they are given by the creator and he shines a ton of
light on how SQLite is built and why.
129
Full Stack Python: 2020 Supporter's Edition
• How SQLite is tested digs into the nitty-gritty behind the quality
assurance practices for testing potential SQLite releases.
• Using the SQLite JSON1 and FTS5 Extensions with Python shows how
to compile SQLite 3.9.0+ with json1 and fts5 (full-text search) support
to use these new features.
• SQLite with a fine-toothed comb digs into the internals of SQLite and
shows some bugs found (and since fixed) while the author was
researching the SQLite source code.
• Going Fast with SQLite and Python shares essential knowledge for
working effectively with SQLite in Python, particularly when it comes
to transactions, concurrency and commits.
• Extending SQLite with Python uses the Peewee object-relational
mapper (ORM) to implement virtual tables and aggregates on top of
SQLite.
• Use SQLite with Django On AWS Lambda with Zappa provides an
example dev_settings.py file for locally testing a Django application
intended for AWS Lambda.
• SQLite Database Authorization and Access Control with Python covers
how to control access to the SQLite database connection and file even
though SQLite normally allows unauthorized access by design.
• Can I read and write to a SQLite database concurrently from multiple
connections? answers one of the concerns that was an issue in earlier
versions of SQLite that could have issues if more than one connection
was writing to the database at one time.
• Appropriate uses for SQLite is an official documentation page that
explains what types of applications are designed to work well with
SQLite as the backend.
• How to corrupt a SQLite file explains how the database file could
potentially get corrupted if you really work at screwing it up.
130
Full Stack Python: 2020 Supporter's Edition
Object-relational Mappers (ORMs)
An object-relational mapper (ORM) is a code library that automates the
transfer of data stored in relational databases tables into objects that are more
commonly used in application code.
Why are ORMs useful?
ORMs provide a high-level abstraction upon a relational database that allows
a developer to write Python code instead of SQL to create, read, update and
delete data and schemas in their database. Developers can use the
programming language they are comfortable with to work with a database
instead of writing SQL statements or stored procedures.
For example, without an ORM a developer would write the following SQL
statement to retrieve every row in the USERS table where the zip_code
column is 94107:
SELECT * FROM USERS WHERE zip_code=94107;
The equivalent Django ORM query would instead look like the following
Python code:
# obtain everyone in the 94107 zip code and assign to users variable
users = Users.objects.filter(zip_code=94107)
The ability to write Python code instead of SQL can speed up web application
development, especially at the beginning of a project. The potential
development speed boost comes from not having to switch from Python code
into writing declarative paradigm SQL statements. While some software
developers may not mind switching back and forth between languages, it's
131
Full Stack Python: 2020 Supporter's Edition
typically easier to knock out a prototype or start a web application using a
single programming language.
ORMs also make it theoretically possible to switch an application between
various relational databases. For example, a developer could use SQLite for
local development and MySQL in production. A production application could
be switched from MySQL to PostgreSQL with minimal code modifications.
In practice however, it's best to use the same database for local development
as is used in production. Otherwise unexpected errors could hit in production
that were not seen in a local development environment. Also, it's rare that a
project would switch from one database in production to another one unless
there was a pressing reason.
Do I have to use an ORM for my web application?
Python ORM libraries are not required for accessing relational databases. In
fact, the low-level access is typically provided by another library called a
database connector, such as psycopg (for PostgreSQL) or MySQL-python (for
MySQL). Take a look at the table below which shows how ORMs can work
with different web frameworks and connectors and relational databases.
The above table shows for example that SQLAlchemy can work with varying
web frameworks and database connectors. Developers can also use ORMs
without a web framework, such as when creating a data analysis tool or a
batch script without a user interface.
132
Full Stack Python: 2020 Supporter's Edition
What are the downsides of using an ORM?
There are numerous downsides of ORMs, including
1. Impedance mismatch
2. Potential for reduced performance
3. Shifting complexity from the database into the application code
Impedance mismatch
The phrase "impedance mismatch" is commonly used in conjunction with
ORMs. Impedance mismatch is a catch-all term for the difficulties that occur
when moving data between relational tables and application objects. The gist
is that the way a developer uses objects is different from how data is stored
and joined in relational tables.
This article on ORM impedance mismatch does a solid job of explaing what
the concept is at a high level and provides diagrams to visualize why the
problem occurs.
Potential for reduced performance
One of the concerns that's associated with any higher-level abstraction or
framework is potential for reduced performance. With ORMs, the
performance hit comes from the translation of application code into a
corresponding SQL statement which may not be tuned properly.
ORMs are also often easy to try but difficult to master. For example, a
beginner using Django might not know about the select_related()
function and how it can improve some queries' foreign key relationship
performance. There are dozens of performance tips and tricks for every ORM.
It's possible that investing time in learning those quirks may be better spent
just learning SQL and how to write stored procedures.
There's a lot of hand-waving "may or may not" and "potential for" in this
section. In large projects ORMs are good enough for roughly 80-90% of use
cases but in 10-20% of a project's database interactions there can be major
performance improvements by having a knowledgeable database
administrator write tuned SQL statements to replace the ORM's generated
SQL code.
133
Full Stack Python: 2020 Supporter's Edition
Shifting complexity from the database into the app code
The code for working with an application's data has to live somewhere. Before
ORMs were common, database stored procedures were used to encapsulate
the database logic. With an ORM, the data manipulation code instead lives
within the application's Python codebase. The addition of data handling logic
in the codebase generally isn't an issue with a sound application design, but it
does increase the total amount of Python code instead of splitting code
between the application and the database stored procedures.
Python ORM Implementations
There are numerous ORM implementations written in Python, including
1.
2.
3.
4.
5.
6.
SQLAlchemy
Peewee
The Django ORM
PonyORM
SQLObject
Tortoise ORM (source code)
There are other ORMs, such as Canonical's Storm, but most of them do not
appear to currently be under active development. Learn more about the major
active ORMs below.
Django's ORM
The Django web framework comes with its own built-in object-relational
mapping module, generally referred to as "the Django ORM" or "Django's
ORM".
Django's ORM works well for simple and medium-complexity database
operations. However, there are often complaints that the ORM makes
complex queries much more complicated than writing straight SQL or using
SQLAlchemy.
It is technically possible to drop down to SQL but it ties the queries to a
specific database implementation. The ORM is coupled closely with Django so
replacing the default ORM with SQLAlchemy is currently a hack workaround.
Note though it is possible that swappable ORM backends will be possible in
134
Full Stack Python: 2020 Supporter's Edition
the future as it is now possible to change the template engine for rendering
output in Django.
Since the majority of Django projects are tied to the default ORM, it is best to
read up on advanced use cases and tools for doing your best work within the
existing framework.
SQLAlchemy ORM
SQLAlchemy is a well-regarded Python ORM because it gets the abstraction
level "just right" and seems to make complex database queries easier to write
than the Django ORM in most cases. There is an entire page on SQLAlchemy
that you should read if you want to learn more about using the library.
Peewee ORM
Peewee is a Python ORM implementation that is written to be "simpler,
smaller and more hackable" than SQLAlchemy. Read the full Peewee page for
more information on the Python ORM implementation.
Pony
Pony ORM is another Python ORM available as open source, under the
Apache 2.0 license.
SQLObject ORM
SQLObject is an ORM that has been under active open source development
for over 14 years, since before 2003.
Schema migrations
Schema migrations, for example when you need to add a new column to an
existing table in your database, are not technically part of ORMs. However,
since ORMs typically lead to a hands-off approach to the database (at the
developers peril in many cases), libraries to perform schema migrations often
go hand-in-hand with Python ORM usage on web application projects.
Database schema migrations are a complex topic and deserve their own page.
For now, we'll lump schema migration resources under ORM links below.
135
Full Stack Python: 2020 Supporter's Edition
General ORM resources
• This detailed overview of ORMs is a generic description of how ORMs
work and how to use them.
• This example GitHub project implements the same Flask application
with several different ORMs: SQLAlchemy, Peewee, MongoEngine,
stdnet and PonyORM.
• Martin Fowler addresses the ORM hate in an essay about how ORMs
are often misused but that they do provide benefits to developers.
• The Rise and Fall of Object Relational Mapping is a talk on the history
of ORMs that doesn't shy away from some controversy. Overall I found
the critique of conceptual ideas worth the time it took to read the
presentation slides and companion text.
• If you're confused about the difference between a connector, such as
MySQL-python and an ORM like SQLAlchemy, read this
StackOverflow answer on the topic.
• What ORMs have taught me: just learn SQL is another angle in the
ORM versus embedded SQL / stored procedures debate. The author's
conclusion is that while working with ORMs such as SQLAlchemy and
Hibernate (a Java-based ORM) can save time up front there are issues
as a project evolves such as partial objects and schema redundancies. I
think the author makes some valid points that some ORMs can be a
shaky foundation for extremely complicated database-backed
applications. However, I disagree with the overriding conclusion to
eschew ORMs in favor of stored procedures. Stored procedures have
their own issues and there are no perfect solutions, but I personally
prefer using an ORM at the start of almost every project even if it later
needs to be replaced with direct SQL queries.
• The Vietnam of Computer Science provides the perspective from Ted
Neward, the originator of the phrase "Object/relational mapping is the
Vietnam of Computer Science" that he first spoke about in 2004. The
gist of the argument against ORMs is captured in Ted's quote that an
ORM "represents a quagmire which starts well, gets more complicated
as time passes, and before long entraps its users in a commitment that
has no clear demarcation point, no clear win conditions, and no clear
136
Full Stack Python: 2020 Supporter's Edition
exit strategy." There are follow up posts on Coding Horror and another
one from Ted entitled thoughts on Vietnam commentary.
• Turning the Tables: How to Get Along with your Object-Relational
Mapper coins the funny but insightful phrase "database denial" to
describe how some ORMs provide a usage model that can cause more
issues than they solve over straight SQL queries. The post then goes
into much more detail about the problems that can arise and how to
mitigate or avoid them.
SQLAlchemy and Peewee resources
A comprehensive list of SQLAlchemy and Peewee ORM resources can be
found on their respective pages.
Django ORM links
A curated list of resources can be found on the dedicated Django ORM
resources page.
Pony ORM resources
All Pony ORM resources are listed on the dedicated Pony ORM page.
SQLObject resources
SQLObject has been around for a long time as an open source project but
unfortunately there are not that many tutorials for it. The following talks and
posts will get you started. If you take an interest in the project and write
additional resources, file an issue ticket so we can get them added to this list.
• This post on Object-Relational Mapping with SQLObject explains the
concept behind ORMs and shows the Python code for how they can be
used.
• Ian Bicking presented on SQLObject back in 2004 with a talk on
SQLObject and Database Programming in Python.
• Connecting databases to Python with SQLObject is an older post but
still relevant with getting started basics.
137
Full Stack Python: 2020 Supporter's Edition
SQLAlchemy
SQLAlchemy (source code) is a well-regarded database toolkit and objectrelational mapper (ORM) implementation written in Python. SQLAlchemy
provides a generalized interface for creating and executing database-agnostic
code without needing to write SQL statements.
Why is SQLAlchemy a good ORM choice?
SQLAlchemy isn't just an ORM- it also provides SQLAlchemy Core for
performing database work that is abstracted from the implementation
differences between PostgreSQL, SQLite, etc. In some ways, the ORM is a
bonus to Core that automates commonly-required create, read, update and
delete operations.
SQLAlchemy can be used with or without the ORM features. Any given project
can choose to just use SQLAlchemy Core or both Core and the ORM. The
following diagram shows a few example configurations with various
application software stacks and backend databases. Any of these
configurations can be a valid option depending on what type of application
you are coding.
138
Full Stack Python: 2020 Supporter's Edition
A benefit many developers enjoy with SQLAlchemy is that it allows them to
write Python code in their project to map from the database schema to the
applications' Python objects. No SQL is required to create, maintain and
query the database. The mapping allows SQLAlchemy to handle the
underlying database so developers can work with their Python objects instead
of writing bridge code to get data in and out of relational tables.
How does SQLAlchemy code compare to raw SQL?
Below is an example of a SQLAlchemy model definition from the open source
compare-python-web-frameworks project that uses SQLAlchemy with Flask
and Flask-SQLAlchemy.
class Contact(db.Model):
__tablename__ = 'contacts'
id = db.Column(db.Integer, primary_key=True)
first_name = db.Column(db.String(100))
last_name = db.Column(db.String(100))
phone_number = db.Column(db.String(32))
def __repr__(self):
return '<Contact {0} {1}: {2}>'.format(self.first_name,
self.last_name,
self.phone_number)
SQLAlchemy handles the table creation that otherwise we would have had to
write a create table statement like this one to do the work:
CREATE TABLE CONTACTS(
ID INT PRIMARY KEY
FIRST_NAME
CHAR(100)
NOT NULL,
NOT NULL,
139
Full Stack Python: 2020 Supporter's Edition
LAST_NAME
PHONE_NUMBER
CHAR(100)
CHAR(32)
NOT NULL,
NOT NULL,
);
By using SQLAlchemy in our Python code, all records can be obtained with a
line like contacts = Contact.query.all() instead of a plain SQL such as
SELECT * FROM contacts . That may not look like much of a difference in
syntax but writing the queries in Python is often faster and easier for many
Python developers once multiple tables and specific filtering on fields for
queries have to be written. In addition, SQLAlchemy abstracts away
idiosyncratic differences between database implementations in SQLite,
MySQL and PostgreSQL.
SQLAlchemy Extensions, Plug-ins and Related Libraries
Take a look at the SQLAlchemy extensions, plug-ins and related libraries page
for a curated list of useful code libraries to use with SQLAlchemy.
Using SQLAlchemy with Web Frameworks
There is no reason why you cannot use the SQLAlchemy library in any
application that requires a database backend. However, if you are building a
web app with Flask, Bottle or another web framework then take a look at the
following extensions. They provide some glue code along with helper
functions that can reduce the boilerplate code needed to connect your
application's code with the SQLAlchemy library.
• SQLAlchemy is typically used with Flask as the database ORM via the
Flask-SQLAlchemy extension.
• The bottle-sqlalchemy extension for Bottle provides a bridge between
the standard SQLAlchemy library and Bottle. However, from my
experience using the library it does not have quite as many helper
functions as Flask-SQLAlchemy.
• Pyramid uses the alchemy scaffold to make it easy to add SQLAlchemy
to a Pyramid web app.
• While Django does not yet support easy swapping of the default Django
backend ORM with SQLAlchemy (like it does for template engines),
there are hacks for using SQLAlchemy within Django projects.
140
Full Stack Python: 2020 Supporter's Edition
• Morepath has easy-to-use support for SQLAlchemy via its
more.transaction module. There is a morepath-sqlalchemy demo that
serves as a working example.
• Merging Django ORM with SQLAlchemy for Easier Data Analysis has
details on why, how and when you may want to use SQLAlchemy to
augment the Django ORM.
• Building a Simple Birthday App with Flask-SQLAlchemy combines
SQLAlchemy with Flask to create a birthday reminder application.
SQLAlchemy resources
The best way to get comfortable with SQLAlchemy is to dig in and write a
database-driven application. The following resources can be helpful if you are
having trouble getting started or are starting to run into some edge cases.
• There is an entire chapter in the Architecture of Open Source
Applications book on SQLAlchemy. The content is detailed and well
worth reading to understand what is executing under the covers.
• The SQLAlchemy cheatsheet has many examples for querying,
generating database metadata and many other common (and not so
common) operations when working with Core and the ORM.
• 10 reasons to love SQLAlchemy is a bit of a non-critical lovefest for the
code library. However, the post makes some good points about the
quality of SQLAlchemy's documentation and what a pleasure it can be
to use it in a Python project.
• SQLAlchemy and Django explains how one development team uses the
Django ORM for most of their standard queries but relies on
SQLAlchemy for really advanced queries.
• This SQLAlchemy tutorial provides a slew of code examples that cover
the basics for working with SQLAlchemy.
• Implementing User Comments with SQLAlchemy gives a wonderful
walkthrough of how to build your own online commenting system in
Python using SQLAlchemy.
141
Full Stack Python: 2020 Supporter's Edition
• Master SQLAlchemy Relationships in a Performance Friendly Way
dives into code that shows how to improve performance when setting
and accessing relationship-based data in your models.
• SQLAlchemy and data access in Python is a podcast interview with the
creator of SQLAlchemy that covers the project's history and how it has
evolved over the past decade.
• Most Flask developers use SQLAlchemy as an ORM to relational
databases. If you're unfamiliar with SQLAlchemy questions will often
come up such as what's the difference between flush and commit? that
are important to understand as you build out your app.
• SQLAlchemy in batches shows the code that a popular iOS application
runs in background batch scripts which uses SQLAlchemy to generate
playlists. They provide some context and advice for using SQLAlchemy
in batch scripts.
• Getting PostgreSQL transactions under control with SQLAlchemy
provides a quick introduction to the tool Chryso that they are working
on to provide better transaction management in SQLAlchemy
connections.
SQLAlchemy compared to other ORMs
SQLAlchemy is one of many Python object-relational mapper (ORM)
implementations. Several open source projects and articles are listed here to
make it a bit easier to understand the differences between these
implementations.
• Introduction to SQLAlchemy ORM for Django Developers is written by
a developer who typically used the Django ORM at work and then had a
chance to try SQLAlchemy for one project. He covers differences in
how each one handles transactions, models and queries.
• SQLAlchemy vs Other ORMs provides a detailed comparison of
SQLAlchemy against alternatives.
• If you're interested in the differences between SQLAlchemy and the
Django ORM I recommend reading SQLAlchemy and You by Armin
Ronacher.
142
Full Stack Python: 2020 Supporter's Edition
• This GitHub project named PythonORMSleepy implements the same
Flask application with several different ORMs: SQLAlchemy, Peewee,
MongoEngine, stdnet and PonyORM. Looking through the code is
helpful for understanding the varying approaches each library takes to
accomplish a similar objective.
• Quora has several answers to the question of which is better and why:
Django ORM or SQLALchemy based on various developers'
experiences.
Open source code for learning SQLAlchemy
Many open source projects rely on SQLAlchemy. A great way to learn how to
properly work with this tool is to read the code that shows how those projects
use SQLAlchemy. This section alphabetically lists these code examples by
class and function in the SQLAlchemy code base.
Peewee
Peewee (source code) is a object-relational mapper (ORM) implementation
for bridging data stored in relational database tables with Python objects.
What makes Peewee a useful ORM?
Peewee can be an easier library to wrap your head around than SQLAlchemy
and other ORMs. It is designed to be easier to hack on and understand,
similar to how Bottle is a smaller, one-file web framework compared to the
comprehensive Django framework. If you are just getting started with web
development, it may be worth using Peewee for your database mapping and
operations, especially if you use a microframework such as Flask or Bottle.
143
Full Stack Python: 2020 Supporter's Edition
Peewee can be used with pretty much any web framework (although using it
with Django would currently be complicated due to its tight built-in ORM
coupling) or without a web framework. In the latter case Peewee is good for
pulling data out of a relational database in a script or Jupyter notebook.
Any of the common relational database backends such as PostgreSQL, MySQL
or SQLite are supported, although a database driver is still required. The chart
below shows a few example configurations that could use Peewee as an ORM.
How does Peewee compare to other Python ORMs?
The analogy used by the core Peewee author is that Peewee is to SQLAlchemy
as SQLite is to PostgreSQL. An ORM does not have to work for every
exhaustive use case in order to be useful.
Peewee resources
Peewee is a much newer library than several other Python ORMs. For
example, Peewee's first public commit was in 2010, compared to 2005 for
SQLAlchemy. The project is still over five years old though and matured
substantially in development during that time. However, there are typically
less resources and examples available to demonstrate how to use Peewee in
your projects than some other ORMs that have been around for a longer
period of time.
Many of the best resources come from the project's author, Charles Leifer, on
his blog and on the official site. There are also hundreds of questions
144
Full Stack Python: 2020 Supporter's Edition
answered on the Stack Overflow peewee tag, so as usual that can be a rich
source of examples for your Peewee-powered Python applications.
• An encrypted command-line diary with Python is an awesome
walkthrough explaining how to use SQLite, SQLCipher and Peewee to
create an encrypted file with your contents, diary or otherwise.
• An Intro to Peewee – Another Python ORM is a short tutorial that
walks through creating a database model mapping, adding data,
deleting records and querying fields.
• Introduction to peewee uses an example public dataset, loads it into a
SQLite database and shows how to query it using Peewee.
• Shortcomings in the Django ORM and a look at Peewee from the
author of the Peewee ORM explains how some of the design decisions
made in Peewee were in reaction to parts of the Django ORM that
didn't work so well in practice.
• The official Peewee quickstart documentation along with the example
Twitter clone app will walk you through the ins and outs of your first
couple Peewee-powered projects.
• Flask and Peewee 101 has some basic code for querying with Peewee
and populating a drop-down in a Jinja2 template. Note that the Flaskpeewee extension is no longer maintained, although you do not need to
use it to work with both Flask and Peewee in an application.
• How to make a Flask blog in one hour or less is a well written tutorial
that uses the Peewee ORM instead of SQLAlchemy for the blog back
end.
• These posts on querying the top item by group with Peewee ORM and
querying the top N objects per group with Peewee ORM provide
working examples on how to properly query your data via Peewee.
• There was a good discussion in a Python subreddit thread about the
differences between Peewee and SQLAlchemy. Charles Leifer even
chimed in to add his own fair assessment of the differences in the
ORMs.
145
Full Stack Python: 2020 Supporter's Edition
• peewee-async (source code) is an alpha library for using Python 3's
asyncio standard library with Peewee. This library is worth watching if
you use an async web framework and want to have Peewee serve as
your application's ORM.
• Accessing remote MySQL database with peewee debugs a question
where the original author had issues accessing a remote MySQL
database because they did not properly include the Model class from
peewee.py when instantiating a mapper class.
Django ORM
The Django web framework includes a default object-relational mapping layer
(ORM) that can be used to interact with application data from various
relational databases such as SQLite, PostgreSQL and MySQL.
Django ORM resources
The Django ORM has evolved over the past dozen years since it was created
make sure to not only read up on the latest tutorials but also learn about
newer optimizations, such as prefetch_related and select_related, that have
been added throughout the project's history.
• Django models, encapsulation and data integrity is a detailed article by
Tom Christie on encapsulating Django models for data integrity.
• Django Debug Toolbar is a powerful Django ORM database query
inspection tool. Highly recommended during development to ensure
you're writing reasonable query code. Django Silk is another inspection
tool and has capabilities to do more than just SQL inspection.
146
Full Stack Python: 2020 Supporter's Edition
• Django QuerySet Examples (with SQL code included) teaches how
QuerySets work and shows the corresponding SQL code behind the
Python code you write to use the ORM.
• Making a specific Django app faster is a Django performance blog post
with some tips on measuring performance and optimizing based on the
measured results.
• Why I Hate the Django ORM is Alex Gaynor's overview of the bad
designs decisions, some of which he made, while building the Django
ORM.
• Going Beyond Django ORM with Postgres is specific to using
PostgreSQL with Django.
• How to view Django ORM SQL queries along with django-sql-explorer
allow you to better understand the SQL code that is generated from the
Django ORM.
• Migrating a Django app from MySQL to PostgreSQL is a quick look at
how to move from MySQL to PostgreSQL. However, my guess is that
any Django app that's been running for awhile on one relational
database will require a lot more work to port over to another backend
even with the power of the ORM.
• Adding basic search to your Django site shows how to write generic
queries that'll allow you to provide site search via the Django ORM
without relying on another tool like ElasticSearch. This is great for
small sites before you scale them up with a more robust search engine.
• The Django ORM Cookbook provides code recipes for various ways to
use the Django ORM to insert and query data.
• How to use Django's Proxy Models is a solid post on a Django ORM
concept that doesn't frequently get a lot of love or explanation.
• Tightening Django Admin Logins shows you how to log authentication
failures, create an IP addresses white list and combine fail2ban with
the authentication failures list.
• How to Turn Django Admin Into a Lightweight Dashboard and How to
Use Grouping Sets in Django are two great posts on how to add custom
147
Full Stack Python: 2020 Supporter's Edition
features to the Django Admin as well as optimize with more advanced
SQL when the first attempt at the queries get slow due to larger
amounts of data.
• Sorting querysets with NULLs in Django shows what to do if you're
struggling with the common issue of sorting columns that contain
NULL values.
• Best Practices working with Django models in Python has a ton of great
advice on proper model naming conventions, quirks to avoid with
ForeignKey field relationships, handling IDs and many other edge
cases that come up when frequently working with Django's ORM.
• Merging Django ORM with SQLAlchemy for Easier Data Analysis
provides rationale for using the SQLAlchemy ORM instead of Django's
default ORM in some situations.
• Working with huge data sets in Django explains how to slice the data
you retrieve by query into pages and then use prefetch_related on a
subset of the data rather than your whole data set.
• Solving performance problems in the Django ORM gives a slew of great
code snippets to use with django.db.connection so you can discover
issues such as unexpected extra queries and problematic key
relationships.
• Full-text search in Django with PostgreSQL is a very detailed example
that shows how to work specifically with a PostgreSQL backend.
• Django Anti-Patterns: Signals explains why you should avoid using
Django ORM's signals feature in your applications if you want to make
them easier to maintain.
• Django ORM optimization story on selecting the least possible goes
through one developer's Django ORM code refactoring to optimize the
performance and results of a single query.
• Fixing your Django async job - database integration is a great article on
how to properly integrate the RQ task queue with a Django backend.
148
Full Stack Python: 2020 Supporter's Edition
Django migrations resources
Django migrations were added in version 1.7. Django projects prior to 1.7 used
the South project, which is now deprecated and merged into Django.
Migrations can be tricky to wrap your head around as you're getting started
with the overall framework but the following resources should get you past
the initial hurdles.
• Django Migrations - a Primer takes you through the new migrations
system integrated in the Django core as of Django 1.7, looking
specifically at a solid workflow that you can use for creating and
applying migrations.
• Django 1.7: Database Migrations Done Right explains why South was
not directly integrated into Django, how migrations are built and shows
how backwards migrations work.
• Executing custom SQL in Django migrations examines how you can
hook in straight SQL that will run during a Django migration.
• Squashing and optimizing migrations in Django shows a simple
example with code for how to use the migrations integrated into
Django 1.7.
• Supporting both Django 1.7 and South explains the difficulty of
supporting Django 1.7 and maintaining South migrations for Django
1.6 then goes into how it can be done.
• Writing unit tests for Django migrations contains a ton of awesome
code examples for testing your migrations to ensure data migrations
work well throughout the lifecycle of your Django project.
• Strategies for reducing memory usage in Django migrations shows the
large memory usage problem that often occurs with Django migrations
at scale and what you can do to mitigate the issue.
• How to Create Django Data Migrations has a straightforward blog
ORM modeling example to show how to perform data migration.
• Keeping data integrity with Django migrations shows two table
modification scenarios, one where a column needs to be added to an
existing table, and another where a Many-to-Many field needs to be
149
Full Stack Python: 2020 Supporter's Edition
converted to a standard ForeignKey column while retaining all of the
data.
• Double-checked locking with Django ORM shows how you can
implement a double-checking locking pattern in the Django ORM with
PostgreSQL, which is useful when you want to prevent multiple
processes from accessing the same data at the same time.
• Using Django Check Constraints for the Sum of Percentage Fields
shows how you can combine several PositiveIntegerField model
fields with a checking constraint and a web form that ensures all of the
fields sum up to a precise amount, such as 100%.
Pony ORM
Pony (source code) is a Python object-relational mapper (ORM) library
(database module source code). Pony can be used to interact and manipulate
data in relational databases, including PostgreSQL, SQLite and MySQL.
Pony resources
• Why you should give Pony ORM a chance explains some of the benefits
of Pony ORM that make it worth trying out.
• An intro to Pony ORM shows the basics of how to use the library, such
as creating databases and manipulating data.
• The Pony ORM author explains on a Stack Overflow answer how Pony
ORM works behind the scenes. Worth a read whether or not you're
using the ORM just to find out how some of the magic coding works.
150
Full Stack Python: 2020 Supporter's Edition
NoSQL Data Stores
Relational databases store the vast majority of web application persistent
data. However, there are several alternative classifications of storage
representations.
1.
2.
3.
4.
Key-value pair
Document-oriented
Column-family table
Graph
These persistent data storage representations are commonly used to augment,
rather than completely replace, relational databases. The underlying
persistence type used by the NoSQL database often gives it different
performance characteristics than a relational database, with better results on
some types of read/writes and worse performance on others.
Key-value Pair
Key-value pair data stores are based on hash map data structures.
Key-value pair data stores
• Redis is an open source in-memory key-value pair data store. Redis is
often called "the Swiss Army Knife of web application development." It
can be used for caching, queuing, and storing session data for faster
access than a traditional relational database, among many other use
cases. Learn more on the Redis page.
• Memcached is another widely used in-memory key-value pair storage
system.
Key-value pair resources
• What is a key-value store database? is a Stack Overflow Q&A that
straight on answers this subject.
Redis resources
• How to Use Redis with Python 3 and redis-py on Ubuntu 16.04
contains detailed steps to install and start using Redis in Python.
151
Full Stack Python: 2020 Supporter's Edition
• "How To Install and Use Redis" is a guide for getting up with the
extremely useful in-memory data store.
• This video on Scaling Redis at Twitter is a detailed look behind the
scenes with a massive Redis deployment.
• Walrus is a higher-level Python wrapper for Redis with some caching,
querying and data structure components build into the library.
• Real World Redis Tips provides some guidance from Heroku's
engineers from deploying Redis at scale. The tips include setting an
explicit idle connection timeout, using a connection pooler and
avoiding using KEYS in favor of SCAN .
• Writing Redis in Python with Asyncio shows a detailed example for
how to use the new Asyncio standard library in Python 3.4+ for
working with Redis.
• How to collect Redis metrics shows how to use the Redis CLI client to
grab key metrics on latency.
• You should revise your Redis max connections setting is a retrospective
from a hard web application failure due to Redis connections maxing
out on Heroku, and how to avoid this in your own applications by
modifying your redis.conf settings.
Document-oriented
A document-oriented database provides a semi-structured representation for
nested data.
Document-oriented data stores
• MongoDB is an open source document-oriented data store with a
Binary Object Notation (BSON) storage format that is JSON-style and
familiar to web developers. PyMongo is a commonly used client for
interfacing with one or more MongoDB instances through Python code.
MongoEngine is a Python ORM specifically written for MongoDB that
is built on top of PyMongo.
• Riak is an open source distributed data store focused on availability,
fault tolerance and large scale deployments.
152
Full Stack Python: 2020 Supporter's Edition
• Apache CouchDB is also an open source project where the focus is on
embracing RESTful-style HTTP access for working with stored JSON
data.
Document-oriented data store resources
• The creator and maintainers of PyMongo review four decisions they
regret from building the widely-used Python MongoDB driver.
1. start_request
2. use_greenlets
3. copy_database
4. MongoReplicaSetClient
• The Python and MongoDB Talk Python to Me podcast has a great
interview with the maintainer of the Python driver for MongoDB.
• MongoDB queries don’t always return all matching documents! is a
walkthrough of discovering how MongoDB queries actually work, and
shows some potential pitfalls of relying on technologies where you do
not fully understand how they operate.
• Introduction to MongoDB and Python shows how to use Python to
interface with MongoDB via PyMongo and MongoEngine.
Column-family table
A column-family table class of NoSQL data stores builds on the key-value pair
type. Each key-value pair is considered a row in the store while the column
family is similar to a table in the relational database model.
Column-family table data stores
• Apache Cassandra
• Apache HBase
Graph
A graph database represents and stores data in three aspects: nodes, edges
and properties.
A node is an entity, such as a person or business.
153
Full Stack Python: 2020 Supporter's Edition
An edge is the relationship between two entities. For example, an edge could
represent that a node for a person entity is an employee of a business entity.
A property represents information about nodes. For example, an entity
representing a person could have a property of "female" or "male".
Graph data stores
• Neo4j is one of the most widely used graph databases and runs on the
Java Virtual Machine stack.
• Cayley is an open source graph data store written by Google primarily
written in Go.
• Titan is a distributed graph database built for multi-node clusters.
Graph data store resources
• Introduction to Graph Databases covers trends in NoSQL data stores
and compares graph databases to other data store types.
• Graph search algorithm basics explains the methods for searching
nodes for data in a graph database.
NoSQL third-party services
• Compose provides MongoDB as a service. It's easy to set up with either
a standard LAMP stack or on Heroku.
NoSQL data store resources
• NoSQL databases: an overview explains what NoSQL means, how data
is stored differently than in relational systems and what the
Consistency, Availability and Partition-Tolerance (CAP) Theorem
means.
• NoSQL Explained is a good high-level overview of considerations and
features when choosing a type of NoSQL database compared to a
relational database.
• CAP Theorem overview presents the basic constraints all databases
must trade off in operation.
154
Full Stack Python: 2020 Supporter's Edition
• This post on What is a NoSQL database? Learn By Writing One in
Python is a detailed article that breaks the mystique behind what some
forms of NoSQL databases are doing under the covers.
• The CAP Theorem series explains concepts related to NoSQL such as
what is ACID compared to CAP, CP versus CA and high availability in
large scale deployments.
• NoSQL Weekly is a free curated email newsletter that aggregates
articles, tutorials, and videos about non-relational data stores.
• NoSQL comparison is a large list of popular, BigTable-based, special
purpose, and other datastores with attributes and the best use cases for
each one.
• Relational databases such as MySQL and PostgreSQL have added
features in more recent versions that mimic some of the capabilities of
NoSQL data stores. For example, check out this blog post on storing
JSON data in PostgreSQL.
NoSQL data stores learning checklist
1. Understand why NoSQL data stores are better for some use cases than
relational databases. In general these benefits are only seen at large
scale so they may not be applicable to your web application.
2. Integrate Redis into your project for a speed boost over slower
persistent storage. Storing session data in memory is generally much
faster than saving that data in a traditional relational database that
uses persistent storage. Note that when memory is flushed the data
goes away so anything that needs to be persistent must still be backed
up to disk on a regular basis.
3. Evaluate other use cases such as storing transient logs in a documentoriented data store such as MongoDB.
Redis
Redis is an in-memory key-value pair database typically classified as a NoSQL
database. Redis is commonly used for caching, transient data storage and as a
holding area for data during analysis in Python applications.
155
Full Stack Python: 2020 Supporter's Edition
Redis tutorials
Redis is easy to install and start using compared to most other persistent
backends, but it's useful to follow a walkthrough if you have never previously
used Redis or any NoSQL data store.
• How to Use Redis with Python 3 and redis-py on Ubuntu 16.04
contains detailed steps to install and start using Redis in Python.
• How To Install and Use Redis is a straightforward starter guide that
includes installation instructions.
Redis with Python
Redis is easier to use with Python if you have a code library client that bridges
from your code to your Redis instace. The following libraries and resources
provide more information on handling data in a Redis instance with your
Python code.
• Redis-py is a solid Python client to use with Redis.
• Walrus is a higher-level Python wrapper for Redis with some caching,
querying and data structure components build into the library.
• Writing Redis in Python with Asyncio shows a detailed example for
how to use the new Asyncio standard library in Python 3.4+ for
working with Redis. There is also a EuroPython video of the talk that
goes along with the code.
• Cache_deco is a generic Python caching decorator library.
• Write your own miniature Redis with Python doesn't actually use Redis
but shows how you can write a simplified version of Redis' in-memory
156
Full Stack Python: 2020 Supporter's Edition
data store with Python. It's a good article to understand more about
how NoSQL data stores can work under the covers.
• Introduction to Redis streams with Python shows how to use the new
(as of Redis version 5.0) append-only streams functionality via Python
code.
Redis tools and examples
Redis' wide applicability can be a downside if you don't know what to start
using it for in your application. The following code and posts provide common
use cases for Redis.
• redis-labs-use-cases has a couple of examples of using Redis to analyze
geospatial data and tweets.
• redis-migrate-tool is a library to make it easier to move data between
redis clusters and groups.
• redis-rdb-tools parses the Redis' database storage files and can dump
the contents to JSON files.
Redis Security
Redis should be customized out of its default configuration to secure it against
unauthorized and unauthenticated users. These resources provide some
advice on Reids security and guarding against data breaches.
• Pentesting Redis servers shows that security is important not only on
your application but also the databases you're using as well.
• Redis, just as with any relational or NoSQL database, needs to be
secured based on security guidelines. There is also a post where the
main author of Redis cracks its security to show the tradeoffs purposely
made between ease of use and security in the default settings.
• For God’s sake, secure your Mongo/Redis/etc! digs into the
unfortunate default security settings that come with many NoSQL
databases which can be used to compromise your systems. Make sure
to not only install your dependencies such as Redis, but automate
modifying default settings to lock them down against attackers.
157
Full Stack Python: 2020 Supporter's Edition
Specific Redis topics
Once you have configured Redis, become comfortable using it and locked it
down against malicious actors, you will want to learn more about operating,
scaling and collecting metrics. The following resources should help you get
started in those areas.
• A Key Expired In Redis, You Won't Believe What Happened Next is a
well-written post on issues one team found with their configuration
when they were using Redis for a lot of data caching.
• Redis-playbook is an Ansible playbook for installing, configuring and
securing a Redis instance.
• Monitoring Redis shows common commands for accessing meta data
about your Redis databases, such as info and slowlog .
• GitHub wrote a retrospective on moving persistent data out of Redis
and into MySQL that is worth a read as you scale up your Redis usage.
• Learn Redis the hard way (in production) investigates problems found
with a development team's Redis infrastructure, how they went about
debugging them and ultimately fixing some of the issues, while being
aware of limitations that could cause them issues in the future.
• This video on Scaling Redis at Twitter is a detailed look behind the
scenes with a massive Redis deployment.
• Real World Redis Tips provides some guidance from Heroku's
engineers from deploying Redis at scale. The tips include setting an
explicit idle connection timeout, using a connection pooler and
avoiding using KEYS in favor of SCAN .
• How to collect Redis metrics shows how to use the Redis CLI client to
grab key metrics on latency.
• You should revise your Redis max connections setting is a retrospective
from a hard web application failure due to Redis connections maxing
out on Heroku, and how to avoid this in your own applications by
modifying your redis.conf settings.
• A Speed Guide To Redis Lua Scripting shows how to use the Lua
programming language to create extensions for Redis.
158
Full Stack Python: 2020 Supporter's Edition
• Better tests for Redis integrations with redislite shows how to mock out
a Redis instance using the redislite library and clean up existing hacks
you may be using to test your Redis usage.
• Our journey from Redis 2 to Redis 3 while not taking the site down
explains their infrastructure and uptime demands in the gambling
industry, and how they were able to roll their upgraded Redis versions.
MongoDB
MongoDB is a document-oriented NoSQL database that is often used for
storing, querying and analyzing persistence data in Python applications.
General MongoDB tutorials
It is worth taking some time to learn the ins and outs of MongoDB before
connecting it to your Python application. The following tutorials are not
specific to Python and will have you work directly with the MongoDB
command line and query language.
• Getting Started with MongoDB - Part 1 and Part 2 are programming
language agnostic tutorials that show how to interact via querying and
various operators such as $in , $lte and $gte .
• An Introduction to MongoDB examines common commands frequently
used to perform data operations.
• MongoDB In 30 Minutes goes over the basics for creating, querying,
updating and deleting data in MongoDB.
• MongoDB queries do not always return all matching documents! walks
through discovering that potential pitfalls on how MongoDB queries
159
Full Stack Python: 2020 Supporter's Edition
operate that were non-intuitive to developers who are new to using this
database.
• On MongoDB is not a tutorial but instead discusses the culture and
adoption patterns around how MongoDB became such a common
NoSQL database. This is a must read to understand both the strengths
and many weaknesses Mongo has despite what you may read in other
introductory tutorials.
• This 3-part series on monitoring MongoDB with WiredTiger MMAP
and Datadog explains how to install and configure agents and gather
metrics out of your MongoDB instances.
• How to Investigate MongoDB Query Performance shows how to work
with the MongoDB profiler, use the explain method and check
execution plans.
• How to Optimize Performance of MongoDB covers schema design,
replication lag, resource provisioning and query efficiency.
• Getting Started With Google Cloud Functions and MongoDB shows
how to connect Mongo with a Google Cloud Function to store
persistent data while running on a serverless platform.
MongoDB security
NoSQL databases can be a weak spot in a production deployment
environment, especially when default settings are built for ease of
development instead of proper access control. MongoDB is no exception with
its loose default security controls so make sure to lock down your instances.
• For God's sake, secure your Mongo/Redis/etc! explains the weak
default security settings provided by many NoSQL databases, including
MongoDB. Make sure to automate locking down your NoSQL
databases just as you would any other component in your stack.
• Before deploying a MongoDB instance to production, be sure to go
through each of the items on the official MongoDB security checklist.
• The definitive guide to MongoDB security is a high-level overview for
the multitude of tasks you must perform to lock down your MongoDB
160
Full Stack Python: 2020 Supporter's Edition
instances, such as appropriately using SSL certificates and accesscontrol lists.
• MongoDB Security Basics For Your Deployments in AWS is primarily a
guide on AWS security from the perspective of using installing and
using MongoDB on your own instance. The post covers authentication,
SSL and firewalls.
• Securing MongoDB using Let's Encrypt certificate gives a configuration
that encrypts that traffic coming from and going to your MongoDB
instances using free Let's Encrypt certificates.
• This 4 post securing MongoDB series covers Data Security
Requirements for Regulatory Compliance, Database Access Control,
Database Auditing and Encryption and Environmental Control &
Database Management.
• Lightweight Directory Access Protocol (LDAP) is common in many
established company environments for security. This post on How to
Configure LDAP Authentication for MongoDB goes over how to
authenticate users via LDAP who are using MongoDB.
Python with MongoDB resources
MongoDB is straightforward to use in a Python application when a driver
such as PyMongo is installed. The following tutorials show how to install,
configure and start using MongoDB with Python.
• Introduction to MongoDB and Python shows how to use Python to
interface with MongoDB via PyMongo and MongoEngine.
• How To Set Up Flask with MongoDB and Docker combines the Flask
web framework with MongoDB then containerizes the example
application with Docker.
• The PyMongo project creators wrote a retrospective focusing on four
decisions they would have done differently with the benefit of
hindsight:
1. start_request
2. use_greenlets
3. copy_database
161
Full Stack Python: 2020 Supporter's Edition
4. MongoReplicaSetClient
• Python and MongoDB on the Talk Python to Me podcast has a great
interview with the MongoDB Python driver maintainer.
• PyMongo Monday: Setting Up Your PyMongo Environment is an
introduction to using MongoDB with Python code. This first part of the
series shows how to set up the development environment required for
working with Mongo.
• Testing MongoDB Failover in Your Python App shows show to switch
to a MongoDB replica in production failure scenarios using the
PyMongo library.
Apache Cassandra
Apache Cassandra is a column-family NoSQL data store designed for writeheavy persistent storage in Python web applications and data projects.
Python with Cassandra resources
Cassandra is commonly used with Python for write-heavy application
demands. The following tutorials walk through several of the helper libraries
that can be used to interact with Cassandra, with and without web
frameworks such as Django.
• DataStax's Python Cassandra driver can be installed as an application
dependency to make it easier to access and work with Cassandra in
your Python applications.
• Async Python and Cassandra with Gevent explains how you
monkeypatch gevent into a Python 2.7 application and work with
Cassandra using gevent's coroutines. Note that this post could have
instead been written with asycnio if it were coded with Python 3.
162
Full Stack Python: 2020 Supporter's Edition
• How to Install and Use Cassandra on Django instructs how to use
Cassandra with Django 1.8 but it should still be relevant for newer
Django versions as well.
• Using Cassandra with Python and uWSGI gives some short example
code for connecting to a Cassandra cluster outside the HTTP requestresponse cycle to prevent timeouts and blocking issues with WSGI
servers.
• The Stack Overflow thread asking about the best Cassandra library/
driver for Python? has a good answer on why to use the datastax/
python-driver project due to its CQL support and active development.
• Cassandra performance in Python: Avoid namedtuple covers the
performance penalty of using the namedtuple type with the DataStax
Cassandra Python driver and how you can work around it.
How Companies Use Cassandra
These resources are written by engineering teams at organizations that have
large scale Cassandra deployments. The posts cover topics such as
monitoring, scaling and usage with billions of records.
• How Discord Stores Billions of Messages talks about the evolution of
Discord's very large scale message store system from a MongoDB
instance to Cassandra for storing messages in a distributed, replicated
cluster.
• Monitoring Cassandra at Scale explains how the Yelp engineering team
uses Cassandra to complement their MySQL and ElasticSearch
instances. The post does a nice job of enumerating the warning signs to
monitor and provides a short example of an issue with replication that
could be caught by their approach.
• How Uber Manages A Million Writes Per Second Using Mesos And
Cassandra Across Multiple Datacenters shows why Uber needs
accurate real-time data at large scale to make their driver and
passenger operations run properly. The post goes into the overall
architecture they use including cluster size, tolerable latency and other
libraries in their stack.
163
Full Stack Python: 2020 Supporter's Edition
General Cassandra resources
Apache Cassandra can be used independently of Python applications for data
storage and querying. The learning curve for getting started is similar to other
NoSQL data stores but scaling, performance and monitoring can be
challenging. The following resources focus on addressing those issues based
on teams that have felt the pain and often released their resulting tools as
open source projects.
• The official getting started documentation for Cassandra provides
installation, configuration, and basic querying information.
• How Not To Use Cassandra Like An RDBMS (and what will happen if
you do) gives examples in Cassandra's query language CQL of
operations that are typical with relational databases but go terribly
wrong with Cassandra, due to its NoSQL architecture that is optimized
for other types of operations.
• Cassandra Query Language (CQL) Tutorial explains the concepts and
syntax behind the data management language that is Cassandra's
equivalent to relational database SQL.
• Backup and Recovery for Apache Cassandra and Scale-Out Databases
covers issues encountered when trying to take snapshot backups of
Cassandra due to partitions and consistency lag time that occur with
just about every Cassandra setup.
• Getting the Most Out of Cassandra is a video for on data modeling and
application development for developers new to Cassandra.
• The Total Newbie’s Guide to Cassandra compares Cassandra to
traditional relational databases.
• On Cassandra Collections, Updates, and Tombstones and Undetectable
tombstones in Apache Cassandra present how developers often use
Cassandra collections incorrectly when they are not experienced with
how the data store operates.
• When to use Cassandra and when to steer clear explains the advantages
Cassandra provides such as high throughput on writes (versus reads)
and availability. The disadvantages are also given such as strong
consistency, typical relational database-style (ACID) transactions and
164
Full Stack Python: 2020 Supporter's Edition
reads without knowing the primary key of the record you want to
access. These are common database tradeoffs you need to understand
based on your workload and decide upon before you build out your
whole data architecture!
• Analyzing Cassandra Performance with Flame Graphs and Garbage
Collection Tuning for Apache Cassandra are two posts in a series on
how to debug issues in operational Cassandra deployments using
appropriate data visualization, especially when the issue is due to the
Java Virtual Machine (JVM)'s garbage collection methods.
Neo4j
Neo4j (source code) is a NoSQL graph database that can be used to persist
data in Python web applications and data projects. Neo4j has both a
commercial version and a community version of the database.
Comparing Neo4j with relational databases
• RDBMS & Graphs: Relational vs. Graph Data Modeling
• Building social network with Neo4j and Python explores varying
approaches for performing social data analysis in relational databases
and graph databases.
• The Newest RDBMS-to-Neo4j ETL Tool explains the differences
between traditional relational database models and the graph-based
structure Neo4j provides. The article also covers how to use an Extract,
Transform and Load (ETL) tool to move your data from one database
such as MySQL into Neo4j.
165
Full Stack Python: 2020 Supporter's Edition
Neo4j resources
• Building a Recommendation Engine with Neo4j and Python shows how
to use Neo4j's Cypher query language to retrieve and process data.
• Using Neo4j from Python is the official page with Python-based
database drivers.
• Getting started with Neo4j and Python is a short tutorial for installing
Neo4j and running your first query.
• impfuzzy for Neo4j is a Python script that uses Neo4j as a backend to
analyze malware.
• Neo4j runs an online monthly developer "meetup" and records the
talks. Here are a few that stand out:
◦ Natural Language Understanding with Python and Neo4j
◦ Analysing football transfers with Neo4j
◦ Efficient Graph Algorithms in Neo4j
◦ An introduction to Neo4j Bolt Drivers
• A Pythonic Tour of Neo4j and the Cypher Query Language is a PyData
conference talk that gives a Python view of Neo4j's query language.
• Analyzing the Paradise Papers with Neo4j: A Closer Look at Queries,
Data Models & More uses the Neo4j Cypher Query Language to
perform analysis on the leaked Paradise Papers data.
• How to Import the Bitcoin Blockchain into Neo4j shows how to use an
existing cryptocurrency data set within Neo4j to perform analysis on
the graph structure.
• Getting Started with Data Analysis using Neo4j is a programming
language agnostic tutorial that explains how to do analysis directly in
the Neo4j Cypher Query Language.
Data analysis
Data analysis involves a broad set of activities to clean, process and transform
a data collection to learn from it. Python is commonly used as a programming
language to perform data analysis because many tools, such as Jupyter
Notebook, pandas and Bokeh, are written in Python and can be quickly
applied rather than coding your own data analysis libraries from scratch.
166
Full Stack Python: 2020 Supporter's Edition
Data analysis resources
• The following series on data exploration uses Python as the
implementation language while walking through various stages of how
to analyze a data set.
◦ Part 1 gives insight into how you should think about data and
clarify what you are looking to learn.
◦ Part 2 explains categorization and transforming a data set into
one that is easier to analyze.
◦ Part 3 shows how to visualize the results of your data
exploration.
• PyData 101 presents the slides for one of the leading developers in the
Python ecosystem on how to orient yourself if you are new to data
science.
• The Python Data Science Handbook is available to read for free online,
although I also recommend buying the book as it is a great resource for
learning the topic.
• PyData TV contains all the videos from the PyData conference series.
The conference talks are often given by professional data scientists and
the developers who write these analysis libraries, so there is a wealth of
information not necessarily captured anywhere else.
• Python Plotting for Exploratory Data Analysis is a great tutorial on how
to use simple data visualizations to bootstrap your understanding of a
data set. The walkthrough covers histograms, time series analysis,
scatter plots and various forms of bar charts.
• This series entitled "Agile Analytics" has three parts that cover how to
work in a data science team and how to operate one if you are a
manager:
◦ Part 1: The Good Stuff
◦ Part 2: The Bad Stuff
◦ Part 3: The Adjustments
• Learning Seattle's Work Habits from Bicycle Counts provides a great
example of using open data, in this case from the city of Seattle,
messing with it using Python and pandas, then charting it using skikit-
167
Full Stack Python: 2020 Supporter's Edition
learn. You can do this type of analysis on almost any data set to find out
its patterns.
• Exploring the shapes of stories using Python and sentiment APIs is a
wonderful read with context for the problem being solved, plenty of
insight into how to reproduce the results with your own code and a
good number of charts that show how sentiment analysis can extract
information from blocks of text.
• How to automate creating high end virtual machines on AWS for data
science projects walks through setting up a development environment
on Amazon Web Services so that you can perform data analysis without
owning a high-end computer. Also check out the Introduction to AWS
for Data Scientists for another tutorial that shows you how to set up
additional commonly-used data science tools on AWS.
• Analyzing bugs.python.org uses extracted data from CPython
development to show the most-commented issues and issues by
version number throughout the project's history.
• Divergent and Convergent Phases of Data Analysis examines the flow
most people doing data science and analysis projects go through during
the exploration, synthesis, modeling and narration phases.
• Forget privacy: you're terrible at targeting anyway is a different type of
article. It is a strong piece of commentary rather than a tutorial on a
specific data analysis topic. The author argues that collecting data is
typically easy but doing the dirty analysis work often yields little in the
way of definitive, actionable insight. Overall it's a well-written thought
piece that will make you at least stop and ask yourself, "do we really
need to collect this user data?"
• Gender Distribution in North Korean Posters with Convolutional
Neural Networks is a fascinating post that uses convolutional neural
networks as a mechanism to identify gender by faces in North Korean
posters. The article's analysis on this messy data set and the results it
produces using some Python glue code with various open source
libraries is a great example of how data analysis can answer questions
that would be very time consuming for a person to figure out without a
computer.
168
Full Stack Python: 2020 Supporter's Edition
• Time Series Analysis in Python: An Introduction shows how to use the
open source Prophet library to perform time series analysis on a data
set.
• Python Data Wrangling Tutorial: Cryptocurrency Edition uses the
pandas library to clean up a messy cryptocurrency data set and shift the
data into a structure that is useful for analysis the author wantds to
perform.
• Handy Python Libraries for Formatting and Cleaning Data provides a
short overview of the libraries such as Arrow and Dora that make it
easier to wrangle your data before doing analysis.
• Analyzing one million robots.txt files explains what a robots.txt file
is, why it matters, how to download a bunch of them and then perform
some analysis with NumPy.
• Safely Analyzing Popular Licenses on GitHub Projects uses a Google
BigQuery Python helper library to work with a massive 3 terabyte data
set provided by GitHub.
• Cleaning and Preparing Data in Python shows how to uses pandas to
do the "boring" part of a data analysis job and convert dirty data into a
more consistent, structured format.
• 9 obscure Python libraries for data science presents several lesserknown but still very useful libraries for performing data analysis such
as fuzzywuzzy and gym.
• Nvidia's series on defining data analysis, machine learning and deep
learning are worth reading for the background and how they break
down the problem domains:
◦ What’s the Difference Between Artificial Intelligence, Machine
Learning, and Deep Learning?
◦ Deep Learning in a Nutshell: History and Training
◦ Deep Learning in a Nutshell: Core Concepts
pandas
The Python Data Analysis Library (pandas) is a data structures and analysis
library.
169
Full Stack Python: 2020 Supporter's Edition
pandas resources
• Intro to pandas data structures, working with pandas data frames and
Using pandas on the MovieLens dataset is a well-written three-part
introduction to pandas blog series that builds on itself as the reader
works from the first through the third post.
• pandas exercises is a GitHub repository with Jupyter Notebooks that
let you practice sorting, filtering, visualizing, grouping, merging and
more with pandas.
• A simple way to anonymize data with Python and Pandas is a good
tutorial on removing sensitive data from your unfiltered data sets.
• Learn a new pandas trick every day! is a running list of great pandas
tips that the author originally posted on Twitter and then aggregated
onto a single webpage.
• Time Series Analysis with Pandas show you how to combine Python
3.6, pandas, matplotlib and seaborn to analyze and visualize open data
from Germany's power grid. This is a great tutorial to learn these tools
with a realistic data set.
• Analyzing a photographer's flickr stream using pandas explains how
the author grabbed a bunch of Flickr data using the flickr-api library
then analyzed the EXIF data in the photos using pandas.
• Pandas Crosstab Explained shows how to use the crosstab function
in pandas so you can summarize and group data.
• Calculating streaks in pandas shows how to measure and report on
streaks in data, which is where several events happen in a row
consecutively.
170
Full Stack Python: 2020 Supporter's Edition
• How to Convert a Python Dictionary to a Pandas DataFrame is a
straightforward tutorial with example code for loading and adding data
stored in a typical Python dictionary into a DataFrame.
• This two-part series on loading data into a pandas DataFrame presents
what to do when CSV files do not match your expectations and how to
handle missing values so you can start performing your analysis rather
than getting frustrated with common issues at the beginning of your
workflow.
• Building a financial model with pandas explains how to create an
amortization schedule with corresponding table and charts that show
the pay off period broken down by interest and principal.
• tabula-py: Extract table from PDF into Python DataFrame presents
how to use the Python wrapper for the Tabula library that makes it
easier to extract table data from PDF files.
• Analyzing Browser History Using Python and Pandas shows how to
take data from Google Chrome and start to visualize it with pandas and
matplotlib.
• Time Series Forecast Case Study with Python: Monthly Armed
Robberies in Boston walks through the data wrangling, analysis and
visualization steps with a public data set of murders in Boston from
1966 to 1975. This particular data problem may not be your thing but
by going through the process you can learn a lot that can be applied to
any data set.
• A Gentle Visual Intro to Data Analysis in Python Using Pandas
presents spreadsheet-like pictures to show conceptually what pandas is
doing with your data as you apply various functions like groupby and
loc .
• Data Manipulation with Pandas: A Brief Tutorial uses some example
data sets to show how the most commonly-used functions in pandas
work.
• Analyzing Pronto CycleShare Data with Python and Pandas uses Seattle
bikeshare data as a source for wrangling, analysis and visualization.
171
Full Stack Python: 2020 Supporter's Edition
• Stylin' with pandas shows how to add colors and sparklines to your
output when using pandas for data visualization.
• Python and JSON: Working with large datasets using Pandas is a welldone detailed tutorial that shows how to mung and analyze JSON data.
• Fun with NFL Stats, Bokeh, and Pandas uses National (American)
Football League data as a source for wrangling and visualization.
• Analyzing my Spotify Music Library With Jupyter And a Bit of Pandas
shows how to grab all of your user data from the Spotify API then
analyze it using pandas in Jupyter Notebook.
• Scalable Python Code with Pandas UDFs explains that pandas
operations can often be parallelized for better performance using the
Pandas UDFs feature in PySpark version 2.3 or greater.
• How to use Pandas read_html to Scrape Data from HTML Tables has a
bunch of great code examples that show how to load data from HTML
directly into your DataFrames.
• How to download fundamentals data with Python shows how to obtain
and use financial data, such as balance sheets, stock prices, and various
ratios to perform your own analysis on.
• How to convert JSON to Excel with Python and pandas provides
instructions for creating a spreadsheet out of JSON file.
SciPy and NumPy
SciPy is a collection of open source code libraries for math, science and
engineering. NumPy, Matplotlib and pandas are libraries that fall under the
SciPy project umbrella.
172
Full Stack Python: 2020 Supporter's Edition
NumPy (source code) is a Python code library that adds scientific computing
capabilities such as N-dimensional array objects, FORTRAN and C++ code
integration, linear algebra and Fourier transformations. NumPy serves as a
required dependency for many other scientific computing packages such as
pandas.
Blaze is a similar, but separate, ecosystem with additional tools for wrangling,
cleaning, processing and analyzing data.
SciPy resources
Take a look at the pages on Matplotlib and pandas for tutorials specific to
those projects. The following resources are broader walkthroughs for the
SciPy ecosystem:
• SciPy Lecture notes goes into the overall Python scientific computing
ecosystem and how to use it.
173
Full Stack Python: 2020 Supporter's Edition
• The SciPy Cookbook contains instructions for various SciPy packages
that were previously hosted on the SciPy wiki.
• Robots and Generative Art and Python, oh my! uses Scipy, Numpy, and
Matplotlib to generate some nice looking art that can even be written to
paper using a plotter. This is a very cool example project that ties
together the scientific world and the art world.
• Lectures in Quantitative Economics: SciPy provides a good overview of
SciPy compared to the specific NumPy project, as well as explanations
for the wrappers SciPy provides over lower-level FORTRAN libraries.
• A plea for stability in the SciPy ecosystem presents concerns from one
scientist's perspective about how fast the Python programming
ecosystem changes and that code can become backwards incompatible
in only a few years. The issue is that many science projects last decades
and therefore cannot follow the rate of change as easily as typical
software development projects.
NumPy resources
• From Python to NumPy is an awesome resource that shows how to use
your basic Python knowledge to learn how to do vectorization with
NumPy.
• The ultimate beginner's guide to NumPy explains how to install and
import NumPy, then digs into using arrays for computation and how to
perform operations that get the results you need for your data analysis.
• Math to Code provides an interactive tutorial to learn how to
implement math in NumPy.
• 101 NumPy Exercises for Data Analysis
• NumPy: creating and manipulating numerical data
• Python NumPy Array Tutorial is a starter tutorial specifically focused
on using and working with NumPy's powerful arrays.
• Beyond Numpy Arrays in Python is a predecessor to a Numpy
Enhancement Proposal that recommends how to prepare the scientific
computing ecosystme for GPU, distributed and sparse arrays.
174
Full Stack Python: 2020 Supporter's Edition
• Probability distribution explorer contains graphs for understanding
how different probabilities look when plotted. There is also code for
implementing the visuals in NumPy and SciPy.
Example NumPy code
• SmoothLife is an implementation of Conway's Game of Life using
NumPy. The project uses a continuous space rather than the traditional
discrete board.
• Advanced Numpy Techniques is a Jupyter Notebook with code on
beyond-the-basics NumPy features.
Data Visualization
Data visualizations transform raw numbers into graphic formats that make it
easier for humans to see patterns, trends and other useful information.
Python data visualization tools
• Bokeh
• HoloViews
• Matplotlib
• Chartify (source code)
• Graphviz
Python-specific data viz resources
• Python Data Visualization 2018: Why So Many Libraries? is an indepth article on the Python data visualization tools landscape. A mustread whether you are new to the space or have been using one or more
of these libraries for awhile.
• The Python Graph Gallery has a slew of visualizations created with
Python and includes the code used to produced each one.
• Python & OpenGL for Scientific Visualization is a free book that shows
how to combine open source tools such as PyOpenGL with Python data
analysis libraries to generate interactive scientific data visualizations.
175
Full Stack Python: 2020 Supporter's Edition
• 10 Useful Python Data Visualization Libraries for Any Discipline is a
straightforward overview of Python packages that create Python
visualizations.
• Introduction to Data Visualization with Altair is a starter post for the
wonderful Altair visualization tool written in Python.
• The Next Level of Data Visualization in Python uses the Plotly graphing
library to draw more complex visualizations.
• An introduction to Altair provides another wonderful tutorial on this
data visualization tool.
• A Dramatic Tour through Python’s Data Visualization Landscape
provides examples with the ggplot and Altair libraries. The questionand-answer format for what you can do with the data is a really good
model that keeps your attention throughout the post.
• How to Generate FiveThirtyEight Graphs in Python gives a great
tutorial on generating a specific style graph with pandas and Matplotlib
that is similar to FiveThirtyEight's plots.
• Intro to pdvega - Plotting for Pandas using Vega-Lite shows how to
generate plots from your pandas-structured data using pdvega.
• Sorting Algorithms Visualized in Python uses Python, numpy and
scikit-image to animate how sorting algorithms work.
• How to Build a Reporting Dashboard using Dash and Plotly explains
how to use the Dash library to take a bunch of data and turn it into a
nice-looking dashboard.
• The reason I am using Altair for most of my visualization in Python
explains why this wrapper for Vega-lite is awesome and that the author
uses Altair because Matplotlib can be very complicated for whipping up
quick visualizations.
Beautiful example visualizations
Sometimes you need inspiration from other sources to figure out what you
want to build. The following links have made me excited about data
visualization and gave me ideas for what to build.
176
Full Stack Python: 2020 Supporter's Edition
• Roads to Rome is a beautiful visualization showing the data behind the
expression "all roads lead to Rome" and whether or not there is a
"Rome" central city in every country.
• Monarchs is a wonderful 1,000 year history visual of European rulers.
The developer also wrote an in-depth article on how Monarchs was
created using d3.js.
• Star Wars: The Force Accounted is Bloomberg's way of breaking down
on-screen action between light and dark sides, the main characters,
various bits about the Force and other data extracted from the movies.
• Big League Graphs presents a bunch of creative ways to view data for
sports such as basketball, baseball and hockey.
• What do numbers look like? is a Python 3 dimensional visualization of
millions of integers, colored by special factors such as prime and
Fibonacci numbers.
• Bay Area Housing Marketing Analysis: Part 1 and Part 2 are a
combination of inspiration and tutorial. These posts contain a ton of
data analysis and graphing and show numerous ways to slice and
present information.
• How We Animated Trillions of Tons of Flowing Ice breaks down the
process that the NY Times data team used to create the beautiful
Antarctic Dispatches articles that show how glaciers and ice are
moving.
• Who knew good old histograms could be so fascinating? Check out this
post titled What's so hard about histograms? and scroll through to
learn a ton about the details you can think about when creating these
types of data visuals.
• Optimized Brewery Road Trip, With Genetic Algorithm shows how a
heuristic solving approach such as a genetic algorithm can be used to
handle a version of the Traveling Salesman Problem (TSP), but with
the more fun Top 100 American brewery locations.
177
Full Stack Python: 2020 Supporter's Edition
Data visualization resources
• Guides for visualizing data provides the thought processes that you can
use when you are showing uncertainty, incompletet data, differences,
outliers, and other common scenarios that occur when your data is
messy - and it usually is!
• Data visualization, from 1987 to today is a wonderful reference about
the pre-computer age era of visualization which was a combination of
cartography, art and statistics rather than any cohesive field as it is
often seen today. The images showing how people worked with paper
to build their visuals add fantastic context to the story.
• Xenographics presents uncommon and unusual visualization formats
such as the Manhattan Plot and Time Curve.
• Engineering Intelligence Through Data Visualization at Uber explains
how Uber's data visualization team grew from 1 person to 15 and the
output they created along the way, including the open source tool
react-vis.
• The Practitioner's Guide to System Dashboard Design series covers a
lot of ground for what you should consider when building one form of
visualization, the data dashboard:
◦ Part 1: Structure and Layout
◦ Part 2: Presentation and Accessibility
◦ Part 3: What Charts to Use
◦ Part 4: Context Improvement
• Truncating the Y-Axis: Threat or Menace? is a great in-indepth
explanation of why something that seems as straightforward as laying
out the Y-Axis requires thought and care otherwise the visualization
could end up being misleading.
Bokeh
Bokeh is a data visualization library that allows a developer to code in Python
and output JavaScript charts and visuals in web browsers.
178
Full Stack Python: 2020 Supporter's Edition
Why is Bokeh a useful library?
Web browsers are ideal clients for consuming interactive visualizations.
However, libraries such as d3.js can be difficult to learn and time consuming
to connect to your Python backend web app. Bokeh instead generates the
JavaScript for your application while you write all your code in Python. The
removal of context switching between the two programming languages can
make it easier and faster to create charts and visualizations.
What do Bokeh visualizations look like?
Bokeh can create any type of custom graph or visualization. For example, here
is a screenshot of a bar chart created with the figure plot:
For more references, including interactive live demonstrations, check out
these sites:
• The official Bokeh gallery has many example Bokeh visual formats.
• Bokeh Applications hosts numerous data visualizations built with
Bokeh.
179
Full Stack Python: 2020 Supporter's Edition
Bokeh resources
Bokeh is under heavy development ahead of the upcoming 1.0 release. Note
that while all of the following tutorials are useful, it is possible some of the
basic syntax will change as the library's API is not yet stable.
• Integrating Bokeh Visualisations Into Django Projects does a nice job
of walking through how to use Bokeh to render visualizations in Django
projects.
• Responsive Bar Charts with Bokeh, Flask and Python 3 is my
recommended tutorial for those new to Bokeh who want to try out the
library and get an example project running quickly with Flask.
• Fun with NFL Stats, Bokeh, and Pandas takes an NFL play-by-play
data set, shows how to wrangle the data into an appropriate format
then explains the code that uses Bokeh to visualize it.
• Data is beautiful: Visualizing Roman imperial dynasties provides a
walkthrough for creating a gorgeous visualization based on historical
Roman data. The post is about more than just the visual, it also goes
into the ideation, data wrangling and analysis phases that came before
using Bokeh to show the results.
• Visualizing with Bokeh gives a detailed explanation with the code for
number Bokeh visuals you can output while working with a pandas
data set.
• Interactive Data Visualization in Python With Bokeh is a great
beginners tutorial that shows you how to structure your data, draw
your first figures and add interactivity to the visualizations.
• Creating Bar Chart Visuals with Bokeh, Bottle and Python 3 is a tutorial
that combines the Bottle web framework
• Building Bullet Graphs and Waterfall Charts with Bokeh covers
buildings two types of useful visualizations into your applications using
Bokeh.
• Interactive Visualization of Australian Wine Ratings builds a nontrivial visualization with a nice sample set of data based on wine
ratings.
180
Full Stack Python: 2020 Supporter's Edition
• Visualization with Bokeh
• Drawing a Brain with Bokeh is a fun example of a chord diagram that
represents neural connections in the brain.
• Bryan Van de Ven on Bokeh is a podcast episode by one of the main
Bokeh maintainers.
• The Python Visualization Landscape by Jake VanderPlas at PyCon 2017
covers many Python data visualization tools, including Bokeh.
• This flask-bokeh-example project has the code to create a simple chart
with Bokeh and Flask.
• Bokeh vs Dash — Which is the Best Dashboard Framework for Python?
contains a single project that was written in both Dash and Bokeh. The
author gives his subjective view on the implementation difficulty
although the web application only contained a single type of data
visualization so it is hard to drawn any real conclusions from his
opinion.
• Realtime Flight Tracking with Pandas and Bokeh provides a great
example of combining pandas for structuring data with Bokeh for
visualization.
• How to Create an Interactive Geographic Map Using Python and Bokeh
shows how to use a GeoJSONDataSource as input for Bokeh and draw
a map with the data.
d3.js
Data-Driven Documents (d3.js) is a JavaScript visualization library used to
create interactive visuals for web browsers.
d3.js tutorials
d3.js has a steep learning curve so it is a good idea to read several tutorials
before diving in and trying to create your own visualization from scratch.
181
Full Stack Python: 2020 Supporter's Edition
• The Hitchhiker’s Guide to d3.js is a wonderfully-written resource that
explains the context for how d3.js works and how all the pieces can be
used to create your desired visualizations.
• d3.js first steps contains the code and markup for building your first
d3.js visual.
• Let's Make a D3 Plugin shows you how to create your own reusable
plugin that you can use across multiple visualizations as a separate
JavaScript library. d3.js version 4 is now widely used so don't worry
about the note at the start of the article.
• Reusable and extendable d3 charts is a natural extension of the d3
plugin post. It shows how to reuse visualization code between multiple
visuals.
• Visualizing Movement Data - Part I provides a detailed example of how
to draw a complex visualization.
• This Fantasy Map Generator is such a cool example of what d3.js can
procedurally generate based on a set of inputs.
• Building dashboards with Django and D3
• Argyle in d3? Oh yes, the library can do that, and here is the code to
prove it.
• How and why to use D3 with React is an awesome overview of the D3.js
plugin ecosystem and how to use the tool with a React-based
JavaScript front end.
• D3 is not a data visualization library breaks down the parts to D3 and
why it's not directly comparable to a typical charting library.
Charts with d3.js
• Responsive D3js Charts shows how to take a static line chart and make
it responsive when the browser size changes.
• Resize to Scale with d3.js gives code for a render function that adjusts
the size of the viewing window based on the parent element for the
visualization.
182
Full Stack Python: 2020 Supporter's Edition
• Responsive Data Visualization provides another approach for making
responsive D3.js charts.
• Make great-looking d3.js charts in Python without coding a line of
JavaScript combines a Python backend with the python-nvd3 library to
generate d3.js charts without having to hand-write the JavaScript code.
If you are interested in a solution like this for your own visualizations
then you should also check out Bokeh.
• How to make a modern dashboard with NVD3.js uses the NVD3.js
library that works as an abstraction on top of d3.js to create charts. The
post puts together several charts to show how to build a dashboard
based on public JSON data.
• d3-regression is a module for calculating statistical regressions from
two-dimensionala data.
D3 ecosystem
• The trouble with D3 is not a tutorial but it's an important read because
it discusses why D3 can be very difficult to learn: the learning curve
depends on your background. If you are a front-end developer you will
likely have the easiest time if you already understand JavaScript, SVG
and the browser Document Object Model (DOM). Non-technical
designers and analysts typically have the hardest time with using D3
because the not only have to learn the tool itself but all the concepts
and web browser technologies that it is built upon.
• This visualization of d3.js modules and resources is a wonderful map
for understanding how various parts of the library and the ecosystem
fit together. The visual provides a useful map for where to concentrate
your learning depending on what type of visualization you are working
to build.
• D3.js in Action, Second Edition is partially an announcement for the
authors book but also contains good context for who uses D3 and why
its usage continues to grow.
183
Full Stack Python: 2020 Supporter's Edition
Matplotlib
Matplotlib is a data visualization plotting library where a developer can code
visuals in Python and output them as part of Jupyter Notebooks, web
applications and graphical user interface (GUI) toolkits.
Matplotlib resources
• Effectively using Matplotlib is an awesome getting started tutorial that
breaks through the confusing beginner steps so you can quick start
using the plotting library.
• Web Scraping With Python: Scrapy, SQL, Matplotlib To Gain Web Data
Insights is a long and comprehensive tutorial that walks through
obtaining, cleaning and visualizing data.
• Matplotlib Cheat Sheet: Plotting in Python contains some handy
snippets of code to perform common plotting operations in Matplotlib.
• 5 Quick and Easy Data Visualizations in Python shows several code
examples with explanations for performing exploratory data analysis
using Matplotlib.
• Introduction to Matplotlib — Data Visualization in Python explains how
to install and start using Matplotlib. The post has a ton of detail on
customizing your plots and graphs after creating the initial visuals.
• Visualize World Trends using Seaborn in Python shows world life
expectancy in plots generated by Matplotlib and Seaborn.
• Pandas & Seaborn - A guide to handle & visualize data in Python builds
a visualization by starting with pandas for data wrangling then outputs
charts with Matplotlib and Seaborn.
• Matplotlib Tips and Demos is a long Jupyter Notebook with a ton of
example code that shows how to use Matplotlib in many ways.
184
Full Stack Python: 2020 Supporter's Edition
• Animation with Matplotlib explains the animation base class and the
main interfaces for creating animations in your visualizations.
• Matplotlib: Creating Plots is a video tutorial series where each 15-ish
minute episode covers one important topic such as shading areas on
line plots, drawing pie charts or plotting a stream of updating data.
Markup Languages
Markup languages provide pre-defined, parsable syntax within text
documents that is used for annotations. For example, within a Markdown
document, the syntax
[this is a link to Full Stack Python](https://www.fullstackpython.com)
indicates the text "this is a link to Full Stack Python" should be annotated
with a link to "https://www.fullstackpython.com/" when run through a
Markdown parser and then transformed into HTML output.
Markup language resources
• Reach for Markdown, not LaTeX argues for the simplicity of Markdown
versus the steep learning curve and less easily adopted LaTeX for
creating documents.
• Yet Another Markup LOL? explains the virtues and the significant
downsides of tooling for the YAML markup language. Mistakes in
configuration files that use YAML or any markup language often fly
past testing and continuous integration services that catch errors in
regular code. The post also introduces a couple of tools that can help
with specific YAML issues, especially when using Kubernetes.
Markdown
Markdown is a common markup language frequently used by developers to
write Python project documention.
185
Full Stack Python: 2020 Supporter's Edition
Markdown's origin
Markdown was originally developed by John Gruber in 2004. The markup
language's lightweight design helped it gain rapid adoption by software
developers and designers. The format's simplicity also makes it easier to write
parsers to convert the structured syntax into other formats such as HTML and
JSON.
Markdown resources
Markdown does not have an extensive set of strict rules like some other text
formats so you should be able to read up on the basics with these articles then
write a few practice documents to be comfortable with it. The following
resources are really helpful when you are getting started or need a quick
reference on a less commonly-used feature such as tables or block quotes.
• Say yes to Markdown, no to MS Word provides a really awesome
overview of why Markdown is a more usable file format than Microsoft
Word and similar proprietary file types. The article also has a good list
of useful Markdown-related tools such as a Markdown-to-PDF
converter (a NodeJS package but easy enough to use with a basic
development environment).
186
Full Stack Python: 2020 Supporter's Edition
• Markdown syntax is the defacto standard and wonderful reading for
both initial learning and random reference.
• Markdown cheatsheet is a quick reference that is a shortened version of
the above Markdown syntax page.
• Markdown parsers in Python reviews many of the most common
Python Markdown parser implementations to give insight into the
advantages and disadvantages of each one.
• reStructuredText vs Markdown for documentation brings up some
really good points about the downsides to Markdown's simplicity. First,
a lot of documentation needs more complex output that is not possible
with vanilla Markdown so you need to drop into plain old HTML,
which defeats the purpose of using a markup language. Second, some
of the syntax around inserting blank lines by adding spaces at the end
of lines is confusing if someone is using a text editor or development
environment that is not configured to show blank spaces. Worse yet, if
your editor is set to remove blank spaces at the end of lines, which is
fairly common among developers, then you can mistakenly break the
formatting intended by the original author. Overall this is a good piece
to read for a balanced view of Markdown and the reasons it provides
are one reason why I use both Markdown and reStructuredText
depending on the project.
• The Python Package Index (PyPI) supports Markdown as of 2018
although there are still some tweaks being made to the flavors that can
be used such as GitHub-flavored Markdown.
• PowerShell and Markdown shows how to work with Markdown in
PowerShell including customizing colors and listing some quirks you
may need to get around.
• reStructuredText vs. Markdown for technical documentation compares
Markdown and reStructuredText specifically for documenting software
and explains where each one has advantages.
• Reach for Markdown, not LaTeX examines the virtues of using straight
Markdown along with tools such as pandoc to convert from one file
format to another, including how to use Markdown for presentations
and not just regular documentation.
187
Full Stack Python: 2020 Supporter's Edition
• Markdown page is a JavaScript file that makes it easy to render plain
old Markdown as a webpage.
reStructuredText
reStructuredText, sometimes abbreviated as "RST" or "reST", is a markup
language implementation that is often used to document Python projects.
reStructuredText resources
• A brief tutorial on parsing reStructuredText (reST)
• reStructuredText vs. Markdown for technical documentation
• RestructuredText (reST) and Sphinx CheatSheet
• reStructuredText cheat sheet
188
Full Stack Python: 2020 Supporter's Edition
Web Development
Web development is the umbrella term for conceptualizing, creating,
deploying and operating web applications and application programming
interfaces for the Web.
Why is web development important?
The Web has grown a mindboggling amount in the number of sites, users and
implementation capabilities since the first website went live in 1989. Web
development is the concept that encompasses all the activities involved with
websites and web applications.
How does Python fit into web development?
Python can be used to build server-side web applications. While a web
framework is not required to build web apps, it's rare that developers would
not use existing open source libraries to speed up their progress in getting
their application working.
Python is not used in a web browser. The language executed in browsers such
as Chrome, Firefox and Internet Explorer is JavaScript. Projects such as pyjs
can compile from Python to JavaScript. However, most Python developers
write their web applications using a combination of Python and JavaScript.
Python is executed on the server side while JavaScript is downloaded to the
client and run by the web browser.
Web development resources
To become an experienced web developer you need to know the foundation
principles that the web is built with, such as HTTP requests and responses,
client (typically web browsers) and server (web servers such as Nginx and
Apache architectures, HTML, CSS and JavaScript, among many other topics.
The following resources provide a range of perspectives and when combined
together should get you oriented in the web development world.
• How the Internet works is a must-read to get a quick overview of all the
pieces that go into a network connection from one machine to another.
The example explains how an email is sent and the story is just as
190
Full Stack Python: 2020 Supporter's Edition
useful for learning about other connections such as downloading a
webpage.
• If you want to be a web developer it's important to know the
foundational tools used to build websites and web applications. It is
also important to understand that the core concepts such as HTTP,
URLs and HTML were all there at the beginning and then were
expanded with new specifications over time. This article on the History
of the Web succinctly explains the origins of the web starting from Tim
Berners-Lee's origin vision and release at CERN.
• Web Architecture 101 is a great high-level overview of the technologies
that run the modern web, such as DNS, load balancers, web application
servers (for Python that equates to WSGI servers), data bases, task
queues, caching and several other critical concepts.
• The Evolution of the Web visualizes how web browsers and related
technologies have changed over time as well as the overall growth of
the Internet in the amount of data transferred. Note that the
visualization unfortunately stops around the beginning of 2013 but it's
a good way to explore what happened in the first 24 years.
• What happens when? is an incredibly detailed answer to the questions
"What happens when you type google.com into your browser's address
box and press enter?" that seems straightforward on the surface until
you really dig in.
• How browsers work provides an overview with solid detail on how
browsers take the HTML, CSS, JavaScript, images and other files as
input and render webpages as output. It is well worth your time to
know this stuff as a web developer.
• The history of the URL explains how the growth of ARPANET to
hundreds of nodes eventually led to the creation of the URL. This is a
great read that provides historical context for why things are the way
they are with the web.
• Web app checklist presents good practices that developers building and
deploying web applications should follow. Don't worry about having
every single one of these recommendations implemented before getting
your site live, but it is worthwhile to review the list to make sure there
191
Full Stack Python: 2020 Supporter's Edition
is not something obvious you can handle in a few minutes that will
improve your site's security, performance or usability.
• Web application development is different and better provides some
context for how web development has evolved from writing static
HTML files into the complex JavaScript client-side applications
produced today.
• The Browser Hacker's Guide to Instantly Loading Everything is a
spectacular technical talk given by Addy Osmani at JSConf EU 2017
that has great bits of developer knowledge for both beginner and
experienced web developers alike.
• Build a web application from scratch and its follow on posts for request
handling middleware explores the fundamentals of web development.
Learning these foundational concepts is critical for a web developer
even though you should still plan to use an established web framework
such as Django or Flask to build real-world applications. The open
source code for these posts is available on GitHub.
• While not Python-specific, Mozilla put together a Learning the Web
tutorial for beginners and intermediate web users who want to build
websites. It's worth a look for general web development learning.
• Web development involves HTTP communication between the server,
hosting a website or web application, and the client, a web browser.
Knowing how web browsers works is important as a developer, so take
a look at this article on what's in a web browser.
• Ping at the speed of light dives into the computer networking weeds
with how fast packets travel through the internet plumbing. The author
created a Python script that scrapes network speeds from disparate
locations to see what the network speed is in fiber optic cables as a
percentage of the speed of light.
• The critical path: optimizing load times with the Chrome DevTools
provides a well-written explanation about using Chrome's developer
features to improve the performance of your websites and web
applications.
192
Full Stack Python: 2020 Supporter's Edition
• Three takeaways for web developers after two weeks of painfully slow
Internet is a must-read for every web developer. Not everyone has fast
Internet service, whether because they are in a remote part of the world
or they're just in a subway tunnel. Optimizing sites so they work in
those situations is important for keeping your users happy.
• The History of the URL: Path, Fragment, Query, and Auth gives a
comprenhensive historical perspective on the fundamental way to link
to resources on the web. This post should be required reading for web
developers.
• Quantum Up Close: What is a browser engine? explains how a browser
takes in HTML, JavaScript, CSS, images and any other data and files to
produce a webpage as output.
• How to understand performance tests is an important topic because
many websites are slow and bloated. Learning about improving the
performance of your site is one of the best ways to become a better web
developer. Another great article on website performance is The average
web page is 3MB. How much should we care?. The visuals alone tell a
compelling story about how large webpage sizes have grown in recent
years.
Web Frameworks
A web framework is a code library that makes web development faster and
easier by providing common patterns for building reliable, scalable and
maintainable web applications. After the early 2000s, professional web
development projects always use an existing web framework except in very
unusual situations.
193
Full Stack Python: 2020 Supporter's Edition
Why are web frameworks useful?
Web frameworks encapsulate what developers have learned over the past
twenty years while programming sites and applications for the web.
Frameworks make it easier to reuse code for common HTTP operations and to
structure projects so other developers with knowledge of the framework can
quickly build and maintain the application.
Common web framework functionality
Frameworks provide functionality in their code or through extensions to
perform common operations required to run web applications. These
common operations include:
1. URL routing
2. Input form handling and validation
3. HTML, XML, JSON, and other output formats with a templating
engine
4. Database connection configuration and persistent data manipulation
through an object-relational mapper (ORM)
5. Web security against Cross-site request forgery (CSRF), SQL Injection,
Cross-site Scripting (XSS) and other common malicious attacks
6. Session storage and retrieval
Not all web frameworks include code for all of the above functionality.
Frameworks fall on the spectrum from executing a single use case to
providing every known web framework feature to every developer. Some
frameworks take the "batteries-included" approach where everything possible
comes bundled with the framework while others have a minimal core package
that is amenable to extensions provided by other packages.
For example, the Django web application framework includes the Django
ORM layer that allows a developer to write relational database read, write,
query, and delete operations in Python code rather than SQL. However,
Django's ORM cannot work without significant modification on nonrelational (NoSQL) databases such as MongoDB or Cassandra.
Some other web frameworks such as Flask and Pyramid are easier to use with
non-relational databases by incorporating external Python libraries. There is a
spectrum between minimal functionality with easy extensibility on one end
194
Full Stack Python: 2020 Supporter's Edition
and including everything in the framework with tight integration on the other
end.
Do I have to use a web framework?
Whether or not you use a web framework in your project depends on your
experience with web development and what you're trying to accomplish. If
you are a beginner programmer and just want to work on a web application as
a learning project then a framework can help you understand the concepts
listed above, such as URL routing, data manipulation and authentication that
are common to the majority of web applications.
On the other hand if you're an experienced programmer with significant web
development experience you may feel like the existing frameworks do not
match your project's requirements. In that case, you can mix and match open
source libraries such as Werkzeug for WSGI plumbing with your own code to
create your own framework. There's still plenty of room in the Python
ecosystem for new frameworks to satisfy the needs of web developers that are
unmet by Django, Flask, Pyramid, Bottle and many others.
In short, whether or not you need to use a web framework to build a web
application depends on your experience and what you're trying to accomplish.
Using a web framework to build a web application certainly isn't required, but
it'll make most developers' lives easier in many cases.
Comparing web frameworks
Are you curious about how the code in a Django project is structured
compared with Flask? Check out this Django web application tutorial and
then view the same application built with Flask.
Talk Python to Me had a podcast episode with a detailed comparison of the
Django, Flask, Tornado and Pyramid frameworks.
There is also a repository called compare-python-web-frameworks where the
same web application is being coded with varying Python web frameworks,
templating engines and object-relational mappers.
195
Full Stack Python: 2020 Supporter's Edition
Web framework resources
• Building Your Own Python Web Framework is an awesome way to
learn how the WSGI works and the many other pieces that combine to
make web frameworks useful to web developers.
• When you are learning how to use one or more web frameworks it's
helpful to have an idea of what the code under the covers is doing. This
post on building a simple Python framework from scratch shows how
HTTP connections, routing, and requests can work in just 320 lines of
code. This post is awesome even though the resulting framework is a
simplification of what frameworks such as Django, Flask and Pyramid
allow developers to accomplish.
• There is also another, more recent multi-part tutorial about building
your own web framework in Python. This series is based on the alcazar
project the author is coding for learning purposes:
◦ Part 1: Handling requests
◦ Part 2: Routes, Class-Based Handlers and Unit Testing
◦ Part 3: Test Client and Templating Support
◦ Part 4: Exception Handling, Static Files and Middleware
• Check out the answer to the "What is a web framework and how does it
compare to LAMP?" question on Stack Overflow.
• Another great series that digs behind the web framework magic is
"Web Application from Scratch". The four parts are:
◦ Part 1: handling HTTP requests and responses
◦ Part 2: abstracting Requests, Responses and Servers
◦ Part 3: request handlers and middleware
◦ Part 4: abstracting applications
• Frameworks is a really well done short video that explains how to
choose between web frameworks. The author has some particular
opinions about what should be in a framework. For the most part I
agree although I've found sessions and database ORMs to be a helpful
part of a framework when done well.
• "What is a web framework?" is an in-depth explanation of what web
frameworks are and their relation to web servers.
196
Full Stack Python: 2020 Supporter's Edition
• Django vs Flask vs Pyramid: Choosing a Python Web Framework
contains background information and code comparisons for similar
web applications built in these three big Python frameworks.
• This fascinating blog post takes a look at the code complexity of several
Python web frameworks by providing visualizations based on their
code bases.
• Python's web frameworks benchmarks is a test of the responsiveness of
a framework with encoding an object to JSON and returning it as a
response as well as retrieving data from the database and rendering it
in a template. There were no conclusive results but the output is fun to
read about nonetheless.
• What web frameworks do you use and why are they awesome? is a
language agnostic Reddit discussion on web frameworks. It's
interesting to see what programmers in other languages like and dislike
about their suite of web frameworks compared to the main Python
frameworks.
• This user-voted question & answer site asked "What are the best
general purpose Python web frameworks usable in production?". The
votes aren't as important as the list of the many frameworks that are
available to Python developers.
• Django vs. Flask in 2019: Which Framework to Choose looks at the best
use cases for Django and Flask along with what makes them unique,
from an educational and development standpoint.
• 11 new Python web frameworks has a quick blurb on several newer
frameworks that are still emerging, such as Sanic, Masonite and
Molten.
Web frameworks learning checklist
1. Choose a major Python web framework (Django or Flask are
recommended) and stick with it. When you're just starting it's best to
learn one framework first instead of bouncing around trying to
understand every framework.
197
Full Stack Python: 2020 Supporter's Edition
2. Work through a detailed tutorial found within the resources links on
the framework's page.
3. Study open source examples built with your framework of choice so you
can take parts of those projects and reuse the code in your application.
4. Build the first simple iteration of your web application then go to the
deployment section to make it accessible on the web.
Django
Django is a widely-used Python web application framework with a "batteriesincluded" philosophy. The principle behind batteries-included is that the
common functionality for building web applications should come with the
framework instead of as separate libraries.
For example, authentication, URL routing, a template engine, an objectrelational mapper (ORM), and database schema migrations are all included
with the Django framework. Compare that included functionality to the Flask
framework which requires a separate library such as Flask-Login to perform
user authentication.
The batteries-included and extensibility philosophies are simply two different
ways to tackle framework building. Neither philosophy is inherently better
than the other one.
Why is Django a good web framework choice?
The Django project's stability, performance and community have grown
tremendously over the past decade since the framework's creation. Detailed
tutorials and good practices are readily available on the web and in books. The
198
Full Stack Python: 2020 Supporter's Edition
framework continues to add significant new functionality such as database
migrations with each release.
I highly recommend the Django framework as a starting place for new Python
web developers because the official documentation and tutorials are some of
the best anywhere in software development. Many cities also have Djangospecific groups such as Django District, Django Boston and San Francisco
Django so new developers can get help when they are stuck.
There's some debate on whether learning Python by using Django is a bad
idea. However, that criticism is invalid if you take the time to learn the Python
syntax and language semantics first before diving into web development.
Django books and tutorials
There are a slew of free or low cost resources out there for Django. Make sure
to check the version numbers used in each post you read because Django was
released over 10 years ago and has had a huge number of updates since then.
These resources are geared towards beginners. If you are already experienced
with Django you should take a look at the next section of resources for more
advanced tutorials.
• Tango with Django is an extensive set of free introductions to using the
most popular Python web framework. Several current developers said
this book really helped them get over the initial framework learning
curve.
• The Django Girls Tutorial is a great tutorial that doesn't assume any
prior knowledge of Python or Django while helping you build your first
web application.
• A Complete Beginner's Guide to Django is a wonderful seven-part
series that incrementally builds out a Django project and handles
deploying the app in the final post. The seven parts are:
◦
◦
◦
◦
◦
◦
Getting Started
Fundamentals
Advanced Concepts
Authentication
Django ORM
Class-Based Views
199
Full Stack Python: 2020 Supporter's Edition
◦ Deployment
• Test-Driven Development with Python focuses on web development
using Django and JavaScript. This book uses the development of a
website using the Django web framework as a real world example of
how to perform test-driven development (TDD). There is also coverage
of NoSQL, WebSockets and asynchronous responses. The book can be
read online for free or purchased in hard copy via O'Reilly.
• Django OverIQ is a project-based tutorial for beginners that covers the
required features such as the Django ORM and Django Templates.
• The Django subreddit often has links to the latest resources for
learning Django and is also a good spot to ask questions about it.
• This Django tutorial shows how to build a project from scratch using
Twitter Bootstrap, Bower, Requests and the Github API.
• The recommended Django project layout is helpful for developers new
to Django to understand how to structure the directories and files
within apps for projects.
• Django for Beginners: Build websites with Python and Django by
William S. Vincent is perfect if you are just getting started with Django
and web development, taking you from total beginner to confident web
developer with Django and Python.
Django videos
Are you looking for Django videos in addition to articles? There is a special
section for Django and web development on the best Python videos page.
Intermediate and advanced Django topics
These books and tutorials assume that you know the basics of building Django
and want to go further to become much more knowledgeable about the
framework.
• 2 Scoops of Django by Daniel Greenfeld and Audrey Roy is well worth
the price of admission if you're serious about learning how to correctly
develop Django websites.
200
Full Stack Python: 2020 Supporter's Edition
• The Test-Driven Development with Django, Django REST Framework,
and Docker course details how to set up a development environment
with Docker in order to build and deploy a RESTful API powered by
Python, Django, and Django REST Framework.
• User Interaction With Forms explains general web form input, how
Django handles forms via POST requests, different types of input such
as CharFields, DateFields and EmailFields, and validating that input.
• This 3-part Django project optimization guide covers a wide range of
advanced topics such as Profiling and Django settings, working with
databases and caching.
• Caching in Django is a detailed look at the configuration required for
caching and how to measure the performance improvements once you
have it in place.
• Mental Models for Class Based Views provides some comparison points
between class based views (CBVs) and function based views and the
author's opinions for how you can better understand CBVs.
• Working with time zones is necessary for every web application. This
blog post on pytz and Django is a great start for figuring out what you
need to know.
• A Guide to ASGI in Django 3.0 and its Performance covers the new
Asynchronous Server Gateway Interface (ASGI) that was introduced in
Django 3.0 and explains some of the nuances and gotchas that you
should consider if you decide to use it for your web apps.
• REST APIs with Django: Build powerful web APIs with Python and
Django by William S. Vincent is the book for you if you are just moving
beyond the basics of Django and looking to get up speed with Django
REST Framework (DRF) and service-oriented architecture (SOA). It
also dives into more advanced topics like token-based authentication
and permissions.
• Django Stripe Tutorial details how to quickly add Stripe to accept
payments in a Django web app.
• This Python Social Auth for Django tutorial will show you how to
integrate social media sign in buttons into your Django application.
201
Full Stack Python: 2020 Supporter's Edition
• Upgrading Django provides a version-by-version guide for updating
your Django projects' code.
• The Django Admin Cookbook and Building Multi Tenant Applications
with Django are two solid "code recipe-style" free open source books
that will teach you more about the admin interface as well as building
projects that will be used by more than a single customer so their data
needs to be properly separated.
• How to Create Custom Django Management Commands explains how
to expand the default manage.py commands list with your own
custom commands in your projects. The tutorial has a bunch of great
examples with expected output to make it easy to follow along and
learn while you work through the post.
• Luke Plant writes about his approach to class based views (CBVs),
which often provoke heated debate in the Django community for
whether they are a time saver or "too much magic" for the framework.
• Django Apps Checklist gives some good practices rules for building
reusable Django apps.
Django migrations
• Paul Hallett wrote a detailed Django 1.7 app upgrade guide on the
Twilio blog from his experience working with the django-twilio
package.
• Real Python's migrations primer explores the difference between
South's migrations and the built-in Django 1.7 migrations as well as
how you use them.
• Andrew Pinkham's "Upgrading to Django 1.7" series is great learning
material for understanding what's changed in this major release and
how to adapt your Django project. Part 1, part 2 and part 3 and part 4
are now all available to read.
• Django migrations without downtimes shows one potential way of
performing on-line schema migrations with Django.
202
Full Stack Python: 2020 Supporter's Edition
• How to Extend Django User Model presents four main ways to expand
upon the built-in User model that is packaged with Django. This
scenario is very common for all but the simplest Django projects.
• Creating a Custom User Model in Django looks at how to create a
custom User model in Django so that an email address can be used as
the primary user identifier instead of a username for authentication.
Django Channels
Channels are a new mechanism in Django 1.9 provided as a standalone app.
They may be incorporated into the core framework in 2.0+. Channels provide
"real-time" full-duplex communication between the browser and the server
based on WebSockets.
• This tutorial shows how to get started with Django Channels in your
project.
• The channels examples repository contains a couple of good starter
projects such as a live blog and a chat application to use as base code.
• The Developing a Real-Time Taxi App with Django Channels and
Angular course details how to create a ride-sharing app with Django
Channels, Angular, and Docker. Along the way, you'll learn how to
manage client/server communication with Django Channels, control
flow and routing with Angular, and build a RESTful API with Django
REST Framework.
Django testing
• Integrating Front End Tools with Django is a good post to read for
figuring out how to use Gulp for handling front end tools in
development and production Django sites.
• Django Testing Cheat Sheet covers many common scenarios for Django
applications such as testing POST requests, request headers,
authentication, and large numbers of model fields in the Django ORM.
• Getting Started with Django Testing will help you stop procrastinating
on testing your Django projects if you're uncertain where to begin.
203
Full Stack Python: 2020 Supporter's Edition
• Testing in Django provides numerous examples and explanations for
how to test your Django project's code.
• Django views automated testing with Selenium gives some example
code to get up and running with Selenium browser-based tests.
Django with JavaScript MVC frameworks
There are resources for JavaScript MVC frameworks such as Angular, React
and Vue.js on their respective pages.
Django ORM tutorials
Django comes with its own custom object-relational mapper (ORM) typically
referred to as "the Django ORM". Learn more about the Django ORM on the
its own page and more broadly about ORMs on the Python object-relational
mappers page.
Static and media files
Deploying and handling static and media files can be confusing for new
Django developers. These resources along with the static content page are
useful for figuring out how to handle these files properly.
• Using Amazon S3 to Store your Django Site's Static and Media Files is a
well written guide to a question commonly asked about static and
media file serving.
• Loading Django FileField and ImageFields from the file system shows
how to load a model field with a file from the file system.
• Storing Django Static and Media Files on Amazon S3 shows how to
configure Django to load and serve up static and media files, public and
private, via an Amazon S3 bucket.
Django project templates
Project templates, not to be confused with a template engine, generate
boilerplate code for a base Django project plus optional libraries that are often
used when developing web applications.
• Caktus Group's Django project template is Django 2.2+ ready.
204
Full Stack Python: 2020 Supporter's Edition
• Cookiecutter Django is a project template from Daniel Greenfeld, for
use with Audrey Roy's Cookiecutter. The template results are Heroku
deployment-ready.
• Two Scoops Django project template is also from the PyDanny and
Audrey Roy. This one provides a quick scaffold described in the Two
Scoops of Django book.
• Sugardough is a Django project template from Mozilla that is
compatible with cookiecutter.
Open source Django example projects
Reading open source code can be useful when you are trying to figure out how
to build your own projects. This is a short list of some real-world example
applications, and many more can be found on the Django example projects
and code page.
• Browser calls with Django and Twilio shows how to build a web app
with Django and Twilio Client to turn a user's web browser into a fullfledged phone. Pretty awesome!
• Openduty is a website status checking and alert system similar to
PagerDuty.
• Courtside is a pick up sports web application written and maintained
by the author of PyCoder's Weekly.
• These two Django Interactive Voice Response (IVR) system web
application repositories part 1 and part 2 show you how to build a really
cool Django application. There's also an accompanying blog post with
detailed explanations of each step.
• Taiga is a project management tool built with Django as the backend
and AngularJS as the front end.
Open source code to learn Django
There are many open source projects that rely on Django. One of the best
ways to learn how to use this framework is to read how other projects use it in
real-world code. This section lists these code examples by class and method in
Django's code base.
205
Full Stack Python: 2020 Supporter's Edition
Flask
Flask (source code) is a Python web framework built with a small core and
easy-to-extend philosophy.
Why is Flask a good web framework choice?
Flask is considered more Pythonic than the Django web framework because in
common situations the equivalent Flask web application is more explicit.
Flask is also easy to get started with as a beginner because there is little
boilerplate code for getting a simple app up and running.
For example, here is a valid "Hello, world!" web application with Flask:
from flask import Flask
app = Flask(__name__)
@app.route('/')
def hello_world():
return 'Hello, World!'
if __name__ == '__main__':
app.run()
The above code shows "Hello, World!" on localhost port 5000 in a web
browser when run with the python app.py command and the Flask library
installed.
The equivalent "Hello, World!" web application using the Django web
framework would involve significantly more boilerplate code.
Flask was also written several years after Django and therefore learned from
the Python community's reactions as the framework evolved. Jökull Sólberg
206
Full Stack Python: 2020 Supporter's Edition
wrote a great piece articulating to this effect in his experience switching
between Flask and Django.
How does Flask relate to the Pallets Projects?
Flask was originally designed and developed by Armin Ronacher as an April
Fool's Day joke in 2010. Despite the origin as a joke, the Flask framework
became wildly popular as an alternative to Django projects with their
monolithic structure and dependencies.
Flask's success created a lot of additional work in issue tickets and pull
requests. Armin eventually created The Pallets Projects collection of open
source code libraries after he had been managing Flask under his own GitHub
account for several years. The Pallets Project now serves as the communitydriven organization that handles Flask and other related Python libraries such
as Lektor, Jinja and several others.
Flask tutorials
The "Hello, World!" code for Flask is just seven lines of code but learning how
to build full-featured web applications with any framework takes a lot of work.
These resources listed below are the best up-to-date tutorials and references
for getting started.
• The Flask mega tutorial by Miguel Grinberg is a perfect starting
resource for using this web framework. Each post focuses on a single
topic and builds on previous posts. The series includes 18 parts: #1
Hello World, #2 Templates, #3 Web Forms, #4 Database, #5 User
Logins, #6 Profile Page and Avatars, #7 Unit Testing, #8 Followers,
Contacts, and Friends, #9 Pagination, #10 Full Text Search, #11 Email
Support, #12 Facelift, #13 Dates and Times, #14 I18n and L10n, #15
Ajax, #16 Debugging, Testing and Profiling, #17 Deployment on Linux
and #18 Deployment on the Heroku Cloud. Miguel also wrote and
recorded numerous Flask Web Development content including a great
book and video book that are excellent resources worth the price,
especially to support his continuous revisions to the content.
• Armin Ronacher, the creator of Flask, presented the technical talk
Flask for Fun and Profit at PyBay 2016 where he discusses using the
framework to build web apps and APIs.
207
Full Stack Python: 2020 Supporter's Edition
• Explore Flask is a public domain book that was previously backed on
Kickstarter and cost money for about a year before being open sourced.
The book explains best practices and patterns for building Flask apps.
• Learn to Build Web Applications with Flask and Docker is a video
course by Nick Janetakis that shows how to build a Software-as-aService (SaaS) application that he open sourced which uses Flask for
the web framework and Docker for the local development environment.
• Flask by Example: Part 1 shows the basic first steps for setting up a
Flask project. Part 2 explains how to use PostgreSQL, SQLAlchemy and
Alembic. Part 3 describes text processing with BeautifulSoup and
NLTK. Part 4 shows how to build a task queue with Flask and Redis.
• The blog post series "Things which aren't magic" covers how Flask's
ubiquitous @app.route decorator works under the covers. There are
two parts in the series, part 1 and part 2.
• How to Structure Large Flask Applications covers a subject that comes
up quickly once you begin adding significant functionality to your Flask
application.
• Flask Blueprint templates shows a way of structuring your
__init__.py file with blueprints for expanding projects into many
files and modules.
• If you're not sure why DEBUG should be set to False in a production
deployment, be sure to read this article on how Patreon got hacked.
• Developing a Single Page App with Flask and Vue.js step-by-step
walkthrough of how to set up a basic CRUD app with Vue and Flask.
Intermediate to advanced Flask resources
Once you move past the beginner tutorials and have created a few Flask
projects you will want to learn how to use Flask extensions, deploy your code
and integrate web APIs to build more extensive functionality. The following
tutorials will guide you through more advanced topics and provide solid
learning materials, especially when combined with the example real-world
projects listed in the next section.
208
Full Stack Python: 2020 Supporter's Edition
• Microservices with Docker, Flask, and React is an awesome course for
beyond-the-basics work with Flask. There are a couple of free chapters
and the rest of the course is well worth paying for to learn a bunch of
valuable tools such as Docker, React and microservices architectures.
• Visualize your trip with Flask and Mapbox along with the open source
flask_mapbox GitHub repository provides a fantastic example
visualization of a trip to Iceland with Flask as the backend web
framework.
• Microservices with Flask, Docker, and React teaches how to spin up a
reproducible Flask development environment with Docker. It shows
how to deploy it to an Amazon EC2 instance then scale the services on
Amazon EC2 Container Service (ECS).
• Build a Video Chat Application with Python, JavaScript and Twilio
Programmable Video shows how to use Twilio Programmable Video to
build cross-platform (web, iOS and Android) video into Flask
applications.
• Why and how to handle exceptions in Python Flask has some great
example code and reasons why you should code defensively by
anticipating and handling the unhappy path exceptions in your Flask
applications. The examples are relevant to any web framework you will
use and are easy to copy and paste to test in your own applications.
• The Flask Extensions Registry is a curated list of the best packages that
extend Flask. It's the first location to look through when you're
wondering how to do something that's not in the core framework.
• How I Structure My Flask Application walks through how this
developer organizes the components and architecture for his Flask
applications.
• Adding phone calling to your web application is a killer Flask tutorial
with all the code needed to create a web app that can dial phones and
receive inbound calls.
• Jeff Knupp provides some solid advice on how to productionize a Flask
app.
209
Full Stack Python: 2020 Supporter's Edition
• If you're looking for a fun tutorial with Flask and WebSockets, check
out my blog post on creating Choose Your Own Adventure
Presentations with Reveal.js, Python and WebSockets. Follow up that
tutorial by building an admin interface in part 1, part 2 and part 3
that'll show you how to use forms and SQLAlchemy. There is also a
companion open source GitHub repository for the app with tags for
each step in the blog posts.
• One line of code cut our Flask page load times by 60% is an important
note about optimizing Flask template cache size to dramatically
increase performance in some cases.
• Unit Testing Your Twilio App Using Python’s Flask and Nose covers
integrating the Twilio API into a Flask application and how to test that
functionality with nose.
• The Flask documentation has some quick examples for how to deploy
Flask with standalone WSGI containers.
• Serverless Python Web Applications With AWS Lambda and Flask is a
spectacular post that walks through how to run Flask applications on
AWS Lambda's serverless offering. The tutorial has instructions on
how to include application dependencies and handle your deployment
workflow.
• Visualize your trip with Flask and Mapbox uses geographic GeoJSON
data and presents it in a Flask application that uses Mapbox.
• Handling Email Confirmation in Flask is a great walkthrough for a
common use case of ensuring an email address matches with the user's
login information.
• Static websites with Flask shows how to use Flask with Frozen-Flask to
generate a static website from a backend data source.
• Running Flask on Docker Swarm details how to run a Flask app on
Docker Swarm.
• Running Flask on Kubernetes step-by-step walkthrough of how to
deploy a Flask-based microservice (along with Postgres and Vue.js) to a
Kubernetes cluster.
210
Full Stack Python: 2020 Supporter's Edition
• Dynamic Secret Generation with Vault and Flask looks at how to use
Hashicorp's Vault and Consul to create dynamic Postgres credentials
for a Flask web app.
Open source Flask example projects
Flask's lack of standard boilerplate via a commandline interface for setting up
your project structure is a double edged sword. When you get started with
Flask you will have to figure out how to scale the files and modules for the
code in your application. The following open source projects range from
simple to complex and can give you ideas about how to working on your
codebase.
• Skylines is an open source flight tracking web application built with
Flask. You can check out a running version of the application.
• Flask JSONDash is a Flask blueprint that creates JavaScript Object
Notiation (JSON) APIs for data dashboards.
• Microblog is the companion open source project that goes along with
Miguel Grinberg's O'Reilly Flask book.
• Flaskr TDD takes the official Flask tutorial and adds test driven
development and JQuery to the project.
• Charles Leifer (author of Peewee and Pony ORM) built a note-taking
app along with the source code in Gists.
• Reddit Job Search uses the Reddit API for a jobs data set and presents
them via a Flask web app.
• Bean Counter is an open source Flask app for tracking coffee.
• FlaskBB is a Flask app for a discussion forum.
• psdash is an app built with Flask and psutils to display information
about the computer it is running on.
Flask project templates
Flask's wide array of extension libraries comes at the cost of having a more
complicated project setup. The following project templates provide a starter
211
Full Stack Python: 2020 Supporter's Edition
base that you can either use for your own applications or just learn various
ways to structure your code.
• Use the Flask App Engine Template for getting set up on Google App
Engine with Flask.
• Flask Foundation is a starting point for new Flask projects. There's also
a companion website for the project that explains what extensions the
base project includes.
• Cookiecutter Flask is a project template for use with Cookiecutter.
• Flask-Boilerplate provides another starting project with sign up, log in
and password reset.
• The company Sunscrapers provides this Flask boilerplate project with
SQLAlchemy, py.test and Celery baked into the Flask project structure.
• flask-webpack-cookiecutter combines a Flask framework project
structure with Webpack, a module bundler frequently used in the
JavaScript world.
Open source code for learning Flask
There are many open source projects that rely on Flask to operate. One of the
best ways to learn how to use this framework is to read how other projects use
it in real-world code. This section lists these code examples by class and
method in Flask.
Bottle
Bottle (source code) is a WSGI-compliant single source file web framework
with no external dependencies other than the Python standard library (stdlib).
212
Full Stack Python: 2020 Supporter's Edition
Should I use Bottle for web development?
Bottle is awesome for a few web development situations:
1. Prototyping ideas
2. Learning how web frameworks are built
3. Building and running simple personal web applications
Prototyping
Prototyping simple ideas is often easier with Bottle than a more opinionated
web framework like Django because Django projects start with a significant
amount of boilerplate code. The Model-View-Template structure for Django
apps within projects makes maintaining projects easier, but it can be
cumbersome on starter projects where you're just playing with random ideas
so you aren't worried about your application's long-term code structure.
Learning about frameworks
Bottle is contained within a single large source file named bottle.py so it
provides great reading when learning how WSGI web frameworks work.
Everything you need to learn about how your web application's code connects
with the Bottle framework is contained within that single source code.
Personal projects
Personal projects can be deployed with Bottle as the only dependency. If
you've never performed a Python web app deployment before, the number of
concepts and steps can be daunting. By packaging bottle.py with your
app's source code, you can skip some of the steps to more easily get your web
application up and running.
Bottle resources
• Configuring Python 3, Bottle and Gunicorn for Development on
Ubuntu 16.04 LTS is a quick tutorial for getting an out-of-the-box
default Ubuntu 16.04 image ready for Bottle development with Green
Unicorn as the WSGI server.
• Check out these Full Stack Python Bottle tutorials that'll teach you how
to write a few small but very useful Bottle web apps:
◦ Creating Bar Chart Visuals with Bokeh, Bottle and Python 3
◦ Dialing Outbound Phone Calls with a Bottle Web App
213
Full Stack Python: 2020 Supporter's Edition
◦ How to Monitor Python / Bottle Web Apps
◦ Replying to SMS Text Messages with Python and Bottle
• Digital Ocean provides an extensive introductory post on Bottle.
• First Steps with Python and Bottle is a quick 4 minute introduction that
I created for developers so they can get the simplest possible Bottle web
app running. There is also a companion blog post with the code.
• Getting Started with Python, Bottle and Twilio SMS / MMS shows how
to build a simple Bottle web application that can send and receive text
and picture messages. How to Make and Receive Phone Calls with
Python, Bottle and Twilio Voice is a similar beginner's tutorial for
handling phone calls with a Bottle app using Twilio.
• Developing With Bottle details how to create a basic application with
Bottle.
• The official Bottle tutorial provides a thorough view of basic concepts
and features for the framework.
• Running a Bottle app with Gunicorn shows how to execute a simple
Bottle web app with Green Unicorn.
• Here's a short code snippet for creating a RESTful API with Bottle and
MongoDB.
• Bottle, full stack without Django does a nice job of connecting
SQLAlchemy with Bottle and building an example application using the
framework.
• Using bottle.py in Production has some good tips on deploying a Bottle
app to a production environment.
• Jinja2 Templates and Bottle shows how to use Jinja instead of the
built-in templating engine for Bottle page rendering.
• Python patterns contains a setup that combines Bottle, Celery and
Peewee as the developer's choice for backend web development. The
developer also uses Vim as the primary editor for working with Bottle.
Open source Bottle example projects
• Pattle is a pastebin clone built with Bottle.
214
Full Stack Python: 2020 Supporter's Edition
• Decanter is a library for structuring Bottle projects.
• compare-python-web-frameworks provides an example application
using Bottle as one of the implementations.
Bottle framework learning checklist
1. Download Bottle or install via pip with pip install bottle on your
local development machine.
2. Work through the official Bottle tutorial.
3. Start coding your Bottle app based on what you learned in the official
tutorial plus reading open source example applications found above.
4. Move on to the deployment section to get your initial Bottle application
on the web.
Pyramid
Pyramid is an open source WSGI web framework based on the Model-ViewController (MVC) architectural pattern.
Open source Pyramid example apps
These projects provide solid starting code to learn from as you are building
your own applications.
• pyramid_blogr is an example project that shows how to build a blog
with Pyramid modeled on the Flaskr tutorial.
215
Full Stack Python: 2020 Supporter's Edition
• pyramid-blogr-cf is another Pyramid web app that has a similar title to
the one above but this one is intended for teaching web development to
new developers.
• pyramid_appengine provides a project skeleton for running Pyramid
on Google App Engine.
Pyramid-specific packages
The following packages are designed to make Pyramid play nicely with
existing open source libraries by reducing the boilerplate you need to add to
your project.
• pyramid_celery and pyramid_rq make it easier to use the Celery task
queue in your Pyramid applications for handling asynchronous work.
• pyramid_zipkin provides distributed tracing via the Zipkin library.
• Ramses (source code) is a RESTful web API generation framework,
similar in concept (but not in implementation details) to how Django
REST Framework works with Django.
Pyramid resources
Pyramid has fantastic official project documentation on its site. Other
resources are harder to come by compared to other established web
frameworks such as Django and Flask, but there are enough tutorials out
there for you to learn Pyramid if you choose to build your web applications
with it.
• Try Pyramid is the official marketing website for Pyramid, with
resources for extending your Pyramid apps. It also provides some
sample "hello world!" code.
• An introduction to the Pyramid web framework for Python provides a
detailed configuration and more than trivial code to build a "to do"
application. This post is one in a four part series that compares
Pyramid to Flask, Django and Tornado so there is some commentary
on how this framework compares to the others.
• The first Pyramid app is a good place to start getting your hands dirty
with an example project.
216
Full Stack Python: 2020 Supporter's Edition
• Six Feet Up explains why Pyramid is their choice for rapid development
projects in that blog post.
• Build a chat app with Pyramid, SQLDB, and Bluemix is a Pyramid
application walkthrough specific to IBM's Bluemix platform.
• Developing Web Apps Using the Python Pyramid Framework is a video
from San Francisco Python with an overview of how to install, get
started and build a web app with the Pyramid framework.
• This podcast interview with the primary author of the Pyramid
framework explains how Pyramid sprang from Pylons and how
Pyramid compares to other modern frameworks.
• Anyone using the Pyramid framework? is a good /r/Python thread with
responses by current users as well as frustrations by developers who
tried and decided against sticking with the framework.
TurboGears
TurboGears, born as a full stack layer on top of Pylons, is now a standalone
WSGI web framework that can act both as a full stack framework (like
Django) or as a micro framework (like Flask)
Originally inspired by RubyOnRails it's based on MVC where the controller
dispatches the request to a set of actions exposed from the controller itself.
TurboGears, in its full stack mode, provides all the features you would require
during development of a web application:
•
•
•
•
Identification and Authentication
Authorization
Autogenerated Admin and CRUD
Sessions
217
Full Stack Python: 2020 Supporter's Edition
•
•
•
•
•
•
•
Caching
Schema Migrations
Master/Slave Database Queries Balancing
Request Bound Transactions
Interactive Debugger
Builtin Profiling
Pluggable Applications
It's also one of the few web frameworks officially supporting MongoDB as one
of the primary storage backends, including support into the TurboGears
Admin to autogenerate CRUDs from MongoDB models.
While TurboGears has always been a full stack framework with same scope of
projects like Django, it differentiates from other frameworks due the its
philosophy on two major parts of a web framework: Templating and Routing
Templating
While TurboGears provides support for multiple template engines, the
primary one has always been a fully validated XML template engine.
Currently TurboGears ships with the Kajiki template engine, which was
developed within the project itself, but in the past it relied on the Genshi and
Kid template engines which were mostly syntax compatible with Kajiki.
Historically validated xml template engines has always been slower than text
template engines, but the Kajiki project was able to create a very fast template
engine that usually renders faster than Mako or Django Template while still
retaining all the expected features.
The fact that it relies on a validated XML engine provides some benefits
compared to plain text engines like Django Template, Jinja2 and Mako:
Automatic Escaping
It automatically escapes content rendered into the template, thus making
easier to avoid XSS and injection security issues:
<div>${value}</div>
with value='<script>alert("hello")</script>' will render as
218
Full Stack Python: 2020 Supporter's Edition
<div>&lt;script&gt;alert(&quot;hello&quot;)&lt;/script&gt;</div>;
thus preventing any form of injection from user provided content.
Automatic Internationalization
The template engine parses the provided template document and recognises
the nodes that contain static text.
As the engine is able to distinguish text from markup it's able to flag the text
for translation.
Content like <div>Hello World</div> would get automatically translated if
a translation for "Hello World" is provided, without having to wrap text in
gettext calls.
Compatibility with WYSIWYG Editors
As the template engine syntax is purely valid XHTML the template itself can
be opened with WYSIWYG editors and as far as they don't strip unknown
attributes the template can be edited and saved back from those editors.
Routing
Most web frameworks have been relying on regular expressions to declare
routing, through decorators or through a routing map.
TurboGears supports regular expressions through the tgext.routes
extension, but the preferred way of routing is through the Object Dispatch
system.
In Object Dispatch a root controller object is traversed while resolving the
URL. Each part of the url path is mapped to a property of the controller
(Which might point to a sub controller) until a final callable action is
encountered.
This leads to a very natural mapping between URLs and the code serving
them, allowing people with minimal knowledge of a project to jump in and
quickly find actions in charge of serving a specific page.
In Object Dispatch an URL like /users/new?name=MyName would be served
by a hierarchy of objects like:
219
Full Stack Python: 2020 Supporter's Edition
class UsersController(TGController):
@expose()
def new(self, name=None):
return 'Hi, %s' % name
class RootController(TGController):
users = UsersController()
It's easy to see how /users/new actually resolves to
RootController.users.new and all options provided to the URL are passed
to the action serving the response as arguments.
TurboGears Resources
• TurboGears Introduction Video An overview of TurboGears2 features
presented at the PyConWeb
• TurboGears Documentation The official TurboGears documentation
• Microframework Mode Tutorial The official tutorial that focuses on
starting TurboGears in microframework mode and leads to
developement of a single file web application
• FullStack Tutorial The Wiki in 20 minutes tutorial that showcases how
to create a fully functional wiki application with TurboGears in full
stack mode.
• The CogBin The CogBin is a list of the most common pluggable
applications for TurboGears, it enlists ready made pieces you can plug
into your web application to provide features like Facebook Login,
Comments, Registration and so on...
• React in Pure Python An article showcasing how to create web
applications relying on React without the need to have NodeJS
installed at all. The article uses TurboGears as the web framework to
develop the example application.
• Turbogears and the future of Python web frameworks is a Talk Python
to Me podcast episode featuring an interview with one of the
TurboGears core developers.
220
Full Stack Python: 2020 Supporter's Edition
Falcon
Falcon is a WSGI-compliant web framework designed to build RESTful APIs
without requiring external code library dependencies.
Falcon resources
• Building Very Fast App Backends with Falcon Web Framework on
PyPy provides a walkthrough of a web API in Falcon that runs with
PyPy and Nginx.
• Asynchronous Tasks with Falcon and Celery shows how to configure
Celery with the framework.
• The official Falcon tutorial has a meaty guide for building and
deploying your first Falcon web application.
• Building Scalable RESTful APIs with Falcon and PyPy shows a to-do
list example with Falcon running on PyPy.
Morepath
Morepath is a micro web framework with a model-driven approach to creating
web applications and web APIs.
221
Full Stack Python: 2020 Supporter's Edition
Morepath's framework philosophy is that the data models should drive the
creation via the web framework. By default the framework routes URLs
directly to model code, unlike for example Django which requires explicit URL
routing by the developer.
Why is Morepath an interesting web framework?
Simple CRUD web applications and APIs can be tedious to build when they
are driven straight from data models without much logic between the model
and the view. Learn more about how Morepath compares with other web
frameworks from the creator.
With the rise of front end JavaScript frameworks, many Python web
frameworks are first being used to build RESTful APIs that return JSON
instead rendering HTML via a templating system. Morepath appears to have
been created with the RESTful API model approach in mind and cuts out the
assumption that templates will drive the user interface.
Morepath resources
Morepath, with its first commit using the Morepath name in 2013, is a much
newer web framework than Django, Flask or Pyramid, which results in fewer
tutorials. There is also a lot of opportunity for newer Python developers to fill
the gaps with their own Morepath tutorials. However, these resources below
are a good place to get started.
• On the Morepath is a blog post by Startifact on how they use Morepath
and some of the features of the framework.
• Build a better batching UI with Morepath and Jinja2 is an introductory
post on building a simple web application with the framework. The
code for the application is also open source and available on GitHub.
• podcast.__init__ interviewed Martijn Faassen about Morepath and he
described what makes the framework different from other existing web
frameworks as well as why someone should be convinced to switch for
a new project.
• Morepath's creator gave a great talk on the motivation and structure
for the new framework at EuroPython 2014.
222
Full Stack Python: 2020 Supporter's Edition
• Is Morepath Fast Yet? includes some benchmarks and discusses
performance implications of using a Python-based web framework for
your web application.
• The Morepath-cookiecutter project handles project creation templates
using
Cookiecutter and recommended file structure from the Morepath
documentation.
Sanic
Sanic is a Python web framework built on uvloop and designed for fast HTTP
responses via asynchronous request handling.
What are the tradeoffs of using Sanic?
Sanic cannot be developed or deployed on Windows due to its necessary
uvloop dependency.
There was an excellent discussion on the /r/python subreddit about using one
of the newer async frameworks such as Sanic or Japronto compared with a
traditional web framework like Django. One of the major tradeoff of adopting
a newer framework is simply that the code library ecosystem has not, and may
never, grow up around that framework. You have to accept the risk that you
will need to build a significant amount of the plumbing yourself rather than
pip installing existing, well-tested libraries.
223
Full Stack Python: 2020 Supporter's Edition
Sanic tutorials
Sanic is under very active development and is still in its infancy as a web
framework. The following tutorials will get you started but there is a chance
you will have to work through errors as Sanic is regularly updated.
• Getting started with Sanic: the asynchronous, uvloop based web
framework for Python 3.5+ is a "Hello, World!" style post for the
framework and also shows how to respond to SMS text messages using
Twilio.
• Fixing bugs and handling 186k requests/second using Python is a fun
benchmarking exercise that a developer ran when testing out Sanic on
a Digital Ocean droplet.
• Exploring Asyncio - uvloop, sanic and motor explains why asyncio is
important to the Python community and how uvloop & sanic fit into
the bigger picture.
• Python Sanic Tutorial is a video tutorial on how to write your first
Sanic web apps.
• A Guide to Instrumenting Sanic Applications, Part 1 shows how to add
Prometheus-based monitoring to Sanic applications.
Sanic open source projects and examples
There are not many example applications and extensions for Sanic compared
to Flask, Django or other web frameworks because Sanic is still so new.
However, there are some initial projects that are useful for figuring out how to
build your first applications with this framework.
• Gutenberg-HTTP is a web application and API built with Sanic. It's a
solid clean example of how to build a decent-sized project with Sanic.
There is even a demo that was deployed to Azure to show how it works.
• Practical Log Viewers with Sanic and Elasticsearch - Designing CI/CD
Systems shows how to build a log viewer using Sanic that collects data
from various Docker containers being created through a build system.
• Sanic comes with a slew of examples in the official repository.
224
Full Stack Python: 2020 Supporter's Edition
• Sanic starter bundles Sanic with SQLAlchemy and Alembic (for data
migrations) as a starter project.
• Sanic-limiter is an extension for rate-limiting the number of requests
from a single user on Sanic APIs.
• Sanic-GraphQL adds GraphQL support to a Sanic web application.
• Sanic OpenAPI provides a user interface for Sanic APIs.
• This Sanic & Nginx & docker-compose example has boilerplate code for
setting up a Sanic project using Docker and Nginx.
• Sanic JWT (source code) adds support for authentication via JSON
Web Tokens (JWT).
Other Web Frameworks
Python has a significant number of newer and less frequently-used web
frameworks that are still worth your time to investigate. The list on this page
does not include the following web frameworks that have their own dedicated
pages:
• Django
• Flask
• Pyramid
• TurboGears
• Bottle
• Falcon
• Morepath
• Sanic
web.py
web.py is a Python web framework designed for simplicity in building web
applications.
• See this Reddit discussion on reasons why to not use web.py for some
insight into the state of the project.
225
Full Stack Python: 2020 Supporter's Edition
web2py
Web2py is a batteries-included philosophy framework with project structure
based on model-view-controller patterns.
CherryPy
CherryPy is billed as a minimalist web framework, from the perspective of the
amount of code needed to write a web application using the framework. The
project has a long history and made a major transition between the second
and third release.
Muffin
Muffin is a web framework built on top of the asyncio module in the Python
3.4+ standard library. Muffin takes inspiration from Flask with URL routes
defined as decorators upon view functions. The Peewee ORM is used instead
of the more common SQLAlchemy ORM.
Ray
Ray is a framework for building RESTful APIs, similar to Falcon. The
introductory post provides some initial code to get started with creating
endpoints, adding authentication and protecting against malicious clients.
Vibora
Vibora is an asynchronous model framework similar to Sanic that was
inspired by Flask's syntax. However, the framework's author rewrote many
parts like the template engine to maximize performance.
Pecan
Pecan is inspired by CherryPy and TurboGears. It purely focuses on HTTP
requests and responses via Python objects and does not integrate session
handling or database access.
Masonite
Masonite is a modern, developer centric, batteries-included Python web
framework. It uses the MVC (Model-View-Controller) architecture pattern
226
Full Stack Python: 2020 Supporter's Edition
and comes with a lot of functionality out of the box with an extremely
extendable architecture.
Check out the following resources to lean more:
1. 5 reasons why people are choosing Masonite over Django
2. MasoniteCasts
3. Dockerizing Masonite with Postgres, Gunicorn, and Nginx
Other web framework resources
• This roundup of 14 minimal Python frameworks contains both familiar
and less known Python libraries.
• The web micro-framework battle presentation goes over Bottle, Flask,
and many other lesser known Python web frameworks.
• A Python newcomer asked the Python Subreddit to explain the
differences between numerous Python web frameworks and received
some interesting responses from other users.
Other frameworks learning checklist
1. Read through the web frameworks listed above and check out their
project websites.
2. It's useful to know what other web frameworks exist besides Django
and Flask. However, when you're just starting to learn to program
there are significantly more tutorials and resources for Django and
Flask on the web. My recommendation is to start with one of those two
frameworks then expand your knowledge from there.
Template Engines
Template engines take in tokenized strings and produce rendered strings with
values in place of the tokens as output. Templates are typically used as an
intermediate format written by developers to programmatically produce one
or more desired output formats, commonly HTML, XML or PDF.
227
Full Stack Python: 2020 Supporter's Edition
Why are template engines important?
Template engines allow developers to generate desired content types, such as
HTML, while using some of the data and programming constructs such as
conditionals and for loops to manipulate the output. Template files that are
created by developers and then processed by the template engine consist of
prewritten markup and template tag blocks where data is inserted.
For example, look at the first ten source lines of HTML of this webpage:
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="author" content="Matt Makai">
<meta name="description" content="Template engines provide programmatic output of formatted conte
nt such as HTML, XML or PDF.">
<link rel="shortcut icon" href="//static.fullstackpython.com/fsp-fav.png">
Every one of the HTML lines above is standard for each page on Full Stack
Python, with the exception of the <meta name="description"... line
which provides a unique short description of what the individual page
contains.
The base.html Jinja template used to generate Full Stack Python allows every
page on the site to have consistent HTML but dynamically generate the pieces
that need to change between pages when the static site generator executes.
The below code from the base.html template shows that the meta
description is up to child templates to generate.
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="author" content="Matt Makai">
{% block meta_header %}{% endblock %}
<link rel="shortcut icon" href="//static.fullstackpython.com/fsp-fav.png">
In a typical WSGI application, the template engine would generate the HTML
output response when an HTTP request comes in for a particular URL.
228
Full Stack Python: 2020 Supporter's Edition
Python template engines
There are several popular Python template engines. A template engine
implementation will fall somewhere on the spectrum between allowing
arbitrary code execution and granting only a limited set of capabilities via
template tags. A rough visual of the code in template spectrum can be seen
below for four of the major Python template engines.
Jinja (Jinja2)
Jinja, also known and referred to as "Jinja2", is a popular Python template
engine written as a self-contained open source project. Some template
engines, such as Django templates are provided as part of a larger web
framework, which can make them difficult to reuse in projects outside their
coupled library.
Major Python open source applications such as the configuration
management tools Ansible and SaltStack as well as the static site generator
Pelican use the Jinja template engine by default for generating output files.
There is a whole lot more to learn about Jinja on the Jinja2 page.
Django templating
Django comes with its own template engine in addition to supporting (as of
Django 1.9) drop-in replacement with other template engines such as Jinja.
Mako template engine
Mako was the default templating engine for the Pylons web framework and is
one of many template engines supported by Pyramid. Mako has wide support
as a replacement template engine for many other web frameworks as well.
229
Full Stack Python: 2020 Supporter's Edition
Other Python template engine implementations
There are numerous Python template engine implementations that range
from weekend hacks to actively developed mature libraries. These template
engines are listed alphabetically:
• Chameleon is an HTML and XML template engine that supports both
Python 2 and 3.
• Cheetah
• Diazo
• evoque
• Genshi
• Juno
• Myghty
• pyratemp
• pystache
Template engine implementation comparisons
There are many Python template engine implementations in addition to the
ones listed above. These resources can help you select a Python template
engine implementation that works well for your project.
• This template engines site contains a range of information from what
templates engines are to listing more esoteric Python template engines.
• Python Template Engine Comparison is an older but still relevant post
by the creator of Jinja that explains why and how he switches between
Mako, Jinja and Genshi for various projects he works on.
• Python Web Frameworks: What are the advantages and disadvantages
of using Mako vs. Jinja2? has some good answers from developers on
Quora about using Mako compared with Jinja2.
Template engine resources
Template engines are often used with web frameworks a black box where
input goes in, and rendered text magically appears out the other side.
230
Full Stack Python: 2020 Supporter's Edition
However, when something unexpected returns from a template engine it is
useful to know how they work to aid your debugging. The following resources
examine existing template engine design as well as how to build your own
engine when that's necessary for your projects.
• How a template engine works uses the template module in Tornado as
an example to step through how a template engine produces output,
from parsing the incoming string to rendering the final output.
• A template engine in 500 lines or less is an article by Ned Batchelder
provides a
template engine in 252 lines of Python that can be used to understand
how template engines work under the cover.
• The world's simplest Python template engine shows how the
format() function can implement a simple template engine with
conditionals, loops and method invocations.
• When to use a Templating Engine (in Python)? is a Stack Overflow
question with a useful answer on why and when to use an existing
template engine.
• Template Engines uses Jinja as an implementation example to explain
the tasks that template engines can be used to perform.
• Approach: Building a toy template engine in Python walks through how
to create your own simple template engine in Python to understand the
basics of how most template engines work.
Jinja2
Jinja, also commonly referred to as "Jinja2" to specify the newest release
version, is a Python template engine used to create HTML, XML or other
markup formats that are returned to the user via an HTTP response.
231
Full Stack Python: 2020 Supporter's Edition
Why is Jinja2 useful?
Jinja2 is useful because it has consistent template tag syntax and the project is
cleanly extracted as an independent open source project so it can be used as a
dependency by other code libraries.
Jinja2 strikes a thoughtful balance on the template engine spectrum where on
one end you can embed arbitrary code in the templates and the other end a
developer can code whatever she wants.
Jinja2 origin and development
The first recorded public released of Jinja2 was in 2008 with 2.0rc1. Since
then the engine has seen numerous updates and remains in active
development.
Jinja2 engine certainly wasn't the first template engine. In fact, Jinja2's
syntax is inspired by Django's built-in template engine, which was released
several years earlier. There were many template systems, such as JavaServer
Pages (JSPs), that originated almost a decade before Jinja2. Jinja2 built upon
the concepts of other template engines and today is widely used by the Python
community.
What projects depend on Jinja2?
Jinja2 is a commonly-used templating engine for web frameworks such as
Flask, Bottle, Morepath and, as of its 1.8 update, optionally Django as well.
Jinja2 is also used as a template language by configuration management tool
Ansible and the static site generator Pelican, among many other similar tools.
232
Full Stack Python: 2020 Supporter's Edition
The idea is that if a developer already knows Jinja2 from working with one
project then the exact same syntax and style can be used in another project
that requires templating. The re-use reduces the learning curve and saves the
open source project author from having to reinvent a new templating style.
Jinja2 resources
• Real Python has a nice Jinja2 primer with many code examples to show
how to use the template engine.
• The second part of the Flask mega tutorial is all about Jinja2 templates.
It walks through control flow, template inheritance and other standard
features of the engine.
• Upgrading to Jinja2 Templates in Django 1.8 With Admin shows how
to fix an issue that can occur with Django 1.8 and using Jinja2 as the
template engine.
• The official Jinja2 template designer documentation is exceptionally
useful both as a reference as well as a full read-through to understand
how to properly work with template tags.
• When you want to use Jinja2 outside of a web framework or other
existing tool, here's a handy quick load function snippet so the
template engine can be easily used from a script or the REPL.
• When working with Jinja2 in combination with LaTeX, some of
Jinja2's blocks can conflict with LaTeX commands. Check out this post
on LaTeX templates with Python and Jinja2 to generate PDFs to
resolve those issues.
• When you use Jinja2 for long enough, eventually you'll want to escape
large blocks of Jinja2-like text in your templates. To do so, you'll need
the "raw" template tag.
• Python Templating Performance Showdown: Django vs Jinja puts
together some benchmarks for how Django templates compare with
Jinja templates. The usual benchmarking caveats apply here but there
are some interesting tests that examine how the template engines
handle large numbers of variables and other factors.
233
Full Stack Python: 2020 Supporter's Edition
• Universal Jinja presents a high-level overview of what you could do
using the Jinja-like Nunchuks library to perform server-side template
rendering for Django applications.
Mako
Mako is a template engine built in Python that is used to generate output
HTML, XML and similar formats.
Mako resources
• Exploring Mako explains a bit about the template engines Myghty and
Mason, which influenced Mako's design. The post then shows a few
basic examples for how to use Mako.
• Configuration Templates with Python and Mako shows some basic
situations for how to use Mako in an example project.
• Flask-Mako is a Flask extension that makes it easier to use Mako as the
template engine in your Flask web app projects.
• The Stack Overflow question on What is the fastest template system for
Python? provides some basic benchmarks comparing Mako, Jinja and
other template engines. Any benchmark should be taken as a data point
rather than a rule on which engine is actually the fastest in real world
scenarios. In addition, if you are using Mako or any other template
engine as part of a static website generator then it will not really matter
which one is the fastest because the output is created before the
234
Full Stack Python: 2020 Supporter's Edition
website is deployed rather than during the web server's HTTP requestresponse cycle.
Django Templates
The Django web framework contains its own template engine for generating
HTML, XML and other output formats.
What's the difference between a "project template" and
Django templates?
A project template contains the files and code to start a new web application.
For example, when you run django-admin.py startproject abc , the
Django admin script creates a new abc directory along with several Python
configuration so the web app can be run by a WSGI server.
Django templates are different from a project template because they live
within a project and are written by the developer to generate output, most
commonly HTML.
Django template resources
• Make ALL Your Django Forms Better presents some tricks for
customizing Django templates to handle the widgets on your site.
• Python Templating Performance Showdown: Django vs Jinja provides
some benchmarks for how Django templates compare with Jinja
templates. Note that as with any benchmarks these only provide a few
data points that can be useful rather than a definitive statement that
one tool is always faster than the other.
235
Full Stack Python: 2020 Supporter's Edition
• Reconciling Backend Templates with Frontend Components explains
how to use React components with the traditional server-side Django
templates despite some mismatch in how each tool approaches the end
goal of rendering a webpage.
• When and how to use Django TemplateView is not specifically about
using the Django template engine, but instead how to use the
TemplateView in your views which lead directly into rendering a
template.
Web Design
Web design is the creation of a web application's style and user interaction
using CSS and JavaScript.
Why is web design important?
You wouldn’t use a web application that looked like the following screenshot,
would you?
Creating web pages with their own style and interactivity so users can easily
accomplish their tasks is a major part of building modern web applications.
236
Full Stack Python: 2020 Supporter's Edition
Getting started if you have no "eye" for design
Design can feel like something "creative" people understand intuitively, but
like all skills design is something that can be learned. Some people are faster
learners in design just like some folks are quicker in picking up programming.
But anyone can learn how to be a better designer by learning the basic
principles and practicing them.
One of the best mental models for basic design is C.R.A.P., which helped me
grasp why some designs look good while others do not. CRAP is an acronym
for:
*
*
*
*
Contrast: noticeable differences from one element to another
Repetition: elements' consistency
Alignment: order among all elements
Proximity: placement between elements and how they are organized
These basic principles all you to start breaking down the problem into
digestible pieces that you can work on rather than feeling like you "just don't
have an eye for design".
Designing for various screen sizes
Separating the content from the rules for how to display the content allows
devices to render the output differently based on factors such as screen size
and device type. Displaying content differently based on varying screen
attributes is often called responsive design. The responsiveness is
accomplished by implementing media queries in the CSS.
For example, a mobile device does not have as much space to display a
navigation bar on the side of a page so it is often pushed down below the main
content. The Bootstrap Blog example shows that navigation bar relocation
scenario when you resize the browser width.
Fantastic design resources
There are way too many design resources on the web, so I picked out this
short list as my absolute favorites that help developers become (hopefully
much) better with design.
• Clean up your mess: A Guide to Visual Design for Everyone walks
through the basic principles for clean and effective design. You can
237
Full Stack Python: 2020 Supporter's Edition
make a website go from terrible to well-designed often by following a
few principles on spacing, alignment, contrast and repetition of page
elements.
• Resilient web design is an incredible online book that teaches how to
create websites that are accessible to every reader and look great while
doing it.
• Design 101 for Developers gives away the "secrets" to good design that
designers follow but that can be similarly accessible to developers who
understand what they want their design to accomplish.
• Laws of UX provides a beautiful overview of design principles for
building user experiences. Highly recommended even if just to see how
the information is presented.
• How I Work with Color is a fantastic article from a professional
designer on how he thinks about color and uses it for certain effects in
his designs.
• Building dark mode on Stack Overflow explains the thought process
and work that the team at Stack Overflow had to do with colors, styling
and the code implementation so they could offer a dark mode design to
their users.
• A short history of body copy sizes on the Web provides a useful
examination of how originally the web's typography mirrored what was
in traditional print. Eventually, the designs shifted upwards in font size
because screen resolutions changed. However, even in 2020 there is no
consensus for what font size to use in various situations so designers
simply have to do what they've always done and try their sites on
various devices and screen sizes.
• Web bloat is the story of traveling and using the web with low
bandwidth, high latency internet connections that often drop packets.
The author explains how many websites are barely usable and that if
you truly want your site to work well you need to ensure it works for
connections much worse than the fiber connection you may have at the
home or office.
238
Full Stack Python: 2020 Supporter's Edition
• Fundamental design principles for non-designers summarizes
numerous design tips into four principles that those without prior
design knowledge can follow. The author gives a bunch of great
examples and further details for the four principles, which are contrast,
consistency, Occam's Razor and space.
• Gallery of web design history is a collection of websites from between
1991 and 2006 that show the evolution of what the web looked like
before the modern CSS era. This is a great resource to see how websites
evolved, such as Microsoft in 1996 and YouTube in 2005.
• The Average Web Page (Data from Analyzing 8 Million Websites)
shows the most frequently used HTML elements, metadata, text
content and other statistics from a large scale analysis of the web.
• How to design delightful dark themes explains some subtle tactics to
make dark themes work well for users.
• Setting height and width on images is important again describes how
browser layout and resizing settings could affect your images so
manually setting the size is more useful than it has been in the past few
years.
• Kuler is a complementary color picker by Adobe that helps choose
colors for your designs.
• If you want to learn more about how browsers work behind the scenes
to render a webpage's design, here is a blog post series on building a
browser engine that will show you how to build a simple rendering
engine.
• How to Use C.R.A.P. Design Principles for Better UX has a good
summary of what contrast, repetition, alignment and proximity means
for designing user interfaces.
• Defining Colors in CSS presents how to define color in your Cascading
Style Sheets (CSS) and breaks down the differences between specifying
predefined color values, hexadecimal values, red-green-blue (RGB) and
Hue-Saturation-Lightness (HSL).
239
Full Stack Python: 2020 Supporter's Edition
• Easy to Remember Color Guide for Non-Designers gives guidance to
less aesthetically-inclined folks like myself who need rules for picking
groups of colors to use together in your designs.
• Styling HTML checkboxes is hard - here's why explains why form
elements like checkboxes are more difficult to style than other HTML
elements, The post shows how to perform the styling in various ways
such as CSS-only and then with JavaScript plus CSS.
• 13 Terrible Web Trends From the 90s, and How to Recreate Them
revisits a simpler and perhaps more... ugly time on the web where
designs were a bit out of control. Learn more about the history of web
design and styling techniques with this hilarious but useful blog post.
Checklists and design guidelines
• Frontend Guidelines is an amazing write up of good practices for
HTML, CSS and JS.
• Learn Design Principles is a well thought out clear explanation for how
to think about design according to specific rules such as axis,
symmetry, hierarchy and rhythm.
• Front-end performance checklist is a comprehensive checklist useful
when you are implementing a web application's client side code.
• Front-end Developer Handbook 2018 provides a high-level overview of
many of the tools developers use in the browser and some other useful
information on average salaries and where to search for front-end
development jobs.
• Who Can Use shows how color contrast can affect different people with
visual impairments and can help you improve your site's accessibility to
these groups.
Hypertext Markup Language (HTML)
Hypertext Markup Language (HTML) is the standard markup language that is
downloaded from a web server to a web browser and is used to display
website and web application content.
240
Full Stack Python: 2020 Supporter's Edition
HTML resources
• Mozilla's HTML page and introduction to HTML break down HTML by
elements, metadata and forms before diving into more advanced web
development topics.
• A few HTML tips is written for beginners who are learning HTML. The
article gives guidance on common mistakes to avoid and what to do
instead to write proper HTML.
• CodeAcademy's HTML basics course provides an interactive
environment for learning the gist of the markup language.
• A list of everything that could go in the head of your document provides
a comprehensive reference for elements that are required or optional in
the <head> element of your webpage.
• (Why) Some HTML is "optional" gives historical context for the <p>
paragraph element as an example to explain how HTML was originally
designed. The backwards-compatibility remains because there is not
enough optimization juice to squeeze from changing the
implementation compared to the backwards-breaking changes in
rendering existing sites.
• A Complete Guide to Links and Buttons extensively covers what might
be thought of as a simple topic: the a and button elements in HTML,
along with their many attributes and quirks.
• HTML: the inaccessible parts explains how even basic HTML elements
can cause accessibility issues for screen readers and other devices that
help people with impairments to use the web.
241
Full Stack Python: 2020 Supporter's Edition
Cascading Style Sheets (CSS)
Cascading Style Sheet (CSS) files contain rules for how to display and lay out
the HTML content when it is rendered by a web browser.
Why is CSS necessary?
CSS separates the content contained in HTML files from how the content
should be displayed. It is important to separate the content from the rules for
how it should be rendered primarily because it is easier to reuse those rules
across many pages. CSS files are also much easier to maintain on large
projects than styles embedded within the HTML files.
How is CSS retrieved from a web server?
The HTML file sent by the web server contains a reference to the CSS file(s)
needed to render the content. The web browser requests the CSS file after the
HTML file as shown below in a screenshot captured of the Chrome Web
Developer Tools network traffic.
242
Full Stack Python: 2020 Supporter's Edition
That request for the fsp.css file is made because the HTML file for Full Stack
Python contains a reference to theme/css/fsp.css which is shown in the
view source screenshot below.
CSS preprocessors
A CSS preprocessor compiles a processed language into plain CSS code. CSS
preprocessing languages add syntax such as variables, mixins and functions to
reduce code duplication. The additional syntax also makes it possible for
243
Full Stack Python: 2020 Supporter's Edition
designers to use these basic programming constructs to write maintainable
front end code.
• Sass is currently the favored preprocessor in the design community.
Sass is considered the most powerful CSS preprocessor in terms of
advanced language features.
• LESS is right up there with Sass and has an ace up its sleeve in that the
Bootstrap Framework is written in LESS which brings up its
popularity.
• Stylus is often cited as the third most popular CSS preprocessing
language.
CSS preprocessor resources
• The Advanced Guide to HTML and CSS book has a well-written
chapter on preprocessors.
• Sass vs LESS provides a short answer on which framework to use then
a longer more detailed response for those interested in understanding
the details.
• How to choose the right CSS preprocessor has a comparison of Sass,
LESS and Stylus.
• Musings on CSS preprocessors contains helpful advice ranging from
how to work with preprocessors in a team environment to what apps
you can use to aid your workflow.
CSS libraries and frameworks
CSS frameworks provide structure and a boilerplate base for building a web
application's design.
• Bootstrap
• Foundation
• Gumby
• Compass
• Profound Grid
244
Full Stack Python: 2020 Supporter's Edition
• Skeleton
• HTML5 Boilerplate
• Spectre
CSS resources
• The languages which were almost CSS contains the history of what
might have been if other styling proposals were adopted instead of CSS,
such as RRP, PWP, FOSI, DSSSL, PSL96 and CHSS. Many of those
proposals came before CSS was first published as a specification in
1996 so the article is a wonderful view into the Web in its infancy.
• Frontend Development Bookmarks is one of the largest collections of
valuable resources for frontend learning both in CSS as well as
JavaScript.
• This series on how CSS works including How CSS works: Parsing &
painting CSS in the critical rendering path and How CSS works:
Understanding the cascade examines the rendering methods browsers
use to display web pages along with details of the algorithms they use
to cascade style rules.
• CSS Reference provides much-needed visual examples for every CSS
property to show you what they are actually going to look like on your
pagee when you use them.
• CSS coding techniques provides advice on how to write simpler, easierto-maintain CSS code to reduce your need to rely on CSS preprocessors
and build pipelines.
• CSS refresher notes is incredibly helpful if you've learned CSS in bits
and pieces along the way and you now want to fill in the gaps in your
knowledge.
• Mozilla Developer Network's CSS page contains an extensive set of
resources, tutorials and demos for learning CSS.
• CSS Positioning 101 is a detailed guide for learning how to do element
positioning correctly with CSS.
245
Full Stack Python: 2020 Supporter's Edition
• Did CSS get more complicated since the late nineties? is a solid look
back at how CSS evolved and where it has ended up today compared to
its origins.
• Using feature queries in CSS covers the @supports rule and how to
use it in your stylesheets.
• Learn CSS layout is a simple guide that breaks CSS layout topics into
chapters so you can learn each part one at a time.
• How well do you know CSS display? zooms into a single CSS property,
display , to teach the reader about it in-depth.
• Google's Web Fundamentals class shows how to create responsive
designs and performant websites.
• Tailoring CSS for performance is an interesting read since many
developers do not consider the implications of CSS complexity in
browser rendering time.
• Can I Use... is a compatibility table that shows which versions of
browsers implement specific CSS features.
• How do you remove unused CSS from a site? covers tools for
identifying unnecessary CSS and the process for eliminating rules that
are overwritten by other rules and therefore do not need to be sent to
the client.
• The Web Design Museum is an amazing look back at how web design
has evolved over the past 25+ years. Some of the designs can still be
seen in their current site's presentation such as the top navigation of
Apple's 2001 site.
• The invisible parts of CSS asks the question "can you describe what the
display:block property and value do? Most developers would have
some sense of what it is for but could not explain it to someone else
beyond that. The article helps fix this situation with display as well
as other less visible properties such as floats and auto width.
• A brief history of CSS until 2016 explains how CSS originated at CERN
in 1994 as a solution to the problem of HTML not having reasonable
styling features (in-line style attributes on elements don't count).
246
Full Stack Python: 2020 Supporter's Edition
• Old CSS, New CSS tells the wonderful and painful story of web design
in the early days of the Web, when inline styling was required, HTML
CAPS were mandatory, and most websites wished they had design
styles like the amazing Space Jam pages. Oh also, lots and lots of talk
about tables, because that was the only way to position anything back
in the day.
• 30 seconds of CSS provides short useful code snippets for you to learn
from and use for building your own web applications.
• CSS: The bad bits examines global scope, implicit percentage styling
rules and the z-index which can be difficult to use and require some
restraint to ensure they do not cause issues for the rest of your
stylesheet rules as you create and maintain your frontend.
• Improving Your CSS with Parker shows how to use the static CSS
analysis tool Parker to improve your stylesheets.
• CSS and network performance analyzes how splitting your CSS can
affect browser render times and how you can improve your site loading
performance by changing how you structure your CSS files. My
recommendation: there's a lot you can do with these techniques but it
is probably a better idea to make your CSS simpler and cut down the
massive bloat that can accumulate as you build your site as a first step
to improving your performance.
• A Guide To CSS Support In Browsers analyzes features versus bugs in
different browser versions and how to test for support so that issues are
less likely to hit your web app.
• That Time I Tried Browsing the Web Without CSS is an enlightening
look at how crucial CSS is to the modern web. You can view examples
of what it is like to use Amazon, DuckDuckGo, GitHub and several
other sites.
• Third party CSS is not safe is a good reminder that any code you did
not write yourself, especially code served through 3rd party sources not
under your control can contain potentially malicious applications, such
as the experimental CSS keylogger hack that made the rounds in early
2018.
247
Full Stack Python: 2020 Supporter's Edition
• Understanding specificity in CSS provides some wonderful detail on
the various ways to select elements via CSS selectors, including type
selectors, pseudo-elements, class selectors, attribute selectors and ID
selectors.
• Realistic Water Effect without JavaScript - HTML/CSS Only is one of
the coolest tutorials I have seen that uses CSS to create a water effect
over an image. This video provides an example of how there are so
many incredible ways to use CSS and web development
technologies.
CSS learning checklist
1. Create a simple HTML file with basic elements in it. Use the
python -m SimpleHTTPServer command to serve it up. Create a
<style></style> element within the <head> section in the HTML
markup. Play with CSS within that style element to change the look and
feel of the page.
2. Check out front end frameworks such as Bootstrap and Foundation and
integrate one of those into the HTML page.
3. Work through the framework's grid system, styling options and
customization so you get comfortable with how to use the framework.
4. Apply the framework to your web application and tweak the design
until you have something that looks much better than generic HTML.
Responsive Design
Responsive design is a method of using media queries in Cascading Style
Sheets (CSS) to ensure a site is usable on various devices with different screen
sizes, from very small to very large.
248
Full Stack Python: 2020 Supporter's Edition
Full Stack Python uses responsive design to improve readability across a
broad range of devices that people use to read this site.
Responsive design resources
• The Responsive Design podcast and accompanying email newsletter
interview web designers who've dealt with creating responsive designs
from both a blank slate and retrofitting into an existing site.
• This site is entirely on responsive design and has many articles on how
to achieve readability across devices.
• Using Media Queries For Responsive Design In 2018 contains a bunch
of great examples for how to use media queries to achieve a responsive
design.
• Making Tables Responsive With Minimal CSS steps through a couple of
ways to responsively handle tables.
• Smashing Magazine's article from 2011 on responsive design - what it is
and how to use it remains a definitive source for understanding why
sites should be responsive.
• Fundamentals of responsive images shows the code behind several
ways to make your web images resize according to browser window
size.
• ResponsiveViewer is a Chrome plugin in that allows you to view
multiple sizes and devices in the same browser window. This is a handy
249
Full Stack Python: 2020 Supporter's Edition
tool to avoid constantly flipping between tabs or resizing the same tab
to see the changes you are making to a design.
Minification
Minification is the process of reducing the size of static content assets that
need to be transferred from a web server to the web browser client. The goal
of minification is to make webpages and web applications load faster, without
changing how the assets are ultimately used after being downloaded.
Cascading Style Sheet (CSS) files, JavaScript and even HTML can be minified.
The techniques to minify an HTML file differ from a CSS file because the file's
contents and syntax are different.
CSS Minification example
Let's say your web application has a CSS file with the following four lines to
add some padding around every element with the pad-me class:
.pad-me { padding-top: 10px;
padding-right: 5px;
padding-left: 5px;
padding-bottom: 10px; }
That example has 122 characters. A CSS minifier would take the above four
lines and convert them to the following single line:
.pad-me{padding:10px 5px 5px 10px}
The result would have only a single line with 35 characters, that's 87 less
characters that would need to be sent from the web server to the web browser!
The minification process reduced the single CSS class by 71% but kept the
exact same result when the web browser applies pad-me to all elements with
that class.
The file size savings can be a major benefit when applied across hundreds or
thousands of lines in a typical CSS file. When you also multiply that savings
across every client that downloads the CSS from your web server or CDN it
becomes clear that minification is a powerful way to improve the efficiency of
your production web application.
250
Full Stack Python: 2020 Supporter's Edition
CSS minification resources
• CSS minifier is an online form to copy and paste CSS then retrieve the
minified results on the same page.
JavaScript minification resources
• JavaScript minifier is similar to the above CSS minifier but works with
JavaScript syntax.
CSS Frameworks
A Cascading Style Sheet (CSS) framework is a code library that abstracts
common web designs and makes the designs easier for a developer to
implement in their web apps.
CSS framework implementations
• Bootstrap
• Foundation
• Screen
• Materialize
• Concise
CSS framework resources
• Which Responsive Design Framework Is Best? Of Course, It Depends.
• awesome-css-frameworks
• Bulma: CSS framework you should consider in 2018
• An Introduction to Tailwind CSS explains the basics of getting started
and using this newer CSS framework.
Bootstrap
Bootstrap is a CSS framework that makes it easier to create website and web
application user interfaces. Bootstrap is especially useful as a base layer of
CSS to build sites with responsive web design.
251
Full Stack Python: 2020 Supporter's Edition
Should I use Bootstrap 3 or 4?
Bootstrap 3 was out for almost five years before Bootstrap 4 was finally
released at the start of 2018. When learning Bootstrap you will likely come
across many tutorials that cover Bootstrap 3 but not version 4.
If you are completely new to Bootstrap and CSS frameworks in general then I
recommend learning Bootstrap 4. If you have already been working with
Bootstrap 3 then there is no major rush to upgrade to the latest version.
Bootstrap 4 is more complicated than version 3 because it has a lot more
features so the learning curve is a bit steeper.
Full Stack Python is actually built with an early version of Bootstrap 3.
However, this site is so heavily customized with my own CSS that I likely will
never upgrade to Bootstrap 4 because there are no new features that I feel will
be useful in my specific situation.
Bootstrap resources
• Getting Started with Bootstrap provides a walkthrough of a sample
starter template so you can understand the CSS classes that Bootstrap
uses. There is also a follow up post that shows how to create an
effective sales page with Bootstrap 3.
• What is Bootstrap and how do I use it? is an awesome tutorial that goes
through many of the main Bootstrap elements such as icons, navigation
bars and "jumbotron"-style webpages.
• Bootstrap 5 tutorial covers the alpha version of the fifth major revision
for Bootstrap.
252
Full Stack Python: 2020 Supporter's Edition
• Bootstrap 4: What's New? shows a slew of comparisons for how user
interface elements have changed design over the past several Bootstrap
major version iterations.
• Tabler is a free, open source admin theme for Bootstrap 4. You can also
check out a demo of the theme.
• How to use Bootstrap 4 forms with Django shows how to combine the
popular django-crispy-forms library with Django to create and obtain
user data through web forms that are styled with Bootstrap 4 CSS.
• React Bootstrap (source code replaces the existing Bootstrap
JavaScript with React components that do not rely on jQuery.
Foundation
Foundation is a Cascading Style Sheet (CSS) framework that makes it easier to
create website and web application user interfaces. Bootstrap is especially
useful as a base layer of CSS to build sites with responsive web design.
JavaScript
JavaScript is a scripting programming language interpretted by web browsers
that enables dynamic content and interactions in web applications.
Why is JavaScript necessary?
JavaScript executes in the client and enables dynamic content and interaction
that is not possible with HTML and CSS alone. Every modern Python web
application uses JavaScript on the front end.
253
Full Stack Python: 2020 Supporter's Edition
Front end frameworks
Front end JavaScript frameworks move the rendering for most of a web
application to the client side. Often these applications are informally referred
to as "one page apps" because the webpage is not reloaded upon every click to
a new URL. Instead, partial HTML pages are loaded into the document object
model or data is retrieved through an API call then displayed on the existing
page.
Examples of these front end frameworks include:
• React
• Angular.js
• Vue.js
• Backbone.js
• Ember.js
Front end frameworks are rapidly evolving. Over the next several years
consensus about good practices for using the frameworks will emerge.
How did JavaScript originate?
JavaScript is an implementation of the ECMAScript specification which is
defined by the Ecma International Standards Body. Read this paper on the
first 20 years of JavaScript by Brendan Eich, the original creator of the
programming language, and Allen Wirfs-Brock for more context on the
language's evolution.
JavaScript resources
• How Browsers Work is a great overview of both JavaScript and CSS as
well as how pages are rendered in a browser.
• This step-by-step tutorial to build a modern JavaScript stack is useful
to understanding how front-end JS frameworks such as Angular and
Ember.js work.
• A re-introduction to JavaScript by Mozilla walks through the basic
syntax and operators.
254
Full Stack Python: 2020 Supporter's Edition
• The Cost of Javascript Frameworks presents a balanced view of how
JavaScript libraries and frameworks impact web performance on both
desktop and mobile. The author acknowledges that these libraries are
useful for developers but in extreme cases there are significant
downsides to including so much JavaScript in pages.
• The State of JavaScript report contains a wealth of data on what
libraries developers are using in the JavaScript ecosystem. There are
also reports from previous years which show how the community's
preferences have changed over time.
• Coding tools and JavaScript libraries is a huge list by Smashing
Magazine with explanations for each tool and library for working with
JavaScript.
• Superhero.js is an incredibly well designed list of resources for how to
test, organize, understand and generally work with JavaScript.
• Unheap is an amazing collection of reusable JQuery plugins for
everything from navigation to displaying media.
• The Modern JavaScript Developer's Toolbox provides a high-level
overview of tools frequently used on the client and server side for
developers using JavaScript in their web applications.
• Front-end Walkthrough: Designing a Single Page Application
Architecture covers what a single page app (SPA) architecture looks
like, what the tools are that you can use and some comparisons when
deciding between Angular and React.
• Developing a Single Page App with Flask and Vue.js is a step-by-step
walkthrough of how to set up a basic CRUD app with Vue.js and Flask.
• A Guide to Console Commands shows off what JavaScript commands
you can use in your browser's console, which is a typical debugging
pattern for JavaScript development.
• is-website-vulnerable is an open source tool that identifies security
vulnerabilities based on the front end JavaScript code a web
application runs.
255
Full Stack Python: 2020 Supporter's Edition
• SPAs are harder, and always will be talks about the inherent complexity
in building client-side user interfaces with JavaScript.
• The Deep Roots of Javascript Fatigue starts by covering the non-stop
library churn in the JavaScript ecosystem and then relates JavaScript
evolution since the mid-90s to explain the history of the problem.
• How JavaScript works: the rendering engine and tips to optimize its
performance is one particularly relevant post in a multi-part series that
explains how you can optimize slower JavaScript code to better suit the
JS engines in common web browsers.
• How to reduce the impact of JavaScript on your page load time
provides insight into minimizing the size and improving the download
and execution speed for JavaScript libraries, especially ones that are
used at scale by many websites.
• Learn JS data teaches how to manipulate data using JavaScript in a
browser or on the server using Node.js.
• A Beginner's Guide to JavaScript's Prototype explains the
fundamentals of JavaScript's object system, which is a prototype-based
model and different from many other common programming
languages' object models.
• Understanding Data Types in JavaScript examines JavaScript's
dynamic data type model and how it manifests in the way numbers,
string, Booleans and arrays are used.
JavaScript learning checklist
1. Create a simple HTML file with basic elements in it. Use the
python -m SimpleHTTPServer command to serve it up. Create a
<script type="text/javascript"></script> element at the end
of the <body> section in the HTML page. Play with JavaScript within
that element to learn the basic syntax.
2. Download JQuery and add it to the page above your JavaScript
element. Start working with JQuery and learning how it makes basic
JavaScript easier.
256
Full Stack Python: 2020 Supporter's Edition
3. Work with JavaScript on the page. Incorporate examples from open
source projects listed below as well as JQuery plugins. Check out
Unheap to find a large collection of categorized JQuery plugins.
4. Check out the JavaScript resources below to learn more about
advanced concepts and open source libraries.
5. Integrate JavaScript into your web application and check the static
content section for how to host the JavaScript files.
React
React is a JavaScript web application framework for building rich user
interfaces that run in web browsers.
Beginner React tutorials
Generally before you start working with React you will want to learn how to
build your Python backend with a web framework such as Django, Flask or
Pyramid. Once you get comfortable with the web development basics with one
of those frameworks as well as JavaScript then it will be much easier to tack
on React to build your client-side user interfaces.
• JavaScript fundamentals before learning React provides a gut check
that you have the prerequisite knowledge to avoid getting impossibly
stuck while trying to learn React.
• The official React tutorial is one of the best ways to start using React
because many other tutorials quickly fall out of date while this one
tends to stick to the basics that are relevant to beginners.
257
Full Stack Python: 2020 Supporter's Edition
• This Modern Django 4-part tutorial series is well-done, has freely
available source code and includes:
1. Setting up Django and React
2. Redux and React Router setup
3. Creating an API and integrating with React
4. Adding authentication to React SPA using DRF
• Django REST with React (Django 2.0 and a sprinkle of testing)
combines a Django plus Django REST Framework (DRF) backend with
React on the front end and shows how to stich it all together.
• Build a Simple CRUD App with Python, Flask, and React shows how to
combine a Flask backend with React.
• Learn React app is a Git repository with a code tutorial and instructions
for how to follow along, as well as exercises to ensure you are tested as
you go.
• 9 things every React.js beginner should know is not a tutorial but
instead the author gives some strong opinions for what beginners
should know as they start learning React.
• React Bootstrap (source code replaces the existing Bootstrap
JavaScript with React components that do not rely on jQuery.
Other React resources
• React interview questions is a good quiz to see what you know or still
need to learn about the fundamentals of using React.
• Under-the-hood-ReactJS examines the React codebase itself rather
than teaching you how to use it.
Vue.js
Vue.js (source code) is a JavaScript web application framework for building
rich apps that run in web browsers.
258
Full Stack Python: 2020 Supporter's Edition
Vue.js-related projects
• flask-vue-spa is an example project with a Flask API on the backend
and Vue on the front.
• django-vue-template contains example code for a Django backend.
There is also another project named django-vue-template by a different
developer so take a peek at that one as well if the first one does not suit
your needs.
• vuepress (source code) is a static site generator that uses Vue and
Markdown to create pre-rendered static HTML pages.
Vue.js resources
• A friendly introduction to Vue.js contains the code and brief
explanations of what it's doing so you can learn to create your first Vue
app.
• Developing a Single Page App with Flask and Vue.js walks through all
of the environment configuration, project setup and coding you need to
do to build a legitimate Vue.js application that uses a Flask API on the
backend.
• Building a SaaS using Django and Vue.js shows how to create a
software-as-a-service application with a Django back end and Vue on
the front end via a series of videos that are each between 10-15 minutes
in length.
259
Full Stack Python: 2020 Supporter's Edition
• Learn Vue by Building and Deploying a CRUD App is an awesome
course that walks through the Vue basics and building a whole
application and deploying it to Netlify.
• The official Vue.js getting started documentation is fantastic and highly
recommended as a top starting resource.
• Replacing jQuery With Vue.js: No Build Step Necessary explains why
you may want to replace existing jQuery code with Vue and how to do
so with minimal steps, depending on the complexity of your
applicationi code, of course!
• Why we chose Vue.js covers GitLab's reasons for using this JavaScript
framework over other options.
• The Vue.js publishes their own documentation page on how Vue
compares with other frameworks. It is refreshing to see a
straightforward technical analysis without the posturing that often
comes from authors of one project discussing other work in the same
space.
• Build a Crud application using Vue and Django combines Django and
Django REST Framework with Vue.js in this tutorial that shows how to
build a simple web app for managing subscriptions.
• Building Modern Applications with Django and Vue.js combines
Django, Django REST Framework, Vue.js and Auth0 (an
authentication web API) in an introductory web application.
• Vue.js And SEO: How To Optimize Reactive Websites For Search
Engines And Bots shows how pre-rendering and various other
attributes of JavaScript MVC frameworks like Vue.js can negatively
impact your SEO. It then walks through the most important items to
address to mitigate these problems on your own Vue-based sites.
• Exploring Data with Serverless and Vue: Automatically Update GitHub
Files With Serverless Functions combines a serverless backend with
Vue.js on the front.
260
Full Stack Python: 2020 Supporter's Edition
Angular
Angular is a JavaScript web application framework for building rich apps that
run in web browsers.
Angular resources
• This end to end web app with Django-Rest-Framework & AngularJS
part 1 tutorial along with part 2, part 3 and part 4 creates an example
blog application with Djangular. There is also a corresponding GitHub
repo for the project code.
• Django-angular is a code library that aims to make it easier to pair
Django with AngularJS on the front end.
• Developing a Real-Time Taxi App with Django Channels and Angular
this course teaches you how to build and test a real-time ride-sharing
app with Django Channels and Angular.
Task Queues
Task queues manage background work that must be executed outside the
usual HTTP request-response cycle.
Why are task queues necessary?
Tasks are handled asynchronously either because they are not initiated by an
HTTP request or because they are long-running jobs that would dramatically
reduce the performance of an HTTP response.
For example, a web application could poll the GitHub API every 10 minutes to
collect the names of the top 100 starred repositories. A task queue would
handle invoking code to call the GitHub API, process the results and store
them in a persistent database for later use.
261
Full Stack Python: 2020 Supporter's Edition
Another example is when a database query would take too long during the
HTTP request-response cycle. The query could be performed in the
background on a fixed interval with the results stored in the database. When
an HTTP request comes in that needs those results a query would simply fetch
the precalculated result instead of re-executing the longer query. This
precalculation scenario is a form of caching enabled by task queues.
Other types of jobs for task queues include
• spreading out large numbers of independent database inserts over time
instead of inserting everything at once
• aggregating collected data values on a fixed interval, such as every 15
minutes
• scheduling periodic jobs such as batch processes
Task queue projects
The defacto standard Python task queue is Celery. The other task queue
projects that arise tend to come from the perspective that Celery is overly
complicated for simple use cases. My recommendation is to put the effort into
Celery's reasonable learning curve as it is worth the time it takes to
understand how to use the project.
• The Celery distributed task queue is the most commonly used Python
library for handling asynchronous tasks and scheduling.
• The RQ (Redis Queue) is a simple Python library for queueing jobs and
processing them in the background with workers. RQ is backed by
Redis and is designed to have a low barrier to entry.
• Taskmaster is a lightweight simple distributed queue for handling large
volumes of one-off tasks.
• Huey is a Redis-based task queue that aims to provide a simple, yet
flexible framework for executing tasks. Huey supports task scheduling,
crontab-like repeating tasks, result storage and automatic retry in the
event of failure.
262
Full Stack Python: 2020 Supporter's Edition
• Kuyruk is simple and easy to use task queue system built on top of
RabbitMQ. Although feature set is small, new features can be added by
extensions.
• Dramatiq is a fast and reliable alternative to Celery. It supports
RabbitMQ and Redis as message brokers.
• django-carrot is a simple task queue specifically for Django that can
serve when Celery is overkill.
• tasq is a brokerless task queue for simple use cases. It is not
recommended for production unless further testing and development is
done.
Hosted message and task queue services
Task queue third party services aim to solve the complexity issues that arise
when scaling out a large deployment of distributed task queues.
• Iron.io is a distributed messaging service platform that works with
many types of task queues such as Celery. It also is built to work with
other IaaS and PaaS environments such as Amazon Web Services and
Heroku.
• Amazon Simple Queue Service (SQS) is a set of five APIs for creating,
sending, receiving, modifying and deleting messages.
• CloudAMQP is at its core managed servers with RabbitMQ installed
and configured. This service is an option if you are using RabbitMQ
and do not want to maintain RabbitMQ installations on your own
servers.
Open source examples that use task queues
• Take a look at the code in this open source Flask application and this
Django application for examples of how to use and deploy Celery with a
Redis broker to send text messages with these frameworks.
• flask-celery-example is a simple Flask application with Celery as a task
queue and Redis as the broker.
263
Full Stack Python: 2020 Supporter's Edition
• django_dramatiq_example and flask_dramatiq_example are simple
apps that demo how you can use Dramatiq with Django and Flask,
respectively.
Task queue resources
• International Space Station notifications with Python and Redis Queue
(RQ) shows how to combine the RQ task queue library with Flask to
send text message notifications every time a condition is met - in this
blog post's case that the ISS is currently flying over your location on
Earth.
• Evaluating persistent, replicated message queues is a detailed
comparison of Amazon SQS, MongoDB, RabbitMQ, HornetQ and
Kafka's designs and performance.
• Queues.io is a collection of task queue systems with short summaries
for each one. The task queues are not all compatible with Python but
ones that work with it are tagged with the "Python" keyword.
• Why Task Queues is a presentation for what task queues are and why
they are needed.
• Asynchronous Processing in Web Applications Part One and Part Two
are great reads for understanding the difference between a task queue
and why you shouldn't use your database as one.
• Flask by Example Implementing a Redis Task Queue provides a
detailed walkthrough of setting up workers to use RQ with Redis.
• Heroku has a clear walkthrough for using RQ for background tasks.
• How to use Celery with RabbitMQ is a detailed walkthrough for using
these tools on an Ubuntu VPS.
• Celery - Best Practices explains things you should not do with Celery
and shows some underused features for making task queues easier to
work with.
• Celery in Production on the Caktus Group blog contains good practices
from their experience using Celery with RabbitMQ, monitoring tools
and other aspects not often discussed in existing documentation.
264
Full Stack Python: 2020 Supporter's Edition
• A 4 Minute Intro to Celery is a short introductory task queue
screencast.
• This Celery tasks checklist has some nice tips and resources for using
Celery in your applications.
• Heroku wrote about how to secure Celery when tasks are otherwise
sent over unencrypted networks.
• Miguel Grinberg wrote a nice post on using the task queue Celery with
Flask. He gives an overview of Celery followed by specific code to set up
the task queue and integrate it with Flask.
• Ditching the Task Queue for Gevent explains how in some cases you
can replace the complexity of a task queue with concurrency. For
example, you can remove Celery in favor of gevent.
• 3 Gotchas for Working with Celery are things to keep in mind when
you're new to the Celery task queue implementation.
• Setting up an asynchronous task queue for Django using Celery and
Redis is a straightforward tutorial for setting up the Celery task queue
for Django web applications using the Redis broker on the back end.
• Three quick tips from two years with Celery provides some solid advice
on retry delays, the -Ofair flag and global task timeouts for Celery.
• Asynchronous Tasks with Flask and Redis Queue looks at how to
configure Redis Queue to handle long-running tasks in a Flask app.
• Developing an Asynchronous Task Queue in Python looks at how to
implement several asynchronous task queues using Python's
multiprocessing library and Redis.
Task queue learning checklist
1. Pick a slow function in your project that is called during an HTTP
request.
2. Determine if you can precompute the results on a fixed interval instead
of during the HTTP request. If so, create a separate function you can
call from elsewhere then store the precomputed value in the database.
265
Full Stack Python: 2020 Supporter's Edition
3. Read the Celery documentation and the links in the resources section
below to understand how the project works.
4. Install a message broker such as RabbitMQ or Redis and then add
Celery to your project. Configure Celery to work with the installed
message broker.
5. Use Celery to invoke the function from step one on a regular basis.
6. Have the HTTP request function use the precomputed value instead of
the slow running code it originally relied upon.
Celery
Celery is a task queue implementation for Python web applications used to
asynchronously execute work outside the HTTP request-response cycle.
Why is Celery useful?
Task queues and the Celery implementation in particular are one of the
trickier parts of a Python web application stack to understand.
If you are a junior developer it can be unclear why moving work outside the
HTTP request-response cycle is important. In short, you want your WSGI
server to respond to incoming requests as quickly as possible because each
request ties up a worker process until the response is finished. Moving work
off those workers by spinning up asynchronous jobs as tasks in a queue is a
straightforward way to improve WSGI server response times.
What's the difference between Celeryd and Celerybeat?
Celery can be used to run batch jobs in the background on a regular schedule.
A key concept in Celery is the difference between the Celery daemon (celeryd),
266
Full Stack Python: 2020 Supporter's Edition
which executes tasks, Celerybeat, which is a scheduler. Think of Celeryd as a
tunnel-vision set of one or more workers that handle whatever tasks you put
in front of them. Each worker will perform a task and when the task is
completed will pick up the next one. The cycle will repeat continously, only
waiting idly when there are no more tasks to put in front of them.
Celerybeat on the other hand is like a boss who keeps track of when tasks
should be executed. Your application can tell Celerybeat to execute a task at
time intervals, such as every 5 seconds or once a week. Celerybeat can also be
instructed to run tasks on a specific date or time, such as 5:03pm every
Sunday. When the interval or specific time is hit, Celerybeat will hand the job
over to Celeryd to execute on the next available worker.
Celery tutorials and advice
Celery is a powerful tool that can be difficult to wrap your mind around at
first. Be sure to read up on task queue concepts then dive into these specific
Celery tutorials.
• A 4 Minute Intro to Celery is a short introductory task queue
screencast.
• This blog post series on Celery's architecture, Celery in the wild: tips
and tricks to run async tasks in the real world and dealing with
resource-consuming tasks on Celery provide great context for how
Celery works and how to handle some of the trickier bits to working
with the task queue.
• How to use Celery with RabbitMQ is a detailed walkthrough for using
these tools on an Ubuntu VPS.
• Celery - Best Practices explains things you should not do with Celery
and shows some underused features for making task queues easier to
work with.
• Celery Best Practices is a different author's follow up to the above best
practices post that builds upon some of his own learnings from 3+
years using Celery.
267
Full Stack Python: 2020 Supporter's Edition
• Common Issues Using Celery (And Other Task Queues) contains good
advice about mistakes to avoid in your task configurations, such as
database transaction usage and retrying failed tasks.
• Asynchronous Processing in Web Applications Part One and Part Two
are great reads for understanding the difference between a task queue
and why you shouldn't use your database as one.
• My Experiences With A Long-Running Celery-Based Microprocess
gives some good tips and advice based on experience with Celery
workers that take a long time to complete their jobs.
• Checklist to build great Celery async tasks is a site specifically designed
to give you a list of good practices to follow as you design your task
queue configuration and deploy to development, staging and
production environments.
• Heroku wrote about how to secure Celery when tasks are otherwise
sent over unencrypted networks.
• Unit testing Celery tasks explains three strategies for testing code
within functions that Celery executes. The post concludes that calling
Celery tasks synchronously to test them is the best strategy without any
downsides. However, keep in mind that any testing method that is not
the same as how the function will execute in a production environment
can potentially lead to overlooked bugs. There is also an open source
Git repository with all of the source code from the post.
• Rollbar monitoring of Celery in a Django app explains how to use
Rollbar to monitor tasks. Super useful when workers invariably die for
no apparent reason.
• 3 Gotchas for Working with Celery are things to keep in mind when
you're new to the Celery task queue implementation.
• Dask and Celery compares Dask.distributed with Celery for Python
projects. The post gives code examples to show how to execute tasks
with either task queue.
• Python+Celery: Chaining jobs? explains that Celery tasks should be
dependent upon each other using Celery chains, not direct
dependencies between tasks.
268
Full Stack Python: 2020 Supporter's Edition
Celery with web frameworks
Celery is typically used with a web framework such as Django, Flask or
Pyramid. These resources show you how to integrate the Celery task queue
with the web framework of your choice.
• How to Use Celery and RabbitMQ with Django is a great tutorial that
shows how to both install and set up a basic task with Django.
• Miguel Grinberg wrote a nice post on using the task queue Celery with
Flask. He gives an overview of Celery followed by specific code to set up
the task queue and integrate it with Flask.
• Setting up an asynchronous task queue for Django using Celery and
Redis is a straightforward tutorial for setting up the Celery task queue
for Django web applications using the Redis broker on the back end.
• A Guide to Sending Scheduled Reports Via Email Using Django And
Celery shows you how to use django-celery in your application. Note
however there are other ways of integrating Celery with Django that do
not require the django-celery dependency.
• Flask asynchronous background tasks with Celery and Redis combines
Celery with Redis as the broker and Flask for the example application's
framework.
• Celery and Django and Docker: Oh My! shows how to create Celery
tasks for Django within a Docker container. It also provides some
• Asynchronous Tasks With Django and Celery shows how to integrate
Celery with Django and create Periodic Tasks.
• Getting Started Scheduling Tasks with Celery is a detailed walkthrough
for setting up Celery with Django (although Celery can also be used
without a problem with other frameworks).
• Introducing Celery for Python+Django provides an introduction to the
Celery task queue with Django as the intended framework for building
a web application.
• Asynchronous Tasks with Falcon and Celery configures Celery with the
Falcon framework, which is less commonly-used in web tutorials.
269
Full Stack Python: 2020 Supporter's Edition
• Custom Celery task states is an advanced post on creating custom
states, which is especially useful for transient states in your application
that are not covered by the default Celery configuration.
• Asynchronous Tasks with Django and Celery looks at how to configure
Celery to handle long-running tasks in a Django app.
Celery deployment resources
Celery and its broker run separately from your web and WSGI servers so it
adds some additional complexity to your deployments. The following
resources walk you through how to handle deployments and get the right
configuration settings in place.
• The "Django in Production" series by Rob Golding contains a post
specifically on Background Tasks.
• How to run celery as a daemon? is a short post with the minimal code
for running the Celery daemon and Celerybeat as system services on
Linux.
• Celery in Production on the Caktus Group blog contains good practices
from their experience using Celery with RabbitMQ, monitoring tools
and other aspects not often discussed in existing documentation.
• Three quick tips from two years with Celery provides some solid advice
on retry delays, the -Ofair flag and global task timeouts for Celery.
Redis Queue (RQ)
Redis Queue (RQ) is a Python task queue implementation that uses Redis to
keep track of tasks in the queue that need to be executed.
270
Full Stack Python: 2020 Supporter's Edition
RQ resources
• Asynchronous Tasks in Python with Redis Queue is a quickstart-style
tutorial that shows how to use RQ to fetch data from the Mars Rover
web API and process URLs for each of the photos taken by NASA's
Mars rover. There is also a follow-up post on Scheduling Tasks in
Python with Redis Queue and RQ Scheduler that shows how to
schedule tasks in advance, which is a common way of working with task
queues.
• The RQ intro post contains information on design decisions and how to
use RQ in your projects.
• International Space Station notifications with Python and Redis Queue
(RQ) shows how to combine the RQ task queue library with Flask to
send text message notifications every time a condition is met - in this
blog post's case that the ISS is currently flying over your location on
Earth.
• Asynchronous Tasks with Flask and Redis Queue looks at how to
configure RQ to handle long-running tasks in a Flask app.
• How We Spotted and Fixed a Performance Degradation in Our Python
Code is a quick story about how an engineering team moving from
Celery to RQ fixed some deficiencies in their RQ performance as they
started to understand the difference between how the two tools execute
workers.
• Flask by Example - Implementing a Redis Task Queue explains how to
install and use RQ in a Flask application.
• rq-dashboard is an awesome Flask-based dashboard for viewing
queues, workers and other critical information when using RQ.
• Sending Confirmation Emails with Flask, Redis Queue, and Amazon
SES shows how RQ fits into a real-world application that uses many
libraries and third party APIs.
• Background Tasks in Python using Redis Queue gives a code example
for web scraping data from the Goodreads website. Note that the first
sentence in the post is not accurate: it's not the Python language that is
linear, but instead the way workers in WSGI servers handle a single
271
Full Stack Python: 2020 Supporter's Edition
request at a time by blocking. Nevertheless, the example is a good one
for understanding how RQ can execute.
Dramatiq
Dramatiq (source code) is a Python-based task queue created as an alernative
to the ubiquitous Celery project Dramatiq supports RabbitMQ and Redis as
message brokers.
Dramatiq resources
• django_dramatiq is an app for integrating Dramatiq with Django.
• django_dramatiq_example is a simple Django project demoing how
you can combine Django and Dramatiq.
• flask_dramatiq_example is an example Flask app that shows how to
use Dramatiq.
Static Site Generators
A static website generator combines a markup language, such as Markdown
or reStructuredText, with a templating engine such as Jinja, to produce
HTML files. The HTML files can be hosted and served by a web server or
content delivery network (CDN) without any additional dependencies such as
a WSGI server.
Why are static site generators useful?
Static content files such as HTML, CSS and JavaScript can be served from a
content delivery network (CDN) for high scale and low cost. If a statically
generated website is hit by high concurrent traffic it will be easily served by
the CDN without dropped connections.
For example, when Full Stack Python was on the top of Hacker News for a
weekend, GitHub Pages was used as a CDN to serve the site and didn't have
any issues even with close to 400 concurrent connections at a time, as shown
in the following Google Analytics screenshot captured during that traffic
burst.
272
Full Stack Python: 2020 Supporter's Edition
How do static website generators work?
Static site generators allow a user to create HTML files by writing in a markup
language and coding template files. The static site generator then combines
the markup language and templates to produce HTML. The output HTML
does not need to be maintained by hand because it is regenerated every time
the markup or templates are modified.
For example, as shown in the diagram below, the Pelican static site generator
can take in reStructuredText files and Jinja2 template files as input then
combine them to output a set of static HTML files.
273
Full Stack Python: 2020 Supporter's Edition
What's the downside of using static site generators?
The major downside is that code cannot be executed after a site is created.
You are stuck with the output files so if you're used to building web
applications with a traditional web framework you'll have to change your
expectations.
Content that is typically powered by a database, such as comments, sessions
and user data can only be handled through third party services. For example,
if you want to have comments on a static website you'd need to embed
Disqus's form and be completely reliant upon their service.
Many web applications simply cannot be built with only a static site
generator. However, a static website generator can create part of a site that
will be served up by a web server while other pages are handled by the WSGI
server. If done right, those web applications have the potential to scale better
than if every page is rendered by the WSGI server. The complexity may or
may not be worth it for your specific application.
274
Full Stack Python: 2020 Supporter's Edition
Python implementations
Numerous static website generators exist in many different languages. The
ones listed here are primarily coded in Python.
• Pelican is a commonly used Python static website generator which is
used to create Full Stack Python. The primary templating engine is
Jinja and Markdown, reStructuredText and AsciiDoc are supported
with the default configuration.
• Lektor is a static content management system and site generator that
can deploy its output to any webserver. It uses Jinja2 as its template
engine.
• MkDocs uses a YAML configuration file to take Markdown files and an
optional theme to output a documentation site. The templating engine
is Jinja, but a user doesn't have to create her own templates unless a
custom site is desired at which point it might make more sense to use a
different static site generator instead.
• mynt (source code) is built to create blogs and uses Jinja to generate
HTML pages.
• Nikola (source code) takes in reStructuredText, Markdown or Jupyter
(IPython) Notebooks and combines the files with Mako or Jinja2
templates to output static sites. It is compatible with both Python 2.7
and 3.3+. Python 2.7 will be dropped in early 2016 while Python 3.3+
will continue to be supported.
• Acrylamid (source code) uses incremental builds to generate static sites
faster than recreating every page after each change is made to the input
files.
• Hyde (source code) started out as a Python rewrite of the popular
Ruby-based Jekyll static site generator. Today the project has moved
past those "clone Jekyll" origins. Hyde supports Jinja as well as other
templating languages and places more emphasis on metadata within
the markup files to instruct the generator how to produce the output
files. Check out the Hyde-powered websites page to see live examples
created with Hyde.
275
Full Stack Python: 2020 Supporter's Edition
• Grow SDK (source code) uses projects, known as pods, which contain a
specific file and directory structure so the site can be generated. The
project remains in the "experimental" phase.
• Complexity (source code) is a site generator for users who like to work
in HTML. It uses HTML for templating but has some functionality
from Jinja for inheritance. Works with Python 2.6+, 3.3+ and PyPy.
• Cactus (source code) uses the Django templating engine that was
originally built with front-end designers in mind. It works with both
Python 2.x and 3.x.
Open source Python static site generator examples
• This site is all open source in its own GitHub repository under the MIT
license. Fork away!
• Django REST Framework uses MkDocs to create its documentation
site. Be sure to take a look at the mkdocs.yml file to see how large, wellwritten docs are structured for that project.
• Practicing web development uses Acrylamid to create its site. The code
is open source on GitHub.
• Linux Open Admin Days (Loadsys) has their site open source and
available for viewing.
• The Pythonic Perambulations blog has a fairly standard theme but is
also open source on GitHub.
Static site generator resources
Static site generators can be implemented in any programming language. The
following resources either are general to any programming ecosystem or
provide a unique angle on how to use a static site generator.
• Static vs Dynamic Websites does an excellent job of showing the
differences between a dynamic website that uses a database backend to
produce content in response to a request compared with static sites
that are pregenerated. There is also a second part in the series where
generic static site generator concepts are explained.
276
Full Stack Python: 2020 Supporter's Edition
• Staticgen lists static website generators of all programming languages
sorted by various attributes such as the number of GitHub stars, forks
and issues.
• The title is a big grandiose, but there's some solid detail in this article
on why static website generators are the next big thing. I'd argue static
website generators have been big for a long time now.
• Static site generators can be used for a range of websites from side
projects up to big sites. This blog post by WeWork on why they use a
static site generator explains it from the perspective of a large business.
• Ditching Wordpress and becoming one of the cool kids is one
developer's experience moving away from Wordpress and onto Pelican
with reStructuredText for his personal blog.
• Static websites with Flask explains how to use Flask-Frozen to generate
a static site based on content from the web framework and a data
source backend. This approach is an alternative to using a purposebuilt static website generator such as Pelican, Lektor or MkDocs.
• Building A Serverless Contact Form For Your Static Site shows how to
use HTML and JavaScript deployed to AWS Lambda to collect input
with a form on a static site.
• 5 ways to handle forms on your static site gives a good overview of
options like Google Forms for when you absolutely must get input from
users on a static site.
Static site deployment resources
Deploying a static site is far less complicated than a traditional web
application deployment, but you still need to host the files somewhere
accessible. You'll also to set up DNS to point a domain name to your site as
well as provide HTTPS support. These guides walk through various ways of
handling the static site deployment.
• Static site hosting with S3 and Cloudflare shows how to set up an S3
bucket with Cloudflare in front as a CDN that serves the content with
HTTPS. You should be able to accomplish roughly the same situation
277
Full Stack Python: 2020 Supporter's Edition
with Amazon Cloudfront, but as a Cloudflare user I like their service for
these static site configurations.
• Google Cloud provides a tutorial on how to use them to host your static
site. Note that you cannot currently use HTTPS on Google Storage
servers, which is a major downside.
• scar is an open source tool for making static site deployments and
redeployments to Amazon Web Services easier.
• Deploying a Static Blog with Continuous Integration uses a Hugo (a
Golang-based static site generator) generated site as an example but
the instructions can easily be used to deploy a Python-based static site
generator output as well.
• How to Make an AWS S3 Static Website With SSL explains the
configuration required to use SSL for HTTPS on an AWS-hosted static
site.
• Hosting your static site with AWS S3, Route 53, and CloudFront is
another solid tutorial that uses the AWS stack to deploy a globallyhosted site.
• Why your static website needs HTTPS shows all of the malicious
activity that bad actors can cause if you do not use HTTPS as part of
your static site deployment. There are few excuses for having an
insecure site without required HTTPS in today's world of free Let's
Encrypt certificates.
• How to Build a Low-tech Website? takes static site deployments to the
extreme. The site served with solar power and customized hardware
setup. This is a great read even though it will not be remotely practical
for most organizations.
Pelican
Pelican is a static site generator implemented in Python that combines Jinja
templates with content written in Markdown or reStructuredText to produce
websites.
Pelican's source code is available on GitHub under the AGPL 3 license.
278
Full Stack Python: 2020 Supporter's Edition
Why is Pelican a useful tool?
Static websites are easier to deploy than full web applications built with a web
framework that rely on a persistent database backend. In addition, static sites
are typically much faster to load because there are no database queries or
middleware code to execute during the HTTP request-response cycle.
A web server that hosts a static website simply responds to inbound HTTP
requests with the file being requests - no dynamic data is populated on the
server during the response.
Pelican resources
Static site generators like Pelican are a simple compared to web frameworks
so most tutorials focus on creating simple sites that you can style yourself, as
well as deploying to hosting services such as Amazon S3 and GitHub Pages.
• How to Create Your First Static Site with Pelican and Jinja2 walks
through installing, generating the boilerplate and customizing your
first static site using Pelican.
• How I built this website, using Pelican walks through getting your first
Pelican site generated and running.
• A Pelican Tutorial to Build A Static, Python-Powered Blog with Search
& Comments provides a walkthrough for how to build a great
combination of useful features into your static site such as search and
comments with the Staticman library. Bonus points at the end for
showing how to deploy to Netlify as an alternative to GitHub Pages or
S3.
• Creating your own blog with Pelican covers the decision-making
process with building a static versus dynamic website. The post then
279
Full Stack Python: 2020 Supporter's Edition
dives into using Pelican as a static site generator with a blog structure
and basic theme.
• Using Pelican to generate and manage static websites explains how to
use the pelican-quickstart command to generate an initial site
then adds the Tipue Search plugin to provide content search despite
the static site limitations.
• Pelican Folder Structure explains how the pages and posts structure
under content works when using Pelican.
• Pelican's official Plugin creation documentation gives a great starting
point for building your own plugins that can take in new input markup
formats, modify the generator process and add handy features such as
a custom table of contents.
• Pelican Sitemap and Pagination explains how to generate a
sitemap.xml file for your static site that includes all pages instead of
just auto-included top-level pages.
• Getting started with Pelican and GitHub pages is a tutorial I wrote to
use the Full Stack Python source code to create and deploy your first
static site.
• Moving blogs to Pelican talks about one developer's transition from
Jekyll to Pelican for his own sites.
• Using Travis & GitHub to deploy static sites shows how to automate
deployments of a Pelican-based static site using Travis CI.
Lektor
Lektor is a static website generator with content management system (CMS)
and web framework features for creating websites.
Lektor's source code is available on GitHub under the BSD 3-clause license.
280
Full Stack Python: 2020 Supporter's Edition
How is Lektor different from other static site generators?
Most static site generators such as Pelican are built with programmers as the
primary user. Lektor tries to be more accessible to non-programmers by
providing an admin panel for creating and updating site content similar to
Django or Wordpress.
Lektor example projects
• PyCon Colombia's 2018 site was built with Lektor and is freely
available on GitHub.
• freedombox.org is also available for reference.
Lektor resources
Lektor is a young project and therefore has a nascent community compared
with Pelican and Jekyll (the most popular Ruby-based static site generator).
However, the official documentation and initial quickstarts for the project are
wonderful so getting a basic site up and running is painless.
• Introducing Lektor is the background story for what motivated Armin
Ronacher to start hacking on his own static site generator project after
jumping around from Django to Wordpress for hosting content. The
post also includes details on the differences in the project compared to
other static site generators.
• Hello, Lektor is a wonderful getting started and overview post. The post
walks through the files Lektor generates, the admin content
management system and pulling data into pages from the Meetup API.
281
Full Stack Python: 2020 Supporter's Edition
• The official Lektor quickstart contains the first commands to use to
generate a new project scaffold. There is also a getting started
screencast that walks through installing and initial steps for getting set
up with the project.
• Lektor Static CMS, put the fun back into Content Management is a
short overview as the first part in what aims to be a continuing series
on how to use Lektor as a content management system.
• In Experiences Migrating to Lektor the author gives his impression of
Lektor after moving his 400+ articles over from a home-grown
blogging engine. He talks a bit about how he went from deploying on
GitHub Pages to surge.sh and finally over to Netlify.
• Automating deployment of Lektor blog sites covers using OpenShift to
deploy a static site. Seems like a lot of work when AWS S3 deployments
are a lot easier but OpenShift has its own ecosystem to keep you away
from AWS world if that's your thing.
MkDocs
MkDocs is a Python-based static site generator that combines Markdown
content with Jinja2 templates to produce websites. MkDocs can be
pronounced "McDocs" or "M-K Docs", although the core committers do not
have a strong preference one way or the other on the name's pronunciation.
MkDocs' source code is available on GitHub under the BSD 2-clause license.
What's great about MkDocs?
MkDocs uses a YAML configuration file and can optionally use themes to easy
change the look-and-feel of the documentation output.
282
Full Stack Python: 2020 Supporter's Edition
In addition to the easy configuration via a YAML file and the drop-in themes,
MkDocs also has a wonderful search feature. Search is often not possible out
of the box in other static site generators. With MkDocs search can easily be
added without plugins or code changes to the static site generator.
MkDocs resources
• The official Getting Started with MkDocs is likely the best place to go
when you are just getting set up with your first site that uses this
project.
• Mkdocs documentation is a quick tutorial to get MkDocs installed and
modify the initial mkdocs.yml file.
• MkDocs making strides is a post from one of the project's core
commiters on some changes that greatly improved the project such as
site regeneration during development when a file is modified, search,
the command-line client and packageable theming.
Testing
Testing determines whether software runs correctly based on specific inputs
and identifies defects that need to be fixed.
Why is testing important?
As software scales in codebase size, it's impossible for a person or even a large
team to keep up with all of the changes and the interactions between the
changes. Automated testing is the only proven method for building reliable
software once they grow past the point of a simple prototype. Many major
software program development failures can be traced back to inadequate or a
complete lack of testing.
It's impossible to know whether software works properly unless it is tested.
While testing can be done manually, by a user clicking buttons or typing in
input, it should be performed automatically by writing software programs that
test the application under test.
There are many forms of testing and they should all be used together. When a
single function of a program is isolated for testing, that is called unit testing.
Testing more than a single function in an application at the same time is
283
Full Stack Python: 2020 Supporter's Edition
known as integration testing. User interface testing ensures the correctness of
how a user would interact with the software. There are even more forms of
testing that large programs need, such as load testing, database testing, and
browser testing (for web applications).
Testing in Python
Python software development culture is heavy on software testing. Because
Python is a dynamically-typed language as opposed to a statically-typed
language, testing takes on even greater importance for ensuring program
correctness.
Python testing tools
The Python ecosystem has a wealth of tools to make it easier to run your tests
and interpret the results. The following tools encompass test runners,
coverage reports and related libraries.
• green is a test runner that has pretty printing on output to make the
results easier to read and understand.
• requestium merges the Requests library with Selenium to make it
easier to run automated browser tests.
• coverage.py is a tool for measuring code coverage when running tests.
Testing resources
• The Minimum Viable Test Suite shows how to set unit tests and
integration tests for a Flask example application.
• BDD Testing a Restful Web Application in Python is an introduction to
behavior-driven development (BDD) and uses a Flask web application
as an example project for learning.
• Testing, for people who hate testing provides examples for how to
improve your testing environment such as using a new test harness and
getting your test suite to run fast.
• Getting started with Pytest goes over some code challenges as examples
for how to use Pytest in your own projects.
284
Full Stack Python: 2020 Supporter's Edition
• Good test, bad test explains the difference between a "good" test case
and one that is not as useful. Along the way the post breaks down some
myths about common testing subjects such as code coverage, assertions
and mocking.
• Python Testing is a site devoted to testing in - you guessed it - the
Python programming language.
• In praise of property-based testing is an article by the author of the
Hypothesis testing tool written in Python. The article explains the
shortcomings of example-based tests and how property-based testing
can augment or replace those example-based tests to find more defects
in software. There is also a great introductory post on Hypothesis that
goes into further detail about getting your first Hypothesis tests up and
running.
• Google has a testing blog where they write about various aspects of
testing software at scale.
• A beginner's guide to Python testing covers test-driven development
for unit, integration and smoke tests.
• Testing a Twilio Interactive Voice Response (IVR) System With Python
and pytest is an incredibly in-depth tutorial on testing a Pythonpowered IVR using pytest. There is also a tutorial on how to build the
IVR before this testing tutorial, although you can just clone it to jump
into the testing walkthrough.
• Still confused about the difference between unit, functional and
integration tests? Check out this top answer on Stack Overflow to that
very question.
• Testing Python applications with Pytest walks through the basics of
using Pytest and some more advanced ways to use it such as
continuous testing through Semiphore CI.
• The cleaning hand of Pytest provides a couple of case studies for how
companies set up their testing systems. It then gives the boilerplate
code the author uses for Pytest and goes through a bunch of code
samples for common situations.
285
Full Stack Python: 2020 Supporter's Edition
• Using pytest with Django shows how to get a basic pytest test case
running for a Django project and explains why the author prefers
pytest over standard unittest testing.
• Distributed Testing with Selenium Grid and Docker shows how to
distribute automated, browser tests with Selenium Grid and Docker
Swarm. It also looks at how to run tests against a number of browsers
and automate the provisioning and deprovisioning of machines to keep
costs down.
• Principles of Automated Testing explains how to prioritize what to test,
goes through some levels of testing from unit to integration and
examines when to use example and bulk tests.
• How to run tests continuously while coding contains a Python script
that uses the watchdog to check for changes to source code files in your
project directory. If changes are detected then your tests will be run to
check that everything is still working as intended.
• 8 great pytest plugins is a list of plugins that go with PyTest and help
with tasks like reducing the friction around testing Django, as well as
running tests in parallel.
• This test-driven development series shows you how to write an
interpreter in Python and contains a ton of great code samples to learn
from:
◦ A game of tokens: write an interpreter in Python with TDD Part 1
◦ A game of tokens - part 2
◦ A game of tokens - part 3
◦ A game of tokens - part 4
• Pytest leaking examines situations where tests leak memory and can
cause abnormal results if they are not fixed.
Mocking resources
Mocking allows you to isolate parts of your code under test and avoid areas
that are not critical to the tests you are writing. For example, you may want to
test how you handle text message responses but do not want to actually
receive text messages in your application. You can mock a part of the code
286
Full Stack Python: 2020 Supporter's Edition
that would provide a happy-path scenario so you can run your tests properly.
The following resources show you how to use mocks in your test cases.
• Getting Started with Mocking in Python provides a whole code example
based on a blog project that shows how to use mock when testing.
• Python Mocking 101: Fake It Before You Make It explains what
mocking is and is not, and shows how to use the patch function to
accomplish it in your project. Revisiting Unit Testing and Mocking in
Python is a follow-up post that expands upon using the patch
function along with dependency injection.
• Mocks and Monkeypatching in Python uses the mock and
monkeypatch tools to show how to mock your test cases.
• Mock yourself, not your tests examines when mocks are necessary and
when they are not as useful so you can avoid them in your test cases.
• Mocking Redis & Expiration in Python is a specific scenario where you
would want to test your Redis-dependent code but prefer to mock it
rather than ensure an installation and connection are present
whenever you run your tests.
• Better tests for Redis integrations with redislite is a great example of
how using the right mocking library can clean up existing hacky testing
code and make it more straightforward for any developer that happens
upon the tests in the future.
Unit Testing
Unit testing is a method of determining the correctness of a single function
isolated from a larger codebase. The idea is that if all the atomic units of an
application work as intended in isolation, then integrating them together as
intended is much easier.
Why is unit testing important?
Unit testing is just one form of testing that works in combination with other
testing approaches to wring out the bugs from a piece of software being
developed. When several functions and classes are put together it's often
difficult to determine the source of a problem if several bugs are occurring at
287
Full Stack Python: 2020 Supporter's Edition
the same time. Unit testing helps eliminate as many of the individual bugs as
possible so when the application comes together as a whole the separate parts
work as correct as possible. Then when issues arise they can often be tracked
down as unintended consequences of the disparate pieces not fitting together
properly.
Unit testing tools
There are many tools for creating tests in Python. Some of these tools, such as
pytest, replace the built-in unittest framework. Other tools, such as nose, are
extensions that ease test case creation. Note that many of these tools are also
used for integration testing by writing the test cases to exercise multiple parts
of code at once.
• unittest is the built-in standard library tool for testing Python code.
• pytest is a complete testing tool that emphasizes backwardscompatibility and minimizing boilerplate code.
• nose is an extension to unittest that makes it easier to create and
execute test cases.
• Hypothesis is a unit test-generation tool that assists developers in
creating tests that exercise edge cases in code blocks. The best way to
get started using Hypothesis is by going through the well-written
quickstart.
• mimesis generates synthetic test data which is useful to apply in your
tests.
• testify was a testing framework meant to replace the common
unittest+nose combination. However, the team behind testify is
transitioning to pytest so it's recommended you do not use testify for
new projects.
Unit testing resources
Unit tests are useful in every project regardless of programming language.
The following resources provide a good overview of unit testing from several
viewpoints and follow up with additional depth in testing Python-specific
applications.
288
Full Stack Python: 2020 Supporter's Edition
• Introduction to Unit Testing provides a broad introduction to unit
testing, its importance and how to get started in your projects.
• Unit Testing Your Twilio App Using Python’s Flask and Nose is a
detailed tutorial for using the nose test runner for ensuring a Flask
application is working properly.
• Understanding unit testing explains why testing is important and
shows how to do it effectively in your applications.
• Unit testing with Python provides a high-level overview of testing and
has diagrams to demonstrate what's going on in the testing cycle.
• The Python wiki has a page with a list of Python testing tools and
extensions.
• An Extended Introduction to the nose Unit Testing Framework shows
how this test runner can be used to write basic test suites. While the
article is from 2006, it remains relevant today for learning how to use
nose with your projects.
• Revisiting Unit Testing and Mocking in Python is a wonderful post with
many code examples showing how and why to use dependency
injection and @property to mock unit tests.
• Unit Testing Doesn’t Affect Codebases the Way You Would Think
presents research on how unit testing impacts project code and ways
that it does not. It is only one research report but the findings on more
unit tests leading to higher Cyclomatic Complexity per method are
interesting. Perhaps more tests are needed to keep a project running
due to the increased complexity.
• Python unittest with Robert Collins is the transcript of an interview
with Robert Collins who is a core committer of unittest .
• Precise Unit Tests with PyHamcrest is a short tutorial on using the
PyHamcrest assertion tool for unit testing.
• Why most unit testing is a waste discusses how low risk unit tests
rarely fail even as the code changes and why they do not matter as
much to a project's health as many developers are led to believe based
on the test-driven development dogma.
289
Full Stack Python: 2020 Supporter's Edition
• Writing Unit Tests for Django Migrations digs into code examples for
testing Django data migrations.
• The business case for unit testing makes arguments for management
should buy into unit testing, including risk management and faster
development timelines.
• Unit testing often requires a bunch of fake data to use as inputs that
otherwise are generated by users or oother parts of a system not under
test. Fake data generation tools can help create data instead of having
too write it all yourself. This post on generating fake data with Faker
shows how to use the Faker tool with various seeds and outputs into
your tests.
• Unit Testing Applications that use Flask-Login and Flask-SocketIO
explains how to set up a WebSockets unit test with a couple of common
Flask libraries.
Integration Testing
Integration testing exercises two or more parts of an application at once,
including the interactions between the parts, to determine if they function as
intended. This type of testing identifies defects in the interfaces between
disparate parts of a codebase as they invoke each other and pass data between
themselves.
How is integration testing different from unit testing?
While unit testing is used to find bugs in individual functions, integration
testing tests the system as a whole. These two approaches should be used
together, instead of doing just one approach over the other. When a system is
comprehensively unit tested, it makes integration testing far easier because
many of the bugs in the individual components will have already been found
and fixed.
As a codebase scales up, both unit and integration testing allow developers to
quickly identify breaking changes in their code. Many times these breaking
changes are unintended and wouldn't be known about until later in the
development cycle, potentially when an end user discovers the issue while
using the software. Automated unit and integration tests greatly increase the
290
Full Stack Python: 2020 Supporter's Edition
likelihood that bugs will be found as soon as possible during development so
they can be addressed immediately.
Integration testing resources
• Integration testing with Context Managers gives an example of a
system that needs integration tests and shows how context managers
can be used to address the problem.
• Pytest has a page on integration good practices that you'll likely want to
follow when testing your application.
• Integration testing, or how to sleep well at night explains what
integration tests are and gives an example. The example is coded in
Java but still relevant when you're learning about integration testing.
• What is an integration test exactly? is an awesome Stack Exchange
thread that defines the differences in testing approaches like unit tests
versus integration and other tests. There is also some practical advice
like "It’s not important what you call it, but what it does" which as a
pragmatic programmer I am keen to agree on.
• Consistent Selenium Testing in Python gives a spectacular code-driven
walkthrough for setting up Selenium along with SauceLabs for
continuous browser-based testing.
• Where do our flaky tests come from? presents Google's data on where
their integration tests fail and how the tools you use can sometimes
lead to higher incidents of failed tests than other testing tools.
• Unleash the test army covers the author's first impressions of using
Hypothesis for testing the properties of a system under test.
Debugging
Developers often find themselves in situations where the code they've written
is not working quite right. When that happens, a developer debugs their code
by instrumenting, executing and inspecting the code to determine what state
of the application does not match the assumptions of how the code should be
correctly running.
291
Full Stack Python: 2020 Supporter's Edition
Why is debugging important?
There are bugs in every modest sized or larger application. Every developer
has to learn how to debug code in order to write programs that work as
correctly as time and budget allow.
Debugging tools
There are many debugging tools, some of which are built into IDEs like
PyCharm and others that are standalone applications. The following list
contains mostly standalone tools that you can use in any development
environment.
• pdb is a debugger built into the Python standard library and is the one
most developers come across first when trying to debug their
programs.
• Web-PDB provides a web-based user interface for pdb to make it easier
to understand what is happening while inspecting your running code.
• wdb uses WebSockets to allow you to debug running Python code from
a web browser.
• Pyflame (source code) is a profiling tool that generates flame graphs for
executing Python program code.
• objgraph (source code) uses graphviz to draw the connections between
Python objects running in an application
pdb tutorials
pdb is the most commonly-used debugger for Python because it is built into
the standard library. The following walkthroughs will show you how to use
pdb while fixing your own code.
• How to Use Pdb to Debug Your Code is a wonderful code-first tutorial
on getting started with pdb.
• pdb - Interactive Debugger is featured on the Python Module of the
Week blog and has some great detail on using the program effectively.
• pdb: Using the Python debugger in Django is a tutorial specific to
working with pdb in Django projects.
292
Full Stack Python: 2020 Supporter's Edition
• Debugging your Python code walks through a scenario where pdb can
be used to find a defect in a block of Python code.
• pdb Tutorial is a code-heavy beginners tutorial for pdb.
• Debugging in Python elaborates on what pdb does and how it can be
used.
Python-specific debugging tutorials
The Python ecosystem has a range of tools to help with debugging your code.
These tutorials show you how to either use a tool other than pdb or provide an
overview of the debugging ecosystem for Python.
• Python debugging tools provides a list of tools such as pdb and its
derivatives ipdb, pudb and pdb++ along with how they can be used in
the hunt for defects.
• Profiling Python web applications with visual tools details a
configuration for visualizing code execution using KCachegrind.
• My Startling Encounter With Python Debuggers along with the followup second post are a fantastic couple of posts that walk through a
specific scenario of how a well-tested distributed web crawler failed
and how tools like gdb, top and Winpdb were used to debug a
multithreaded application.
• Debugging Python like a boss covers several Python debuggers such as
pudb, pydbgr and ipdb.
• The case of the mysterious Python crash explains the symptoms that
happened during a crash and what steps the author took to figure out
what was going on.
General debugging resources
The following resources are not specific to Python development but give solid
programming language-agnostic debugging advice.
• The art of debugging provides a whirlwind overview for how to fix
issues in your code.
293
Full Stack Python: 2020 Supporter's Edition
• How to debug JavaScript errors introduces some key debugging tools
such as source maps that make identifying errors on the client side
much easier during development.
• Linux debugging tools you'll love is an awesome comic that covers the
Linux ecosystem for debugging.
Code Metrics
Code metrics can be produced by static code analysis tools to determine
complexity and non-standard practices.
Why are code metrics important?
Code metrics allow developers to find problematic codebase areas that may
need refactoring. In addition, some metrics such as technical debt assist
developers in communicating to non-technical audiences why issues with a
system are occurring.
Open source code metrics projects
• Radon is a tool for obtaining raw metrics on line counts, Cyclomatic
Complexity, Halstead metrics and maintainability metrics.
• Pylint contains checkers for PEP8 code style compliance, design,
exceptions and many other source code analysis tools.
• PyFlakes parses source files for errors and reports on them.
• Pyntch is a static code analyzer that attempts to detect runtime errors.
It does not perform code style checking.
• Prospector inspects Python source code files to give data on type and
location of classes, methods and other related source information.
• Flake8 is a code format style guideline enforcer. Its goal is not to gather
metrics but ensure a consistent style in all of your Python programs for
maximum readability. The rules for Flask8 are all defined in this list,
which rolls up the Flake8 dependencies of pycodestyle, pyflakes and
McCabe.
294
Full Stack Python: 2020 Supporter's Edition
• Black is a Python code formatter with strong, uncompromising
assumptions about how your code must be formatted.
• dlint is a linter for secure coding practices.
• pylintdb puts pylint results into a SQLite database for programmatic
access and searching. Ned Batchelder coded it and wrote about how he
uses the program in this bite-sized command line tools: pylintdb post.
• Flask8-eradicate (source code) is a Flask8 plugin for identifying dead
code.
Hosted code metrics services
The following tools are ready to use by going to the service, punching in the
URL to your site, letting them perform their analysis and then reading the
results.
• Coveralls shows code coverage from test suites and other metrics to
help developers improve the quality of their code.
• webhint, formerly Sonarwhal scans your website for accessibility, speed
and security. There is both an online version that you can point at an
arbitrary URL as well as a command-line version for running yourself.
• Codecov hooks into GitHub, BitBucket or GitLab and reports code
coverage on your code repositories.
Code metrics resources
Code metrics are really useful when you have a team working on a project for
awhile and want to keep the code quality from degrading. However, you can
easily go overboard instrumenting everything and overanalyzing the results.
The following resources will introduce code metrics topics to you as well as
give perspective when metrics are useful to the point of diminishing returns.
• How do Ruby & Python profilers work? explains the difference between
sampling and tracing profilers then digs into how they work and their
advantages and disadvantages.
• Moving Fast With High Code Quality provides Quora's code quality
goals and how they handle code reviews with their internal tools.
295
Full Stack Python: 2020 Supporter's Edition
• Lessons from Building Static Analysis Tools at Google is a fantastic indepth read that explains why workflow integration and adapting to
developer feedback are critical for static analysis tools adoption in
development environments.
• Static Code Analizers for Python is an older article but goes over the
basics of what Python static code analyzers do.
• Getting Started with Pylint goes over setting up Pylint, generating the
.pylintrc file and what's in the configuration.
• Linting as Lightweight Defect Detection for Python presents a
straightforward code example of how linters can detect certain classes
of errors in code that especially in dynamically-typed languages are not
caught at compile time. The post then shows how to use Flake8 in your
own code reviews.
• Static Analysis at Scale explains how Instagram's extremely hightrafficked Python-powered site is enabled by linting, codemods, and
type-checking with Pyre.
• This /r/Python poll on what linters the community uses provides some
input on using PyCharm just for its linting features as well as some
other approaches.
• Automatically PEP8 & Format Your Python Code shows how to use the
autopep8 library, including an Emacs plugin that lints your code for
PEP8 compliance as you work.
• How to use Flask8 explains what Flask8 is, its usage and expected
output.
• Pylint false positives is a walkthrough of issues that Pylint detects in an
example project, which ones cannot be fixed and the ones where the
tool was incorrect. The author concludes that with all of the false
positives that were found the signal to noise ratio was not useful
enough to use the tool on a typical project. However, on a brand new
project without many dependencies it might be helpful to keep your
code in a pristine state before the code base grows beyond the nosiness
false positives threshold.
296
Full Stack Python: 2020 Supporter's Edition
• What is Flake8 and why we should use it? covers why using a linting
tool like Flake8 can improve the quality of your Python code and how
to install and configure it for your environment.
• Which Python static analysis tools should I use? covers Pylint, PyFlakes
and mypy with a short description of the advantages and disadvantage
for each one.
• This Stack Overflow question on Python static code analysis tools
contains comparison discussions of PyLint, PyChecker and PyFlakes.
• Consistent Python code with Black covers how to use Black and add it
as a pre-commit hook in Git to ensure consistency in repository
updates.
• Format Python however you like with Black is a super short
introduction on what Black does to keep your code formatting
consistent.
• Intro to Black – The Uncompromising Python Code Formatter contains
some introductory examples for using Black and shows how to use it on
your source files.
• Python static analysis tools is a quick overview of the features for Black,
Pylint, Pyflakes, Mypy, Bandit and Prospector.
Networking
Computing networking is critical to building reliable, performant Python web
applications.
Resources about networking
• Monitoring and Tuning the Linux Networking Stack: Receiving Data
along with Monitoring and Tuning the Linux Networking Stack:
Sending Data are incredibly detailed technical posts on the networking
layer within Linux operating systems.
• Computer networking is a free book that explains how networking
between computer systems works. There are also exercises for testing
what you learned along the way.
297
Full Stack Python: 2020 Supporter's Edition
• What's the history behind 192.168.1.1? Why not 192.169.1.1 or any
other IP address? When did it start being used? Who started it? Why?
Why not 1.1.1.1? What is the relation to 127.0.0.1? What about 10.0.0.1
(Apple)? is a nice answer on the history of IPv4 addressing and why
various IP addresses such as 192.168.1.1 became standards for localhost
or other local networking.
• We built network isolation for 1,500 services to make Monzo more
secure provides the thinking, processes and data analysis behind how
one team took a complex environment, separated the dependencies
and was able to improve their network. It's a great case study-style
article that has more detail than a lot of similar operations posts.
• Dropbox traffic infrastructure: Edge network explains how Dropbox
uses edge-of-the-network resources closer to the end user to optimize
performance of their service.
• On the shoulders of giants: recent changes in Internet traffic examines
how the COVID-19 pandemic of 2020 changed the times during the day
and locations of how the majority of internet traffic was routed and
consumed.
HTTPS
The HTTP Secure (HTTPS) protocol encyrpts data between a web server and
the client web browser.
HTTPS tutorials
• How To Secure Nginx with Let's Encrypt on Ubuntu 18.04 walks
through how to configure your Nginx server with HTTPS using a free
Let's Encrypt domain certificate.
• Switching Your Site to HTTPS on a Shoestring Budget shows the steps
for moving a GitHub Pages-based site or any site that can be behind
Cloudflare's free tier to HTTPS.
298
Full Stack Python: 2020 Supporter's Edition
Additional HTTPS resources
• The 6-Step "Happy Path" to HTTPS covers how to obtain a free SSL
certificate, permanently redirect HTTP to HTTPS and fix insecure
references to non-HTTPS resources.
• HTTPS on Stack Overflow: the end of a long road is a wonderfully indepth post on transitioning from HTTP to HTTPS, including
redirecting all HTTP requests to HTTPS, for a very high trafficked
website.
• TLS stats from 1.6 billion connections to mozilla.org provides realworld data for which ciphersuites to use based on mozilla.org
connections.
• HTTPS in the real world is both an accesible read and gives the nitty
gritty details beyond how the HTTPS handshake and transmission
work, along the assumptions it breaks down about where the most
likely attack vectors really come from.
• How Let's Encrypt Works is a primer on how the free and now widelyused certificate service grants and revokes domain certificates.
• Performing & Preventing SSL Stripping: A Plain-English Primer
explains the KRACK Attack for stealing data despite an HTTPS
connection and how your site needs to use HSTS to prevent the attack.
• HTTPS adoption has reached the tipping point shows data about the
growth of HTTPS and how most sites now serve more than half their
traffic via secure connections.
• Google's HTTPS transparency report card shows the growth of HTTPS
connections across Google's properties. There is a clear upward trend
but even some of Google's properties still are not 100% using HTTPS
over unencrypted HTTP connections.
• An overview of TLS 1.3 and TLS 1.3 - enhanced performance, hardened
security cover the high-level information on the latest approved version
of Transport Security Layer (TLS) 1.3.
• How https works is a fun cartoon illustration that demonstrates the
basic concepts of a secure HTTP connection.
299
Full Stack Python: 2020 Supporter's Edition
WebSockets
A WebSocket is a standard protocol for two-way data transfer between a client
and server. The WebSockets protocol does not run over HTTP, instead it is a
separate implementation on top of TCP.
Why use WebSockets?
A WebSocket connection allows full-duplex communication between a client
and server so that either side can push data to the other through an
established connection. The reason why WebSockets, along with the related
technologies of Server-sent Events (SSE) and WebRTC data channels, are
important is that HTTP is not meant for keeping open a connection for the
server to frequently push data to a web browser. Previously, most web
applications would implement long polling via frequent Asynchronous
JavaScript and XML (AJAX) requests as shown in the below diagram.
Server push is more efficient and scalable than long polling because the web
browser does not have to constantly ask for updates through a stream of
AJAX requests.
300
Full Stack Python: 2020 Supporter's Edition
While the above diagram shows a server pushing data to the client,
WebSockets is a full-duplex connection so the client can also push data to the
server as shown in the diagram below.
The WebSockets approach for server- and client-pushed updates works well
for certain categories of web applications such as chat room, which is why
that's often an example application for a WebSocket library.
301
Full Stack Python: 2020 Supporter's Edition
Implementing WebSockets
Both the web browser and the server must implement the WebSockets
protocol to establish and maintain the connection. There are important
implications for servers since WebSockets connections are long lived, unlike
typical HTTP connections.
A multi-threaded or multi-process based server cannot scale appropriately for
WebSockets because it is designed to open a connection, handle a request as
quickly as possible and then close the connection. An asynchronous server
such as Tornado or Green Unicorn monkey patched with gevent is necessary
for any practical WebSockets server-side implementation.
On the client side, it is not necessary to use a JavaScript library for
WebSockets. Web browsers that implement WebSockets will expose all
necessary client-side functionality through the WebSockets object.
However, a JavaScript wrapper library can make a developer's life easier by
implementing graceful degradation (often falling back to long-polling when
WebSockets are not supported) and by providing a wrapper around browserspecific WebSocket quirks. Examples of JavaScript client libraries and Python
implementations are shown in a section below.
Nginx WebSocket proxying
Nginx officially supports WebSocket proxying as of version 1.3. However, you
have to configure the Upgrade and Connection headers to ensure requests are
passed through Nginx to your WSGI server. It can be tricky to set this up the
first time.
Here are the configuration settings I use in my Nginx file as part of my
WebSockets proxy.
# this is where my WSGI server sits answering only on localhost
# usually this is Gunicorn monkey patched with gevent
upstream app_server_wsgiapp {
server localhost:5000 fail_timeout=0;
}
server {
# typical web server configuration goes here
# this section is specific to the WebSockets proxying
location /socket.io {
proxy_pass http://app_server_wsgiapp/socket.io;
302
Full Stack Python: 2020 Supporter's Edition
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 600;
}
}
Note if you run into any issues with the above example configuration you'll
want to scope out the official HTTP proxy module documentation.
The following resources are also helpful for setting up the configuration
properly.
• Nginx has an official page for WebSocket proxying.
• Proxying WebSockets with Nginx shows how to proxy with Socket.io.
Python WebSockets implementations
The following projects either implement WebSockets in Python or provide
example code you can follow to use WebSockets in your own projects.
• Autobahn uses Twisted and asyncio to create the server-side
WebSockets component while AutobahnJS assists on the client web
browser side.
• Django Channels is built on top of WebSockets and is easy to integrate
with existing or new Django projects.
• Flask-SocketIO is a Flask extension that relies upon eventlet or gevent
to create server-side WebSockets connections.
• websocket-client provides low level APIs for WebSockets and works
with both Python 2 and 3.
• Crossbar.io builds upon Autobahn and includes a separate server for
handling the WebSockets connections if desired by the web app
developer.
303
Full Stack Python: 2020 Supporter's Edition
JavaScript client libraries
JavaScript is used to create the client side of the WebSocket connection
because the client is typically a web browser. While you do not need one of
these client-side libraries to use WebSockets, they are useful for minimizing
the amount of boilerplate code to establish and handle your connections.
• Socket.io's client side JavaScript library can be used to connect to a
server side WebSockets implementation.
• web-socket-js is a Flash-based client-side WebSockets implementation.
• AutobahnJS assists on the client web browser side.
Example code for WebSockets in Python
There are numerous Python-based implementations for WebSockets so
sometimes it's just easiest to examine an example so you can build something
for your own project.
• The Flask-SocketIO project has a chat web application that demos
sending server generated events as well as input from users via a text
box input on a form.
• The realtime codenames game source code is a full-featured example
for using WebSockets via Flask-SocketIO. There is also a multi-part
tutorial that walks through the code.
• The python-websockets-example contains code to create a simple web
application that provides WebSockets using Flask, Flask-SocketIO and
gevent.
Python-specific WebSockets resources
• The "Async Python Web Apps with WebSockets & gevent" talk I gave at
San Francisco Python in January 2015 is a live-coded example Flask
web app implementation that allows the audience to interact with
WebSockets as I built out the application.
• Creating a Real-time Web-based Application using Flask, Vue, and
Socket.io: part 1, part 2 and part 3 are a complete front-to-backend
WebSockets, Python and JavaScript front end framework example with
open source code.
304
Full Stack Python: 2020 Supporter's Edition
• Real-time in Python provides Python-specific context for how the
server push updates were implemented in the past and how Python's
tools have evolved to perform server side updates.
• websockets is a WebSockets implementation for Python 3.3+ written
with the asyncio module (or with Tulip if you're working with Python
3.3).
• Speeding up Websockets 60X is a cool experiment in coding loops
different ways to eek out more performance from WebSockets
connections. It is unclear how generalizable the results in the blog post
are to other programs but it is a good example of how tweaking and
tuning can produce outsized returns in some applications.
• The Choose Your Own Adventure Presentations tutorial uses
WebSockets via gevent on the server and socketio.js for pushing vote
count updates from the server to the client.
• Adding Real Time to Django Applications shows how to use Django
and Crossbar.io to implement a publish/subscribe feature in the
application.
• Async with Bottle shows how to use greenlets to support WebSockets
with the Bottle web framework.
• If you're deploying to Heroku, there is a specific WebSockets guide for
getting your Python application up and running.
• The Reddit thread for this page has some interesting comments on
what's missing from the above content that I'm working to address.
• Creating Websockets Chat with Python shows code for a Twisted server
that handles WebSockets connections on the server side along with the
JavaScript code for the client side.
• Synchronize clients of a Flask application with WebSockets is a quick
tutorial showing how to use Flask, the Flask-SocketIO extension and
Socket.IO to update values between web browser clients when changes
occur.
305
Full Stack Python: 2020 Supporter's Edition
General WebSockets resources
WebSockets have wide browser support and therefore many web frameworks
across all major programming languages have libraries to make creating
WebSockets connections easier. The following resources are general guides
and tutorials that provide context for the protocol without getting into the
weeds of how to use WebSockets in Python.
• The official W3C candidate draft for WebSockets API and the working
draft for WebSockets are good reference material but can be tough for
those new to the WebSockets concepts. I recommend reading the
working draft after looking through some of the more beginner-friendly
resources list below.
• WebSockets 101 by Armin Ronacher provides a detailed assessment of
the subpar state of HTTP proxying in regards to WebSockets. He also
discusses the complexities of the WebSockets protocol including the
packet implementation.
• The "Can I Use?" website has a handy WebSockets reference chart for
which web browsers and specific versions support WebSockets.
• WebSockets for fun and profit has a nice concise overview of
WebSockets alternatives like long polling and Server-Sent Events (SSE)
before it goes into a WebSockets example that includes JavaScript code
for the client-side implementation.
• Mozilla's Developer Resources for WebSockets is a good place to find
documentation and tools for developing with WebSockets.
• WebSockets from Scratch gives a nice overview of the protocol then
shows how the lower-level pieces work with WebSockets, which are
often a black box to developers who only use libraries like Socket.IO.
• websocketd is a WebSockets server aiming to be the "CGI of
WebSockets". Worth a look.
• How WebSockets Work – With Socket.io Demo walks through the
HTTP-to-WebSocket upgrade handshake and explains a bit about how
WebSockets work. The provided code is NodeJS on the backend but the
SocketIO client side JavaScript is the same as you would implement in
a Python-backed web application.
306
Full Stack Python: 2020 Supporter's Edition
• Can WebSockets and HTTP/2 Co-exist? compares and contrasts the
two protocols and shows how they have differences which will likely
lead to WebSockets sticking around for awhile longer.
• A Brief Introduction to WebSockets and Socket.io by Saleh Hamadeh is
a video on WebSockets basics and using the Socket.io JavaScript
library to wrap WebSockets functionality in web browsers.
• Benchmarking and Scaling WebSockets: Handling 60000 concurrent
connections is a detailed examination of how WebSockets connections
can scale to tens of thousands of users.
• Writing WebSocket servers gets into the nitty-gritty of how
WebSockets work. Well worth reading to get a deep understanding of
WebSockets connections.
WebRTC
Web Real-Time Communications (WebRTC) is a specification for a protocol
implementation that enables web apps to transmit video, audio and data
streams between client (typically a web browser) and server (usually a web
server).
WebRTC tutorials
• How to Get Started Learning WebRTC Development explains what you
do and do not need to know as prerequisites for building with WebRTC
along with some sources for learning.
• This post titled WebRTC: a working example and the companion open
source repository provides a simple working example of WebRTC
technology, without any 3rd party dependencies. It allows 2 web
browsers to exchange audio and video streams by using the aiohttp
and python-socketio modules.
• A real world guide to WebRTC goes through WebRTC fundamentals
such as data channels, audio and video, screen sharing and file
transfers with the JavaScript code provided for each concept.
307
Full Stack Python: 2020 Supporter's Edition
• The Introduction to WebRTC video series (part 2 and part 3) can be a
bit dry at points but overall has a ton of good information that gives a
solid overview of the technology.
• Building a Snapchat-like app with WebRTC in the browser walks
through the front end JavaScript for building a photo filter application
using the WebRTC browser APIs.
• WebRTC issues and how to debug them explains the various ways that
implementations can go wrong and where to start looking when you
run into errors.
Other WebRTC resources
• A Study of WebRTC Security gives a great overview of WebRTC and the
new security concerns it can bring as it is integrated into more web
applications.
• How Discord Handles Two and Half Million Concurrent Voice Users
using WebRTC provides detailed insight into the what and why of the
highly scalable Discord technical architecture that relies upon WebRTC
for communication. There are a bunch of great examples here for why
some of the service must be centralized (to prevent client IP addresses
from leaking to other clients) while others are decentralized to assist
with scaling the number of possible connections.
• Architectures for a kickass WebRTC application is a video of a technical
talk that covers some of the tools and protocols that can be used to
create your WebRTC projects and why you would choose one tool other
another.
• WebRTC connection times and the power of playing around with data
provides data on connection times and potential reasons for WebRTC
connection quality suffers in some cases.
• A closer look into WebRTC covers the Safari WebRTC implementation
in WebKit and explains some of the nuances for that specific web
browser's implementation.
• STUN/TURN servers are used to relay data to a non-public IP address
in a WebRTC application. This blog post on Do you still need TURN if
308
Full Stack Python: 2020 Supporter's Edition
your media server has a public IP address? answers some frequently
asked questions about when a TURN server is truly required.
• An Intro to WebRTC and Accessing a User’s Media Devices goes into
the JavaScript needed to use a computer's media devices such as the
microphone and video camera through the web browser's APIs.
• AIORTC: An Asynchronous WebRTC Framework is an interview with
the developer of an async WebRTC framework that is built upon
asyncio.
Application Programming Interfaces
Application programming interfaces (APIs) provide machine-readable data
transfer and signaling between applications.
Why are APIs important?
HTML, CSS and JavaScript create human-readable webpages. However, those
webpages are not easily consumable by other machines.
Numerous scraping programs and libraries exist to rip data out of HTML but
it's simpler to consume data through APIs. For example, if you want the
content of a news article it's easier to get the content through an API than to
scrap the text out of the HTML.
Key API concepts
There are several key concepts that get thrown around in the APIs world. It's
best to understand these ideas first before diving into the API literature.
• Representation State Transfer (REST)
• Webhooks
• JavaScript Object Notation (JSON) and Extensible Markup Language
(XML)
• Endpoints
309
Full Stack Python: 2020 Supporter's Edition
What are Webhooks?
A webhook is a user-defined HTTP callback to a URL that executes when a
system condition is met. The call alerts the second system via a POST or GET
request and often passes data as well.
Webhooks are important because they enable two-way communication
initiation for APIs. Webhook flexibility comes in from their definition by the
API user instead of the API itself.
For example, in the Twilio API when a text message is sent to a Twilio phone
number Twilio sends an HTTP POST request webhook to the URL specified
by the user. The URL is defined in a text box on the number's page on Twilio
as shown below.
API open source projects
• Swagger is an open source project written in Scala that defines a
standard interface for RESTful APIs.
API resources
• Zapier has an APIs 101 free guide for what APIs are, why they are
valuable and how to use them properly.
310
Full Stack Python: 2020 Supporter's Edition
• What is REST? is a well-written overview of the REpresentational State
Transfer (REST) architecture proposed by Roy Fielding in his
dissertation.
• The list of public APIs in this Git repository is incredible and worth
examining if you are looking to find data sources for your projects.
• GET PUT POST is a newsletter just about APIs. Past issues have
included interviews with the developers behind Stripe, Dropbox and
Coinbase.
• Designing robust and predictable APIs with idempotency discusses
designing APIs for idempotency, which means guaranteeing that side
effects only occur once. This topic is especially important with web
APIs because network connections are and will always be unreliable so
you need to build knowing network problems will happen.
• What RESTful actually means does a fantastic job of laying out the
REST principles in plain language terms while giving some history on
how they came to be.
• What is a webhook? by Nick Quinlan is a plain English explanation for
what webhooks are and why they are necessary in the API world.
• Simplicity and Utility, or, Why SOAP Lost provides context for why
JSON-based web services are more common today than SOAP which
was popular in the early 2000s.
• API tools for every occasion provides a list of 10 tools that are really
helpful when working with APIs that are new in 2015.
APIs learning checklist
1. Learn the API concepts of machine-to-machine communication with
JSON and XML, endpoints and webhooks.
2. Integrate an API such as Twilio or Stripe into your web application.
Read the API integration section for more information.
3. Use a framework to create an API for your own application. Learn
about web API frameworks on the API creation page.
311
Full Stack Python: 2020 Supporter's Edition
4. Expose your web application's API so other applications can consume
data you want to share.
Microservices
Microservices are an application architecture style where independent, selfcontained programs with a single purpose each can communicate with each
other over a network. Typically, these microservices are able to be deployed
independently because they have strong separation of responsibilities via a
well-defined specification with significant backwards compatibility to avoid
sudden dependency breakage.
Why are microservices getting so much buzz?
Microservices follow in a long trend of software architecture patterns that
become all the rage. Previously, CORBA and (mostly XML-based) serviceoriented architectures (SOA) were the hip buzzword among ivory tower
architects.
However, microservices have more substance because they are typically based
on RESTful APIs that are far easier for actual software developers to use
compared with the previous complicated XML-based schemas thrown around
by enterprise software companies. In addition, successful applications begin
with a monolith-first approach using a single, shared application codebase
and deployment. Only after the application proves its usefulness is it then
broken down into microservice components to ease further development and
deployment. This approach is called the "monolith-first" or "MonolithFirst"
pattern.
Microservice resources
• Martin Fowler's microservices article is one of the best in-depth
explanations for what microservices are and why to consider them as
an architectural pattern.
• Why microservices? presents some of the advantages, such as the
dramatically increased number of deployments per day, that a welldone microservices architecture can provide in the right situation.
Many organizational environments won't allow this level of flexibility
but if yours is one that will, it's worth considering these points.
312
Full Stack Python: 2020 Supporter's Edition
• On monoliths and microservices provides some advice on using
microservices in a fairly early stage of a software project's lifecycle.
• Why Microservices? presents advantages microservices can bring to an
existing monolithic application where it is clear what needs to be
broken down into smaller components to make it easier to iterate and
maintain.
• Developing a RESTful microservice in Python is a good story of how an
aging Java project was replaced with a microservice built with Python
and Flask.
• Microservices: The essential practices first goes over what a monolith
application looks like then dives into what operations you need to
support potential microservices. For example, you really need to have
continuous integration and deployment already set up. This is a good
high-level overview of the topics many developers aren't aware of when
they embark on converting a monolith to microservices.
• Using Nginx to Load Balance Microservices explains how an Nginx
instance can use configuration values from etcd updated by confd as
the values are modified. This setup can be useful for load balancing
microservices as the backend services are brought up and taken down.
• How Microservices have changed and why they matter is a high level
overview of the topic with some quotes from various developers around
the industry.
• The State of Microservices Today provides some general trends and
broad data showing the increasing popularity of microservices heading
into 2016. This is more of an overview of the term than a tutorial but
useful context for both developers and non-developers.
• bla bla microservices bla bla is a transcript for a killer talk on
microservices that breaks down the important first principles of
distributed systems, including asynchronous communication, isolation,
autonomicity, single responsibility, exclusive state, and mobility. The
slides along with the accompanying text go into how reality gets messy
and how to embrace the constraints inherent in distributed systems.
313
Full Stack Python: 2020 Supporter's Edition
• In the Microservices with Docker, Flask, and React course bundle, you
will learn how to quickly spin up a reproducible development
environment with Docker to manage a number of microservices. Once
the app is up and running locally, you'll learn how to deploy it to an
Amazon EC2 instance. Finally, we'll look at scaling the services on
Amazon EC2 Container Service (ECS).
• Should I use microservices? contains a high-level perspective on why or
why not use microservices as an architectural choice.
• Zuul is open source proxy for combining multiple microservices into a
unified API call. Check out this post on Using Netflix Zuul to Proxy
your Microservices to learn more and get started using it.
• The Majestic Monolith explains the advantages of a monolithic
architecture and how it's worked amazingly well for the Basecamp
small development team.
• Developing a RESTful micro service in Python goes into detail on how
one development team rebuilt an existing Java application as a
microservice in Python with Flask.
• Documenting microservices has some good thoughts on how to explain
your microservice API to other developers such as clearly showing all of
the endpoints as well as the intersection of multiple endpoints.
• Best practices for building a microservice is an exhaustive (and
somewhat exhausting to read!) list with what you should think about as
you build your microservice.
• The Hardest Part About Microservices: Your Data presents a datacentric view on how to structure and transport data in a microservices
architecture.
• Deleting data distributed throughout your microservices architecture
examines how Twitter handles issues with discoverability, access and
erasure in their microservices-heavy production environment.
Webhooks
Webhooks are user-defined HTTP callbacks that allow interactions between
otherwise independent web applications.
314
Full Stack Python: 2020 Supporter's Edition
Webhooks projects
• Thorn is a Python framework for building webhooks and event-driven
applications. There is also an introduction post with more information
on using Thorn.
• PyWebhooks is a proof-of-concept library for building webhooks-based
services.
• Webhook Tester
Webhook resources
• What's a webhook? is a high-level explanation of this concept that also
contains some basic security considerations when using them.
• How to Listen for Webhooks with Python has code examples in both
Flask and Django for how to receive an HTTP POST webhook request,
as well as how to test it locally with Ngrok.
• Should you build a webhooks API?
• Webhooks do’s and dont’s: what we learned after integrating +100
APIs
• Why Are Webhooks Better Than Serverless Extensibility?
• Webhooks Provide an Efficient Alternative to API Polling is a high-level
overview of the advantages of webhooks over alternatives such as
constant polling for updates.
• Webhooks: the devil in the details
• How to use webhooks for recovering customers
Bots
Bots are software programs that combine requests, which are typically
provided as text, with contextual data, such as geolocation and payment
information, to appropriately handle the request and respond. Bots are often
also called "chatbots", "assistants" or "agents."
315
Full Stack Python: 2020 Supporter's Edition
Open source bot examples
• Limbo is an awesome Slack chatbot that provides a base for Python
code that otherwise would require boilerplate to handle the Slack API
events firehose.
• python-rtmbot is the bot framework for building Slack bots with the
Real Time Messaging (RTM) API over WebSockets.
• The GitHub bots search results and the bots GitHub topic contain tens
of thousands of example bots you take analyze to see how they are
built.
• Errbot can work with multiple backends such as Hipchat, Discord,
Slack and Telegram. It's designed to be deployed "as is" except for your
credentials but the Python source code can also be customized.
Python-specific bot resources
• How to Buid an SMS Slack Bot is a tutorial on using SMS text messages
to communicate with a Slack bot that can post and receive messages.
The bot is a good base for a more complicated Slack bot that could use
natural language processing or other more advanced parsing
techniques. Either Python 2 or 3 can be used with the code which is
also available on GitHub.
• Dropbox open sourced their security Slack bot, which is built in
Python. The bot converses with a user when backend systems detect
strange behavior on one of their accounts to check if there has been a
security breach.
• Making a Reddit + Facebook Messenger Bot builds a bot for two
platforms and shows how to deploy it to Heroku.
• Build a Slack Bot that Mimics Your Colleagues with Python is a
humorous post that uses the markovify Markov Chains library to
generate responses that are similar to ones other Slack users have said.
• A Slack bot with Python's asyncio shows how to connect a bot to Slack
via the web API using the Python 3 asyncio standard library.
• Facebook-Message-Bot is an open source Facebook Messenger bot
written in Python.
316
Full Stack Python: 2020 Supporter's Edition
• Build a Reddit bot is a four part tutorial series that starts with reading
posts, continues with replying to posts, automating the bot and finally
adding behavior and a personality to the bot.
Additional Bots resources
• Bots: An introduction for developers explains the technical details of
how to create Telegram bots.
• Building better bots with AWS Lex: Part 1 and part 2 show how to use
Amazon's service offering for better natural language processing in
your bots.
• Slack bot token leakage exposing business critical information is a
detailed look at a search on GitHub for Slack tokens that are used
mostly for bots but must be kept secret. Otherwise those tokens expose
the entire Slack team's messaging to outside parties.
• The Economist wrote a general piece on why bots look like they'll gain
adoption in various market segments. The piece doesn't have much
technical depth but it's a good overview of how some businesses are
looking at the opportunity.
• Three challenges you’re going to face when building a chatbot provides
insightful thoughts on problems to anticipate based on the author's
experience building, deploying and scaling chatbots.
• Bots won't replace apps is a fantastic piece by WeChat's product
manager on how text-based bots alone typically do not provide a good
user experience. Instead, chat apps with automated responses, user
data and basic web browser functionality are what has allowed bot
concepts to bloom in Asian markets. There's a lot of good information
in this post to unpack.
• Principles of bot design contains some general, common-sense ideas to
keep in mind when building bots such as do not pretend to be a human
(because it will be quickly discovered that your bot is not a human) and
keep it as simple as possible so people can actually use the damn thing.
• 6 things I learned creating my own Messenger chatbot contains some
solid general advice for building your custom bots.
317
Full Stack Python: 2020 Supporter's Edition
API Creation
Creating and exposing APIs allows your web application to interact with other
applications through machine-to-machine communication.
API creation frameworks
• Django REST framework and Tastypie are the two most widely used
API frameworks to use with Django. The edge currently goes to Django
REST framework based on rough community sentiment. Django REST
framework continues to knock out great releases after the 3.0 release
mark when Tom Christie ran a successful Kickstarter campaign.
• Flask-RESTful is widely used for creating web APIs with Flask. It was
originally open sourced by Twilio then moved into its own GitHub
organization so engineers from outside the company could be core
contributors.
• Flask API is another common library for exposing APIs from Flask web
applications.
• Sandman is a widely used tool to automatically generate a RESTful API
service from a legacy database without writing a line of code (though
it's easily extensible through code).
• Cornice is a REST framework for Pyramid.
• Restless is a lightweight API framework that aims to be framework
agnostic. The general concept is that you can use the same API code for
Django, Flask, Bottle, Pyramid or any other WSGI framework with
minimal porting effort.
• Eve is a Python REST framework built with Flask, MongoDB and
Redis. The framework's primary author Nicola Iarocci gave a great talk
at EuroPython 2014 that introduced the main features of the
framework.
• Falcon is a fast and lightweight framework well suited to create
RESTful APIs.
• Hug built on-top of Falcon and Python3 with an aim to make
developing Python driven APIs as simple as possible, but no simpler.
318
Full Stack Python: 2020 Supporter's Edition
Hug leverages Python3 annotations to automatically validate and
convert incoming and outgoing API parameters.
• Pycnic is a JSON-API-only framework designed with REST in mind.
API testing projects
Building, running and maintaining APIs requires as much effort as building,
running and maintaining a web application. API testing frameworks are the
equivalent of browser testing in the web application world.
• zato-apitest invokes HTTP APIs and provides hooks for running
through other testing frameworks.
• Tavern is a pytest plugin for automated API testing.
Hosted API testing services
• Runscope is an API testing SaaS application that can test both your
own APIs and external APIs that your application relies upon.
• API Science is focused on deep API testing, including multi-step API
calls and monitoring of external APIs.
API creation resources
• An API is only as good as its documentation is a strongly held mantra
in the web API world because so many APIs have poor documentation
that prevents ease-of-use. If an API is not well documented then
developers who have options to use something else will just skip it.
• 8 Open-Source Frameworks for Building APIs in Python presents a
high-level overview of the options for building APIs in Python.
• Adventures in running a free, public API is a quick story of a
developer's geolocation API being abused and his lack of resources for
preventing further abuse. Eventually he had to shut down the free plan
and only provide a paid plan in addition to allowing others to host the
open source code. Fraud and malware prevention are difficult
problems so keep an eye on server utilization and endpoint calls growth
to separate legitimate from illegitimate traffic.
319
Full Stack Python: 2020 Supporter's Edition
• API versioning is a wonderful article on the tradeoffs between
including an API version in the URL compared to other common ways
to version APIs.
• API Doc JS allows a developer to embed markup in their
documentation that will generate a site based on the endpoints
available in the API.
• 10 Reasons Why Developers Hate Your API (And what to do about it)
goes through the top difficulties and annoyances developers face when
working with APIs and how you can avoid your API falling into the
same traps.
• Versioning of RESTful APIs is a difficult and contentious topic in the
web API community. This two-part series covers various ways to
version your API and how to architect a version-less API.
• NARWHL is a practical API design site for developers confused about
what is appropriate for RESTful APIs.
• 18F's API standards explains the details behind their design decisions
on creating modern RESTful APIs.
• Design a beautiful REST API reviews common design decisions
regarding endpoints, versioning, errors and pagination. There is also a
source material YouTube video where this blog post derives its
recommendations from.
• Move Fast, Don't Break Your API are slides and a detailed blog post
from Amber Feng at Stripe about building an API, separating layers of
responsibility, hiding backwards compatibility and a whole slew of
other great advice for developers and API designers.
• Self-descriptive, isn't. Don't assume anything. is an appeal that
metadata makes a difference in whether APIs are descriptive or not.
• Designing the Artsy API has their recommendations list for building an
API based on their recent experiences.
• Hacker News had a discussion on what's the best way to write an API
spec? that provides a few different viewpoints on this topic.
320
Full Stack Python: 2020 Supporter's Edition
• Apigee's Web API Design ebook is free and contains a wealth of
practical advice for what design decisions to make for your web API.
• Documenting APIs: A guide for technical writers and engineers is a
guide that covers good practices for thinking like a developer who will
use your API, as well as what the documentation for endpoints and
other important pieces should look like.
• 1-to-1 Relationships and Subresources in REST APIs tells the story of
design decisions that were made during an API's creation and why
those choices were made.
• How many status codes does your API need? gives an answer from a
Dropbox API developer as to their decision making process.
• These two Stack Overflow questions and answers on Is it better to place
a REST API on a subdomain or in a subfolder? and subdomain vs.
subdirectory in web programming provide reasons and opinions on the
debate around using a subdomain, for example
api.fullstackpython.com versus www.fullstackpython.com/api/. There
is also a nice summary of endpoint configurations in this article from
ProgrammableWeb.
• This API Design Guide is based on Heroku's best practices for the
platform's API.
Python-specific API creation resources
• Deploying a Machine Learning Model as a REST API
• Choosing an API framework for Django by PyDanny contains questions
and insight into what makes a good API framework and which one you
should currently choose for Django.
• Creating Web APIs with Python and Flask is a free book on building
APIs with Flask as the core web framework.
• RESTful web services with Python is an interesting overview of the
Python API frameworks space.
• Implementing a RESTful Web API with Python & Flask is a good
walkthrough for coding a Flask app that provides standard web API
321
Full Stack Python: 2020 Supporter's Edition
functionality such as proper HTTP responses, authentication and
logging.
• REST Hooks is an open source Python project that makes it easier to
implement subscription-based "REST hooks". These REST hooks are
similar to webhooks, but provide a different mechanism for subscribing
to updates via a REST interface. Both REST hooks and webhooks are
far more efficient than polling for updates and notifications.
• Rate limiters provides a great overview of how limiting access in both
the number of requests per second as well as the number of concurrent
open connections can help keep your API alive during times of heavy
traffic.
• Writing HTTP files to test HTTP APIs shows how to perform
automated testing of APIs using HTTP file formats provided by VS
Code REST Client and the JetBrains HTTP Client Editor.
• Serialization is common for transforming objects into web API JSON
results. One company found the serialization performance of Django
REST framework was lacking so they created Serpy and wrote a blog
post with the results of its performance.
• Designing a Web API gives a detailed walkthrough of concepts and
design decisions you need to make when building an API.
• Microsoft's REST API Guidelines are a detailed set of considerations
for when you are building your own APIs that you want to be easilyconsumable by other developers.
• Designing Good Static REST API Documentation is about
documentation not APIs themselves, but it covers a critical topic if you
want your API to succeed: how to use the damn thing.
• Building better API docs shows how Square used Swagger with React to
create more helpful docs.
• Best Practices For Creating Useful API Documentation covers standard
but important topics such as knowing your audience, ensuring your
documentation covers the error codes, and providing a changelog as
well as terms of service.
322
Full Stack Python: 2020 Supporter's Edition
Django REST Framework resources
• This multi-part series on getting started with Django REST framework
and AngularJS (part 1) along with its second part do a good job of
showing how a RESTful API can serve as the backend for a client front
end built with a JavaScript MVC framework.
• If you're looking for a working example of a Django REST framework
project, check out the PokeAPI, open sourced under the BSD license.
API creation learning checklist
1. Pick an API framework appropriate for your web framework. For
Django I recommend Django REST framework and for Flask I
recommend Flask-RESTful.
2. Begin by building out a simple use case for the API. Generally the use
case will either involve data that users want in a machine-readable
format or a backend for alternative clients such as an iOS or Android
mobile app.
3. Add an authentication mechanism through OAuth or a token scheme.
4. Add rate limiting to the API if data usage volume could be a
performance issue. Also add basic metrics so you can determine how
often the API is being accessed and whether it is performing properly.
5. Provide ample documentation and a walkthrough for how the API can
be accessed and used.
6. Figure out other use cases and expand based on what you learned with
the initial API use case.
API Frameworks
API frameworks are code libraries that provide commonly-used functionality
when building your own web application programming interfaces (APIs).
Python API Frameworks
• Django REST Framework
• API Star
323
Full Stack Python: 2020 Supporter's Edition
• Starlette
• Flask RESTful
Django REST Framework
Django REST Framework (source code), typically abbreviated "DRF", is a
Python library for building web application programming interfaces (APIs).
Django REST Framework resources
• How to Developer APIs with Django REST Framework covers the steps
for creating a development environment for your Django+DRF project
then creating API endpoints with the test-driven development (TDD)
approach.
• The official DRF tutorial is one of the best first-party pieces of
documentation on any open source project. The rest of the DRF docs
are spectacular as well.
• Django: Building REST APIs is the first part in an excellent multi-part
series on DRF:
◦
◦
◦
◦
◦
Getting started with DRF
Serializers
ModelSerializer and Generic Views
ViewSet, ModelViewSet and Router
Authentication and Permissions
324
Full Stack Python: 2020 Supporter's Edition
◦ JSON Web Tokens (JWT)
• How to optimize your Django REST Viewsets provides a good step-bystep example about using select_related and prefetch_related
in the Django ORM layer to avoid large numbers of unnecessary
queries in your views. Also, props to the author for wearing a UVA tshirt in his picture when his blog says he works as a developer in
Blacksburg, Virginia (where Virginia Tech is located).
• How to Save Extra Data to a Django REST Framework Serializer is a
concise, handy tutorial for combining additional data with the alreadydefined DRF serializer fields before saving everything to a database or
similar action.
• Django polls api using Django REST Framework gives a great
walkthrough for creating a question polling application backend with
code and the explanation as you build it.
• Django REST Framework Permissions in Depth has code examples and
explains permission classes versus authentication classes in DRF.
• Optimizing slow Django REST Framework performance
• TLT: Serializing Authenticated User Data With Django REST
Framework
• Building an API with Django REST Framework and Class-Based Views
• Simple Nested API Using Django REST Framework
• Building APIs with Django and Django Rest Framework
API Integration
The majority of production Python web applications rely on several externally
hosted application programming interfaces (APIs). APIs are also commonly
referred to as third party services or external platforms. Examples include
Twilio for messaging and voice services, Stripe for payment processing and
Disqus for embedded webpage comments.
There are many articles about proper API design but best practices for
integrating APIs is less commonly written about. However, this subject
325
Full Stack Python: 2020 Supporter's Edition
continuously grows in importance because APIs provide critical functionality
across many implementation areas.
API Integration Resources
• APIs for Beginners is an awesome free video course that shows how to
use APIs and add them to applications with a bunch of useful code
examples.
• Some developers prefer to use Requests instead of an API's helper
library. In that case check out this tutorial on using requests to access
web APIs.
• How to Debug Common API Errors talks about the 1xx through 5xx
status codes you will receive when working with HTTP requests and
what to do when you get some of the more common ones such as 403
Forbidden and 502 Bad Gateway.
• Product Hunt lists many commonly used commercial and free web
APIs to show "there's an API for everything".
• There's a list of all government web APIs at 18F's API-All-the-X list.
The list is updated whenever a new API comes online.
• The Only Type of API Services I'll Use explains how alignment between
usage and pricing is crucial to a solid, long-lasting API experience.
Other models such as tiered usage and enterprise upsells typically lead
to terrible developer experiences and should generally be avoided when
building applications with APIs.
• John Sheehan's "Zen and the Art of API Maintenance" slides are
relevant for API integration.
• This post on "API Driven Development" by Randall Degges explains
how using APIs in your application cuts down on the amount of code
you have to write and maintain so you can launch your application
faster.
• Retries in Requests is a nice tutorial for easily re-executing failed HTTP
requests with the Requests library.
326
Full Stack Python: 2020 Supporter's Edition
• My DjangoCon 2013 talk dove into "Making Django Play Nice With
Third Party Services."
• If you're looking for a fun project that uses two web APIs within a
Django application, try out this tutorial to Build your own Pokédex
with Django, MMS and PokéAPI.
• vcr.py is a way to capture and replay HTTP requests with mocks. It's
extremely useful for testing API integrations.
• Caching external API requests is a good post on how to potentially limit
the number of HTTP calls required when accessing an external web
API via the Requests library.
• Working with APIs the Pythonic way does a nice job walking through
creating a naive web API client, building on it with better error
handling and then explaining how to test it by mocking out the service.
API integration learning checklist
1. Pick an API known for top notch documentation. Here's a list of ten
APIs that are a good starting point for beginners.
2. Read the API documentation for your chosen API. Figure out a simple
use case for how your application could be improved by using that API.
3. Before you start writing any code, play around with the API through
the commandline with cURL or in the browser with Postman. This
exercise will help you get a better understanding of API authentication
and the data required for requests and responses.
4. Evaluate whether to use a helper library or work with Requests. Helper
libraries are usually easier to get started with while Requests gives you
more control over the HTTP calls.
5. Move your API calls into a task queue so they do not block the HTTP
request-response cycle for your web application.
327
Full Stack Python: 2020 Supporter's Edition
Twilio
Twilio is a web application programming interface (API) that software
developers can use to add communications such as phone calling, messaging,
video and two-factor authentication into their Python applications.
Why is Twilio a good API choice?
Interacting with the standard telephone networks to send and receive phone
calls and text messages without Twilio is extremely difficult if you do not
know the unique telecommunications protocols such as Session Initiation
Protocol (SIP). Twilio's API abstracts the telecommunications pieces so as a
developer you can simply use your favorite programming languages and
frameworks in your application. For example, here's how you can send an
outbound SMS using a few lines of Python code:
# import the Twilio helper library (installed with pip install twilio)
from twilio.rest import TwilioRestClient
# replace the placeholder strings in the following code line with
# your Twilio Account SID and Auth Token from the Twilio Console
client = TwilioRestClient("ACxxxxxxxxxxxxxx", "zzzzzzzzzzzzz")
# change the "from_" number to your Twilio number and the "to" number
# to any phone number you want to send the message to
client.messages.create(to="+19732644152", from_="+12023358536",
body="Hello from Python!")
Learn more about the above code in the How to Send SMS Text Messages with
Python tutorial.
328
Full Stack Python: 2020 Supporter's Edition
How is Twilio's documentation for Python developers?
Twilio is a developer-focused company, rather than a traditional "enterprise
company", so their tutorials and documentation are written by developers for
fellow developers.
More Twilio resources
• Most Twilio Tutorials have idiomatic Python versions with entire open
source applications in Django and Flask.
• Automate the Boring Stuff with Python includes a chapter on sending
text messages that uses Twilio to get the job done.
• Build a Simpsons Quote-Bot with Twilio MMS, Frinkiac, and Python
combines picture messages sent via Twilio MMS with Frinkiac to create
and sent Simpsons cartoon quotes to any phone number.
• Finding free food with Python is a fun web scraping tutorial that uses
Beautiful Soup 4 to obtain some data from websites then uses the
Twilio SMS API via some Python code to send a text message with the
results.
• IBM's Bluemix blog contains a nice tutorial on building an IoT Python
app with a Raspberry Pi and Bluemix that uses Twilio to interact with
the Raspberry Pi.
• The Python tag on the Twilio blog provides walkthroughs for Django,
Flask and Bottle apps to learn from while building your own projects.
• Serverless Phone Number Validation with AWS Lambda, Python and
Twilio walks through using the Python runtime on AWS Lambda and
Twilio's APIs for validating phone number information.
• Receive Flask Error Alerts by Email with Twilio SendGrid shows how to
use Twilio to add email error reports to Flask web applications.
• Google Cloud recommends developers use Twilio for communications
in their apps and provides a short walkthrough for Python developers.
• SMS Tracking Notifications is a fun tutorial that combines two APIs
together - Twilio and Easypost to track packages sent through the
329
Full Stack Python: 2020 Supporter's Edition
Easypost service. There is also another tutorial on shipment
notifications specifically for Flask.
• Build a Video Chat Application with Python, JavaScript and Twilio
Programmable Video shows how to use Flask and the Twilio
Programmable Video API to build cross-platform video into new and
existing applications.
• This video titled "We're No Strangers to VoIP: Building the National
Rick Astley Hotline)" presents a hilarious hack that uses Python and
AWS Lambda as a serverless backend to power thousands of Rick
Rolling calls in several countries with Twilio.
• Stripe SMS Notifications via Twilio, Heroku, and Python is a quick
tutorial that demonstrates how to combine multiple services to receive
helpful notifications via SMS.
Disclaimer
I currently work at Twilio on the Developer Network and run the Developer
Voices team.
Stripe
Stripe is a web application programming interface (API) for processing
payments.
330
Full Stack Python: 2020 Supporter's Edition
Stripe tutorials
• How to Create a Subscription SaaS Application with Django and Stripe
shows how to build a Django application with models for the
subscription data in the Django ORM and create a pricing page.
• Switching from Braintree to Stripe covers one development team's
experience with moving payment providers.
• Dirt Cheap Recurring Payments with Stripe and AWS Lambda explains
how to use the Stripe API with AWS Lambda to handle recurring
payments instead of using a more expensive service like Chargify or
Recurly if you only have minimal requirements.
• Django Stripe Tutorial looks at how to quickly add Stripe to accept
payments on a Django/Python website.
• Setting up Stripe Connect with Django this tutorial looks at how to
integrate Stripe Connect into a Django application.
• Adding a Custom Stripe Checkout to a Flask App looks at how to add a
custom Stripe checkout to a Flask application for processing payments.
Resources about Stripe
• How Stripe Designs Beautiful Websites explains the process for how
Stripe creates their gorgeous design that makes people want to use the
service and explore what else they can build with it.
• Creating a Culture of Observability is a technical talk about monitoring
systems at scale. The presenter works at Stripe so much of his
• Implementing API Billing with Stripe covers billing and invoicing
requirements for a video calling API product. They explain how they
matched their requirements to what Stripe offers then what they had to
build themselves to get everything working the way they intended.
Slack
Slack provides a web application programming interface (API) for
programmatically interacting with its messaging service.
331
Full Stack Python: 2020 Supporter's Edition
Slack resources
• How to Build Your First Slack Bot with Python contains all the code for
getting a Slack bot up and running with Python even if you have not
previously worked with their API or built other bots.
• Use a Slack bot to deploy your app gives the sample code to a simplified
bot that you can engage with in your chat channels to perform
application deployments.
• How I built a Slack bot to help me find an apartment in San Francisco
is a story about how the author had issues finding an apartment while
moving from Boston to San Francisco. He started scraping Craigslist to
gather apartment data and built a Slack bot to message him as soon as
something that matched his criteria became available so he could take a
look at it.
• Slack on an SNES is not a Python tutorial but it provides a crazy hack
for communicating with Slack using a Super Nintendo.
• Hacking Slack accounts: As easy as searching GitHub explains how
secret Slack API keys are often committed to public GitHub
repositories which allows malicious actors to easily break into an
organization's messaging systems. Secret credentials in public
repositories is a problem for any API and it is a particular problem for
ones that are critical to a business' private communications.
• Serverless Slash Commands with Python shows how to build a
serverless Flask plus Zappa framework web app that is hosted on AWS
Lambda and can use the Slack API.
332
Full Stack Python: 2020 Supporter's Edition
• Hacking Slack using postMessage and WebSocket-reconnect to steal
your precious token examines a bug that the author found in Slack's
WebSockets reconnection operation that he reported to Slack. Slack
fixed the issue and paid him a bug bounty for his work.
• Posting messages to Slack using incoming webhooks and Python3
Requests API is a short script that uses the Requests library instead of
the Slack-provided Python helper libraries to interact with the API.
• Build a Google Analytics Slack Bot with Python walks through creating
a bot that posts Google Analytics data into Slack channels by
combining the Slack and Google APIs.
Example Slack bots
• python-rtmbot is the Slack-provided library for working with the Slack
API and WebSockets connection.
• slackify is a lightweight framework for building bots and the quickstart
walks you through how to create a simple example bot.
• slack-starterbot
• slack-api-python-examples contains the example code from several
Slack bot blog posts.
• slackbot another popular Slack bot implementation.
Okta
Okta is an application programming interface (API) for authentication and
identity management in web applications.
333
Full Stack Python: 2020 Supporter's Edition
Okta resources
• How to Add User Authentication to Flask Apps with Okta covers using
OpenID Connect with the Okta API for handling user authentication in
Flask applications.
Web Application Security
Website security must be thought about while building every level of the web
stack. However, this section includes topics that deserve particular treatment,
such as cross-site scripting (XSS), SQL injection, cross-site request forgery
and usage of public-private keypairs.
Security tools
• Bro is a network security and traffic monitor.
• lynis (source code) is a security audit tool that can run as a shell script
on a Linux system to find out its vulnerabilities so that you can fix them
instead of allowing them to be exploited by malicious actors.
• Charles is an HTTP proxy for inspecting headers, requests and
responses for all traffic that flows through it.
• TLS Observatory provides a suite of security tools for analyzing and
inspecting Transport Layer Security (TLS) services. There is also a
hosted version you can use at observatory.mozilla.org.
• WIG contains tools for gathering wireless data via Wifi protocols.
• HTTP Evader is an automated testing tool for checking firewalls to
ensure they are protecting the appropriate ports and payloads.
• Security monkey monitors for changes to AWS, Google Cloud, GitHub
and other infrastructure systems.
Specific vulnerabilities
• httpoxy is a set of vulnerabilities that can affect Python web application
servers via HTTP requests.
• Heartbleed is a vulnerability in OpenSSL implementations that must be
patched for any systems you run otherwise you are at serious risk for
data leakage.
334
Full Stack Python: 2020 Supporter's Edition
• Meltdown and Spectre are x86 architecture problems caused by
exploiting CPU branch-prediction implementations.
HTTPS resources
SSL over HTTP (HTTPS) is mandatory for securing web data traffic in transit.
There is a page dedicated to HTTPS and the following resources can also give
you a good overview of how HTTPS works.
• How does HTTPS actually work? is a well-written overview of the
protocol including certificates, signatures, signing and related topics.
• These introduction to HTTPS videos explain what HTTPS is and how to
implement it.
• This question asking what is the difference between TLS and SSL?
explains that TLS is a newer version of SSL and should be used because
SSL through version 3.0 is insecure.
• If you have wondered what all the SSL/TLS acronyms and settings
mean, read the Security/Server Side TLS guide which Mozilla uses to
operationalize its servers.
• If you're having users submit sensitive information to your site you
need to use SSL/TLS. Anything before TLS is now insecure. Check out
this handy guide that goes over some of the nuances of the subject.
• The Sorry State of SSL details the history and evolution of SSL/TLS.
There are important differences between the versions and Hynek
explains why TLS should always be used. The talk prompted work to
improve Python's SSL in 2.7.9 based on the upgrades in Python 3
outlined in The not-so-sorry state of SSL in Python.
• How HTTPS Secures Connections is a guide for what HTTPS does and
does not secure against.
• The first few milliseconds of an HTTPS connection provides a detailed
look at the SSL handshake process that is implemented by browsers
based on the RFC 2818 specification.
• Qualy SSL Server Test can be used to determine what's in place and
what is missing for your server's HTTPS connection. Once you run the
335
Full Stack Python: 2020 Supporter's Edition
test read this article on Getting an A+ on Qualy's SSL Labs Tester to
improve your situation.
General security resources
• The Open Web Application Security Project (OWASP) has cheat sheets
for security topics.
• Stanford's CS253 class is available for free online, including lecture
slides, videos and course materials to learn about web browser
internals, session attacks, fingerprinting, HTTPS and many other
fundamental topics.
• The SaaS CTO Security Checklist is an awesome list of steps for
securing your infrastructure and employees as well as what stage and
size company it is recommended that you put those procedures in
place.
• Reckon you've seen some stupid security things? Here, hold my beer...
provides hilarious, and terribly sad, security vulnerabilities and
weaknesses around encryption and password storage.
• This page contains a fantastic curated list of security reading material
from beginning to advanced topics.
• How to protect your infrastructure against the basic attacker presents a
good overview of what you need to think about when hardening your
system against reasonablely competent malicious attackers.
• The /r/netsec subreddit is one place to go to learn more about network
and application security.
• Hacking Tools Repository is a great list of password cracking, scanning,
sniffing and other security penetration testing tools.
• The EFF has a well written overview on what makes a good security
audit. It's broad but contains some of their behind the scenes thinking
on important considerations with security audits.
• Ubuntu system hardening guide provides step-by-step instructions for
hardening the most recent three Ubuntu LTS releases.
336
Full Stack Python: 2020 Supporter's Edition
• Ars Technica wrote posts on securing your website along with how to
set up a safe and secure web server: part 1 and part 2 to explain HTTPS
and SSL without much required pre-existing knowledge.
• Crypto 101 is an introductory course on cryptography for programmers.
• The first answer to the question "Why are salted hashes more secure
for password storage?" on Stack Overflow gives a wonderful
explanation for why this is an important technique to use to keep your
database passwords and other secrets more secure if the hashed strings
are leaked.
• An in-depth analysis of SSH attacks on Amazon EC2 shows how
important it is to secure your web servers, especially when they are
hosted in IP address ranges that are commonly scanned by malicious
actors.
• Cloud Security Auditing: Challenges and Emerging Approaches is a
high-level overview of some of security auditing problems that come
with cloud deployments.
• Wondering how the common buffer overflow attack works? Check out
this article on buffer overflows that explains the attack in layman's
terms.
• 7 Security Measures to Protect Your Servers provides a good overview
of the fundamentals for how servers should be configured for baseline
security.
• As you're developing on Linux, you'll want to read and follow this
Linux workstation security document to make sure your code and
environment are not compromised. If you're on Mac OS X, check out
this securing Yosemite guide which covers that environment.
• Timing attacks are one form of vulnerability that can be used to defeat
HTTPS in certain configurations. Understanding how those attacks
work is important in keeping your users' connections secure.
• Let's Encrypt at Scale shows an implementation for securing thousands
of sites with SSL certificates to support HTTPS everywhere.
337
Full Stack Python: 2020 Supporter's Edition
Web security learning checklist
1. Read and understand the major web application security flaws that are
commonly exploited by malicious actors. These include cross-site
request forgery (CSRF), cross-site scripting (XSS), SQL injection and
session hijacking. The OWASP top 10 web application vulnerabilities
list is a great place to get an overview of these topics.
2. Determine how the framework you've chosen mitigates these
vulnerabilities.
3. Ensure your code implements the mitigation techniques for your
framework.
4. Think like an attacker and actively work to break into your own system.
If you do not have enough experience to confidently break the security
consider hiring a known white hat attacker. Have her break the
application's security, report the easiest vulnerabilities to exploit in
your app and help implement protections against those weaknesses.
5. Recognize that no system is ever totally secure. However, the more
popular an application becomes the more attractive a target it is to
attackers. Reevaluate your web application security on a frequent basis.
SQL Injection
SQL injections are a category of web application security vulnerabilities that
can affect both relational databases and NoSQL data stores.
SQL Injection resources
• How security flaws work: SQL injection is an approachable primer on
the history and danger of how unsanitized inputs to a database work.
• Preventing SQL injections provides a PostgreSQL and psycopg2
example for how to avoid getting bit by a SQL injection vulnerability.
• Securing your site like it's 1999 covers a bunch of common web
application vulnerabilities including SQL injection.
338
Full Stack Python: 2020 Supporter's Edition
Cross-Site Request Forgery (CSRF)
Cross-Site Request Forgery is a type of web app vulnerability that forces users
to execute unwanted actions when authenticated to an application.
Cross-Site Request Forgery (CSRF) resources
• Preventing cross-site attacks using same-site cookies explains how
Dropbox's engineering team rolled out their same-site cookie defense
that augments other CSRF protections for users.
• Securing your site like it's 1999 covers many common web application
vulnerabilities including Cross-Site Request Forgery issues.
339
Full Stack Python: 2020 Supporter's Edition
Deployment
Deployment involves packaging up your web application and putting it in a
production environment that can run the app.
Why is deployment necessary?
Your web application must live somewhere other than your own desktop or
laptop. A production environment is the canonical version of your current
application and its associated data.
Deployment topics map
Python web application deployments are comprised of many pieces that need
to be individually configured. Here is a map that visually depicts how each
deployment topic relates to each other. Click the image to pull up a PDF
version.
341
Full Stack Python: 2020 Supporter's Edition
342
Full Stack Python: 2020 Supporter's Edition
Deployment hosting options
There are four options for deploying and hosting a web application:
1. "Bare metal" servers
2. Virtualized servers
3. Infrastructure-as-a-service
4. Platform-as-a-service
The first three options are similar. The deployer needs to provision one or
more servers with a Linux distribution. System packages, a web server, WSGI
server, database and the Python environment are then installed. Finally the
application can be pulled from source and installed in the environment.
Note that there are other ways of installing a Python web application through
system-specific package management systems. We won't cover those in this
guide as they are considered advanced deployment techniques.
Deployment tools
• teletraan is the deploy system used by the development teams at
Pinterest, a huge Python shop!
• pants is a build system originally created at Twitter and now split out
as its own sustainable open source project.
• Screwdriver is an open source build system originally developed at
Yahoo! that is now open source. Learn more about it in the
introduction post that contains the rationale for its creation.
Deployment resources
• Hello, production lays out the powerful philosophy of putting a project
into production as soon as possible in a project's lifecycle to establish
the pipeline, identify issues and bottlenecks, and build the foundation
for continuous delivery. The post also covers common objections and
provides some arguments to help you convince others that this strategy
is the right way to go on all projects.
343
Full Stack Python: 2020 Supporter's Edition
• Automated Continuous Deployment at Heroku explains Heroku's
deployment system, the checks they use to ensure code quality and
what they have learned from building the pipeline and process.
• Deploying Python web applications is an episode of the great Talk
Python to Me podcast series where I discuss deploying web
applications based on a fairly traditional virtual private server, Nginx
and Green Unicorn stack.
• Thoughts on web application deployment walks through stages of
deployment with source control, planning, continuous deployment and
monitoring the results.
• Deploying Software is a long must-read for understanding how to
deploy software properly.
• The evolution of code deploys at Reddit teaches the history, including
the mistakes, that Reddit's development teams learned as they scaled
up the development team and the traffic on one of the most-visited
websites in the world.
• Deployment strategies defined explains various ways that development
teams deploy applications, ranging from reckless to versioned.
• How we release so frequently provides a high-level overview of tactics
for how teams at large scale can deploy changes several times per day
or more with confidence the systems will not completely fail. There will
be bugs, but that does not mean the entire operation will stop.
• Hands-off deployment with Canary explains how SoundCloud
automates their deployment process and uses canary builds to identify
and roll back issues to mitigate reliability issues that can occur with
shipping software at scale.
• Practical continuous deployment defines delivery versus deployment
and walks through a continuous deployment workflow.
• 5 ways to deploy your Python application in 2017 is a talk from PyCon
US 2017 where Andrew Baker deploys the getting started Flask app
using Ngrok, Heroku, Zappa on the serverless AWS Lambda platform,
a virtual machine on Google Cloud and Docker.
344
Full Stack Python: 2020 Supporter's Edition
• Continuous deployment at Instagram is the story of how their
deployment process evolved over time from a large Fabric script to
continuous deployments. Along the way they encountered issues with
code reviews, test failures, canary builds and rollbacks. It's a great read
that sheds some light on how Python deployments can be done well at
large scale.
• Stack Overflow's guide on how they do deployment is an awesome indepth read covering topics ranging from git branching to database
migrations.
• In this free video by Neal Ford, he talks about engineering practices for
continuous delivery. He explains the difference between continuous
integration, continuous deployment and continuous delivery. Highly
recommended for an overview of deployment concepts and as an
introduction to the other videos on those subjects in that series.
• TestDriven.io shows how to deploy a microservices architecture that
uses Docker, Flask, and React with container orchestration on Amazon
ECS.
Deployment learning checklist
1. If you're tight on time look at the platform-as-a-service (PaaS) options.
You can deploy a low traffic project web app for free or low cost. You
won't have to worry about setting up the operating system and web
server compared to going the traditional server route. In theory you
should be able to get your application live on the web sooner with PaaS
hosting.
2. Traditional server options are your best bet for learning how the entire
Python web stack works. You'll often save money with a virtual private
server instead of a platform-as-a-service as you scale up.
3. Read about servers, operating systems, web servers and WSGI servers
to get a broad picture of what components need to be set up to run a
Python web application.
345
Full Stack Python: 2020 Supporter's Edition
Hosting
A Python application must be deployed and hosted on one or more servers, or
a platform-as-a-service, to run.
Hosting resources
• An engineer’s guide to cloud capacity planning is a wonderful article
that discusses on-demand provisioning, horizontal and vertical scaling
and how to estimate performance of your infrastructure.
• Selecting a cloud provider reviews Etsy's decisionmaking around selfhosted infrastructure versus cloud hosting. They put together an
architectural model and ultimately decided to start migrating over to
Google Cloud Platform.
• Going Multi-Cloud with AWS and GCP: Lessons Learned at Scale
covers the compute, networking, persistent storage, billing and failover
aspects of using more than one infrastructure provider.
• Auth0 Architecture: Running In Multiple Cloud Providers And Regions
explains their multi-cloud architecture and how it has evolved over the
past several years.
• Choose A Cloud has a few posts with charts for easy cross-cloud
comparisons pricing on features such as persistent storage.
• How Netlify migrated to a fully multi-cloud infrastructure is another
story post about developing a multi-cloud architecture and the
considerations for disaster recovery, databases and testing.
Servers
Servers are the physical infrastructure to run all the layers of software so your
web application can respond to requests from clients such as web browsers.
Why are servers necessary?
Your web application must live somewhere other than your own desktop or
laptop. Servers should ideally be accessible 24 hours a day, 7 days a week,
with no unplanned downtime. The servers that host your web application for
actual users (as opposed to test users) are known as production servers.
346
Full Stack Python: 2020 Supporter's Edition
Production servers hold real data (again as opposed to test data) and must be
secure against unauthorized access.
Bare metal servers
The term bare metal refers to purchasing the actual hardware and hooking it
up to the Internet either through a business-class internet service provider
(ISP) or co-locating the server with other servers. A "business-class" ISP is
necessary because most residential Internet service agreements explicitly
prohibit running web servers on their networks. You may be able to get away
with low traffic volume but if your site serves a lot of traffic it will alert an
ISP's filters.
The bare metal option offers the most control over the server configuration,
usually has the highest performance for the price, but also is the most
expensive upfront option and the highest ongoing maintenance. With bare
metal servers the ongoing operating cost is the electricity the server(s) use as
well as handling repairs when server components malfunction. You're taking
on manual labor working with hardware as well as the rest of the software
stack.
Buy actual hardware from a vendor either pre-built or as a collection of
components that you assemble yourself. You can also buy pre-configured
servers from Dell or HP. Those servers tend to be in smaller case form factors
(called "blades") but are correspondingly more expensive than putting off-theshelf components together yourself in a standard computer case.
Virtualized servers
Virtual private servers (VPSs) are slices of hardware on top of a larger bare
metal server. Virtualization software such as Xen and VMWare allow
providers such as Linode and prgmr (as well as a many others) to provide
fractions of a full server that appear as their own instances. For example, a
server with an 8-core Xeon processor and 16 gigabytes of memory can be
sliced into 8 pieces with the equivalent of 1-core and 2 gigabytes of memory.
The primary disadvantage of virtualized servers is that there is resource
overhead in the virtualization process. In addition, physical constraints such
as heavy I/O operations by a single virtualized instance on persistent storage
347
Full Stack Python: 2020 Supporter's Edition
can cause performance bottlenecks for other virtualized instances on the
shared server. Choosing virtualized server hosting should be based on your
needs for urgency of service ticket requests and the frequency you require for
ongoing maintenance such as persistent storage backups.
Virtualized servers resources
• Choosing a low cost VPS reviews the factors that you should weigh
when deciding on hosting providers.
• How to set up your Linode for maximum awesomeness shows how to
work with a VPS once you've got the server up and running.
• CPU Load Averages explains how to measure CPU load and what to do
about it.
• Which cloud hosting company to choose in 2017? compares
DigitalOcean, Linode, Vultr, OVH and Scaleway in various benchmarks
such as CPUs, memory, disk space, network performance, traffic
capacity and cost. At the end of the article the author also provides
some qualitative feedback on the strengths and weaknesses of each
services' offerings.
Infrastructure-as-a-service
Infrastructure-as-a-service (IaaS) overlaps with virtualized servers because
the resources are often presented in the same way. The difference between
virtualized servers and IaaS is the granularity of the billing cycle. IaaS
generally encourages a finer granularity based on minutes or hours of server
usage instead of on monthly billing cycles.
IaaS can be used in combination with virtualized servers to provide dynamic
upscaling for heavy traffic. When traffic is low then virtualized servers can
solely be used. This combination of resources reduces cost at the expense of
greater complexity in the dynamically scaled infrastructure.
The most common IaaS platforms are Amazon Web Services and Rackspace
Cloud.
The disadvantage to IaaS platforms is the lock-in if you have to write custom
code to deploy, dynamically scale, and generally understand your
348
Full Stack Python: 2020 Supporter's Edition
infrastructure. Every platform has its quirks. For example, Amazon's standard
Elastic Block Store storage infrastructure has at least an order of magnitude
worse I/O throughput than working with your local disk. Your application's
database queries may work great locally but then when you deploy the
performance is inadequate. Amazon has higher throughput EBS instances but
you will pay correspondingly more for them. EBS throughput is just one of
many quirks you need to understand before committing to an IaaS platform.
Infrastructure-as-a-service resources
• The cloud versus dedicated servers
• 5 common server setups for your web application is a great
introduction to how hosting can be arranged.
• Apache Libcloud is a Python library that provides a unified API for
many cloud service providers.
• Amazon Web Services has official documentation for running Python
web applications.
• boto is an extensive and well-tested Python library for working with
Amazon Web Services.
• Poseidon is a Python commandline interface for managing Digital
Ocean droplets (servers).
Servers learning checklist
1. Sign up for a hosting provider. I recommend getting a Linode VPS to
set up your initial infrastructure and deploy your web application there.
Digital Ocean and prgrmr are other VPS options. You can change
hosting providers later after the deployment process is automated.
2. Provision your first server. It will be ready but in a shutdown state
while awaiting your instructions.
3. Move to the operating systems section to learn how to load Ubuntu
14.04 LTS as a base OS for Python web applications.
349
Full Stack Python: 2020 Supporter's Edition
Static Content
Some content on a website does not change and therefore should be served up
either directly through the web server or a content delivery network (CDN).
Examples include JavaScript, image, and CSS files.
Types of static content
Static content can be either assets created as part of your development
process such as images on your landing page or user-generated content. The
Django framework calls these two categories assets and media.
Content delivery networks
A content delivery network (CDN) is a third party that stores and serves static
files. Amazon CloudFront, Akamai, and Rackspace Cloud Files are examples
of CDNs. The purpose of a CDN is to remove the load of static file requests
from web servers that are handling dynamic web content. For example, if you
have an nginx server that handles both static files and acts as a front for a
Green Unicorn WSGI server on a 512 megabyte virtual private server, the
nginx server will run into resource constraints under heavy traffic. A CDN can
remove the need to serve static assets from that nginx server so it can purely
act as a pass through for requests to the Green Unicorn WSGI server.
CDNs send content responses from data centers with the closest proximity to
the requester.
Static Content Resources
• Crushing, caching and CDN deployment in Django shows how to use
django-compressor and a CDN to scale static and media file serving.
• Using Amazon S3 to host your Django static files
• CDNs fail, but your scripts don't have to
• django-storages is a Django library for managing static and media files
on services such as Amazon S3 and other content delivery networks.
• RevSys has a nice article on a range of important static file
optimizations such as setting cache headers, optimizing JavaScript and
reducing the size of images.
350
Full Stack Python: 2020 Supporter's Edition
• Twelve folks with significant experience working on and with CDNs
provide their perspectives in this piece: CDN experts on CDNs.
• Serving Static Files from Flask with WhiteNoise and Amazon
CloudFront looks at how to manage static files with Flask, WhiteNoise,
and Amazon CloudFront.
Static content learning checklist
1. Identify a content delivery network to offload serving static content
files from your local web server. I recommend using Amazon S3 with
CloudFront as it's easy to set up and will scale to high bandwidth
demands.
2. Update your web application deployment process so updated static files
are uploaded to the CDN.
3. Move static content serving from the www subdomain to a static (or
similarly named) subdomain so browsers will load static content in
parallel to www HTTP requests.
Content Delivery Networks (CDNs)
Content delivery networks (CDNs) serve static assets via globally distributed
servers to improve web app loading speed.
CDN resources
• Mastering HTTP Caching is a fantastic post that goes into great
technical detail on how CDNs and caching work.
• MaxCDN vs CloudFlare vs Amazon CloudFront vs Akamai Edge vs
Fastly compares and contrasts the most popular CDN services based on
features, performance and pricing. Note that Full Stack Python uses
Cloudflare to serve all content.
• Crushing, caching and CDN deployment in Django explains how to use
the django-compressor package with the django-storages library to
deploy static assets for a Django application to a CDN.
351
Full Stack Python: 2020 Supporter's Edition
• Building your own CDN for Fun and Profit is a great high-level
overview of how CDNs work and shows you how to create your own,
albeit simplified CDN.
• Do not let your CDN betray you: Use Subresource Integrity describes
the security implications for CDNs with unexpectedly modified content
and how Subresource Integrity in modern web browsers can mitigate
this vulnerability if used properly.
Virtual Private Servers (VPS)
Virtual private servers (VPSs) are sandboxed slices of hardware run with a
hypervisor running on top of a physical server. Virtualization software such as
Xen and VMWare allow a providers' customers to use fractions of a full server
that appear as their own independent instances. For example, a server with an
8-core processor and 16 gigabytes of memory can be roughly virtualized into 8
pieces with the equivalent of 1-core and 2 gigabytes of memory.
The primary disadvantage of virtualized servers is that there is resource
overhead in the virtualization process. But for our web application
deployment, a single well-configured virtual private server provides more
than enough performance and represents a huge cost savings over purchasing
dedicated hardware.
VPS providers
There are many VPS providers and their cost ranges dramatically based on
reliability, support, security and uptime. Make sure to choose a provider that
has a solid reputation unless you are willing to rebuild your server on another
provider whenever issues hit your service.
A few providers I currently use to host my Python web applications:
• Linode
• Digital Ocean
• Amazon Web Services' Lightsail
352
Full Stack Python: 2020 Supporter's Edition
VPS comparisons
• Ready. Steady. Go! The speed of VM creation and SSH access on AWS,
DigitalOcean, Linode, Vexxhost, Google Cloud, Rackspace and
Microsoft Azure and Comparing the speed of VM creation and SSH
access of cloud providers are one way to measure some of the
infrastructure speed provided by several cloud vendors. The virtual
machine and SSH access data points are taken in multiple regions. It's
unclear how these metrics would change over time based on backend
tweaks made by each provider.
• The State of Cloud Instance Provisioning explains the tools and
operations behind how AWS, DigitalOcean, Google Cloud and
Microsoft Azure stand up virtual machine instances for you to use.
• VPS $5 Showdown - October 2018 - DigitalOcean vs. Lightsail vs.
Linode vs. Vultr compares and contrasts the cheapest options for four
popular virtual private server providers.
• VPS comparisons uses Ansible to get some data around provisioning
speed and system performance. The whole README file in that
repository has a ton of useful information and summaries of the tested
providers.
Linode
Linode is a virtual private server service provider that is frequently used for
deploying production Python web applications.
Linode resources
• How I survived going viral on a $5 Linode shows the traffic behind a
high-trafficked site that runs on an inexpensive Linode virtual private
server. The author explains that because the site was a single-page
application with minimal JSON server requests it was able to easily
withstand the load from hundreds of concurrent connections.
DigitalOcean
DigitalOcean is a virtual private server provider and deployment platform that
can be used for running Python applications.
353
Full Stack Python: 2020 Supporter's Edition
DigitalOcean resources
• Creating a Kubernetes Cluster on DigitalOcean with Python and Fabric
shows how to configure a three node Kubernetes cluster using Ubuntu
16.04 LTS.
• Setting up a Digital Ocean server for Selenium, Chrome, and Python
gives the code and instructions for setting up a testing server that uses
Selenium for executing user interface tests.
• The digitalocean tag on GitHub has a slew of open source projects such
as digitalocean-developer-firewall to make it easier to configure
firewalls and other services on your droplets.
Useful tools for working with DigitalOcean
• python-digitalocean is a helper library for interacting with
DigitalOcean's APIs so you can, for example, spin up and shut down
your servers.
• vagrant-digitalocean is a Vagrant provider plugin for managing
DigitalOcean infrastructure.
Lightsail
Lightsail is Amazon Web Services' virtual private server service that can be
used for running production Python applications.
Platform-as-a-service
A platform-as-a-service (PaaS) provides infrastructure and a software layer on
which a web application is deployed. Running your web application from a
PaaS removes the need to know as much about the underlying servers,
operating system, web server, and often the WSGI server.
354
Full Stack Python: 2020 Supporter's Edition
Note: If you are not interested in deploying to a PaaS you can move ahead to
the WSGI servers section.
The PaaS layer defines how the application accesses resources such as
computing time, files, and external services. The PaaS provides a higher-level
abstraction for working with computing resources than deploying an
application to a server or IaaS.
A PaaS makes deployment and operations easier because it forces the
developer to conform applications to the PaaS architecture. For example,
Heroku looks for Python's requirements.txt file in the base directory of the
repository during deployment because that is the file's de facto community
standard location.
If you go the PaaS route, you can skip configuring an operating system and
web server prebaked into PaaS offerings. PaaS offerings generally start at the
WSGI server layer.
355
Full Stack Python: 2020 Supporter's Edition
Platform-as-a-service responsibilities
Although PaaS offerings simplify setting up and maintaining the servers,
operating system, and web server, developers still have responsibilities for
other layers of their web stack.
While it's useful to know the operating system that underpins your PaaS, for
example Heroku uses Ubuntu 10.04, you will not have to know as much about
securing the operating system and server level. However, web applications
deployed to a PaaS are just as vulnerable to security breaches at the
application level as a standard LAMP stack. It's still your responsibility to
ensure the web application framework and your app itself is up to date and
secured. See the security section for further information.
Platforms-as-a-service that support Python
• Heroku
• Google App Engine
• PythonAnywhere
• OpenShift
• AWS Elastic Beanstalk and AWS CodeStar are Amazon Web Services'
PaaS offerings. CodeStar is the newer service and recommended for
new projects.
Platform-as-a-service open source projects
The following open source projects allow you to host your own version of a
platform-as-a-service. Running one of these gives you the advantage of
controlling and modifying the project for your own applications, but prevents
you from offloading the responsibility of keeping servers running to someone
else.
• Kel uses Kubernetes as a foundation for a custom self-hosted PaaS.
Note that it was created by Eldarion, which had one of the first Pythonspecific PaaS offerings on the market around the time that Heroku was
launched.
• Dokku builds on Docker and has hooks for plugins to extend the small
core of the project and customize deployments for your applications.
356
Full Stack Python: 2020 Supporter's Edition
• Convox Rack is open source PaaS designed to run on top of AWS
services.
Platform-as-a-service resources
• The differences between IaaS, PaaS and SaaS explains the abstract
layer differences among "X-as-a-service" offering types and when to
consider using each one.
• PaaS bakeoff: Comparing Stackato, OpenShift, Dotcloud and Heroku
for Django hosting and deployment by Nate Aune.
• Deploying Django by Randall Degges is another great free resource
about Heroku.
• AWS in Plain English shows what current Amazon Web Services
individual services are currently called and what they could've been
called to be more clear to users.
• PAAS comparison - Dokku vs Flynn vs Deis vs Kubernetes vs Docker
Swarm in 2017 covers high-level advantages and disadvantages of
several self-hosted PaaS projects.
• 5 AWS mistakes you should avoid explains how common beginner
practices such as manually managing infrastructure, not using scaling
groups and underutilizing instances can create problems you'd be
better off avoiding altogether.
• Heroku's Python deployment documentation provides clear examples
for how to work with virtualenv, pip and requirements.txt to get a
applications deployed to their platform.
• Miguel Grinberg's Flask tutorial contains an entire post on deploying
Flask applications to Heroku.
• This series on DevOps Django by Randall Degges is great reading for
using the Heroku service:
◦
◦
◦
◦
Part One: Goals
Part Two: The Pain of Deployment
Part Three: The Heroku Way
Part Four: Choosing Heroku
357
Full Stack Python: 2020 Supporter's Edition
• Deploying a Django App to AWS Elastic Beanstalk is a fantastic post
that shows how to deploy to Amazon Web Service's own PaaS.
• Deploy your hack in 3 steps: Intro to AWS and Elastic Beanstalk shows
how to deploy a simple Ruby Sinatra app, but the steps are generally
applicable to Python web apps as well.
• Are you wondering what it will cost to deploy a reasonable sized
production app on a platform-as-a-service like Heroku? Check out
Cushion's transparent costs list where they include their expenses from
using a PaaS as well as other services.
• The beginner's guide to scaling to 11 million users on AWS is a useful
list of services you'll need to look at as you grow an application from 10
to 100 to 1000 to 500,000 and beyond to millions of users.
• How to Separate Your AWS Production and Development Accounts is a
basic post on keeping developer sandbox accounts separate from
production AWS environments.
• Deploying a Django app on Amazon EC2 instance is a detailed
walkthrough for deploying an example Django app to Amazon Web
Services.
• How much is Spotify Paying Google Cloud? provides some insight into
how Spotify runs all of their infrastructure on Google Cloud and posits
what they may be paying to run their service.
• PaaS (false) economics gives some quick back-of-the-envelope
calculations on why running your applications on a PaaS is obviously
going to appear more expensive if you do not take the cost of your own
software engineers into the equation.
• Two blog posts on using AWS Autoscaling in Automatic replacement of
Autoscaling nodes with equivalent spot instances and Autoscaling
nodes: seeing it in action provide a potential approach for making AWS
cheaper via autoscaling. While these posts may look a bit more
dfifficult than the Heroku dyno slider bar, if you're already using AWS
this should prove fairly easy to configure.
358
Full Stack Python: 2020 Supporter's Edition
Platform-as-a-service learning checklist
1. Review the potential Python platform-as-a-service options listed above.
2. Sign up for a PaaS account at the provider that appears to best fit your
application needs. Heroku is the PaaS option recommended for starters
due to their detailed documentation and walkthroughs available on the
web. However, the other options are also viable since their purpose is
to make deploying applications as easy as possible.
3. Check if there are any PaaS-specific configuration files needed for your
app to run properly on the PaaS after it is deployed.
4. Deploy your app to the PaaS.
5. Sync your application's configuration with the database.
6. Set up a content delivery network for your application's static content
unless your PaaS provider already handles this deployment step for
you.
7. Check if the application's functionality is working and tweak as
necessary.
Heroku
Heroku is an implementation of the platform-as-a-service (PaaS) concept that
can be used to more easily deploy Python applications.
359
Full Stack Python: 2020 Supporter's Edition
Heroku resources
• Migrating your Django Project to Heroku is a full tutorial on using
Heroku to run Django web applications. It includes instructions on
converting from MySQL to PostgreSQL if necessary as well as how to
properly handle your settings files.
• Heroku's official Python documentation is fantastic and walks through
deploying WSGI applications in short order.
• Production Django Deployments on Heroku aims to simplify the
process of deploying, maintaining, and scaling production-grade
Django apps on Heroku.
• Deploying Django to Heroku With Docker looks at how to deploy a
Django app to Heroku with Docker via the Heroku Container Runtime.
• Heroku Chatbot with Celery, WebSockets, and Redis is a walkthrough
with available source code to build a Django and Redis-based chat bot
that can be easily deployed to Heroku.
PythonAnywhere
PythonAnywhere is an implementation of the platform-as-a-service (PaaS)
concept and can be used to deploy and operate Python applications.
PythonAnywhere resources
• Turning a Python script into a website shows how to take a simple
script and deploy it to PythonAnywhere so you can have it running
somewhere other than your local machine.
AWS CodeStar
AWS CodeStar is a platform-as-a-service implementation and continuous
delivery deployment pipeline for running Python applications.
360
Full Stack Python: 2020 Supporter's Edition
Operating Systems
An operating system runs on the server or virtual server and controls access to
computing resources. The operating system also includes a way to install
programs necessary for running your Python web application.
Why are operating systems necessary?
An operating system makes many of the computing tasks we take for granted
easy. For example, the operating system enables writing to files,
communicating over a network and running multiple programs at once.
Otherwise you'd need to control the CPU, memory, network, graphics card,
and many other components with your own low-level implementation.
Without using an existing operating system like Linux, Mac OS X or
Windows, you'd be forced to write a new operating system as part of your web
application. It would be impossible to write features for your Python web
application because you'd be too busy hunting down a memory leak in your
assembly code, if you even were able to get that far.
Fortunately, the open source community provides Linux to the Python world
as a rock solid free operating system for running our applications.
Recommended operating systems
The only recommended operating systems for production Python web stack
deployments are Linux and FreeBSD. There are several Linux distributions
commonly used for running production servers. Ubuntu Long Term Support
(LTS) releases, Red Hat Enterprise Linux, and CentOS are all viable options.
361
Full Stack Python: 2020 Supporter's Edition
Mac OS X is fine for development activities. Windows and Mac OS X are not
appropriate for production deployments unless there is a major reason why
you must use them in lieu of Linux.
Canonical's Ubuntu Linux
Ubuntu is a Linux distribution packaged by the Canonical Ltd company.
Ubuntu uses the Debian distribution as a base for packages, including the
aptitude package manager. For desktop versions of Ubuntu, GNOME (until
the 11.04 release, then again in 18.04) or Unity (11.10 until 17.10) is bundled
with the distribution to provide a user interface.
Ubuntu Long Term Support (LTS) releases are the recommended versions to
use for deployments. LTS versions receive five years of post-release updates
from Canonical. Every two years, Canonical creates a new LTS release, which
allows for an easy upgrade path as well as flexibility in skipping every other
LTS release if necessary. As of May 2018, 18.04 Bionic Beaver is the latest
Ubuntu LTS release. Xenial Xerus includes Python 3.6 as its default Python
version, which is a major update compared with 2.7 in Ubuntu 14.04 LTS and
a solid improvement over Python 3.5 included in Ubuntu 16.04 LTS.
Red Hat and CentOS
Red Hat Enterprise Linux (RHEL) and Community ENTerprise Operating
System (CentOS) are the same distribution. The primary difference between
the two is that CentOS is an open source, liberally licensed free derivative of
RHEL.
RHEL and CentOS use a different package manager and command-line
interface from Debian-based Linux distributions: RPM Package Manager
(RPM) and the Yellowdog Updater, Modified (YUM). RPM has a specific .rpm
file format to handle the packaging and installation of libraries and
applications. YUM provides a command-line interface for interacting with the
RPM system.
Learning how operating systems work
• Linux Performance is an incredible site that links to a number of
performance-focused materials that are useful when developing on or
deploying to any Linux distribution.
362
Full Stack Python: 2020 Supporter's Edition
• Linux Journey is a really well designed curriculum for learning Linux
basics such as the command line, package management, text handling.
There are also courses for more advanced topics such as how the kernel
works, setting up logging and device management.
• The Ops School curriculum is a comprehensive resource for learning
about Linux fundamentals and how to perform the work that system
administrators typically handle.
• Since Linux is your go-to production operating system, it's important to
get comfortable with the Unix/Linux commands and philosophy. Study
up on this introduction to Unix tutorial to become more familiar with
the operating system.
• First 5 Minutes on a Server shows the first several security steps that
should be done manually or automatically on any server you stand up.
• How to Use the Command Line for Apple macOS and Linux is useful
for learning the shell and is even helpful for Windows now that the
Windows Subsystem for Linux (WSL) allows you to work with
Widnows as if it is a *nix operating system.
• Linux System Mining with Python shows how to gather system
information using the platform module and some of your own
Python code.
• Digital Ocean has a detailed walkthrough for setting up Python web
applications on Ubuntu.
• linux-internals is a series of posts about how Linux works under the
covers, starting from the low level booting process.
• While not quite necessary to run your Python application, if you want
to dig into how operating systems are built, check out this free book
How to Make a Computer Operating System, which was originally
written by a high school student and later updated as he became a
professional software developer.
• ops-class.org provides an online lecture videos, slides and sample
exams for learning how operating systems are built.
363
Full Stack Python: 2020 Supporter's Edition
• Operating Systems: Three Easy Pieces is a free book by University of
Wisconsin Computer Science professors that teaches how operating
systems are built. Although you do not know exactly how to build your
own OS to use one, understanding the foundation for how software
works is incredibly helpful in unexpected ways while developing and
operating your applications.
• Operating systems: From 0 to 1 is a self-learner resource for writing
your own operating system from scratch.
Choosing an OS resources
macOS and Linux are generally preferred by Python developers over Windows
because many Python packages like gevent simply do not work on Windows.
Others such as Ansible cannot be used as intended on Windows without
major hacks.
The following operating system resources cover perspectives on why
developers chose one operating system over others.
• Finding an alternative to Mac OS X: Part 1, part 2 and part 3: being
productive on Linux explain what alternative applications are available
for common functionality such as the Gnome windowing system, email
and terminal. There are a ton of tips and tricks in there for getting
comfortable with Linux as well as a lot of thought put into what and
why the developer wants his environment set up in a particular way.
• Why I switched from OS X to GNU/Linux explains the rationale for
switching from the Apple-based operating system to Linux along with
what applications the author now uses.
• Ultimate Linux on the Desktop explains one experienced developer's
Linux desktop development environment for getting coding work done.
• Lifehacker's guide to choosing a Linux distro.
• Distro chooser walks you through a set of sixteen questions to
determine which Linux distribution could fit your personal needs.
364
Full Stack Python: 2020 Supporter's Edition
Operating system learning checklist
1. Choose either a Debian-based Linux distribution such as Ubuntu or a
Fedora-based distribution like CentOS.
2. Harden the security through a few basic steps. Install basic security
packages such as fail2ban and unattended-upgrades. Create a new user
account with sudo privileges and disable root logins. Disable passwordonly logins and use a public-private keypair instead. Read more about
hardening systems in the resources listed below.
3. Install Python-specific packages to prepare the environment for
running a Python application. Which packages you'll need to install
depends on the distribution you've selected.
4. Read up on web servers as installing one will be the next step in the
deployment process.
Ubuntu
Ubuntu is a Debian Linux-based operating system distribution often used for
Python development and web application deployment.
Why is Ubuntu important for Python?
Ubuntu is one of the most commonly used Linux distributions for both local
development and server deployments. Some platforms-as-a-service such as
Heroku run Ubuntu as the base operating system, so as a Python developer
you'll often have to work with Ubuntu or a similar Debian-based Linux
operating system.
365
Full Stack Python: 2020 Supporter's Edition
What does "LTS" mean for Ubuntu?
Every two years Ubuntu releases a Long-Term Support (LTS) version that
receives five years of updates instead of only two years for non-LTS releases.
However, there are some issues with the current LTS model, in that you must
only use packages from the main repository unless you're going to manually
handle security updates for non-main repository system packages.
Additional Ubuntu resources
• Get your Python development environment set up with one of these
quick tutorials for Ubuntu 16.04 LTS:
◦ How to set up Python 3, Flask and Green Unicorn on Ubuntu
16.04 LTS
◦ Configuring Python 3, Bottle and Gunicorn for Development on
Ubuntu 16.04 LTS
◦ Setting up Python 3, Django and Gunicorn on Ubuntu 16.04 LTS
• There are also walkthroughs for configuring relational databases and
Redis on Ubuntu:
◦ Setting up PostgreSQL with Python 3 and psycopg on Ubuntu
16.04
◦ How to Install and Use MySQL on Ubuntu 16.04
◦ How to Use Redis with Python 3 and redis-py on Ubuntu 16.04
• Configuring Ubuntu for deep learning with Python is a great tutorial on
which packages you should install and why to use Python 3, OpenCV
and Keras on Ubuntu Linux.
• How to Use the Command Line for Apple macOS and Linux is a
fantastic guide relevant to Ubuntu users who should be able to use the
terminal to accomplish their tasks.
• Linux System Mining with Python shows how to use Python libraries to
gather Linux system information and work with it programmatically in
your applications.
• Canonical, the organization that produces Ubuntu, typically pushes the
boundaries on non-LTS releases, but occasionally rocks the boat with
major changes for an LTS release. 16.04 LTS was one such version,
366
Full Stack Python: 2020 Supporter's Edition
which is described in this article about how Ubuntu 16.04 proves even
an LTS release can live at Linux's bleeding edge.
• What I learned while securing Ubuntu explains how difficult it can be
just to find correct information on how to secure an operating system.
In this case, the author goes over how he went about securing package
management, security standards and file integrity on Ubuntu 14.04
LTS.
• In Beaver We Trust: A Lengthy, Pedantic Review of Ubuntu 18.04 LTS
examines the latest Ubuntu Long Term Support desktop release in
detail.
macOS
macOS is an operating system within the Unix family tree that is developed by
Apple and is often used for developing Python applications.
macOS Python resources
• Using Python on a Macintosh explains how Python 2.7 is installed by
default on Macs. You should start setting up your environment by
installing Python 3 and then follow the instructions in the post to get a
development environment configured.
• Python Development Environment on macOS High Sierra takes the
defualt macOS environment and gives you the step to configure
everything for Python development.
• How to Set Up Your Python Environment is a data science-flavored
configuration guide.
• How to Use the Command Line for Apple macOS and Linux explains
the Terminal. If this guide is useful to you then you should also check
out the shells and Bash pages.
Windows
Windows is a closed-source, proprietary operating system created by
Microsoft that is often used to develop Python applications.
367
Full Stack Python: 2020 Supporter's Edition
Useful resources for Python on Windows
• Windows for Linux nerds gives context for how Windows works
compared to Linux and digs into the Windows Subsystem for Linux
(WSL) which allows the operating system to handle Linux calls.
• How to set up the perfect modern dev environment on Windows
provides a good guide that includes configuring Vagrant, setting up the
Windows toolchain and updating your permissions to handle
development.
• Setting Up Python for Machine Learning on Windows presents detailed
steps for installing Anaconda and using it to install dependencies on
your local machine.
• PyInstaller packages a Python application with its associated
dependencies into a Windows executable file so it can be more easily
distributed and run on other computers.
• Cmder is a beautiful console emulator designed for Windows that can
be useful for when you need to get work done on the commandline.
• Setting up Python on Windows 10 is a short guide to making sure you
have Python 3 installed correctly on your Windows computer.
• Using both Python 2 and 3 in Windows explains how to configure
Windows so you can use both Python 2 and 3 if you still need to dig
into older projects that have not yet been updated.
• Epic Development Environment using Windows Subsystem for Linux
shows how to get your Windows development environment configured.
It is geared towards JavaScript/Node developers but most of the steps
will also be useful to Python developers.
• Windows dev box setup scripts are Powershell programs for setting up
a Windows machine for development.
368
Full Stack Python: 2020 Supporter's Edition
FreeBSD
FreeBSD is an operating system within the Unix family tree that can be used
used for developing Python applications.
FreeBSD Python resources
• How to Get Started with FreeBSD covers the first steps of logging into a
FreeBSD server with SSH, updating the root password and setting
your default shell. There is also a follow-up post for setting up your
SSH keys on FreeBSD.
• The FreeBSD Developer's Handbook shows the extensive
documentation available to you as you work with Python and other
programming languages on this operating system.
• Recommended Steps For New FreeBSD 12.0 Servers explains setting
time zones, setting up the IPFW Firewall and configuring NTP for
accurate time.
Web Servers
Web servers respond to Hypertext Transfer Protocol (HTTP) requests from
clients and send back a response containing a status code and often content
such as HTML, XML or JSON as well.
Why are web servers necessary?
Web servers are the ying to the web client's yang. The server and client speak
the standardized language of the World Wide Web. This standard language is
369
Full Stack Python: 2020 Supporter's Edition
why an old Mozilla Netscape browser can still talk to a modern Apache or
Nginx web server, even if it cannot properly render the page design like a
modern web browser can.
The basic language of the Web with the request and response cycle from client
to server then server back to client remains the same as it was when the Web
was invented by Tim Berners-Lee at CERN in 1989. Modern browsers and
web servers have simply extended the language of the Web to incorporate new
standards.
Web server implementations
The conceptual web server idea can be implemented in various ways. The
following web server implementations each have varying features, extensions
and configurations.
• The Apache HTTP Server has been the most commonly deployed web
server on the Internet for 20+ years.
• Nginx is the second most commonly used server for the top 100,000
websites and often serves as a reverse proxy for Python WSGI servers.
• Caddy is a newcomer to the web server scene and is focused on serving
the HTTP/2 protocol with HTTPS.
• rwasa is a newer web server written in Assembly with no external
dependencies that tuned to be faster than Nginx. The benchmarks are
worth taking a look at to see if this server could fit your needs if you
need the fastest performance trading off for as of yet untested web
server.
Client requests
A client that sends a request to a web server is usually a browser such as
Internet Explorer, Firefox, or Chrome, but it can also be a
•
•
•
•
headless browser, commonly use for testing, such as phantomjs
commandline utility, for example wget and cURL
text-based web browser such as Lynx
web crawler.
370
Full Stack Python: 2020 Supporter's Edition
Web servers process requests from the above clients. The result of the web
server's processing is a response code and commonly a content response.
Some status codes, such as 204 (No content) and 403 (Forbidden), do not
have content responses.
In a simple case, the client will request a static asset such as a picture or
JavaScript file. The file sits on the file system in a location the web server is
authorized to access and the web server sends the file to the client with a 200
status code. If the client already requested the file and the file has not
changed, the web server will pass back a 304 "Not modified" response
indicating the client already has the latest version of that file.
A web server sends files to a web browser based on the web browser's request.
In the first request, the browser accessed the "www.fullstackpython.com"
address and the server responded with the index.html HTML-formatted file.
That HTML file contained references to other files, such as style.css and
script.js that the browser then requested from the server.
Sending static assets (such as CSS and JavaScript files) can eat up a large
amount of bandwidth which is why using a Content Delivery Network (CDN)
to serve static assets is important when possible.
Building web servers
• A Simple Web Server in less than 500 lines of code from the
Architecture of Open Source book provides a great example with
Python as the implementation language..
371
Full Stack Python: 2020 Supporter's Edition
• If you're looking to learn about web servers by building one, here's part
one, part two and part three of a great tutorial that shows how to code a
web server in Python.
• Building a basic HTTP Server from scratch in Python (source code
builds a very simple but insecure web server to show you how HTTP
works.
Web server references
• HTTP/1.1 and HTTP/2 specifications are the source for how web
servers implement the modern web.
• A reference with the full list of HTTP status codes is provided by W3C.
• Usage of web servers broken down by ranking shows how popular
Apache, Nginx and other websites are among the top million through
the top 1,000 sites in the world.
• Apache vs Nginx: practical considerations gives an an overview of each
project, explains how each one handles connections, how it is
configured and the differences between the two web servers in how it
can use custom modules.
• Optimizing web servers for high throughput and low latency is a
wonderful read that shows how detailed knowledge at every layer of the
stack is necessary to optimize web server connections at scale.
• Implementing a tiny web server in a single printf call is an absurd C
language hack that you would never want to use in any real project but
still an amazing little application and wonderful explanation that you
can learn a bit more about web servers by reading.
• Top 5 open source web servers is a short overview of Apache, Nginx,
Lighttpd and two programming language specific servers, Node.js for
JavaScript and Tomcat for Java.
Web servers learning checklist
1. Choose a web server. Nginx is often recommended although Apache is
also a great choice.
372
Full Stack Python: 2020 Supporter's Edition
2. Create an SSL certificate via Let's Encrypt. You will need SSL for
serving HTTPS traffic and preventing myriad security issues that occur
with unencrypted user input.
3. Configure the web server to serve up static files such as CSS, JavaScript
and images.
4. Once you set up the WSGI server you'll need to configure the web
server as a pass through for dynamic content.
Apache HTTP Server
The Apache HTTP Server is a widely deployed web server that can be used in
combination with a WSGI module, such as mod_wsgi or a stand-alone WSGI
server to run Python web applications.
Why is the Apache HTTP Server important?
Apache remains the most commonly deployed web server with a reign of 20+
years. Its wide usage contributes to the large number of tutorials and open
source modules created by developers for anyone to use.
Apache's development began in mid-1994 as a fork of the NCSA HTTP Server
project. By early 1996, Apache overtook the previously dominant but suddenly
stagnant NCSA server as NCSA's progress stalled due to signficantly reduced
development attention.
373
Full Stack Python: 2020 Supporter's Edition
Apache HTTP Server resources
• The official project documentation page contains a section with HowTos and Tutorials to handle authentication, security and dynamic
content.
• Reverse proxies shows how to set up Apache as a reverse proxy using
mod_proxy .
• Deploy Django on Apache with Virtualenv and mod_wsgi provides
instructions for what packages to install to get Apache up and running
with mod_wsgi on Ubuntu.
• Detecting Bots in Apache & Nginx Logs is a great tutorial for filtering
out the significant traffic generated by web crawlers and bots when
using Apache HTTP Server logs for traffic analytics.
• Apache vs Nginx: Practical Considerations is a good comparison post
that covers the differences between Apache and Nginx such as how
they handle connections and serve content.
• Monitoring Apache web server performance gives a really nice
overview of metrics to watch when you are using Apache as your web
server.
• Web Performance 101: HTTP Headers covers the gamut of HTTP
headers and shows how they can impact performance based on your
configuration.
• Apache Web Server on Ubuntu 14.04 LTS explains how to install
Apache on Ubuntu 14.04, which is still a supported release. Note
however, do not install mod_python because it is now insecure and
made obsolete by mod_wsgi and WSGI servers.
Nginx
Nginx, pronounced "engine-X", is the second most common web server
among the top 100,000 websites. Nginx also functions well as a reverse proxy
to handle requests and pass back responses for Python WSGI servers or even
other web servers such as Apache.
374
Full Stack Python: 2020 Supporter's Edition
How is Nginx used in a Python web app deployment?
Nginx is commonly used as a web server to serve static assets such as images,
CSS and JavaScript to web browser clients.
Nginx is also typically configured as a reverse proxy, which passes appropriate
incoming HTTP requests to a WSGI server. The WSGI server produces
dynamic content by running Python code. When the WSGI server passes its
response, which is often in the HTML, JSON or XML format, the reverse
proxy then responds to the client with that result.
The request and response cycle with a reverse proxy server and the WSGI
server can be seen in the following diagram.
Typically the client will not know or need to know that a Python web
application generated the result. The result could have instead been generated
by one or more backend systems written in any programming language, not
just Python.
375
Full Stack Python: 2020 Supporter's Edition
Should I use Nginx or the Apache HTTP Server?
Let's be clear about these two "competing" servers: they are both fantastic
open source projects and either will serve your web application deployment
well. In fact, many of the top global web applications use both servers in their
deployments to function in many steps throughout the HTTP requestresponse cycle.
I personally use Nginx more frequently than Apache because Nginx's
configuration feel easier to write, with less boilerplate than alternatives.
There's also a bit of laziness in the usage: Nginx works well, it never causes me
problems. So I stick with my battle-tested Ansible configuration management
files that set up Nginx with HTTPS and SSL/TLS certificates
Securing Nginx
Nginx's default configuration after a standard installation through a system
package manager or compiling from source is a good base for security.
However, setting up ciphers and redirects can be confusing the first few times
you try it. It's a really good idea to read some of these tutorials to make sure
you are avoiding the most common security errors that plague HTTP(S)
configurations.
• HTTPS with Let's Encrypt and nginx walks throough installing a free
SSL certificate from Let's Encrypt to secure HTTP connects to your
nginx server via HTTPS.
• The Nginx Config tool can generate strong encryption configurations
and ciphers for Nginx.
• Gixy is a static analyzer for your Nginx configuration and can tell you
issues with how you are setup.
• Strong SSL Security on Nginx shows how to mitigate high profile SSL
attacks like Logjam, Heartbleed and FREAK.
Specific Nginx resources
Nginx can be used without Python so there are a massive number of fantastic
resources available for installing, configuring and optimizing this web server
implementation. The following resources are ones that I collected during my
376
Full Stack Python: 2020 Supporter's Edition
own struggle while learning how to use Nginx after I had used Apache HTTP
Server for several years.
• The Nginx chapter in the Architecture of Open Source Applications
book has a great chapter devoted to why Nginx is built to scale a certain
way and lessons learned along the development journey.
• nginx-quick-reference provides fantastic tactical advice for improving
Nginx performance, handling security and many other critical aspects.
• Inside Nginx: How we designed for performance and scale is a blog
post from the developers behind Nginx on why they believe their
architecture model is more performant and scalable than other
approaches used to build web servers.
• Test-driving web server configuration tells a good story for how to
iteratively apply configuration changes, such as routing traffic to Piwik
for web analytics, reverse proxying to backend application servers and
terminately TLS connections appropriately. It is impressive to read a
well-written softare development article like this from a government
agency, although UK's Government Digital Service as well as USA's 18F
and US Digital Service foster a far more credible culture than most
typical agencies.
• Hacker News broke our site – how Nginx and PageSpeed fixed the
problem is primarily about optimizing Nginx's configuration for more
efficient SSL connections. The post also covers configuration
management with Ansible as well as the Pagespeed module that Google
released for both Nginx and the Apache HTTP Server.
• Nginx for Developers: An Introduction provides the first steps to
getting an initial Nginx configuration up and running.
• A faster Web server: ripping out Apache for Nginx explains how Nginx
can be used instead of Apache in some cases for better performance.
• Nginx vs Apache: Our view is a first-party perspective written by the
developers behind Nginx as to the differences between the web servers.
• Rate Limiting with Nginx covers how to mitigate against brute force
password guessing attempts using Nginx rate limits.
377
Full Stack Python: 2020 Supporter's Edition
• Nginx with dynamic upstreams is an important note for setting up your
upstream WSGI server(s) if you're using Nginx as a reverse proxy with
hostnames that change.
• Nginx Caching shows how to set up Nginx for caching HTTP requests,
which is often done by Varnish but can also be handled by Nginx with
the proxy_cache and related directives.
• Dynamic log formats in nginx explains how to use the
HttpSetMiscModule module to transform variables in Nginx and map
input to controlled output in the logs. The author uses this technique
for pixel tracking but there are other purposes this method could be
used for such as advanced debugging.
• Detecting Bots in Apache & Nginx Logs is an awesome tutorial that
shows how to filter web crawlers and bots from your traffic logs when
using them for web traffic analytics.
Caddy
Caddy is a relatively new HTTP server created in 2015 and written in Go. The
server's philosophy and design emphasize HTTPS-everywhere along with the
HTTP/2 protocol.
How can Caddy be used with Python deployments?
Caddy can be used both for testing during local development or as part of a
production deployment as an HTTP server and a reverse proxy with the proxy
directive.
General Caddy resources
• A look inside Caddy shows and explains some of the Go code written to
build the server.
• The official Caddy server docs are the spot to look for what directives
can be placed into a Caddy configuration file
• Caddy a modern web server supporting HTTP/2 is a quick synopsis on
installing Caddy along with a short example configuration file.
378
Full Stack Python: 2020 Supporter's Edition
• HTTP 2.0 on localhost with Caddy shows how to use a self-signed
certificate with Caddy to do local development with an HTTP/2 web
server.
WSGI Servers
A Web Server Gateway Interface (WSGI) server implements the web server
side of the WSGI interface for running Python web applications.
Why is WSGI necessary?
A traditional web server does not understand or have any way to run Python
applications. In the late 1990s, a developer named Grisha Trubetskoy came up
with an Apache module called mod_python to execute arbitrary Python code.
For several years in the late 1990s and early 2000s, Apache configured with
mod_python ran most Python web applications.
However, mod_python wasn't a standard specification. It was just an
implementation that allowed Python code to run on a server. As
mod_python's development stalled and security vulnerabilities were
discovered there was recognition by the community that a consistent way to
execute Python code for web applications was needed.
Therefore the Python community came up with WSGI as a standard interface
that modules and containers could implement. WSGI is now the accepted
approach for running Python web applications.
As shown in the above diagram, a WSGI server simply invokes a callable
object on the WSGI application as defined by the PEP 3333 standard.
379
Full Stack Python: 2020 Supporter's Edition
WSGI's Purpose
Why use WSGI and not just point a web server directly at an application?
• WSGI gives you flexibility
flexibility. Application developers can swap out web
stack components for others. For example, a developer can switch from
Green Unicorn to uWSGI without modifying the application or
framework that implements WSGI. From PEP 3333:
The availability and widespread use of such an API in web servers for
Python [...] would separate choice of framework from choice of web
server, freeing users to choose a pairing that suits them, while freeing
framework and server developers to focus on their preferred area of
specialization.
• WSGI servers promote scaling
scaling. Serving thousands of requests for
dynamic content at once is the domain of WSGI servers, not
frameworks. WSGI servers handle processing requests from the web
server and deciding how to communicate those requests to an
application framework's process. The segregation of responsibilities is
important for efficiently scaling web traffic.
WSGI is by design a simple standard interface for running Python code. As a
web developer you won't need to know much more than
• what WSGI stands for (Web Server Gateway Inteface)
• that a WSGI container is a separate running process that runs on a
different port than your web server
• your web server is configured to pass requests to the WSGI container
which runs your web application, then pass the response (in the form
of HTML) back to the requester
380
Full Stack Python: 2020 Supporter's Edition
If you're using a standard web framework such as Django, Flask, or Bottle, or
almost any other current Python framework, you don't need to worry about
how frameworks implement the application side of the WSGI standard.
Likewise, if you're using a standard WSGI container such as Green Unicorn,
uWSGI, mod_wsgi, or gevent, you can get them running without worrying
about how they implement the WSGI standard.
However, knowing the WSGI standard and how these frameworks and
containers implement WSGI should be on your learning checklist though as
you become a more experienced Python web developer.
Official WSGI specifications
The WSGI standard v1.0 is specified in PEP 0333. As of September 2010,
WSGI v1.0 is superseded by PEP 3333, which defines the v1.0.1 WSGI
standard. If you're working with Python 2.x and you're compliant with PEP
0333, then you're also compliant with 3333. The newer version is simply an
update for Python 3 and has instructions for how unicode should be handled.
wsgiref in Python 2.x and wsgiref in Python 3.x are the reference
implementations of the WSGI specification built into Python's standard
library so it can be used to build WSGI servers and applications.
Example web server configuration
A web server's configuration specifies what requests should be passed to the
WSGI server to process. Once a request is processed and generated by the
WSGI server, the response is passed back through the web server and onto the
browser.
For example, this Nginx web server's configuration specifies that Nginx
should handle static assets (such as images, JavaScript, and CSS files) under
the /static directory and pass all other requests to the WSGI server running
on port 8000:
# this specifies that there is a WSGI server running on port 8000
upstream app_server_djangoapp {
server localhost:8000 fail_timeout=0;
}
# Nginx is set up to run on the standard HTTP port and listen for requests
server {
listen 80;
381
Full Stack Python: 2020 Supporter's Edition
# nginx should serve up static files and never send to the WSGI server
location /static {
autoindex on;
alias /srv/www/assets;
}
# requests that do not fall under /static are passed on to the WSGI
# server that was specified above running on port 8000
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
if (!-f $request_filename) {
proxy_pass http://app_server_djangoapp;
break;
}
}
}
Note that the above code is a simplified version of a production-ready Nginx
configuration. For real SSL and non-SSL templates, take a look at the
Underwear web server templates on GitHub.
WSGI server implementations
There is a comprehensive list of WSGI servers on the WSGI Read the Docs
page. The following are WSGI servers based on community
recommendations.
• Green Unicorn is a pre-fork worker model based server ported from the
Ruby Unicorn project.
• uWSGI is gaining steam as a highly-performant WSGI server
implementation.
• mod_wsgi is an Apache module implementing the WSGI specification.
• CherryPy is a pure Python web server that also functions as a WSGI
server.
WSGI resources
• PEP 0333 WSGI v1.0 and PEP 3333 WSGI v1.0.1 specifications.
• This basics of WSGI post contains a simple example of how a WSGIcompatible application works.
382
Full Stack Python: 2020 Supporter's Edition
• A comparison of web servers for Python web apps is a good read to
understand basic information about various WSGI server
implementations.
• A thorough and informative post for LAMP-stack hosting choices is
presented in the "complete single server Django stack tutorial."
• The Python community made a long effort to transition from
mod_python to the WSGI standard. That transition period is now
complete and an implementation of WSGI should always be used
instead mod_python.
• How to Deploy Python WSGI Applications with CherryPy answers why
CherryPy is a simple combination web and WSGI server along with
how to use it.
• Another Digital Ocean walkthrough goes into How to Deploy Python
WSGI Apps Using Gunicorn HTTP Server Behind Nginx.
• The uWSGI Swiss Army Knife shows how uWSGI can potentially be
used for more than just running the Python web application - it can
also serve static files and handle caching in a deployment.
WSGI servers learning checklist
1. Understand that WSGI is a standard Python specification for
applications and servers to implement.
2. Pick a WSGI server based on available documentation and tutorials.
Green Unicorn is a good one to start with since it's been around for
awhile.
3. Add the WSGI server to your server deployment.
4. Configure the web server to pass requests to the WSGI server for
appropriate URL patterns.
5. Test that the WSGI server responds to local requests but not direct
requests outside your infrastructure. The web server should be the pass
through for requests to and responses from the WSGI server.
383
Full Stack Python: 2020 Supporter's Edition
Green Unicorn (Gunicorn)
Green Unicorn, commonly shortened to "Gunicorn", is a Web Server Gateway
Interface (WSGI) server implementation that is commonly used to run Python
web applications.
Why is Gunicorn important?
Gunicorn is one of many WSGI server implementations, but it's particularly
important because it is a stable, commonly-used part of web app deployments
that's powered some of the largest Python-powered web applications in the
world, such as Instagram.
Gunicorn implements the PEP3333 WSGI server standard specification so
that it can run Python web applications that implement the application
interface. For example, if you write a web application with a web framework
such as Django, Flask or Bottle, then your application implements the WSGI
specification.
How does Gunicorn know how to run my web app?
Gunicorn knows how to run a web application based on the hook between the
WSGI server and the WSGI-compliant web app.
Here is an example of a typical Django web application and how it is run by
Gunicorn. We'll use the django_defaults as an example Django project.
Within the django_defaults project subdirectory, there is a short wsgi.py file
with the following contents:
"""
WSGI config for django_defaults project.
It exposes the WSGI callable as a module-level variable named ``application``.
For more information on this file, see
https://docs.djangoproject.com/en/1.8/howto/deployment/wsgi/
384
Full Stack Python: 2020 Supporter's Edition
"""
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "django_defaults.settings")
application = get_wsgi_application()
That wsgi.py file was generated by the django-admin.py startproject
command when the Django project was first created. Django exposes an
application variable via that wsgi.py file so that a WSGI server can use
application as a hook for running the web app. Here's how the situation
looks visually:
What's a "pre-fork" worker model?
Gunicorn is based on a pre-fork worker model, compared to a worker model
architecture. The pre-work worker model means that a master thread spins up
workers to handle requests but otherwise does not control how those workers
perform the request handling. Each worker is independent of the controller.
Gunicorn resources
• There are three framework-specific posts on the [Full Stack Python
blog](https://www.fullstackpython.com/blog.html for configuring
Gunicorn for development on Ubuntu:
1. Setting up Python 3, Django and Gunicorn on Ubuntu 16.04 LTS
2. How to set up Python 3, Flask and Green Unicorn on Ubuntu
16.04 LTS
3. Configuring Python 3, Bottle and Gunicorn for Development on
Ubuntu 16.04 LTS
385
Full Stack Python: 2020 Supporter's Edition
• gunicorn as your Django development server is a short post with a few
good tips on using Gunicorn for local application development.
• The Full Stack Python Guide to Deployments provides detailed step-bystep instructions for deploying Gunicorn as part of an entire Python
web application deployment.
• Flask on Nginx and Gunicorn combines the Nginx web server with
Gunicorn in a deployment to serve up a Flask application.
• The answers to the question "what's the best practice for running
Django with Gunicorn?" provide some nuance for how Gunicorn
should be invokin the callable application variable provided by Django
within a deployment.
• How to Install Django with Gunicorn and Nginx on FreeBSD 10.2 is a
tutorial for FreeBSD, which is not often used in walkthroughs
compared to the frequency that Ubuntu and CentOS tutorials appear.
• How to make a Scalable Python Web App using Flask, Gunicorn,
NGINX on Ubuntu 14.04 and Deploy a Flask App on Ubuntu both
provide steps for setting up a Flask web app using Gunicorn. There
isn't much explanation provided with each tutorial but they can still be
good concise references in case you're having issues with Ubuntu.
• Deploying a Flask Site Using Nginx, Gunicorn, Supervisor and
Virtualenv on Ubuntu is a similar tutorial to the previous two links. It
provides some good screenshots along the way with what to expect
while you are configuring the deployment server.
• The Django and Flask documentation each contain instructions for
deploying the respective frameworks with Gunicorn.
• Set up Django, Nginx and Gunicorn in a Virtualenv controled by
Supervisor is a GitHub Gist with some great explanations for why we're
setting up virtualenv and what to watch out for while you're doing the
deployment.
• The Gunicorn design document is worth reading because it describes
the server model and gives some context on how to choose the number
of workers for your execution environment.
386
Full Stack Python: 2020 Supporter's Edition
• Configuring Gunicorn for containers explains how to avoid excessive
slowness in your Docker containers that run Gunicorn workers, as well
as some tips on proper logging.
mod_wsgi
mod_wsgi (source code) is an implementation of the Web Server Gateway
Interface specification for running Python code in the Apache HTTP Server.
mod_wsgi is implemented as an Apache HTTP Server module and it supports
both Python 2 and Python 3 code.
mod_wsgi resources
• The official documentation is wonderful for getting started as well as
digging into how to configure your server for better security.
uWSGI
uWSGI (source code), pronounced "mu wiz gee", is a Web Server Gateway
Interface (WSGI) server implementation that is typically used to run Python
web applications.
uWSGI resources
• Configuring uWSGI for Production Deployment explains how
Bloomberg uses uWSGI as a production WSGI server for some of their
Python projects and how to set it up for your own applications.
387
Full Stack Python: 2020 Supporter's Edition
• The official Django framework docs on how to use Django with uWSGI
along with the corresponding official uWSGI Django docs are great
places to start when you are trying to deploy your Django project.
• The uWSGI Swiss Army Knife examines a few lesser-known uWSGI
features that support serving static files, working with SSL, caching and
asynchronous task execution.
• How To Serve Django Applications with uWSGI and Nginx on Debian 8
shows how to set up a Django web app on Debian Linux that uses
Nginx as a web server and reverse proxy for the uWSGI server.
• The official uWSGI quickstart is awesome because it shows you how to
code a quick WSGI application without using a framework then builds
up an example with deploying a traditional Django web app.
• Deployment Notes for Pylons, Nginx, and uWSGI gives the code and
instructions for setting up a Pylons application with uWSGI.
Continuous Integration
Continuous integration automates the building, testing and deploying of
applications. Software projects, whether created by a single individual or
entire teams, typically use continuous integration as a hub to ensure
important steps such as unit testing are automated rather than manual
processes.
Why is continuous integration important?
When continuous integration (CI) is established as a step in a software
project's development process it can dramatically reduce deployment times by
minimizing steps that require human intervention. The only minor downside
to using CI is that it takes some initial time by a developer to set up and then
there is some ongoing maintainence if a project is broken into multiple parts,
such as going from a monolith architecture to microservices.
Automated testing
Another major advantage with CI is that testing can be an automated step in
the deployment process. Broken deployments can be prevented by running a
comprehensive test suite of unit and integration tests when developers check
388
Full Stack Python: 2020 Supporter's Edition
in code to a source code repository. Any bugs accidentally introduced during a
check-in that are caught by the test suite are reported and prevent the
deployment from proceeding.
The automated testing on checked in source code can be thought of like the
bumper guards in bowling that prevent code quality from going too far off
track. CI combined with unit and integration tests check that any code
modifications do not break existing tests to ensure the software works as
intended.
Continuous integration example
The following picture represents a high level perspective on how continuous
integration and deployment can work.
In the above diagram, when new code is committed to a source repository
there is a hook that notifies the continuous integration server that new code
needs to be built (the continuous integration server could also poll the source
code repository if a notification is not possible).
The continuous integration server pulls the code to build and test it. If all tests
pass, the continuous integration server begins the deployment process. The
new code is pulled down to the server where the deployment is taking place.
389
Full Stack Python: 2020 Supporter's Edition
Finally the deployment process is completed via restarting services and
related deployment activities.
There are many other ways a continuous integration server and its
deployments can be structured. The above was just one example of a relatively
simple setup.
Open source CI projects
There are a variety of free and open source continuous integration servers that
are configurable based on a project's needs.
Note that many of these servers are not written in Python but work just fine
for Python applications. Polyglot organizations (ones that use more than a
single language and ecosystem) often use a single CI server for all of their
projects regardless of the programming language the application was written
in.
• Jenkins is a common CI server for building and deploying to test and
production servers. Jenkins source code is on GitHub.
• Go CD is a CI server by ThoughtWorks that was designed with best
practices for the build and test & release cycles in mind. Go CD source
code is on GitHub.
• Bazel is a build tool that works with CI tools to organize large code
bases and provide consistency with a well-defined, automated build
process.
• BuildBot is a continuous integration framework with a set of
components for creating your own CI server. It's written in Python and
intended for development teams that want more control over their
build and deployment pipeline. BuildBot source code is on GitHub.
• TeamCity is JetBrains' closed source CI server that requires a license to
use.
Jenkins CI resources
Jenkins is commonly used as a continuous integration server implementation
for Python projects because it is open source and programming language
390
Full Stack Python: 2020 Supporter's Edition
agnostic. Learn more via the following resources or on the dedicated Jenkins
page.
• My book on deploying Python web applications walks through every
step of setting up a Jenkins project with a WSGI application to enable
continuous delivery. Take a look if you're not grokking all of the steps
provided in these other blog posts.
• Assembling a continuous integration service for a Django project on
Jenkins shows how to set up a Ubuntu instance with a Jenkins server
that'll build a Django project.
• Setting up Jenkins as a continuous integration server for Django is
another solid tutorial that also shows how to send email notifications
as part of the build process.
General continuous integration resources
• What is continuous integration? is a classic detailed article by Martin
Fowler on the concepts behind CI and how to implement it.
• Continuous Deployment For Practical People is not specific to Python
but a great read on what it entails.
• Continuous Integration & Delivery - Illustrated uses well done
drawings to show how continuous integration and delivery works for
testing and managing data.
• The real difference between CI and CD explains what advantages CI
provides, what constraints it operates under (such as total build time)
to work well, and how that is different from the related but distinct
concept of continuous delivery.
• 6 top continuous integration tools gives a high level overview of six CI
tools from a programming language agnostic perspective.
• Updating the GOV.UK Continuous Integration environment explains
the UK's Government Digital Service continuous integration
configuration that relies on Jenkins.
• StackShare's Continuous Integration tag lists a slew of hosted CI
services roughly ranked by user upvotes.
391
Full Stack Python: 2020 Supporter's Edition
• Good practices for continuous integration includes advice on checking
in code, commit tests and reverting to previous revisions.
• Scoring Continuous Integration gives an interesting perspective on
ways to rank the effectiveness of how teams use their CI tooling.
• Why Continuous Integration Is Important is a high-level overview of
how CI can build trust both among developers and between developers
and non-technical people in an organization. The post also discusses
tasks related to setting up reliable CI such as test environments,
integration testing and visibility into the CI results.
• Continuous Intrusion: Why CI tools are a hacker's best friend (PDF)
strongly advises securing your continuous integration server just as you
would every other part of your production application, unless you want
your environment to be vulnerable to malicious actors.
• Measuring and Improving your CI/CD Pipelines provides metrics for
what you should measure with your CI/CD setup to improve the
process for helping your development teams ship code.
• Six rules for setting up continuous integration systems has some solid
general advice for culling problematic tests, ensuring the integration
speed supports the development culture you are building and keeping
all code in source control instead of having complicated logic
configured within the CI server.
• How to Identify Major Blockers in a CI/CD Pipeline gives a high level
overview of concepts such as shipping velocity, test execution and
environment provisioning with regards to CI configurations.
Jenkins
Jenkins is a continuous integration (CI) server often used to automate
building, testing and deploying Python applications.
392
Full Stack Python: 2020 Supporter's Edition
Jenkins resources
• Assembling a continuous integration service for a Django project on
Jenkins shows how to set up a Ubuntu instance with a Jenkins server
that'll build a Django project.
• My book on deploying Python web applications walks through every
step of setting up a Jenkins project with a WSGI application to enable
continuous delivery. Take a look if you're not grokking all of the steps
provided in these other blog posts.
• Revisiting Docker and Jenkins is a fantastic series of posts that
explains how Riot Games combines Jenkins and Docker to test their
back end services.
• Setting up Jenkins as a continuous integration server for Django is
another solid tutorial that also shows how to send email notifications
as part of the build process.
• If you're running into difficulty adding an SSH key to your Jenkins
system account so you can connect to another server or Git repository
this blog post on connecting Jenkins with Git to get the steps to solve
that problem.
• Running Jenkins in Docker Containers is a short tutorial showing how
to use the official Jenkins container on the Docker hub.
• Securing Jenkins is the landing page for Jenkins security. If you're
deploying your own instance, you'll need to lock it down against
unauthorized users.
393
Full Stack Python: 2020 Supporter's Edition
• Updating the GOV.UK Continuous Integration environment describes
the configuration improvements one infrastructure team made to
Jenkins, where they enabled jenkinsfiles to store CI data within project
repositories instead of having to handle the setup through the Jenkins
user interface. The team also published their Puppet files for building
Jenkins infrastructure.
• Jenkins configuration as code details the launch of a new Jenkins tool
for programmatically configuring Jenkins so you can automate the
setup of this part of your deployment infrastructure. The post goes into
the motivations behind creating another tool for code configuration
when other similar libraries already exist.
• Automated servers and deployments with Ansible & Jenkins covers
using Ansible for automating deployments and handling the
coordination via Jenkins builds.
• Building GitHub Pull Requests using Jenkins Pipelines explains how to
use Jenkins 2.0 with Pipelines to create builds that run in Docker
containers off of new GitHub pull requests.
• Automated API testing with Jenkins walks through how to use Jenkins
to tests your API upon each deployment.
• Continuous Delivery with Jenkins and Rollbar is a tutorial on using
Jenkins for continuous integration paired with Rollbar for tracking
deployments and errors.
GoCD
GoCD is a continuous integration (CI) server often used to automatically
build, test and deploy Python applications.
394
Full Stack Python: 2020 Supporter's Edition
GoCD resources
• The official getting started with GoCD guide is a solid place to begin
configuring the CI server and learning about its basic concepts.
• py-gocd is a code library for programmatically interacting with GoCD
from your Python code.
Configuration Management
Configuration management involves modifying servers from an existing state
to a desired state and automating how an application is deployed.
Configuration management tools
Numerous tools exist to modify server state in a controlled way, including
Puppet, Chef, SaltStack, and Ansible. Puppet and Chef are written in Ruby,
while SaltStack and Ansible are written in Python.
Ad hoc tasks
Configuration management tools such as Chef, Puppet, Ansible, and SaltStack
are not useful for performing ad hoc tasks that require interactive responses.
Fabric and Invoke are used for interactive operations, such as querying the
database from the Django manage.py shell.
395
Full Stack Python: 2020 Supporter's Edition
Configuration management tool comparisons
• Moving away from Puppet: SaltStack or Ansible? is an openly biased
but detailed post on why to choose SaltStack over Ansible in certain
situations.
• Ansible vs. Shell Scripts provides some perspective on why using a
configuration management tool is a better choice than venerable but
brittle shell scripts.
• Ansible vs. Chef is a comparsion of Ansible with the Chef configuration
management tool.
• This post on Ansible and Salt: A Detailed Comparison shows the
differences between these two Python-powered tools.
Configuration management resources
• A quick guide to choosing infrastructure tools presents a high-level
overview of the categories of tools you will need to perform
infrastructure-as-code in an organization.
Ansible configuration management
Ansible is an open source configuration management and application
deployment tool built in Python.
Ansible Resources
• An Ansible Tutorial is a fantastically detailed introduction on using
Ansible to set up servers.
• Ansible Text Message Notifications with Twilio SMS is my blog post
with a detailed example for using the Twilio module in core Ansible
1.6+.
• Python for Configuration Management with Ansible slides from PyCon
UK 2013
• First Steps with Ansible
• Red Badger on Ansible
• Getting Started with Ansible
396
Full Stack Python: 2020 Supporter's Edition
• An introduction to Ansible is a tutorial on the basics of getting started
with the tool.
• Ansible and Linode
• Multi-factor SSH authentication with Ansible and Duo Security
• Introducing Ansible into Legacy Projects
• Post-install steps with Ansible
• 6 practices for super smooth Ansible experience
• Create a Couchbase Cluster with Ansible
• Idempotence, convergence, and other silly fancy words we often use
• Testing with Jenkins, Docker and Ansible
Application dependencies learning checklist
1. Learn about configuration management in the context of deployment
automation and infrastructure-as-code.
2. Pick a configuration management tool and stick with it. My
recommendation is Ansible because it is by far the easiest tool to learn
and use.
3. Read your configuration management tool's documentation and, when
necessary, the source code.
4. Automate the configuration management and deployment for your
project. Note that this is by far the most time consuming step in this
checklist but will pay dividends every time you deploy your project.
5. Hook the automated deployment tool into your existing deployment
process.
Ansible
Ansible is a configuration management tool used for application deployment
and environment setup.
397
Full Stack Python: 2020 Supporter's Edition
Example Ansible playbooks
Ansible is far easier to learn when you can read how more full-featured
playbooks are built using many tasks. An interesting note from my own
experience is that when you get more experienced using Ansible there are
many shortcuts in the task syntax so you can often make playbooks that have
fewer lines of code than when you were less experienced yet the readability
does not suffer.
Check out some of these example playbooks to learn more about how you may
be able to structure your playbooks:
• The prod directory under the Full Stack Python Deployment Guide
open source project code contains a full playbook for deploying a
standard Nginx, Gunicorn and PostgreSQL stack.
• mac-dev-playbook configures macOS with various applications and
developer tools such as Docker, Homebrew and Sublime Text.
• ansible-nginx-haproxy-elasticsearch sets up a server with Nginx,
HAProxy and ElasticSearch.
Specific Ansible topics
• An Ansible2 Tutorial is an incredibly detailed look at how one
developer installs and run Ansible.
• This retrospective from a developer on lessons from using Ansible
exclusively for 2 years explains his rationale for choosing Ansible over
Puppet and Chef, then goes through several use cases and best
practices learned over time with the tool.
398
Full Stack Python: 2020 Supporter's Edition
• Using Ansible for deploying serverless applications provides a short
overview with an example playbook how Ansible can also be useful for
configuring serverless applications.
• Painless Immutable Infrastructure with Ansible and AWS covers the
steps needed for the unique authentication complexities that arise from
using Amazon Web Services for your infrastructure.
• DevOps from Scratch, Part 1: Vagrant & Ansible
• Ansible: Post-Install Setup
• How To Use Vault to Protect Sensitive Ansible Data on Ubuntu 16.04
• How to use Ansible Variables and Vaults
• CI for Ansible playbooks which require Ansible Vault protected
variables
• How to use Ansible to manage PostgreSQL
• Deploy A Replicated MongoDB instance on AWS with Terraform and
Ansible
Salt
Salt (source code) is a configuration management tool used for application
deployment and setting up development environments.
Salt resources
• What's new in Salt 3000 Neon covers the latest release and the
significant number of new features and fixes contained within it.
• Introduction to Salt gives a 30 second summary of what the tool can do
for you then provides a collection of links to other resources that plug
399
Full Stack Python: 2020 Supporter's Edition
you into the Salt community, such as the salt-users mailing list and the
Salt Stack company blog.
• Linode has two great beginner guides to Salt, the first one on Getting
Started with Salt - Basic Installation and Setup and the other titled A
Beginner's Guide to Salt.
Containers
Containers are an operating system-level isolation mechanism for running
processes and other system resources from other containers and the base
system.
Are containers new?
Containers are not conceptually new, dating back to around the 1970s, but
they gained rapid adoption starting in 2012 when several Linux distributions
began integrating containers tools generally made it more practical to use
them. Previously, to use containers a developer would need to use a less
common operating system or distribution customized with some sort of
virtual machine feature. Using containers was basically not supported in
typical deployment scenarios.
Containers history and introduction
The following resources do a great job of explaining where the containers
concept came from, how they differ from virtual machines and why they are
useful.
• The Missing Introduction To Containerization truly grants what its title
sets out to do: give a wide-ranging overview and history of container
concepts and tools. The post is dense but well worth the read to start
learning about chroot , Solaris zones, LXC, Docker and how they've
influenced each other throughout the past 40 years.
• A brief history of containers has some solid context for why containers
have taken off in the last several years, including the integration of
operating system container virtualization in most distributions as well
as the creation of management tools such as Docker, Kubernetes,
Docker Swarm and Mesosphere.
400
Full Stack Python: 2020 Supporter's Edition
• Setting the Record Straight: containers vs. Zones vs. Jails vs. VMs
compares and contrasts the designs of Linux containers, zones, jails
and virtual machines. Containers typically take advantage of primitives
but are more complicated because they have more individual parts put
together while zones and jails are designed as top-level operating
system components. There are advantages and disadvantages of these
approaches that you should understand as you use each one.
• Containers and Distributed Systems: Where They Came From and
Where They’re Going is an interview that digs into the past, present
and future of containers based on the experience of Chuck McManis
who has worked on building jails and other process isolation
abstractions into operating systems.
• Linux containers in a few lines of code shows how containers work by
providing some code to run a busybox Docker image but without using
docker. It then explains what's happening under the hood as you run
basic commands such as /bin/sh .
• A Practical Introduction to Container Terminology has both some solid
introductory information on containers as well as a good description of
terms such as container host, registry server, image layer, orchestration
and many others that come up frequently when using containers.
• Containers from scratch explains how Linux features such as cgroups ,
chroot and namespaces are used by container implementations.
• Running containers without Docker reviews a migration path for an
organization that already has a bunch of infrastructure but sees
advantages in using containers. However, the author explains why you
can use containers without Docker even if you eventually plan to use
Docker, Kubernetes or other container tools and orchestration layer.
• mocker is a Docker imitation open source project written in all Python
which is intended for learning purposes.
Working with containers
You can get started using containers once you understand some of the
terminology and work through a couple of introductory tutorials like the ones
401
Full Stack Python: 2020 Supporter's Edition
listed above. Check out the below resources when you want to do more
advanced configurations and dig deeper into how containers work.
• Linux containers in 500 lines of code is a bonkers in-depth post about
building your own simplified, but not simple version of Docker to learn
how it works.
• A Comparison of Linux Container Images presents data on many of the
frequently-used base container images.
• 7 best practices for building containers provides Google's
recommendations for creating containers such as include only a single
application per container, make sure to use descriptive tags and build
the smallest image size possible.
• Building healthier containers examines how Docker containers are
different from virtual machines and digs into dependencies that can be
included in your container image if you do not know how to properly
build them.
• Containers patterns covers common usage patterns that have
developed now that containers have been in development workflows
for a few years.
Container security resources
Container security is a hot topic because there are so many ways of screwing it
up, just like any infrastructure that runs your applications. These resources
explain security considerations specific to containers.
• Building Container Images Securely on Kubernetes discusses some of
the issues with building containers and why the author created img as a
tool to help solve the problems she was seeing.
• Making security invisible is a great presentation that covers sandboxes,
Seccomp and other concepts for isolating potentially unsafe code to
limit attack scope.
• 10 layers of Linux container security explains many of the attack
vectors you need to be aware of when you are working with containers.
402
Full Stack Python: 2020 Supporter's Edition
Docker
Docker (source code for core Docker project) is an infrastructure management
platform for running and deploying software. The Docker platform is evolving
so an exact definition is currently a moving target, but the core idea behind
Docker is that operating system-level containers are used as an abstraction
layer on top of regular servers for deployment and application operations.
Why is Docker important?
Docker can package up applications along with their necessary operating
system dependencies for easier deployment across environments. In the long
run it has the potential to be the abstraction layer that easily manages
containers running on top of any type of server, regardless of whether that
server is on Amazon Web Services, Google Compute Engine, Linode,
Rackspace or elsewhere.
Python projects within Docker images
• This Docker image contains a Flask application configured to run with
uWSGI and Nginx. You can also see the image on Docker hub.
• minimal-docker-python-setup contains an image with Nginx, uWSGI,
Redis and Flask.
Docker resources
• Beginners guide to Docker explains what it is, the difference between
containers and virtual machines, and then provides a hands-on
walkthrough command-driven tutorial.
403
Full Stack Python: 2020 Supporter's Edition
• What is Docker and When to Use It clearly delineates what Docker is
and what it isn't. This is a good article for when you're first wrapping
your head around Docker conceptually.
• rubber-docker is an open source repository and tutorial that shows you
how to recreate a simplified version of Docker to better understand
what it's doing under the hood.
• Andrew Baker presented a fantastic tutorial at PyOhio on beginner and
advanced Docker usage. Andrew also wrote the article what containers
can do for you.
• Docker curriculum is a detailed tutorial created by a developer to show
the exact steps for deploying an application that relies on Elasticsearch.
• How To Install and Use Docker on Ubuntu 16.04 provides a
walkthrough for Ubuntu 16.04 for installing and beginning to use
Docker for development.
• It Really is the Future discusses Docker and containers in the context of
whether it's all just a bunch of hype or if this is a real trend for
infrastructure automation. This is a great read to set the context for
why these tools are important.
• Docker Jumpstart is a comprehensive introduction to what Docker is
and how to get started with using the tool.
• If you want to quickly use Docker on Mac OS X, check out these concise
instructions for setting up your Docker workflow on OS X in 60
seconds.
• What the Bleep is Docker? is a plain English explanation with examples
of what Docker provides and what it can be used for in your
environment.
• Docker in Practice - A Guide for Engineers is an explanation of the
concepts and philosophy by the authors of the new Manning Docker
book in early access format.
• Building Docker containers from scratch is a short tutorial for creating
a Docker container with a specific configuration.
404
Full Stack Python: 2020 Supporter's Edition
• 10 things to avoid in Docker containers provides a lot of "don'ts" that
you'll want to consider before bumping up against the limitations of
how containers should be used.
• Docker Internals presents Linux containers and how Docker uses them
as its base for how the project works. This article is a great way to
bridge what you know about Docker with a more traditional Linux
operating system architecture understanding.
• Improve your Dockerfile, best practices covers image size, layers,
starting scripts and LABEL.
• This post gives an overview and comparison of Docker GUIs which can
be handy for monitoring your Docker containers.
Python-specific Docker resources
• Dockering Django, uWSGI and PostgreSQL the serious way walks
through both the code and the error messages that will likely crop up as
you attempt to container-ize a Django project that uses a PostgreSQL
database on the backend.
• How to deploy Django using Docker assumes you already have the
basic grasp of working with Docker and jumps right into a Django
deployment. The post shows you how to set up your Dockerfile and
explains that GitLab CI can be used to to build this Docker image.
• Hosting Python WSGI applications using Docker shows how to use
Docker in WSGI application deployments specifically using mod_wsgi.
• How to Containerize Python Web Applications is an extensive tutorial
that uses a Flask application and deploys it using a Docker container.
• How to use Django, PostgreSQL, and Docker shows how to get a
Django project that uses PostgreSQL as its back end running in Docker.
• Docker in Action - Fitter, Happier, More Productive is a killer tutorial
that shows how to combine Docker with CircleCI to continuously
deploy a Flask application.
405
Full Stack Python: 2020 Supporter's Edition
• Building smaller Python Docker images examines how to inspect layers
in Dockerfiles and minimize the overhead of what images contain for
better performance.
• The Flask Mega-Tutorial Part XIX: Deployment on Docker Containers
is one post in Miguel Grinberg's absolutely spectacular Flask
application series.
• Deploying Django Applications in Docker explains some of the
concepts behind using Docker for Python deployments and shows how
to specifically use it for deploying Django.
• Using Docker and Docker Compose to replace virtualenv is a tutorial
for using Docker instead of virtualenv for dependency isolation.
• Lincoln Loop wrote up a closer look at Docker from the perspective of
Python developers handling deployments.
• Curious how pip and Docker can be used together? Read this article on
Efficient management Python projects dependencies with Docker.
• Python virtual environments and Docker goes into detail on whether
virtual environments should be used with Docker and how system
packages can generally be a safer route to go.
• Dockerizing Django with Postgres, Gunicorn, and Nginx details how to
configure Django to run on Docker along with Postgres, Nginx and
Gunicorn.
• Dockerizing a Python Django Web Application is another in-depth
tutorial on combining Docker with Django.
• Dockerizing Flask with Postgres, Gunicorn, and Nginx looks at how to
configure Flask to run on Docker along with Postgres, Nginx, and
Gunicorn.
Kubernetes
Kubernetes (source code) is a container orchestration system for deploying,
scaling and operating applications.
406
Full Stack Python: 2020 Supporter's Edition
Kubernetes tools
• Helm (source code) is a package manager for Kubernetes charts, which
are the way to define common types of Kubernetes cluster
arrangements, like MySQL, Cassandra or Jenkins.
• Gitkube (source code) makes it possible to deploy an application to
Kubernetes using git push , similar to how Heroku popularized
making platform-as-a-service deployments easy.
• Kompose (source code) translate Docker Compose files into
Kubernetes configuration resources.
• skaffold. Using Kubernetes for local development is a good starting
place for more information on getting started with Skaffold.
• kubethanos is a tool to kill half of your Kubernetes pods at random, to
test the resilience of your infrastructure under highly chaotic scenarios.
Kubernetes background and retrospectives
• Borg, Omega and Kubernetes goes into the history of Borg and Omega,
projects that preceded Kubernetes' creation. There are a ton of great
notes on why they developed the project in certain ways and what they
knew to avoid based on the prior work on Borg and Omega.
• Kubernetes at GitHub provides a retrospective on transitioning
GitHub's infrastructure from a traditional Ruby on Rails deployment
architecture to a more scalable container-based Kubernetes system.
There are some great details on the steps in the transition and ramping
up capacity until it was the full system for github.com and other critical
services.
407
Full Stack Python: 2020 Supporter's Edition
• Reasons Kubernetes is cool breaks past the "why would I ever need
this?" initial developer reaction and gives solid reasons such as better
visibility into all of the services running on your Kubernetes cluster and
potentially much faster deployment after appropriate configuration.
• How we designed our Kubernetes infrastructure on AWS explains how
the Kubernetes Infrastructure Technology Team (yes, that abbreviates
to KITT in honor of the 1980s Knight Rider TV show) at Atlassian
starting using the tool and how they have built infrastructure around it
for the company to operate their container-ized applications.
• 10 Most Common Reasons Kubernetes Deployments Fail goes over
many of the top technical reasons why issues come up with Kubernetes
and what you need to do to avoid or work through them.
• Draft vs Gitkube vs Helm vs Ksonnet vs Metaparticle vs Skaffold gives a
great overview of the most popular tools that make it easier to use
Kubernetes.
• Architecting applications for Kubernetes is stuffed full of great design
advice that is now available as people having been using Kubernetes for
a couple of years.
• "Let’s use Kubernetes!" Now you have 8 problems is a counterargument for why you should be cautious about introducing the
significant complexity overhead of Kubernetes (or any related tools)
into your environment unless you really need the advantages that they
can provide. Each developer, team and organization should perform an
explicit cost-benefit analysis to make sure the tool's scability, reliability
and related functionality will outweigh the downsides.
• How Zolando manages 140+ Kubernetes clusters covers the
architecture, monitoring and workflow of a team that has to run a
decent number of clusters for their development teams.
Kubernetes tutorials
• Kubernetes The Hard Way is a tutorial that walks you through
manually setting up a Kubernetes cluster. The purpose is to teach you
what is happening at each step instead of performing everything
408
Full Stack Python: 2020 Supporter's Edition
through automation like you normally would after you understand how
to use the tool.
• Kubernetes Any% Speedrun hilariously presents the pain of using
Kubernetes and gives the basic steps for getting a deployment up and
running.
• A Gentle introduction to Kubernetes with more than just the basics is a
Git README tutorial with clear steps for how to get started running a
Kubernetes cluster.
• Anatomy of my Kubernetes Cluster shows how one developer created
their own Raspberry Pi cluster that could run Kubernetes to learn more
about how it works.
• The cult of Kubernetes is a hilarious rant that also manages to teach the
reader a lot about how to avoid some big issues the author ran into
while working with Kubernetes for simple starter projects.
• Kubernetes by Example provides the commands and code for you to
get started with the core Kubernetes concepts.
• Your instant Kubernetes cluster provide a concise set of instructions for
setting up a cluster.
• A tutorial introduction to Kubernetes covers a bunch of introductory
steps using an example Python application.
• An Example Of Real Kubernetes: Bitnami gives instructions for what to
do after you have finished creating a Kubernetes cluster and learned
the "Hello, World!"-style example.
• Kubernetes Production Patterns is a tutorial with good and bad
practices so you can learn what to do and what to avoid in your
Kubernetes infrastructure.
• Django Production Deployment on GCP with Kubernetes uses Helm to
make it easier to deploy the example Django web app with a
PostgreSQL backend.
• K8s YAML Alternative: Python shows how you can use Python scripts
instead of YAML to configure your Kubernetes clusters.
409
Full Stack Python: 2020 Supporter's Edition
Serverless
Serverless is a deployment architecture where servers are not explicitly
provisioned by the deployer. Code is instead executed based on developerdefined events that are triggered, for example when an HTTP POST request is
sent to an API a new line written to a file.
How can code be executed "without" servers?
Servers still exist to execute the code but they are abstracted away from the
developer and handled by a compute platform such as Amazon Web Services
Lambda or Google Cloud Functions.
Think about deploying code as a spectrum, where on one side you build your
own server from components, hook it up to the internet with a static IP
address, connect the IP address to DNS and start serving requests. The
hardware, operating system, web server, WSGI server, etc are all completely
controlled by you. On the opposite side of the spectrum are serverless
compute platforms that take Python code and execute it without you ever
touching hardware or even knowing what operating system it runs on.
In between those extremes are levels that remove the need to worry about
hardware (virtual private servers), up through removing concerns about web
servers (platforms-as-a-service). Where you fall on the spectrum for your
deployment will depend on your own situation. Serverless is simply the
newest and most extreme of these deployment models so it is up to you as to
how much complexity you want to take on with the deployment versus your
control over each aspect of the hardware and software.
410
Full Stack Python: 2020 Supporter's Edition
Serverless implementations
Each major cloud vendor has a serverless compute implementation. These
implementations are under significant active development and not all of them
have Python support.
• AWS Lambda is the current leader among serverless compute
implementations. It has support for both Python 2.7 and Python 3.6/
3.7.
• Azure Functions has second-class citizen support for Python. It's
supposed to be possible but kind of hacky at the moment. Polyglot
support should be quickly coming to Azure to better compete with AWS
Lambda.
• IBM Bluemix OpenWhisk is based on the Apache OpenWhisk open
source project.
• Google Cloud Functions currently only supports JavaScript code
execution.
• Webtask.io also only supports JavaScript but there is a cool prototype
project named webtask-pytask to run Python code in the browser via
webtask. This demo is definitely not for production code use but
awesome to see what the programming community can put together
using existing code and services.
Serverless frameworks
Serverless libraries and frameworks aim to provide reusable code that handles
common or tedious tasks, similar to how web frameworks deal with common
web development tasks. Some of these frameworks are built for a single
service like AWS Lambda, while others attempt to make cross-serverless
operations more palatable.
Frameworks for building Python-based applications on serverless services
include:
• Serverless (source code), which is a useful but generically-named
library that focuses on deployment and operations for serverless
applications.
411
Full Stack Python: 2020 Supporter's Edition
• Zappa (source code) provides code and tools to make it much easier to
build on AWS Lambda and AWS API Gateway than rolling your own on
the bare services.
• Chalice (source code) is built by the AWS team specifically for Python
applications.
• Apex (source code)
General serverless resources
Serverless concepts and implementations are still in their early iterations so
there are many ideas and good practices yet to be discovered. These resources
are the first attempts at figuring out how to structure and operate serverless
applications.
• Serverless software covers a range of topics under serverless and how
deployments have changed as new options such as PaaS have become
widespread.
• Lessons Learned — A Year Of Going “Fully Serverless” In Production is
a retrospective from a small development team that combines a static
site with serverless backend code to easily scale their site without an
operations staff. They discuss the good and the bad of working in this
fashion while generally coming away with a positive experience.
• From bare metal to Serverless gives some historical detail and
background context for how various execution architectures have
evolved, from the invention of the web to software-as-a-service,
infrastructure-as-a-service to today's newer serverless platforms.
• Have you shipped anything serious with a “serverless” architecture?
provides some great answers by Hacker News developers who are using
serverless for large production applications and how they deal with the
limitations of the platforms.
• Serverless cold start war compares startup times of serverless function
instances across Google Cloud, AWS and Azure. This is only one way to
measure the results the author did a great job presenting the data and
elaborating on potential reasons why the results appeared as shown.
412
Full Stack Python: 2020 Supporter's Edition
• Serverless Deployments of Python APIs is a wonderful Python-specific
article on how to use AWS Lambda, API Gateway and DynamoDB to
create and deploy a Python API.
• What's this serverless thing, anyway?
• Serverless architectures - let's ditch the servers?
• Serverless architectures provides a fantastic overview of the subject
with a balanced approach that includes the drawbacks seen in current
serverless platforms.
• Why the fuss about serverless? is a wide-ranging post about the history
of application development and infrastructure. The timeline is a bit
hard to follow but otherwise it's a unique look at why software
deployments are moving to serverless-based architectures and the
advantages that can provide.
• Serverless architectures in short lays out some of the initial thoughts
behind what the advantages and disadvantages of serverless may be.
However, it's early days for serverless so these strengths and
weaknesses may change as the architectures and good practices evolve.
• Building A Serverless Contact Form For Your Static Site uses AWS
Lambda, some HTML and JavaScript to add an input form to a static
website created by a static site generator.
• Serverless architectures, five design patterns goes over the four main
principles of serverless infrastructure and the five major usage patterns
the AWS Lambda team is seeing from initial serverless deployments.
• Serverless Cost Calculator estimates the amount each serverless
platform would charge based on executions, average execution time
and memory needed per execution. AWS Lambda, Google Cloud
Functions, Azure Functions and IBM OpenWhisk are all included in
the results.
Serverless environment comparsions
The "big 3" serverless platforms, AWS Lambda, Azure Functions and Google
Cloud Functions have varying degrees of support for Python. AWS Lambda
has production-ready support for Python 2 and 3.7, while Azure and Google
413
Full Stack Python: 2020 Supporter's Edition
Cloud have "beta" support with unclear production-worthiness. The following
resources are some comparison articles to help you in your decision-making
process for which platform to learn.
• Serverless at scale compares the "Big 3" AWS, Azure and Google Cloud
in serverless performance. The author provides some nice data around
average response times and outliers.
• Serverless hosting comparison is a broad overview of documentation,
community, pricing and other notes for the major platforms as well as
IBM OpenWhisk and the Fission.io project.
• Microsoft Azure Functions vs. Google Cloud Functions vs. AWS
Lambda presents an overview of Azure Functions and how they
compare to Google Cloud Functions and AWS Lambda.
Serverless vendor lock-in?
There is some concern by organizations and developers about vendor lock-in
on serverless platforms. It is unclear if portability is worse for serverless than
other infrastructure-as-a-service pieces, but still worth thinking about ahead
of time. These resources provide additional perspectives on lock-in and using
multiple cloud providers.
• On Serverless, Multi-Cloud, and Vendor Lock In is an opinion piece
that for most cases the additional work of going multi-cloud is not
worth the tradeoffs, therefore at this time it's better to go for a single
vendor such as AWS or Azure and optimize on that platform.
• Why vendor lock-in with serverless isn’t what you think it is
recommends using a single vendor for now and stop worrying about
hedging your bets because it typically makes your infrastructure
significantly more complex.
AWS Lambda
Amazon Web Services (AWS) Lambda is a compute service that executes
arbitrary Python code in response to developer-defined AWS events, such as
inbound API calls or file uploads to AWS' Simple Storage Service (S3).
414
Full Stack Python: 2020 Supporter's Edition
Why is Lambda useful?
Lambda is often used as a "serverless" compute architecture, which allows
developers to upload their Python code instead of spinning and configuring
servers, deploying their code and scaling based on traffic.
Python on AWS Lambda
Lambda only had support for JavaScript, specifically Node.JS, when it was
first released in late 2014. Python 2 developers were welcomed to the platform
less than a year after its release, in October 2015. Lambda now has support for
both Python 2.7, 3.6 and 3.7.
Python-specific AWS Lambda resources
• Serverless Slash Commands with Python shows how to use the Slack
API to build slash commands that run with an AWS Lambda backend.
• Zappa is a serverless framework for deploying Python web
applications. It's a really slick project and used even by internal AWS
developers for their own application deployments. Be sure to read the
Zappa blog as well for walkthroughs and new feature announcements.
• How to Setup a Serverless URL Shortener With API Gateway Lambda
and DynamoDB on AWS builds a non-trivial URL shortener
application as an example Python application that runs on Lambda.
415
Full Stack Python: 2020 Supporter's Edition
• How we built Hamiltix.net for less than $1 a month on AWS walks
through setting up a full website that runs on AWS and scales with the
Lambda free tier to minimize spend despite large traffic spikes.
• Deploying a serverless Flask app to AWS Lambda using Zappa provides
a screen capture of one developer deploying their application to
Lambda.
• Automated SQL Injection Testing of Serverless Functions On a
Shoestring Budget (and Some Good Music) is an awesome operational
security post that uses Python to test for SQL injection vulnerabilities
in serverless functions on AWS Lambda.
• Building Scikit-Learn For AWS Lambda follows up on the Using ScikitLearn In AWS Lambda post which shows how to perform scientific
computing with Python packages on AWS Lambda.
• Creating Serverless Functions with Python and AWS Lambda explains
how to use the Serverless framework to build Python applications that
can be deployed to AWS Lambda.
• Code Evaluation With AWS Lambda and API Gateway shows how to
develop a code evaluation API, to execute arbitrary code, with AWS
Lambda and API Gateway.
• Crawling thousands of products using AWS Lambda gives a real-world
example of where using Python, Selenium and headless Chrome on
AWS Lambda could crawl thousands of pages to collect data with each
crawler running within its own Lambda Function.
• Going Serverless with AWS Lambda and API Gateway
General AWS Lambda resources
• Getting started with serverless on AWS is a wonderful tutorials,
example projects and additional resources guide created by a developer
who used all of these bits to learn AWS services herself.
• AWS Lambda Serverless Reference Architectures provides blueprints
with diagrams of common architecture patterns that developers use for
their mobile backend, file processing, stream processing and web
application projects.
416
Full Stack Python: 2020 Supporter's Edition
• Security Overview of AWS Lambda (PDF file) covers their "Shared
Responsibility Model" for security and compliance. Although the paper
bills itself as an in-depth look at AWS Lambda security it is really more
of a high-level overview, but still worth the read.
• Reverse engineering AWS Lambda is an incredible, in-depth analysis of
the author's work investigating the black box of how Lambda works
and what he learned from it.
• The AWS Lambda tag on the official AWS blog contains all the related
first-party tutorials
• Serverless Cost Calculator estimates the amount that AWS would
charge based on Lambda exeuctions, average execution time and
memory needed per execution.
• Serverless at Nordstrom is an awesome real-world story with the
architecture behind a serverless AWS Lambda application deployment
at Nordstrom.
• How was your experience with AWS Lambda in production? has a good
discussion of some of the benefits and issues that developers had as of
mid-2017 with using Lambda for production applications.
• Passwordless database authentication for AWS Lambda shows how to
use a MySQL backend from your Lambda functions.
• How does language, memory and package size affect cold starts of AWS
Lambda? investigates the performance implications of various Lambda
settings.
• Best Practices for AWS Lambda Timeouts explains some of the current
hard upper limits on AWS timeouts, such as 5 minutes for Lambdas,
when explicitly set that high, as well as 29 seconds for API Gatway
requests. There is also good advice on how the circuit breaker pattern
should be applied to your Lambdas and ultimately why low time outs
are likely the best way to go to prevent your application from becoming
entirely unresponsive.
• X-rays for Flask and Django Serverless Applications is an
instrumentation, monitoring and debugging service built into AWS
417
Full Stack Python: 2020 Supporter's Edition
Lambda specifically for Python web frameworks running on the
service.
• Cutting Through the Layers: AWS Lambda Layers Explained explains
how AWS Lambda now offer a "Bring Your Own Runtime" by exposing
the layers that were previously controlled exclusively by Amazon. There
is an overview of the layers and why they matter for customizing your
functions.
Azure Functions
Azure Functions is a compute service created by Microsoft that can execute
Python code in response to pre-defined events, such as API calls or database
transactions in other Azure services.
Azure Functions resources
• An introduction to Azure Functions is the official quickstart guide by
Microsoft and has some good high-level information on their
platform's services as well.
• Deploy Python to Azure Functions provides the step-by-step
instructions needed to get Python code running on Azure Functions.
• Azure Functions vs AWS Lambda – Scaling Face Off contains metrics
from comparing AWS Lambda with Azure Functions in response time,
user load, requests per second and error rate over various periods of
time.
• How to build a serverless report server with Azure Functions and
SendGrid combines the Sendgrid email API with some configuration
code to have Azure Functions kick off email jobs.
418
Full Stack Python: 2020 Supporter's Edition
• azure-cli are the command line tools for using all of Azure, not just
Functions.
• My (Rough) Start with Azure Functions painstakingly details signing
up for Azure, accessing Functions and finally coding a Function. The
author has some really great points on what is confusing to newcomers
that hopefully will be addressed as Microsoft continues to work on
their Azure platform.
• Azure in Plain English covers all of the Azure services and explains
them because their default names are often too vague to understand
their purpose.
Google Cloud Functions
Google Cloud Functions is a serverless compute service that executes arbitrary
code. However, at this time Google Cloud Functions is a beta service that does
not work with Python.
Amazon Web Services (AWS) Lambda is currently a good alternative until
Google Cloud Functions supports Python.
Google Cloud Functions resources
• Serverless Cost Calculator estimates the amount that AWS would
charge based on Lambda exeuctions, average execution time and
memory needed per execution.
• Microsoft Azure Functions vs. Google Cloud Functions vs. AWS
Lambda gives an overview of these three serverless cloud platforms and
compares their current capabilities.
• Getting Started With Google Cloud Functions and MongoDB explains
how to connect a Google Cloud Function to a persistent MongoDB data
store.
• Running End to End tests as Google Cloud Functions provides the code
and a walkthrough for running a headless Chrome instance in a
serverless Cloud Function to perform browser-based tests.
419
Full Stack Python: 2020 Supporter's Edition
DevOps
DevOps is the combination of application development and operations, which
minimizes or eliminates the disconnect between software developers who
build applications and systems administrators who keep infrastructure
running.
Why is DevOps important?
When the Agile methodology is properly used to develop software, a new
bottleneck often appears during the frequent deployment and operations
phases. New updates and fixes are produced so fast in each sprint that
infrastructure teams can be overwhelmed with deployments and push back on
the pace of delivery. To alleviate some of these issues, application developers
are asked to work closely with operations folks to automate the delivery of
code from development to production.
DevOps tooling resources
DevOps cannot be performed with tools alone, but having the right tools to
augment the culture and processes is important to successful software
delivery. The following resources discuss both Python-specific and general
tools and services for DevOps environments.
• DevOps: Python tools to get started is a presentation slideshow that
explains that while DevOps is a culture, it can be supported by tools
such as Fabric, Jenkins, BuildBot and Git which when used properly
can enable continuous software delivery.
• For an Atlassian-centric perspective on tooling, take a look at this post
on how to choose the right DevOps tools which is biased towards their
tools but still has some good insight such as using automated testing to
provide immediate awareness of defects that require fixing.
General DevOps resources
The following resources give advice and approaches for building the right
teams, culture, processes and tools into software development organizations.
421
Full Stack Python: 2020 Supporter's Edition
• DevOps vs. Platform Engineering considers DevOps an ad hoc
approach to developing software while building a platform is a strict
contract. I see this as "DevOps is a process", while a "platform is code".
Running code is better than any organizational process.
• The open source PagerDuty Incident Response guide is the amazing
result from their company taking the practices they use to keep their
services running and putting them out for other developers to
consume. Highly recommended.
• Operations for software developers for beginners gives advice to
developers who have never done operations work and been on call for
outages before in their career. The advantage of DevOps is greater
ownership for developers who built the applications running in
production. The disadvantage of course is the greater ownership also
leads to much greater responsibility when something breaks!
• Google's Site Reliability Engineering (SRE) book is free online and
required reading for understanding the practices and principles behind
keeping the largest websites alive. Note though that some of the advice
in the book will be considered controversial at more stodgy traditional
organizations that have done operations differently for a long time.
There is also a wonderful interview with Ben Treynor, one of the
authors of the book, that contains additional information.
• The Increment, Stripe's fantastic digital and print magazine, has an
issue dedicated to being on-call which discusses many DevOps-related
topics such as what happens when your pager goes off, ownership and
how startups can be different from large companies with their incident
responses.
• Bing: Continuous Delivery is an impressive visual story that explains
the practices for how their team delivers updates to the search engine.
• Why are we racing to DevOps? is a very high level summary of the
benefits of DevOps to IT organizations. It's not specific to Python and
doesn't dive into the details, but it's a decent start for figuring out why
IT organizations consider DevOps the hot new topic after adopting an
Agile development methodology.
422
Full Stack Python: 2020 Supporter's Edition
• SRE vs. DevOps: competing standards or close friends? covers Google's
take on how Site Reliability Engineering (SRE) fits with the DevOps
world. Roughly speaking, SRE is more closely aligned with metrics and
how to operate infrastructure and applications, rather than the broader
principles embodied by the DevOps philosophy.
Monitoring
Monitoring tools capture, analyze and display information for a web
application's execution. Every application has issues arise throughout all
levels of the web stack. Monitoring tools provide transparency so developers
and operations teams can respond and fix problems.
Why is monitoring necessary?
Capturing and analyzing data about your production environment is critical to
proactively deal with stability, performance, and errors in a web application.
Difference between monitoring and logging
Monitoring and logging are very similar in their purpose of helping to
diagnose issues with an application and aid the debugging process. One way
to think about the difference is that logging happens based on explicit events
while monitoring is a passive background collection of data.
For example, when an error occurs, that event is explicitly logged through
code in an exception handler. Meanwhile, a monitoring agent instruments the
code and gathers data not only about the logged exception but also the
performance of the functions.
This distinction between logging and monitoring is vague and not necessarily
the only way to look at it. Pragmatically, both are useful for maintaining a
production web application.
Monitoring layers
There are several important resources to monitor on the operating system and
network level of a web stack.
1. CPU utilization
2. Memory utilization
423
Full Stack Python: 2020 Supporter's Edition
3. Persistence storage consumed versus free
4. Network bandwidth and latency
Application level monitoring encompasses several aspects. The amount of
time and resources dedicated to each aspect will vary based on whether an
application is read-heavy, write-heavy, or subject to rapid swings in traffic.
1.
2.
3.
4.
5.
Application warnings and errors (500-level HTTP errors)
Application code performance
Template rendering time
Browser rendering time for the application
Database querying performance
Open source monitoring projects
• Sentry started life as a Python-only monitoring project but can now be
used for any programming language.
• Service Canary
• ping.gg (source code)
• glances (source code)
• statsd is a node.js network daemon that listens for metrics and
aggregates them for transfer into another service such as Graphite.
• Graphite stores time-series data and displays them in graphs through a
Django web application.
• Sensu is an open source monitoring framework written in Ruby but
applicable to any programming language web application.
• Graph Explorer by Vimeo is a Graphite-based dashboard with added
features and a slick design.
• Munin is a client plugin-based monitoring system that sends
monitoring traffic to the Munin node where the data can be analyzed
and visualized. Note this project is written in Perl so Perl 5 must be
installed on the node collecting the data.
• Bucky measures the performance of a web application from end user's
browsers and sends that data back to the server for collection.
424
Full Stack Python: 2020 Supporter's Edition
Hosted monitoring services
Hosted monitoring software takes away the burden of deploying and
operating the software yourself. However, hosted monitoring costs (often a
significant amount of) money and take your application's data out of your
hands so these services are not the right fit for every project.
Error Tracking
• Rollbar instruments both the server side and client side to capture and
report exceptions. The pyrollbar code library provides quick integration
for Python web applications. There are also specific instructions for
common web frameworks such as Django and Pyramid.
• Sentry is the hosted version of the open source tool that is used to
monetize and support further development.
Application Performance Monitoring (APM)
• New Relic provides application and database monitoring as well as
plug ins for capturing and analyzing data about other devleoper tools in
your stack, such as Twilio.
• Opbeat Built for django. Opbeat combines performance metrics,
release tracking, and error logging into a single simple service.
• Scout monitors the performance of Django and Flask apps, autoinstrumenting views, SQL queries, templates, and more.
Status Pages
• Status.io focuses on uptime and response metrics transparency for web
applications.
• StatusPage.io (yes, there's both a Status and StatusPage.io) provides
easy set up status pages for monitoring application up time.
Incident Management
• PagerDuty alerts a designated person or group if there are stability,
performance, or uptime issues with an application.
Monitoring resources
• How to Add Hosted Monitoring to Flask Web Applications and How to
Monitor Bottle Web Applications are a couple of posts in a series
425
Full Stack Python: 2020 Supporter's Edition
showing how to add hosted monitoring to Python web apps built with
any of the major Python web frameworks.
• Stack Overflow: How We Do Monitoring - 2018 Edition is a detailed,
long read about how Stack Overflow handles their monitoring, health
checks, alerting and dashboarding of their infrastructure.
• The Virtues of Monitoring
• Effortless Monitoring with collectd, Graphite, and Docker
• Practical Guide to StatsD/Graphite Monitoring is a detailed guide with
code examples for monitoring infrastructure.
• Bit.ly describes the "10 Things They Forgot to Monitor" beyond the
standard metrics such as disk & memory usage.
• The videos from Monitorama, a conference that's all about monitoring
and observability, are recordings of fantastic technical talks from their
events.
• Four Linux server monitoring tools
• How to design useful monitoring and graphing visualizations
• 5 years of metrics and monitoring is a great presentation highlighting
that visualization so humans can understand measurements is a hard
problem. Line graphs are often not the best solution and they are
overused.
• Keeping an eye on our network explains how GitHub uses tagged
metrics to keep better tabs on its infrastructure and network
connections.
• 10 monitoring talks every developer should watch contains a great
collection of relevant monitoring presentations.
• The Collector Highlight Series has an article on StatsD that explains
how to install it and how it works.
• OpenCensuss, OpenTracing and OpenMetrics are three projects that
are trying to create standards for application monitoring metrics. This
article on the 3 projects is a helpful overview to understand what each
one is trying to do and how they compare to each other.
426
Full Stack Python: 2020 Supporter's Edition
Monitoring learning checklist
1. Review the software-as-a-service and open source monitoring tools
below. Third party services tend to be easier to set up and host the data
for you. Open source projects give you more control but you'll need to
have additional servers ready for the monitoring.
2. My recommendation is to install New Relic's free option with the trial
period to see how it works with your app. It'll give you a good idea of
the capabilities for application-level monitoring tools.
3. As your app scales take a look at setting up one of the the open source
monitoring projects such as StatsD with Graphite. The combination of
those two projects will give you fine-grained control over the system
metrics you're collecting and visualizing.
Datadog
Datadog is a hosted monitoring service that can be used with Python web
applications to catch and report errors.
Datadog resources
• Monitoring Django performance with Datadog is a fantastic, detailed
walk through for integrating this service with Django web applications.
• Python log collection provides a framework-agnostic explanation for
how to use the service with Python code.
• Monitoring Flask apps with Datadog walks through adding Datadog to
a Flask application.
427
Full Stack Python: 2020 Supporter's Edition
Prometheus
Prometheus (source code) is an open source monitoring tool that can be used
to instrument and report on Python web applications.
Prometheus resources
• Prometheus-Basics is a newbie's introduction to this tool. It covers
what Prometheus is, the tool's architecture, types of metrics and
contains a walkthrough of how to get it configured.
• This primer on Prometheus walks through installation, configuration
and metrics collection.
• Monitoring synchronous Python web apps with Prometheus and its
asynchronous monitoring counterpart are two tutorials that show how
to add middleware to your web apps that allows Prometheus metrics
collection.
• Monitoring with Prometheus provides an overview of the tool and
explains how to combine it with Grafana to visualize the metrics that
are collected.
• Monitor your applications with Prometheus is a getting started guide
with a walkthrough of how to instrument a simple Golang application.
• Custom Application Metrics with Django, Prometheus, and Kubernetes
shows how to handle the initial configuration with
django-prometheus , deploys the Django web app using Helm and
configures Prometheus to scrape metrics from the application running
on Kubernetes.
• A gentle introduction to the wonderful world of metrics has a quick
summary that compares Prometheus with Nagios, then digs into the
logging format and what you can visualize with this tool.
428
Full Stack Python: 2020 Supporter's Edition
• From Graphite to Prometheus explains some of the differences
between using a StatsD / Graphite monitoring stack and Prometheus,
such as how Prometheus scrapes data instead of the applications
pushing data to a metrics aggregator, and the query languages for each
tool.
Rollbar
Rollbar is a hosted monitoring service that can be used with Python web
applications to catch and report errors.
Rollbar resources
• Rollbar monitoring of Celery in a Django app covers the Celery task
queue and how to discover issues that happen in asynchronous worker
processes.
• How to Add Hosted Monitoring to Flask Web Applications adds
Rollbar's error tracking to a Flask web application.
• How to Monitor Python Web Applications uses the Bottle web
application framework to show how hosted monitoring can be added to
a Python web application.
• The official Rollbar Python docs have tons of great integration
examples for various Python web frameworks.
Sentry
Sentry is an open source monitoring project as well as a service that can be
used to report errors in Python web apps.
Sentry is a monitoring service that you can set up to host yourself or used as a
service to catch and report errors in your Python web applications.
429
Full Stack Python: 2020 Supporter's Edition
Sentry resources
• Sentry's official Python docs explain how to integrate the SDK into an
application to send events to either your own server or the hosted
service.
• Sentry For Data: Error Monitoring with PySpark shows an integration
between the PySpark data analysis tool and Sentry for handling any
issues.
Web App Performance
Web application performance is affected by network latency, bandwidth,
database queries, page size and many other factors.
Performance and load testing tools
• Falco
Load testing resources
• Load Testing with Locust.io & Docker Swarm shows you how to set up
load tests using Docker containers along with AWS for scaling up the
tests.
• HTTP Load Testing with Vegeta (and a dash of Python) covers getting
started with the Vegeta load tester and uses Python to analyze the tool's
results.
• Four reasons developers should write their own load tests and four load
testing mistakes developers love to make are opinionated pieces on
430
Full Stack Python: 2020 Supporter's Edition
how developer should use load testing to ensure their applications
work properly under heavy usage.
• Building a PostgreSQL load tester explains how the pgreplay-go tool
works and how to obtain performance metrics for a PostgreSQL
database.
Web app performance resources
• Web Performance 101 introduces web application loading performance.
There is a ton of great information on JavaScript, CSS and HTTP
optimization as well as tools to use.
• Idle until urgent explains an issue the author found when measuring
First Input Delay (FID) on his site and what techniques he used to fix
the problem.
• How to measure web app performance is a 20 minute code-first demo
that shows how to get a realistic estimate for how many requests per
second your web application will be able to handle.
• How to Interpret Site Performance Tests covers the difference between
client, page and connection speed tests as well as a bit on caching
performance.
• Practical scaling techniques for websites examines how to improve
your website performance with asynchronous task queues, database
optimization and caching.
• The [Performance Testing Guidance for Web
Applications](https://docs.microsoft.com/en-us/previous-versions/
msp-n-p/bb924375(v%3dpandp.10) book from Microsoft is a gem.
There are chapters on foundations of performance testing, modeling
application usage and many other topics that are critical to working on
web app performance.
• awesome-scalability provides a list with a crazy number of scaling and
performance optimization resources and tools by category.
• Every Web Performance Test Tool provides a nice list of tools and
provides short summaries of what each one can help with in identifying
performance problems.
431
Full Stack Python: 2020 Supporter's Edition
• The Infrastructure Behind Twitter: Scale examines the evolution from
having to buy your own hardware from vendors to run a service to the
current days of being able to rely on cloud providers for some or all
workloads regardless of scale.
• Scaling to 100k users covers the architecture scaling techniques
commonly used to move up in serving users by orders of magnitude,
for example from 100 to 1000.
• Web Performance Recipes with Puppeteer digs into tracing through
page rendering to measure performance and how to extract
performance metrics from the Lighthouse tool for further analysis.
Caching
Caching can reduce the load on servers by storing the results of common
operations and serving the precomputed answers to clients.
For example, instead of retrieving data from database tables that rarely
change, you can store the values in-memory. Retrieving values from an inmemory location is far faster than retrieving them from a database (which
stores them on a persistent disk like a hard drive.) When the cached values
change the system can invalidate the cache and re-retrieve the updated values
for future requests.
A cache can be created for multiple layers of the stack.
Caching backends
• memcached is a common in-memory caching system.
• Redis is a key-value in-memory data store that can easily be configured
for caching with libraries such as django-redis-cache and the similarlynamed, but separate project django-redis.
Caching resources
• Caching at Reddit is a wonderful in-depth post that goes into detail on
how they handle caching their Python web app for billions of pageviews
each month.
432
Full Stack Python: 2020 Supporter's Edition
• "Caching: Varnish or Nginx?" reviews some considerations such as SSL
and SPDY support when choosing reverse proxy Nginx or Varnish.
• Caching is Hard, Draw me a Picture has diagrams of how web request
caching layers work. The post is relevant reading even though the
author is describing his Microsoft code as the impetus for writing the
content.
• While caching is a useful technique in many situations, it's important
to also note that there are downsides to caching that many developers
fail to take into consideration.
• Caching at Reddit covers monitoring, tuning and scaling for the very
high scale Reddit.com website.
• Mastering HTTP caching provides more advanced advice on caching
dynamic as well as static content via CDNs and other configurations.
Caching learning checklist
1. Analyze your web application for the slowest parts. It's likely there are
complex database queries that can be precomputed and stored in an inmemory data store.
2. Leverage your existing in-memory data store already used for session
data to cache the results of those complex database queries. A task
queue can often be used to precompute the results on a regular basis
and save them in the data store.
3. Incorporate a cache invalidation scheme so the precomputed results
remain accurate when served up to the user.
Logging
Logging saves output such as errors, warnings and event information to
persistent storage for debugging purposes.
Why is logging important?
Runtime exceptions that prevent code from running are important to log to
investigate and fix the source of the problems. Informational and debugging
433
Full Stack Python: 2020 Supporter's Edition
logging also helps to understand how the application is performing even if
code is working as intended.
Logging levels
Logging is often grouped into several categories:
1.
2.
3.
4.
Information
Debug
Warning
Error
Logging errors that occur while a web framework is running is crucial to
understanding how your application is performing.
Logging aggregators
When you are running your application on several servers, it is helpful to have
a monitoring tool called a "logging aggregator". You can configure your
application to forward your system and application logs to one location that
provides tools for viewing, searching, and monitoring logging events across
your cluster.
Another advantage of log aggregation tools is they allow you to set up custom
alerts and alarms so you can get notified when error rates breach a certain
threshold.
Open source log aggregators
• Sentry started as a Django-only exception handling service but now has
separate logging clients to cover almost all major languages and
frameworks. It still works really well for Python-powered web
applications and is often used in conjunction with other monitoring
tools. Raven is open source Python client for Sentry.
• Graylog2 provides a central server for log aggregation as well as a GUI
for browsing and searching through log events. There are libraries for
most major languages, including python. Saves data in Elasticache.
• Logstash Similar to Graylog2, logstash offers features to
programmatically configure log data workflows.
434
Full Stack Python: 2020 Supporter's Edition
• Scribe A project written by Facebook to aggregate logs. It's designed to
run on multiple servers and scale with the rest of your cluster. Uses the
Thrift messaging format so it can be used with any language.
Hosted logging services
• Loggly is a third party cloud based application that aggregates logs.
They have instructions for every major language, including python. It
includes email alerting on custom searches.
• Splunk offers third party cloud and self hosted solutions for event
aggregation. It excels at searching and data mining any text based data.
• Papertrail is similar to both Loggly and Splunk and provides
integration with S3 for long term storage.
• Raygun logs errors and provides immediate notification when issues
arise.
• Scalyr provides log aggregation, dashboards, alerts and search in a user
interface on top of standard logs.
• There is a hosted version of Sentry in case you do not have the time to
set up the open source project yourself.
Logging resources
• This intro to logging presents the Python logging module and how to
use it.
• Understanding Python's logging module clears up some
misconceptions about how pattern matching with logging hierarchies
works and provides a few clear diagrams to visually explain logging
handlers.
• Logging Cookbook contains useful code snippets to easily add logging
to your own applications.
• Logging in Python explains the logging module in the Python standard
library, configuration settings, handlers and how to log data.
• A Brief Digression About Logging is a short post that gets Python
logging up and running quickly.
435
Full Stack Python: 2020 Supporter's Edition
• Taking the pain out of Python logging provides a logging set up with
uWSGI.
• Good logging practice in Python shows how to use the standard library
to log data from your application. Definitely worth a read as most
applications do not log nearly enough output to help debuggin when
things go wrong, or to determine if something is going wrong.
• Structured Logging: The Best Friend You’ll Want When Things Go
Wrong explains how Grab (a large ridesharing platform in Asia) uses
structured logging to reduce Mean Time To Resolve (MTTR) and how
they arranged their log levels to support their scale.
• How to collect, customize, and centralize Python logs covers the
standard library logging module and how to configure it for various
ways you are likely to use it with one or more Python applications.
• A guide to logging in Python has some clear, simple diagrams to
explain how logging works in Python applications.
• Django Logging, the Right Way covers a few Python logging techniques
and then goes into how you use them with your Django projects.
• Django's 1.3 release brought unified logging into project configurations.
This post shows how to set up logging in a project's settings.py file.
Caktus Group also has a nice tutorial on central logging with graypy
and Graylog2.
• Django Logging Configuration: How the Default Settings Interfere with
Yours explains a problem with the default Django logging configuration
and what to do about in your project.
• The Pythonic Guide To Logging provides a quick introduction to log
levels in Python code.
• Exceptional Logging of Exceptions in Python shows how to log errors
more accurately to pinpoint the problem instead of receiving generic
exceptions in your logs.
• Python and Django logging in plain English is a wonderful overview of
how to configure logging in Django projects.
436
Full Stack Python: 2020 Supporter's Edition
Logging learning checklist
1. Read how to integrate logging into your web application framework.
2. Ensure errors and anomalous results are logged. While these logs can
be stored in monitoring solutions, it's best to have your own log storage
location to debug issues as they arise to complement other monitoring
systems.
3. Integrate logging for system events you may need to use for debugging
purposes later. For example, you may want to know the return values
on functions when they are above a certain threshold.
Web Analytics
Web analytics involves collecting, processing, visualizing web data to enable
critical thinking about how users interact with a web application.
Why is web analytics important?
User clients, especially web browsers, generate significant data while users
read and interact with webpages. The data provides insight into how visitors
use the site and why they stay or leave. The key concept to analytics is
learning about your users so you can improve your web application to better
suit their needs.
Web analytics concepts
It's easy to get overwhelmed at both the number of analytics services and the
numerous types of data points collected. Focus on just a handful of metrics
when you're just starting out. As your application scales and you understand
more about your users add additional analytics services to gain further insight
into their behavior with advanced visualizations such as heatmaps and action
funnels.
User funnels
If your application is selling a product or service you can ultimately build a
user funnel (often called "sales funnel" prior to a user becoming a customer)
to better understand why people buy or don't buy what you're selling. With a
437
Full Stack Python: 2020 Supporter's Edition
funnel you can visualize drop-off points where visitors leave your application
before taking some action, such as purchasing your service.
Open source web analytics projects
• Piwik is a web analytics platform you can host yourself. Piwik is a solid
choice if you cannot use Google Analytics or want to customize your
own web analytics platform.
• Shynet is a lightweight, privacy-friendly cookie-free web analytics
application written in Python.
• Open Web Analytics is another self-hosted platform that integrates
through a JavaScript snippet that tracks users' interactions with the
webpage.
Hosted web analytics services
• Google Analytics is a widely used free analytics tool for website traffic.
• Clicky provides real-time analytics comparable to Google Analytics'
real-time dashboard.
• MixPanel's analytics platform focuses on mobile and sales funnel
metrics. A developer builds what data points need to be collected into
the server side or client side code. MixPanel captures that data and
provides metrics and visualizations based on the data.
• Heap is a recently founded analytics service with a free introductory
tier to get started.
• CrazyEgg is tool for understanding a user's focus while using a website
based on heatmaps generated from mouse movements.
Python-specific web analytics resources
• Building an Analytics App with Flask is a detailed walkthrough for
collecting and analyzing webpage analytics with your own Flask app.
• Build a Google Analytics Slack Bot with Python explains how to
connect the Google Analytics API to a Slack bot, with all the code in
438
Full Stack Python: 2020 Supporter's Edition
Python, so you can query for Google Analytics data from your Slack
channels.
• Automating web analytics through Python is a tutorial for interacting
with your Google Analytics data using pandas and related data analysis
tools.
• The official Google Analytics Python quickstart isn't really the easiest
tutorial to follow due to all of the configuration required to make your
first API call, but it is still the right place to go to get started.
• How Accurately Can Prophet Project Website Traffic? uses the data
forecasting tool Prophet to see if it is possible to predict future trends
in website traffic based on historical data.
General web analytics resources
• The Google Analytics Setup I Use on Every Site I Build is a tutorial
written for developers to better understand the scope of what Google
Analytics can tell you about your site and how to configure it for better
output.
• Roll your own analytics shows you how to use AWS Lambda and some
custom JavaScript to create your own replacement for Google
Analytics. This route is not for everyone but it is really useful if you
want to avoid the Google data trap.
• This beginner's guide to math and stats behind web analytics provides
some context for understanding and reasoning about web traffic.
• An Analytics Primer for Developers by Mozilla explains what to track,
choosing an analytics platform and how to serve up the analytics
JavaScript asynchronously.
• Options for Hosting Your Own Non-JavaScript-Based Analytics has a
few non-Google Analytics web analytics tools that mostly rely on
server-side rather than client-side tracking.
• This post provides context for determining if a given metric is "vanity"
or actionable.
439
Full Stack Python: 2020 Supporter's Edition
• This series on measuring your technical content has a bunch of advice
for figuring out why you want to gather metrics, how to do the
instrumentation and determining your success factors.
◦ Part 1 covers "why"
◦ Part 2 examines success factors
◦ Part 3 digs further into measurement
• awesome-analytics aggregates analytics tools for both web and mobile
applications.
• 10 red flags signaling your analytics program will fail is a more
business-focused piece but it has sosme good information and
visualization on broader themes that developers who work in larger
organizations should think about when it comes to analytics.
Web analytics learning checklist
1. Add Google Analytics or Piwik to your application. Both are free and
while Piwik is not as powerful as Google Analytics you can self-host the
application which is the only option in many environments.
2. Think critically about the factors that will make your application
successful. These factors will vary based on whether it's an internal
enterprise app, an e-commerce site or an information-based
application.
3. Add metrics generated from your web traffic based on the factors that
drive your application's success. You can add these metrics with either
some custom code or with a hosted web analytics service.
4. Continuously reevaluate whether the metrics you've chosen are still the
appropriate ones defining your application's success. Improve and
refine the metrics generated by the web analytics as necessary.
440
Full Stack Python: 2020 Supporter's Edition
What "Full Stack" Means
The terms "full stack" and "Full Stack Python" are ambiguous but I am using a
specific definition here on this site. These term can be defined for a web stack
either to mean
1. Every layer, from the machine code up to the browser, are written in
Python
2. Python code interacts with code written in other languages such as C
and JavaScript to provide a complete web stack
I named this site specifically for the second definition: Python is one
programming language among many that are used to build your application
stack.
Some folks took the title of the site to mean Python runs everything from the
web browser on down. That's simply not practical or possible. While Python is
an amazing programming language, there are many tasks it does not do well.
Python is just one language among many that allows our computers to execute
software and communicate with each other.
For beginners, learning the syntax and libraries in Python necessary to build a
web application or web API is a major undertaking. Even intermediate and
advanced Python software developers need to constantly program and learn
to keep up with our ever evolving ecosystem. I created Full Stack Python to be
just one of many resources that help Python developers build and maintain
their programming skills.
About the Author
Full Stack Python is coded and written by Matt Makai. Matt currently works
in Washington, D.C. for the Twilio Developer Network as the Director of
Developer Content.
Other projects by Matt include The Full Stack Python Guide to Deployments,
Python for Entrepreneurs, Introduction to Ansible and Coding Across
America.
442
Full Stack Python: 2020 Supporter's Edition
You can reach him by email at matthew.makai@gmail.com. Matt can't
respond to every email, but he will do his best to reply when possible.
443