Expand home directory

with open(os.path.expanduser(posts_file)) as f:
    for line in f:

Package creation tutorial


Ternary conditional operator


a if condition else b

Copy directory of files to another directory


from distutils.dir_util import copy_tree
copy_tree("/a/b/c", "/x/y/z")

Create directory if does not exist

import os
if not os.path.exists(directory):

Run unittest


python -m unittest
import unittest

if __name__ == '__main__':

Python 3 http server

python3 -m http.server

How to remove pyenv virtualenv

pyenv uninstall <virtualenv_name>

How to change pyenv python version

pyenv global 3.6.4

GET parameters in Flask

username = request.args.get('username')
password = request.args.get('password')

Get parent directory


>>> path = r'c:\xxx\abc\xyz\fileName.jpg'
>>> import os
>>> os.path.dirname(path)
>>> os.path.basename(os.path.dirname(path))

Get Unix timestamp


ipmort time
timestamp = time.time()

Specify arguments command line

import argparse

parser = argparse.ArgumentParser(description='Process some integers.')
parser.add_argument('integers', metavar='N', type=int, nargs='+',
                    help='an integer for the accumulator')
parser.add_argument('--sum', dest='accumulate', action='store_const',
                    const=sum, default=max,
                    help='sum the integers (default: find the max)')
args = parser.parse_args()
print args.accumulate(args.integers)

parser.add_argument('-s', '--settings')
args = parser.parse_args()

Select random from a list

import random

foo = ['a', 'b', 'c', 'd', 'e']

Saving and reading dictionaries in CSV

with open('filename.csv', 'w') as f:
    fieldnames = ['Email', 'Name', 'Phone Number', 'Detail URL', 'Results URL']
    rows = csv.DictWriter(f, fieldnames=fieldnames)
    for e in emails:
        row = {
            'Email': e,

with open('filename.csv') as f:
    rows = csv.DictReader(f)
    for row in rows:
        email = row['Email'].lower()

Data migration in Google App Engine


Add th to dates when using strftime


from datetime import datetime as dt

def suffix(d):
    return 'th' if 11<=d<=13 else {1:'st',2:'nd',3:'rd'}.get(d%10, 'th')

def custom_strftime(format, t):
    return t.strftime(format).replace('{S}', str(t.day) + suffix(t.day))

print custom_strftime('%B {S}, %Y', dt.now())

Fixing pip issue

vim ~/.pydistutils.cfg


Pure Python stemming library

https://github.com/shibukawa/snowball_py https://pypi.python.org/pypi/snowballstemmer

From timestamp to datetime


>>> from datetime import datetime
>>> datetime.fromtimestamp(1172969203.1)
datetime.datetime(2007, 3, 4, 0, 46, 43, 100000)

Run shell command


from subprocess import call
call(["ls", "-l"])

Make python script executable


In the beginning:

#! /usr/bin/env python

Walk directory


for root, dirs, files in os.walk('python/Lib/email'):
    print root, "consumes",

Itararting through csv

import csv
f = open('cities.csv')
reader = csv.reader(f)
reader.next()  # skip first line
for row in reader:
    print row

Exit with error

exception - exit failed script run (python) - Stack Overflow

import sys

if 1 < 0:
  print >> sys.stderr, "Something is seriously wrong."

The (optional) parameter of exit() is the return code the script will return to the shell. Usually values different than 0 signal an error.

Remove (delete) from a list by value

Is there a simple way to delete a list element by value in python? - Stack Overflow


Wrapping a function that saves some parameters

9.8. functools — Higher-order functions and operations on callable objects — Python v2.7.6 documentation

functools.wraps(wrapped[, assigned][, updated])

This is a convenience function for invoking partial(update_wrapper, wrapped=wrapped, assigned=assigned, updated=updated) as a function decorator when defining a wrapper function. For example:

>>> from functools import wraps
>>> def my_decorator(f):
...     @wraps(f)
...     def wrapper(*args, **kwds):
...         print 'Calling decorated function'
...         return f(*args, **kwds)
...     return wrapper
>>> @my_decorator
... def example():
...     """Docstring"""
...     print 'Called example function'
>>> example()
Calling decorated function
Called example function
>>> example.__name__
>>> example.__doc__

Mocking datetime with mock

Ned Batchelder: Mocking datetime.today How to: Unit testing in Django with mocking and patching - William John Bert


from datetime import datetime

def add(item_title=None, parent_pk=0):
    date_now = datetime.now()
    print date_now


import mock

class MockDateTime(object):
    "A fake replacement for date that can be mocked for testing."
    def __new__(cls, *args, **kwargs):
        return object.__new__(object, *args, **kwargs)

class TestAddition(TestCase):
    @mock.patch('progressio.progressio.datetime', MockDateTime)
    def test_content(self):
        from datetime import datetime
        MockDateTime.now = classmethod(lambda cls: datetime(2014, 3, 20))

Stop nosetests when first fails

Use option: -x, --stop.

Delete an element from dict by key

How to remove a key from dictionary?

my_dict.pop("key", None)

Datetime to UNIX timestamp


from datetime import datetime
d = datetime.utcnow()
import calendar

Now in UTC


from datetime import datetime

Convert from datetime.datetime to time.time timestamp

>>> t = datetime.datetime.now()
>>> t
datetime.datetime(2011, 11, 5, 11, 26, 15, 37496)

>>> time.mktime(t.timetuple()) + t.microsecond / 1E6

Touch reloading Django sites running with uwsgi in emperor mode

To gracefully reload a site on a new deployment of code, you can simply touch the vassal file, e.g. touch /etc/uwsgi/vassals/example.ini


How size of dict changes on deletes


To avoid thrashing when a mix of additions and deletions are made when the table is near a resize threshold, Python doesn't actually check the # of virgins [never used slots] after a delete (in effect, it assumes you'll soon be replacing the turds [slots that are empty but were used] with citizens [currently used slots] again). So, curiously enough, deleting a key never shrinks the table. A long sequence of deletes followed by an add may shrink it, though. A way to force possible shrinkage without adding a key is:

    dict = dict.copy()

Redirecting unicode output from console directly to file


You could redefine sys.stdout so all output is encoded in utf-8:

import sys
import codecs

Or specify encoding while printing (this seems less appropriate when many prints are used).

print (unicode_obj.encode('utf-8'))

Change working directory to file directory during script execution



Multiple files in with statement


with open('a', 'w') as a, open('b', 'w') as b:

Source code file encoding

# -*- coding: utf-8 -*-

http://stackoverflow.com/questions/6289474/working-with-utf-8-encoding-in-python-source http://www.python.org/dev/peps/pep-0263/

Attempt to write a readonly database error in Django


Set permissions both to database file and to the directory where it is stored for user running the process (typically www-data).

Datetime query in Sqlalchemy

subscriptions = UserCourse.query.filter(UserCourse.next_email_time <= datetime.now()).all()

Add parent directory to sys.path to import modules from it

sys.path.append(os.path.join(os.path.dirname(__file__), '..'))

Alembic error with existing ENUM types in database in Postgres

Use ENUM from postgresql and specify create_type=False:

from sqlalchemy.dialects.postgresql import ENUM

category_enum = ENUM('a', 'b', 'c', name='category_enum', create_type=False)


Auto update and auto default for time attributes in Sqlalchemy

from myapp import db

created_on = db.Column(db.DateTime, default=db.func.now())
updated_on = db.Column(db.DateTime, default=db.func.now(), onupdate=db.func.now())

http://stackoverflow.com/questions/13978554/is-possible-to-create-column-in-sqlalchemy-which-is-going-to-be-automatically-po http://docs.sqlalchemy.org/en/rel_0_7/core/schema.html#metadata-defaults

Jinja macro for WTForms form rendering with errors

{% macro render_field(field) %}
<div class="form_field">
    {{ field.label(class="label") }}
    {% if field.errors %}
    {% set css_class = 'has_error ' + kwargs.pop('class', '') %}
    {{ field(class=css_class, **kwargs) }}
    <ul class="errors">{% for error in field.errors %}<li>{{ error|e }}</li>{% endfor %}</ul>
    {% else %}
    {{ field(**kwargs) }}
    {% endif %}
{% endmacro %}

To specify class

{% from "forms/macros.html" import render_field %}
{{ render_field(form.body, class="email-body") }}

from https://github.com/mitsuhiko/flask/wiki/Large-app-how-to#first-template

Also see


Use specific browser if webbrowser library is used


export BROWSER='google-chrome'

Convert from string to datetime (string parse time)

Mnemonic rule for remembering the function - string parse time - strptime.

import datetime

DROPBOX_DATE_FORMAT = "%a, %d %b %Y %H:%M:%S +0000"


d = datetime.datetime.strptime(s['modified'], DROPBOX_DATE_FORMAT)):

There is similar function in time module.

Split a string on several delimiters

>>> import re
>>> string = 'this is, a test of splitting; on two delimiters'
>>> re.split(r'[,;]', string)
['this is', ' a test of splitting', ' on two delimiters']
>>> re.split(r'[, ]', string)
['this', 'is', '', 'a', 'test', 'of', 'splitting;', 'on', 'two', 'delimiters']

InlineModelAdmin: Show partially an inline model and link to the complete model


I create a new template called linked.html that is a copy of tabular.html and I added this code to create the link.

{% if inline_admin_form.original.pk %}
          <td class="{{ field.field.name }}">
              <a href="/admin/{{ app_label }}/{{ inline_admin_formset.opts.admin_model_path }}/{{ inline_admin_form.original.pk }}/">Full record</a>
{% endif %}

then I created a new model LinkedInline inheriting InlineModelAdmin

#override of the InlineModelAdmin to support the link in the tabular inline
class LinkedInline(admin.options.InlineModelAdmin):
    template = "admin/linked.html"
    admin_model_path = None

    def __init__(self, *args):
        super(LinkedInline, self).__init__(*args)
        if self.admin_model_path is None:
            self.admin_model_path = self.model.__name__.lower()

Then when I define a new inline, I have only to use my LinkedInline instead of the normal InlineModelAdmin.

Reseting South migrations

https://groups.google.com/forum/?fromgroups=#!topic/south-users/NKOWJd3xbZE http://balzerg.blogspot.co.il/2012/09/django-app-reset-with-south.html

Add current directory to PYTHONPATH



Delete (remove) files


import os

filelist = [ f for f in os.listdir(".") if f.startswith("progress.") ]
for f in filelist:

XMPP client


Storing (saving) password in keyring


The keyring lib has two functions:

  • get_password(service, username): Returns the password stored in keyring. If the password does not exist, it will return None.
  • set_password(service, username, password): Store the password in the keyring.

Dumping user data in Django

python manage.py dumpdata auth.User --indent 4 > users.json

To use s3 storage in Django


another option:


Create User when creating model

from django.db.models.signals import post_save
from django.dispatch import receiver


@receiver(post_save, sender=Bot)
def bot_save_handler(sender, instance, created, **kwargs):
    if created:
        u = User.objects.create_user(instance.name, 'dudarev@gmail.com', '')
        instance.user = u

current transaction is aborted, commands ignored until end of transaction block


from django.db import connection

Makefile settings for migration with South

ifndef APP
  $(MANAGE) migrate
  @echo Starting of migration of $(APP)
  $(MANAGE) schemamigration $(APP) --auto
  $(MANAGE) migrate $(APP)
  @echo Done

ifndef APP
  @echo Please, specify -e APP=appname argument
  @echo Starting init migration of $(APP)
  $(MANAGE) schemamigration $(APP) --initial
  $(MANAGE) migrate $(APP)
  @echo Done

Setup a model to edit in admin


File admin.py in app directory:

from django.contrib import admin
from myproject.myapp.models import Author


Setting up db with postgres


Change password if needed

psql -U postgres

postgres=# \password postgres
Enter new password:
Enter it again:

Create user if needed

$ createuser -U postgres yournewuser -P
Enter password for new role:
Enter it again:
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) y
Shall the new role be allowed to create more new roles? (y/n) n

Create db

$ createdb -U yournewuser -E utf8 -O yournewuser yournewdb -T template0

Update settings.py

    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'yournewdb',
        'USER': 'yournewuser',
        'PASSWORD': 'whateverpasswordyouenteredearlier',
        'HOST': '',
        'PORT': '',

or using dj_database_url:

import dj_database_url
DATABASES = {'default': dj_database_url.config(default='postgres://user:password@localhost/name')}

local_settings.py for Django


At the very end of your settings.py, add this:

    from local_settings import *
except ImportError:

Storing more information about users


from django.contrib.auth.models import User

class UserProfile(models.Model):
    # This field is required.
    user = models.OneToOneField(User)

    # Other fields here
    accepted_eula = models.BooleanField()
    favorite_animal = models.CharField(max_length=20, default="Dragons.")
AUTH_PROFILE_MODULE = 'accounts.UserProfile'

Django tweek for vim

From http://sontek.net/blog/detail/turning-vim-into-a-modern-python-ide

The only true django tweak I make is before I open vim I'll export the DJANGO_SETTINGS_MODULE environment so that I get code completion for django modules as well:

export DJANGO_SETTINGS_MODULE=project.settings

I put into Makefile:

  DJANGO_SETTINGS_MODULE=project.settings gvim .

Django Registration


Docs: http://django-registration.readthedocs.org/en/latest/index.html

Templates: https://github.com/macdhuibh/django-registration-templates

Generating Sphinx docs from source

Documentation is generated from source when adding a path to config.py:

sys.path.insert(0, os.path.relpath('../../cghub_api'))

sphinx-apidoc generates completely automatic API documentation for a Python package:

sphinx-apidoc -o source/ ../cghub_api/

Not capturing standard output with nose


-s: Not capturing stdout

By default, nose captures all output and only presents stdout from tests that fail. By specifying '-s', you can turn this behavior off.

Selecting tests with Nose


To specify which tests to run, pass test names on the command line:

nosetests only_test_this.py

Test names specified may be file or module names, and may optionally indicate the test case to run by separating the module or file name from the test case name with a colon. Filenames may be relative or absolute. Examples:

nosetests test.module
nosetests another.test:TestCase.test_method
nosetests a.test:TestCase
nosetests /path/to/test/file.py:test_function

Pip install from Github with -e

There is an issue with virtualenv and pip when installing from github with -e flag. It is reported for instance in this issue with py-github code:


I've encountered it with trying to install from https://github.com/fxdgear/pygpx

-e flag means that the source code is editable as described in pip docs.

There are several options:

  • do not use the flag, it works for me for pygpx, in requirements.txt:
  • change directories structure as was done in py-github.

Using nose

cd path/to/project

Testing for exception


def test_afunction_throws_exception(self):
    self.assertRaises(ExpectedException, afunction)

And if afunction takes arguments, just pass them into assertRaises like this:

def test_afunction_throws_exception(self):
    self.assertRaises(ExpectedException, afunction, arg1, arg2)


mkvirtualenv [-a project_path] [-i package] [-r requirements_file] [virtualenv options] ENVNAME



Useful commands

rmvirtualenv ENVNAME

Installation - Shell Startup File


Add three lines to your shell startup file (.bashrc, .profile, etc.) to set the location where the virtual environments should live, the location of your development project directories, and the location of the script installed with this package:

export WORKON_HOME=$HOME/.virtualenvs
source /usr/local/bin/virtualenvwrapper.sh

After editing it, reload the startup file (e.g., run source ~/.bashrc).

Checking that JPEG in PIL works from shell

With a JPG image file_name.jpg in the same directory

from PIL import Image
trial_image = Image.open('file_name.jpg')

Installing PIL with virtualenv

If PIL is required it is better to install it correctly once in the system and then use virtualenv with --site-packages.

Problems with JPEG


sudo apt-get install libjpeg62-dev
/myEnv/bin/pip install --no-install PIL
cd /myEnv/build/PIL

Change in setup.py

JPEG_ROOT = libinclude("/usr/lib")
/myEnv/bin/pip install PIL

Another possibility


For Ubuntu 11.04, what finally worked for me is:

pip uninstall PIL
sudo apt-get install libjpeg8-dev
pip install PIL

And yet another approach:


Rather than mess with the setup.py in PIL, you can also simply create symlinks to the libraries in /usr/lib (assuming you have sudo access).

I did this with the following commands:

sudo ln -s /usr/lib/x86_64-linux-gnu/libfreetype.so /usr/lib/
sudo ln -s /usr/lib/x86_64-linux-gnu/libz.so /usr/lib/
sudo ln -s /usr/lib/x86_64-linux-gnu/libjpeg.so /usr/lib/

And one more good link for Ubuntu 11.10 - http://obroll.com/install-python-pil-python-image-library-on-ubuntu-11-10-oneiric/

Zen of Python



  • Beautiful is better than ugly.
  • Explicit is better than implicit.
  • Simple is better than complex.
  • Complex is better than complicated.
  • Flat is better than nested.
  • Sparse is better than dense.
  • Readability counts.
  • Special cases aren't special enough to break the rules.
  • Although practicality beats purity.
  • Errors should never pass silently.
  • Unless explicitly silenced.
  • In the face of ambiguity, refuse the temptation to guess.
  • There should be one-- and preferably only one --obvious way to do it.
  • Although that way may not be obvious at first unless you're Dutch.
  • Now is better than never.
  • Although never is often better than right now.
  • If the implementation is hard to explain, it's a bad idea.
  • If the implementation is easy to explain, it may be a good idea.
  • Namespaces are one honking great idea -- let's do more of those!

Nohup and Python


See Python output buffering four ways to do the latter:

  1. Use the -u command line switch
  2. Wrap sys.stdout in an object that flushes after every write
  4. sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)

Move from --no-site-packages


Frank Wiles ran into this problem on IRC, where he wanted to add in the site-packages after creating a virtualenv with --no-site-packages. It turns out to be really simple, in that you only have to remove the no-global-site-packages.txt in the lib/python2.x directory inside the virtualenv. After that virtualenv will go ahead and fallback to the global site packages happily.

Test coverage in Python

Coverage utility: http://nedbatchelder.com/code/coverage/

Testing AJAX request with Django


from django.test.client import Client
client = Client()
client.post('http://example.com', {'foo': 'bar'}, **{'HTTP_X_REQUESTED_WITH': 'XMLHttpRequest'})

Conditional expression idiom


Starting with python 2.5, you can use:

x if c else y

In Django 1.4 it will be possible to redefine (override) settings while testing


DEBUG=False in test server in Django


Regardless of the value of the DEBUG setting in your configuration file, all Django tests run with DEBUG=False. This is to ensure that the observed output of your code matches what will be seen in a production setting.

Binding uploaded files to forms


Installing PIP in virtualenv



Template inheritance in Django templates

Using blocks in Django templates


The most powerful -- and thus the most complex -- part of Django's template engine is template inheritance. Template inheritance allows you to build a base "skeleton" template that contains all the common elements of your site and defines blocks that child templates can override.

Step-by-step tutorials for testing in Django



Testing that user logged in and using reverse for URLs


from django.contrib.auth.models import User
from django.core.urlresolvers import reverse
from django.test.client import Client
import unittest

class LoginTestCase(unittest.TestCase):
    def setUp(self):
        self.client = Client()
        self.user = User.objects.create_user('john', 'lennon@thebeatles.com', 'johnpassword')

    def testLogin(self):
        self.client.login(username='john', password='johnpassword')
response = self.client.get(reverse('testlogin-view'))
        self.assertEqual(response.status_code, 200)

I suggest you (if you don't use them already) to use the reverse() function and name your URLs. This way you are sure that you get always the right URL.

Common issue with redirect when testing Django apps


r = self.client.get('/foo')
self.assertEquals(r.status_code, 200)

FAIL Assertion Error: 301 != 200

r = self.client.get('/foo', follow=True)
self.assertEquals(r.status_code, 200)

response.redirect_chain - links visited before a non-redirect was found.

Simple Login Form


By default, the login view renders a template at registration/login.html (you can change this template name by passing an extra view argument ,template_name). This form needs to contain a username and a password field. A simple template might look like this:

{% extends "base.html" %}

{% block content %}

  {% if form.errors %}
    <p class="error">Sorry, that's not a valid username or password</p>
  {% endif %}

  <form action="" method="post">
    <label for="username">User name:</label>
    <input type="text" name="username" value="" id="username">
    <label for="password">Password:</label>
    <input type="password" name="password" value="" id="password">

    <input type="submit" value="login" />
    <input type="hidden" name="next" value="{{ next|escape }}" />

{% endblock %}

When to use django-admin and when manage.py


Why do the Django documentation code examples using django-admin.py instead of manage.py when demonstrating subcommands such as loaddata and dumpdata?

Well, because these scripts are the same in priciple, with the differences, you already mentioned. The Django docs also mention

django-admin.py <subcommand> [options]
manage.py <subcommand> [options]

side by side. Usually you use django-admin.py to start a new project or application and manage.py to do the rest.

Or if really want to use django-admin:

Use virtualenv, and have DJANGO_SETTINGS_MODULE set by bin/activate, and then you can use django-admin.py

Otherwise an error ImportError: Settings cannot be imported, because environment variable DJANGO_SETTINGS_MODULE is undefined.

Django full text search

django-fts on Google code can not be ran right after cloning. I forked and fixed it:


To include with pip in virtualenv add the following line to requirements.txt

-e git+https://github.com/dudarev/django-fts#egg=fts

If you tried Google Code version. Drop fts_ tables. Sync database with manage.py. Restart the server.

How to determine from which class a method was inherited from?

Use the inspect module.


Multiple Inheritance


Python supports a limited form of multiple inheritance as well. A class definition with multiple base classes looks like this:

class DerivedClassName(Base1, Base2, Base3):

For old-style classes, the only rule is depth-first, left-to-right. Thus, if an attribute is not found in DerivedClassName, it is searched in Base1, then (recursively) in the base classes of Base1, and only if it is not found there, it is searched in Base2, and so on.

With new-style classes, dynamic ordering is necessary because all cases of multiple inheritance exhibit one or more diamond relationships (where at least one of the parent classes can be accessed through multiple paths from the bottommost class). For example, all new-style classes inherit from object, so any case of multiple inheritance provides more than one path to reach object. To keep the base classes from being accessed more than once, the dynamic algorithm linearizes the search order in a way that preserves the left-to-right ordering specified in each class, that calls each parent only once, and that is monotonic (meaning that a class can be subclassed without affecting the precedence order of its parents). Taken together, these properties make it possible to design reliable and extensible classes with multiple inheritance. For more detail, see http://www.python.org/download/releases/2.3/mro/.

New-style and classic classes


Classes and instances come in two flavors: old-style (or classic) and new-style.

Up to Python 2.1, old-style classes were the only flavour available to the user. The concept of (old-style) class is unrelated to the concept of type: if x is an instance of an old-style class, then x.__class__ designates the class of x, but type(x) is always . This reflects the fact that all old-style instances, independently of their class, are implemented with a single built-in type, called instance.

New-style classes were introduced in Python 2.2 to unify classes and types. A new-style class is neither more nor less than a user-defined type. If x is an instance of a new-style class, then type(x) is typically the same as x.__class__ (although this is not guaranteed - a new-style class instance is permitted to override the value returned for x.__class__).

The major motivation for introducing new-style classes is to provide a unified object model with a full meta-model. It also has a number of practical benefits, like the ability to subclass most built-in types, or the introduction of “descriptors”, which enable computed properties.

For compatibility reasons, classes are still old-style by default. New-style classes are created by specifying another new-style class (i.e. a type) as a parent class, or the “top-level type” object if no other parent is needed. The behaviour of new-style classes differs from that of old-style classes in a number of important details in addition to what type() returns. Some of these changes are fundamental to the new object model, like the way special methods are invoked. Others are “fixes” that could not be implemented before for compatibility concerns, like the method resolution order in case of multiple inheritance.

While this manual aims to provide comprehensive coverage of Python’s class mechanics, it may still be lacking in some areas when it comes to its coverage of new-style classes. Please see http://www.python.org/doc/newstyle/ for sources of additional information.

Old-style classes are removed in Python 3.0, leaving only the semantics of new-style classes.

User Registartion

https://github.com/lig/django-registration-me - user registration app for Django using Mongoengine.

Exporting data to Excel in Django


Testing Django from terminal

When testing with Python from terminal

import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'myapp.settings'

Python SMTP server for testing sending emails


import smtpd, asyncore
server = smtpd.DebuggingServer(('localhost', 8000), None)