Setting a Full Testing Framework for Django (and more!)

Date Mon 20 July 2015 By Emmanuel Fleury Category tools

Initial Thoughts

Having a production website and making some development on it is sometimes quite dodgy. You really need to know what you are doing before pushing your new developments to the production site. That is why you really need to have a quite wide spectrum of tests and barriers to prevent you to push unwanted code or security issues, or broken features in front of your users.

In fact, the Python language and the Django framework have already quite good tools to help you to do this. My point here will be to list the ones I tried and that I found useful for a few of my projects.

Django Unit Testing

The principle of Unit Testing is to check each unit (feature) of the software in an independent way from the others. Related to websites, it means that we have to identify each service it provides and to check each service independently from the others. Usually, we measure the quality of a test suite (the set of all the tests we have written) by measuring how much code we executed at least once when running the whole set. More you cover code, more your confidence in the code get higher.

Of course, this way of doing is not always bullet-proofed, because the bugs may come from the interaction of services. But, at least, it gives you a basic criteria of quality that you can apply to your code.

The Django framework has already integrated a full unit testing framework borrowed from the Python language itself. If you already developed a few unit testing in Python, things will not change a lot.

First of all, a little bit of vocabulary:

  • Test case: The smallest piece of a test suite, it is a single test (or several very similar tests) on a precise feature of your software.

  • Test suite: A set of test cases that tries to cover as much code as possible of the tested software.

  • Test fixture: Enclose all the operations that are performed before (to get the environment ready for the test) and after (to clean the environment after the test in order to get ready for the next one) the test get executed. The Python Unit Testing framework call the method setUp() before each test case and the method tearDown() after each test case.

  • Test runner: The program that will find all the test cases present in the Django project, that will run it in a given order and that will collect the results and display it.

Writing a test suite is quite easy, take the root of your Django project (where settings.py lies) and write a test.py file as follow:

from django.test import Client, TestCase

class TestLoggedUser(TestCase):
    def setUp(self):
        self.client = Client()

        self.user = User.objects.create_user('test_user', 'user@test.net', 'secret')
        self.user.save()
        self.client.login(username='test_user', password='secret')

    def tearDown(self):
        self.user.delete()

    def test_logged_user_get_homepage(self):
        response = self.client.get(reverse('/'), follow=True)
        self.assertEqual(response.status_code, 200)

    def test_logged_user_get_settings(self):
        response = self.client.get(reverse('/settings/'), follow=True)
        self.assertEqual(response.status_code, 200)

In the previous example, we check if a logged user can get the homepage of the website and its own settings. The fixtures take care of creating, logging and, then, deleting the user before and after each test.

Executing the tests is done through the manage.py command (you may add the option --verbosity=3 to get more output):

$> ./manage.py test
Creating test database for alias 'default'...
.............................................
----------------------------------------------------------------------
Ran 45 tests in 13.102s

OK
Destroying test database for alias 'default'...

Integrated HTML validator with django-html-validator

The django-html-validator is a Django application developed to check automatically if the served HTML pages are valid for the W3C checkers. As usual, installation is quite simple through pip:

$> pip install django-html-validator

Then, add the following to your settings.py file:

HTMLVALIDATOR_ENABLED = True

Note that, by default, the validation is done on-line through a request to the W3C server. If you want to work off-line (or do not want to overload the network), you can use the Nu HTML Checker project to validate locally the webpages. Get the vnu.jar file from the last build and install this file somewhere in your project.

Then, in settings.py, set the variable HTMLVALIDATOR_VNU_JAR to point to this vnu.jar file.

HTMLVALIDATOR_VNU_JAR = './contrib/vnu.jar'

Then, when you define a new test case, the only thing you need is to import a ValidatingClient which behaves exactly like a usual Client but that also check the HTML against W3C verifier. It is written as follow:

from django.test import TestCase
from htmlvalidator.client import ValidatingClient

class CheckExample(TestCase):
    def setUp(self):
    self.client = ValidatingClient()

def tearDown(self):
    pass

def test_example(self):
    response = self.client.get('/example/')
    self.assertEqual(response.status_code, 200)

Then, when running the tests with ./manage.py test:

$> ./manage.py test
Creating test database for alias 'default'...
.......VALIDATION TROUBLE
To debug, see:
         /tmp/htmlvalidator/test_example-619.html
         /tmp/htmlvalidator/test_example-619.txt

And, looking at test_example-619.txt shows:

Arguments to GET:
Error: Start tag "body" seen but an element of the same type was already open.
From line 95, column 8; to line 95, column 13
There were errors. (Tried in the text/html mode.)

After fixing the bug, the tests are running without any problem:

$> ./manage.py test
Creating test database for alias 'default'...
.............................................
----------------------------------------------------------------------
Ran 45 tests in 13.102s

OK
Destroying test database for alias 'default'...

Code coverage with coverage.py

coverage.py is a framework allowing to perform code coverage while your tests cases are walked through.

Installation is quite easy:

$> pip install coverage

Usage is also a child play, just run your test base with the following command:

$> coverage run --source='.' manage.py test
$> coverage report -m
Name                      Stmts   Miss  Cover   Missing
-------------------------------------------------------
my_program                   20      4    80%   33-35, 39
my_other_module              56      6    89%   17-23
-------------------------------------------------------
TOTAL                        76     10    87%

And, generating a full HTML report to visualize which lines are covered and which are still to be covered is as follow (the tests must have been ran before):

$> coverage html
$> firefox htmlcov/index.html

A full example of such HTML report can be seen on this page.

So, here we only follow the covering of the code, but we can also follow the covering of the templates, together with the integrated HTML validator it might be useful to know if we covered most of the generated HTML or not. In fact, the coverage.py module has a django_coverage_plugin module that does exactly this.

Install the module:

$> pip install django_coverage_plugin

Add the following lines to your ~/.coveragerc (create one if you do not have one):

[run]
# The 'timid' line might be unnecessary
# timid = True
plugins = django_coverage_plugin

Then, run the coverage checking with the following command line:

$> coverage run --rcfile='/home/user/.coveragerc' --source='.' manage.py test
$> DJANGO_SETTINGS_MODULE=myproject.settings coverage report -m --rcfile='/home/user/.coveragerc'
Name                                   Stmts   Miss  Cover   Missing
-------------------------------------------------------------------------
myproject/__init__.py                      1      0   100%   
myproject/admin.py                        30      0   100%   
myproject/models.py                       78      3    96%   35, 60, 145
myproject/settings.py                     49      0   100%   
myproject/templates/_account_bar.html     30      2    93%   14-15
myproject/templates/_footer.html           1      0   100%   
myproject/templates/about.html            51      0   100%   
myproject/templates/homepage.html        146      0   100%   
myproject/templates/site_base.html        16      0   100%   
myproject/templates/subnav_base.html       8      2    75%   11-15
myproject/urls.py                         14      0   100%   
myproject/views.py                       306     26    92%   129, 203-228
myproject/wsgi.py                         15     15     0%   16-39
manage.py                                  7      0   100%   
-------------------------------------------------------------------------
TOTAL                                    752     48    94%

Finally…

We got something quite complete, we are able to run unit test cases on the whole project. test for W3C compliance of the generated HTML files and have a coverage report of all these tests over Python files and templates.

I did not check for other web framework than Django, but I believe this one as already quite advanced. What could be improved is that most of theses features come from external development and are not fully integrated in the Django framework. It would be exremely nice to have it from scratch in the Django base libraries.