Setting a Full Testing Framework for Django (and more!)
Initial Thoughts
Having a production website and making some development on it is sometimes quite dodgy. You really need to know what you are doing before pushing your new developments to the production site. That is why you really need to have a quite wide spectrum of tests and barriers to prevent you to push unwanted code or security issues, or broken features in front of your users.
In fact, the Python language and the Django framework have already quite good tools to help you to do this. My point here will be to list the ones I tried and that I found useful for a few of my projects.
Django Unit Testing
The principle of Unit Testing is to check each unit (feature) of the software in an independent way from the others. Related to websites, it means that we have to identify each service it provides and to check each service independently from the others. Usually, we measure the quality of a test suite (the set of all the tests we have written) by measuring how much code we executed at least once when running the whole set. More you cover code, more your confidence in the code get higher.
Of course, this way of doing is not always bullet-proofed, because the bugs may come from the interaction of services. But, at least, it gives you a basic criteria of quality that you can apply to your code.
The Django framework has already integrated a full unit testing framework borrowed from the Python language itself. If you already developed a few unit testing in Python, things will not change a lot.
First of all, a little bit of vocabulary:
-
Test case: The smallest piece of a test suite, it is a single test (or several very similar tests) on a precise feature of your software.
-
Test suite: A set of test cases that tries to cover as much code as possible of the tested software.
-
Test fixture: Enclose all the operations that are performed before (to get the environment ready for the test) and after (to clean the environment after the test in order to get ready for the next one) the test get executed. The Python Unit Testing framework call the method
setUp()
before each test case and the methodtearDown()
after each test case. -
Test runner: The program that will find all the test cases present in the Django project, that will run it in a given order and that will collect the results and display it.
Writing a test suite is quite easy, take the root of your Django
project (where settings.py
lies) and write a test.py
file as follow:
from django.test import Client, TestCase
class TestLoggedUser(TestCase):
def setUp(self):
self.client = Client()
self.user = User.objects.create_user('test_user', 'user@test.net', 'secret')
self.user.save()
self.client.login(username='test_user', password='secret')
def tearDown(self):
self.user.delete()
def test_logged_user_get_homepage(self):
response = self.client.get(reverse('/'), follow=True)
self.assertEqual(response.status_code, 200)
def test_logged_user_get_settings(self):
response = self.client.get(reverse('/settings/'), follow=True)
self.assertEqual(response.status_code, 200)
In the previous example, we check if a logged user can get the homepage of the website and its own settings. The fixtures take care of creating, logging and, then, deleting the user before and after each test.
Executing the tests is done through the manage.py
command (you may
add the option --verbosity=3
to get more output):
$> ./manage.py test
Creating test database for alias 'default'...
.............................................
----------------------------------------------------------------------
Ran 45 tests in 13.102s
OK
Destroying test database for alias 'default'...
Integrated HTML validator with django-html-validator
The
django-html-validator
is a Django application developed to check automatically if the served
HTML pages are valid for the W3C checkers. As usual, installation is
quite simple through pip
:
$> pip install django-html-validator
Then, add the following to your settings.py
file:
HTMLVALIDATOR_ENABLED = True
Note that, by default, the validation is done on-line through a
request to the W3C server. If you want to work off-line (or do not
want to overload the network), you can use the Nu HTML Checker
project to validate locally
the webpages. Get the vnu.jar
file from the last
build and
install this file somewhere in your project.
Then, in settings.py
, set the variable HTMLVALIDATOR_VNU_JAR
to
point to this vnu.jar
file.
HTMLVALIDATOR_VNU_JAR = './contrib/vnu.jar'
Then, when you define a new test case, the only thing you need is to
import a ValidatingClient
which behaves exactly like a usual Client
but that also check the HTML against W3C verifier. It is written as follow:
from django.test import TestCase
from htmlvalidator.client import ValidatingClient
class CheckExample(TestCase):
def setUp(self):
self.client = ValidatingClient()
def tearDown(self):
pass
def test_example(self):
response = self.client.get('/example/')
self.assertEqual(response.status_code, 200)
Then, when running the tests with ./manage.py test
:
$> ./manage.py test
Creating test database for alias 'default'...
.......VALIDATION TROUBLE
To debug, see:
/tmp/htmlvalidator/test_example-619.html
/tmp/htmlvalidator/test_example-619.txt
And, looking at test_example-619.txt
shows:
Arguments to GET:
Error: Start tag "body" seen but an element of the same type was already open.
From line 95, column 8; to line 95, column 13
There were errors. (Tried in the text/html mode.)
After fixing the bug, the tests are running without any problem:
$> ./manage.py test
Creating test database for alias 'default'...
.............................................
----------------------------------------------------------------------
Ran 45 tests in 13.102s
OK
Destroying test database for alias 'default'...
Code coverage with coverage.py
coverage.py
is a
framework allowing to perform code coverage while your tests cases are
walked through.
Installation is quite easy:
$> pip install coverage
Usage is also a child play, just run your test base with the following command:
$> coverage run --source='.' manage.py test
$> coverage report -m
Name Stmts Miss Cover Missing
-------------------------------------------------------
my_program 20 4 80% 33-35, 39
my_other_module 56 6 89% 17-23
-------------------------------------------------------
TOTAL 76 10 87%
And, generating a full HTML report to visualize which lines are covered and which are still to be covered is as follow (the tests must have been ran before):
$> coverage html
$> firefox htmlcov/index.html
A full example of such HTML report can be seen on this page.
So, here we only follow the covering of the code, but we can also
follow the covering of the templates, together with the integrated
HTML validator it might be useful to know if we covered most of
the generated HTML or not. In fact, the coverage.py
module has a
django_coverage_plugin
module that does exactly this.
Install the module:
$> pip install django_coverage_plugin
Add the following lines to your ~/.coveragerc
(create one if you do
not have one):
[run]
# The 'timid' line might be unnecessary
# timid = True
plugins = django_coverage_plugin
Then, run the coverage checking with the following command line:
$> coverage run --rcfile='/home/user/.coveragerc' --source='.' manage.py test
$> DJANGO_SETTINGS_MODULE=myproject.settings coverage report -m --rcfile='/home/user/.coveragerc'
Name Stmts Miss Cover Missing
-------------------------------------------------------------------------
myproject/__init__.py 1 0 100%
myproject/admin.py 30 0 100%
myproject/models.py 78 3 96% 35, 60, 145
myproject/settings.py 49 0 100%
myproject/templates/_account_bar.html 30 2 93% 14-15
myproject/templates/_footer.html 1 0 100%
myproject/templates/about.html 51 0 100%
myproject/templates/homepage.html 146 0 100%
myproject/templates/site_base.html 16 0 100%
myproject/templates/subnav_base.html 8 2 75% 11-15
myproject/urls.py 14 0 100%
myproject/views.py 306 26 92% 129, 203-228
myproject/wsgi.py 15 15 0% 16-39
manage.py 7 0 100%
-------------------------------------------------------------------------
TOTAL 752 48 94%
Finally…
We got something quite complete, we are able to run unit test cases on the whole project. test for W3C compliance of the generated HTML files and have a coverage report of all these tests over Python files and templates.
I did not check for other web framework than Django, but I believe this one as already quite advanced. What could be improved is that most of theses features come from external development and are not fully integrated in the Django framework. It would be exremely nice to have it from scratch in the Django base libraries.