Upgrading Django (to 1.7) Part III: Django 1.7's New Features (original) (raw)

Published October 8, 2014 by

Reviewed by Jacinda Shelly

Edited by Amy Bekkerman

The goal of this part of the series is to offer a more in-depth look at all of the features seen in Part II. We will focus on migrations, the new app registry, and the system check framework but also introduce custom QuerySet classes and the Prefetch object. This article will build upon the Django 1.7 camelot project from last time. The Migrations in Django 1.7 section is recommended reading before starting this article.

The section in this article about migrations is targeted at beginners. The new app registry tries to be as accessible as possible, but uses intermediate Django concepts. The system check framework uses intermediate Python but should be accessible to Django beginners. The final sections about custom QuerySet classes and the Prefetch object are for intermediate Django developers.

Article Contents

Migrations

As detailed in the Understanding Migrations section of Part II, the goal of migrations is to provide a controlled environment for applying changes made in Django models to the database.

Migration History

Prior to Django 1.7, the recommended way to add migration functionality was via the South app. South was created in August 2008 by Django core contributor Andrew Godwin. The project quickly became the go-to app for the job and was widely considered a necessity for production-ready Django sites.

Starting in 2012, Andrew began talking about the future of South. Andrew saw two possibilities: South 2.0 or native Django migrations. Presenting on the topic at DjangoCon 2012, Andrew put forth the idea for both, providing partial code to get it working in Django 1.5 as a means of asking for a public review of the suggested API.

In March 2013, Andrew created a Kickstarter Campaign for native Django migrations. The campaign met its goal of ￡2,500 in just over an hour and all the stretch goals, set at a maximum of ￡7,000, just three hours after that. By the time the two-week campaign ended in April, the Kickstarter had raised ￡17,952 (equivalent to 27,397atthetime),effectivelyraising27,397 at the time), effectively raising 27,397atthetime),effectivelyraising26,027 (Kickstarter takes 5%) for native Django migrations.

The stretch goals included:

  1. ￡3,500: Andrew commits time to releasing Django 1.7 by working on the release blocker bugs
  2. ￡4,500: "More native operations, such as being able to move columns across foreign keys while retaining data, and CREATE INDEX CONCURRENTLY support"
  3. ￡5,500: "Migration plan analyser that tells you which migrations will block which tables, and for roughly how long (PostgreSQL and MySQL only)"
  4. ￡7,000: "A backport of the key features to a new major version of South to support those on Django 1.4 and 1.5"

South vs Native Migrations

When building camelot, we followed the same workflow for both South in Django 1.6 and native migrations in Django 1.7:

  1. Modify the model
  2. Create a schema migration
  3. Apply the schema migration

and / or

  1. Create a data migration
  2. Edit the data migration for our purposes
  3. Apply the data migration

This may lead you to believe that the two migration systems are identical, but this is not the case. Andrew Godwin did not incorporate South into Django; he created a different system with the same workflow. Under the hood, they are very different systems.

A key difference is that native migrations generate files that developers will be more comfortable reading and editing. Native migrations adhere to the DRY (Don't Repeat Yourself) philosophy much more closely, providing only the operations for forwards migrations and symmetrically calculating the reverse. Native migrations also drop the frozen model seen in South, as those should never be modified. Instead, native migrations compute the historical models on the fly thanks to the new App Registry system, which we shall discuss shortly.

Consider that native migrations actually consist of two systems: migration operations and the schema editor. The first uses the second to interact with the database. We have only examined the first, as it typically the only part developers will interact with. However, should you choose to add support for migrations for a new database, you will need to learn how to use the schema editor system.

Django 1.7 and South

Despite some claims on the internet, South does not support Django 1.7. According to the official documentation, there are several steps to upgrade to Django 1.7 from a website using South:

  1. Use South to migrate to the desired (probably latest) database state
  2. Delete the South migrations directory (assuming you have it in a version control system; if not, back it up)
  3. Create a new migrations/ directory in each app you wish support native migrations, along with migrations/__init__.py.
  4. Generate a new migration file for each app you wish to support native migrations
  5. Remove the South app from INSTALLED_APPS in settings.py

You'll note that steps three and four use the term "support native migrations." This is because it is possible to disable migrations in Django 1.7. Termed legacy support, simply removing the migrations/ directory from the app will disable the migration system. This comes with an important caveat. While possible, apps using legacy migration behavior should not have a relationship with an app using native migrations. If your app has a foreign key to the Django User model (migrated), then your app should use native migrations. However, should you wish to have a foreign key in your migrated app to a third party app that has yet to upgrade to native migrations, that will not constitute a problem.

Note that it is possible to package your apps to work with both South and native migrations. South 1.0, released August 1, 2014, will start looking for migration files in the south_migrations/ directory before migrations/. Furthermore, both Django and South know not to use the other's migration files. To that extent, if you must support both South and Django 1.7 migrations, you need only:

  1. Move your South migration files to the south_migrations/ directory
  2. Create a new migrations/ directory
  3. Generate a native migration file

In the event that these default directory names do not suit your purposes, both Django and South provide the ability to override the names of the directories in settings.py, thanks to the MIGRATION_MODULES and SOUTH_MIGRATION_MODULES settings. For instance, should we wish to rename the native migration directory for our roundtable app to db_migrations/, we could specify:

MIGRATION_MODULES = {'roundtable': 'roundtable.db_migrations'}

The upgrade process to Django 1.7 is thus quite flexible, accounting for several different use cases.

Migration Upgrade Caveat

At the time of writing (October 2014), it is possible for native migrations in Django 1.7 to run extremely slowly, as detailed in ticket 22608. The issue appears to affect only projects and apps with a large number of models. For instance, the numbers presented with the ticket are for a website with over 200 models. If your website has a large number of models, your upgrade strategy should account for the possibility that it may not be reasonable for you to upgrade at this stage. While a fix is expected soon---and discussion at DjangoCon US 2014 seemed to indicate a fix in 1.7.1---the ticket itself has not been updated recently.

App Registry

When running a Django website, one of the first actions Django must take is to find the apps associated with the project so that it knows what code to use for the website. Django uses the INSTALLED_APPS setting to find all of the apps in a project and build a list of apps to run. Django refers to lists of apps in this context as an app registry. In version 1.7, the actions Django takes to build this app registry are different than in previous versions.

App Registry History

Prior to Django 1.7, the app registry was stored in the AppCache object, and it was not a terribly popular part of the framework. Django core-committer Andrew Godwin likened the system to the Borg, largely because AppCache instances shared state: any change to one AppCache object was mirrored in all AppCache objects.

Work to improve AppCache started as early as 2007 and has continued on and off since then. In May 2013, Andrew Godwin began to modify AppCache to prepare for migrations. As we discovered during the camelot example, the migration system requires the ability to act on the historical models of the app. The Borg pattern employed by AppCache made this computation impossible and was thus the first thing Andrew changed to allow for the replacement of South's frozen model.

In December 2013, core contributor Aymeric Augustin informed the Django Developer mailing list that he was tackling the app-loading problem. Aymeric detailed all of the previous work that had been put into upgrading the app loader and then listed twelve items that people had previously attempted to fix. Aymeric picked three of the items and set out to rework the loading mechanism over the next two weeks, implementing the basis of Django 1.7's new app registry.

Django 1.7's AppConfig

The pre-Django 1.7 AppCache existed as part of the models section of Django's source code, with code in django/db/models/loading.py. Django's new app registry is now a first-class citizen, with a full package in django/apps/, implemented as a brand new Apps class. Unlike AppCache, each instance of Apps is separate from others, allowing the migration system to build historical apps and models. You can fetch the Apps object for your own project with:

from django.apps import apps as django_apps

Referred to as the master registry (the app registry for your project as it runs), django_apps is an instance of Apps, which contains a list of AppConfig objects. Starting in Django 1.7, all apps now have an AppConfig object attached to them. By default, Django will automatically generate the AppConfig object for every app. However, it is also possible for developers to explicitly define the AppConfig for an app, as we shall see shortly.

The distinction between an app registry and the master registry becomes necessary in Django 1.7. The master registry is what the project is using now. However, the migrations system (or any other) may build any number of app registries to suit its purposes.

Interacting with AppConfig

Because Django will automatically generate AppConfig objects for each app, we can interact with the AppConfig for roundtable right away. Note that in our shell session below, instead of importing the master registry as django_apps, we use camelot_apps, as that makes more sense---we are importing the list of all the apps currently being used in our camelot project. We will see why we rename the import each time shortly.

from django.apps import apps as camelot_apps roundtable = camelot_apps.get_app_config('roundtable') roundtable.name 'roundtable' roundtable.label 'roundtable' roundtable.path 'djangocon2014-updj17/camelot/roundtable' roundtable.get_models() <generator object get_models at 0x104bf3780> list(roundtable.get_models()) [<class 'roundtable.models.Knight'>] roundtable.get_model('Knight') <class 'roundtable.models.Knight'>

Of note, an AppConfig object comes with a label attribute, which is a simple string, and a name attribute, which is a name-spaced string. The difference between the two is not immediately apparent above. A better example would be the admin app, which has for label the string 'admin' but for name the string 'django.contrib.admin'. The difference between these two attributes is key to understanding AppConfig objects.

For the most part, it is unnecessary to customize the AppConfig class of an app. The new app-loading system, like the AppCache, works mostly in the background. However, to better demonstrate the new system, we will override roundtable's AppConfig. By convention, we create a roundtable/apps.py file and write:

roundtable/apps.py

-*- coding: utf-8

from future import unicode_literals from django.apps import AppConfig

class RoundtableConfig(AppConfig): name = 'roundtable' verbose_name = 'Round Table'

To inform camelot of this new class's existence, we must add the following to roundtable/__init__.py.

roundtable/init.py

default_app_config = 'roundtable.apps.RoundtableConfig'

Consider that the file name apps.py is purely convention. The attribute in roundtable/__init__.py allows us to place the AppConfig subclass in any file we wish. That said, I recommend following the new convention as it will make your project more accessible to other developers. This convention further explains our override of django.apps.apps. It is possible to have the apps attribute clash in your Python:

this is wrong!

from django.apps import apps from . import apps

For this reason, the official Django documentation recommends that you set the value of django.apps.apps to something like django_apps or camelot_apps.

name and label attributes

The name attribute is the only attribute that must be specified in an AppConfig class. It is name-spaced according to Python's module import scheme. For this reason we must assign name the value 'roundtable'. Should we use 'camelot.roundtable', for instance, Django will greet us with ImportError: No module named 'camelot.roundtable' when we attempt to use manage.py for anything.

We have not set the label attribute explicitly because it is automatically derived from name. However, we could override label to any desirable value. For instance, we could set it to 'rtable'. As the app methods use the label rather than the name, we would now fetch RoundtableConfig using the following:

from django.apps import apps as camelot_apps roundtable = camelot_apps.get_app_config('rtable') roundtable.name 'roundtable' roundtable.label 'rtable'

Changing an app's label has serious consequences. Consider that most of the app registry APIs find apps using their labels; if we change these labels then we need to change all of our calls. This includes all our migration files, where the dependencies list and RunPython methods use app labels. Fortunately, the command line comes to our rescue.

$ sed -i "" 's/roundtable/rtable/g' roundtable/migrations/*.py

The results of this command may be viewed on github.

Furthermore, the migrate command for roundtable is now:

$ ./manage.py migrate rtable

We have not changed the listing in /camelot/settings.py, as the INSTALLED_APPS list is actually a list of Python module paths, which may point either to the app package or to the AppConfig itself, as we shall see in the next section. The same goes for our definition of default_app_config in /roundtable/__init__.py.

There is no real advantage to overriding the label as we have or even to creating RoundtableConfig. I will nonetheless leave the following in the roundtable/apps.py file, as it will allow us to distinguish between the name and label attributes when we interact with the app registry in the Systems Check section.

roundtable/apps.py

-*- coding: utf-8

from future import unicode_literals from django.apps import AppConfig

class RoundtableConfig(AppConfig): name = 'roundtable' label = 'rtable' verbose_name = 'Round Table'

Recall that we must specify default_app_config in roundtable/__init__.py for Django to use our AppConfig subclass.

ready() method

The main reason for creating an AppConfig class---apart from the potential to override the label and verbose_name attributes---is the ready() method. This method is invoked at the time Django loads the apps into the app registry, making it an ideal place to register signals or system checks.

The trick with ready() is that we cannot import our models when it is called and that the database should not be used. However, as this is an AppConfig object, we can use the get_model() method, as demonstrated at the beginning of this section, allowing us to potentially code:

roundtable/apps.py

class RoundtableConfig(AppConfig): ... def ready(self): Knight = self.get_model('Knight')

In the case of roundtable, we don't currently have any need to implement ready() and no real reason to implement RoundtableConfig. We will instead see an example of ready() in the admin app in the next section.

Admin's AppConfig

To see proper use of the AppConfig ready() method, we can examine the admin app. Recall that the urls.py generated by Django 1.7 in our camelot project lacked the call to admin.autodiscover() seen in Django 1.6. Instead, Django 1.7 now uses the AppConfig class to invoke the function. autodiscover() is still implemented in django/contrib/admin/__init__.py but is now called in the ready() method of the AdminConfig class found in django/contrib/admin/apps.py. The call itself is found on the last line of the file, printed below.

django/contrib/admin/apps.py

from django.apps import AppConfig from django.core import checks from django.contrib.admin.checks import check_admin_app from django.utils.translation import ugettext_lazy as _

class SimpleAdminConfig(AppConfig): """Simple AppConfig which does not do automatic discovery."""

name = 'django.contrib.admin'
verbose_name = _("Administration")

def ready(self):
    checks.register(
        checks.Tags.admin)(check_admin_app)

class AdminConfig(SimpleAdminConfig): """The default AppConfig for admin which does autodiscovery."""

def ready(self):
    super(AdminConfig, self).ready()
    self.module.autodiscover()

The admin app actually features two AdminConfig classes. Django uses the django/contrib/admin/__init__.py file to figure out which one to call, thanks to the following line:

default_app_config = 'django.contrib.admin.apps.AdminConfig'

Should you wish to disable autodiscover(), simply switch the 'django.contrib.admin', string to 'django.contrib.admin.apps.SimpleAdminConfig', in your INSTALLED_APPS tuple in your project's settings.py file. Be aware that this will cause any code in the admin.py files to be ignored. The recommended workaround would be to invoke the necessary code using the new apps.py file, which will always be invoked for any app, provided the necessary code is in __init__.py.

Deprecated Behavior

As detailed in Part I, deprecations occur over the period of two major releases. The code in django/db/models/loading.py is a fantastic example of this in action. Prior to Django 1.7, it was possible to fetch the AppCache instance via:

from django.db.models.loading import cache

On line 315 of django/db/models/loading.py in Django 1.6, you would have found:

In Django 1.7, the file is unsurprisingly very different. On line 3 of django/db/models/loading.py, you will discover an import of the new system.

from django.apps import apps

On line 17, the AppCache is fully replaced by the new app registry.

Importing and using the cache will thus still work as it did before, even though it uses the new system. However, as the cache variable is considered deprecated, any import of this file results in a warning, created on line 7 of django/db/models/loading.py.

warnings.warn( "The utilities in django.db.models.loading are deprecated " "in favor of the new application loading system.", RemovedInDjango19Warning, stacklevel=2)

It will thus be possible to interact with cache as before until Django 1.9, at which point the django/db/models/loading.py module will be fully removed.

Apps Restrictions

The master registry must be built and configured before Django can take certain actions, including loading the custom user model or using parts of the translation system. In this section, we will focus on the custom user model, expanding on the troubleshooting section about Apps in the documentation.

If you are having problems with translation, the troubleshooting section is where you should start. However, note that the AdminConfig code printed earlier demonstrates the best-case use of translation in an AppConfig setting.

Starting in Django 1.5, Django gained the ability to specify a custom user model for the auth-contributed app in the project settings. Starting in Django 1.7, custom user models may only be fetched with django.contrib.auth.get_user_model() after the app registry has finished loading. Compared to Django 1.5 and 1.6, this limits when the custom user model may be referenced. Calls for custom user models must be made at runtime, not before.

However, best practice, as documented here, has always been to use the AUTH_USER_MODEL setting, avoiding direct reference. If you followed the documentation's recommendations, this new restriction will have no effect on your project. However, if you had come up with a different way of handling custom user models, the new app registry effectively forces you to follow recommended best practice.

For reference, you may find ticket 21875 and the discussion on the django developer mailing list to be useful.

Apps in Review

An app registry is simply an Apps instance that contains a list of AppConfig objects. The master registry is an app registry where the AppConfig objects are the list of every app currently in the project, derived from the INSTALLED_APPS list. Django 1.7 allows for developers to modify the attributes of an app's AppConfig by subclassing it. The convention is for developers to place this code in an apps.py file.

The app registry works mostly in the background to build a master registry and performs its key functions without many developers ever knowing (just as many developers never knew about Django 1.6's AppCache). However, understanding the system even a little allows the developer to better edit migrations and create checks, as we will see in the next section.

System Check Framework

The system check framework is brand new in Django 1.7. As demonstrated earlier in this article, it allows Django to warn you of mistakes and to help you during the programming and upgrading process.

System Check Framework History

Django 1.5 was a detailed-oriented upgrade. At deployment, developers discovered that the project settings.py file now expected USE_TZ, ALLOWED_HOSTS, and SECRET_KEY as mandatory settings. Furthermore, every call to the url template tag needed to be subtly altered, as the syntax changed to expect the use of quotation marks.

Django 1.6, probably in an attempt to avoid simple mistakes like the ones above, shipped with the django-admin.py check command, which "perform[ed] a series of checks to verify a given setup (settings/application code) [was] compatible with the current version of Django." If the system found any problem, it would then output a series of warnings, informing the user what the problem was and how to fix it.

A problem with the new check command was that its functionality overlapped with the django-admin.py validate command. The validate command had existed in Django since version 0.9 and its role was to check the validity of the developer's model code, a subset of the purpose of the check command. Further confusion resulted from the fact that many new developers associated the validate command with the validation and clean methods, meant to verify user input for models and forms.

In June 2013, Christopher Medrela proposed reworking the check and validation commands as part of his Google Summer of Code project. The code for both the check and validate commands was centralized and monolithic, and Christopher's goal was to modularize the code and allow for checks to be easily extended, not only within the framework itself but also on a per-project basis. Christopher completed his GSoC in September 2013. His code was merged into Django by Russel Keith-Magee in January 2014.

The history of validate is actually quite interesting. In July 2005, Adrian Holovaty moved code from django/bin/django-admin.py into the new file django/core/management.py. However, the validate command would only be added to django/core/management.py by Adrian in August 2005. This code would be released in November 2005 as part of Django 0.9.In May 2006, Adrian merged the "Magic-Removal Branch" and created _check_for_validation_errors() and get_validation_errors(), extending validate's utility and integrating the process into other commands, such as syncdb. This code was then released as part of Django 0.95 in July 2006.In August 2007, Adrian split django/core/management.py into separate files, creating the django/core/management/ package. The validation behavior was added to django/core/management/validation.py, while the actual validate command now existed in django/core/management/commands/validate.py. This new organization would be released with Django 1.0 in September 2008 and remain the same through Django 1.6. Starting in Django 1.7, django/core/management/validation.py no longer exists. The django/core/management/commands/validate.py file, containing the actual validate command, still exists. As per the deprecations rule, the command warns the user that the validate command will be removed in Django 1.9. The actual code now uses the CheckCommand class.

Using Django 1.7 System Checks

The new checks have two notable features . The first is how persistent the system is. Before Django 1.7, developers had the choice to avoid the check and validate commands entirely. As demonstrated in our camelot project, the new system will warn you as many times as is necessary if there are any problems.

The second notable feature is that checks are now fully extensible. Developers may register custom checks with the framework. These new checks will run just like any of the default checks. The System Check documentation offers an example skeleton check function:

from django.core.checks import register

@register() def example_check(app_configs, **kwargs): errors = [] # ... your check logic here return errors

The app_configs parameter will either pass in None or a list of of AppConfig objects pertaining to the command. We'll see this in just a minute. The kwargs are reserved for potential changes to the command in the future.

The @register() decorator is used by Django to find the check and use it during the system check. It accepts a list of string arguments, called tags. Django comes with four tags out of the box:

$ ./manage.py check --list-tags admin compatibility models signals

Developers may create tags simply by passing a string to register(), as we shall see in the next section.

Note that the development branch of Django also features a security tag. I expect this in Django 1.8, but neither the Django 1.7.1 nor Django 1.8 release notes mention the tag.

This allows developers to limit which checks are called.

$ ./manage.py check -t models

Furthermore, checks may specify which apps to apply the checks to.

$ ./manage.py check roundtable -t models

Creating a Check

Django best practice dictates that all of our models have a __str__() method. In the following section, we will write a check that verifies the existence of the method in all of our models and warns us if a particular model is missing the method.

Creating a Check just for roundtable

To begin, let's build a check that makes sure all of our roundtable models have a __str__() method. We create a new file roundtable/checks.py and add a skeleton check. Our check, named check_model_str, specifies a 'model' tag in the register() decorator. We could just as easily have passed 'model_attr' to register(), which would have created a new 'model_attr' tag.

/roundtable/models.py

-*- coding: utf-8

from future import unicode_literals from django.apps import apps as camelot_apps from django.core.checks import register, Warning

@register('models') def check_model_str(app_configs=None, **kwargs): errors = [ ] return errors

Note our imports. Given the potential namespace clash mentioned eariler, we ensure that our import of the master registry is overridden to camelot_apps. Furthermore, note that we also import the Warning object, which we will see shortly.

For Django to know of the existence of our check, we must let it know this file exists. In /roundtable/__init__.py, we add from . import checks.

roundtable/init.py

from . import checks default_app_config = 'roundtable.apps.RoundtableConfig'

Note that we could also use our AppConfig to register our check. We would first remove the register() decorator from the check, remove the import in /roundtable/__init__.py (created above), and then add the following code to our /roundtable/apps.py:

roundtable/apps.py

from django.core.checks import register from .checks import check_model_str ... class RoundtableConfig(AppConfig): ... def ready(self): super(RoundtableConfig, self).ready() register('models')(check_model_str)

I prefer the first method, however, and will revert the project code, meaning:

  1. Our check has the register() decorator applied
  2. RoundtableConfig does not have a ready() method defined
  3. The new check package is imported via from . import checks in /roundtable/__init__.py.

Our first implementation task is to find all of the models in our project that do not have the __str__() method defined. Below, we use a list comprehension to query the app registry for all such models.

/roundtable/models.py

def check_model_str(app_configs=None, **kwargs): problem_models = [ model for model in camelot_apps.get_models() if 'str' not in model.dict ]

If we have any items in our problem_models list, we want to inform the system checks framework of the problem. The Warning class allows us to print an error via system checks. We use a list comprehension again to generate our errors list. For every item in problem_models we will create a Warning object in the errors list.

/roundtable/models.py

def check_model_str(app_configs=None, **kwargs): ... errors = [ Warning( "All Models must have a str method.", hint=("See https://docs.djangoproject.com/" "en/1.7/ref/models/instances/#str" " for more information."), obj=model, id='rtable.W001', ) for model in problem_models ]

The Warning instantiation process accepts four arguments: message, hint, object, and ID.

The Warning class expects the message and hint arguments. The message is mandatory for all of the classes that inherit CheckMessage, which are Debug, Info, Error, and Critical in addition to Warning. The message should inform the developer of the problem. The hint is intended to provide help for fixing the issue being described by the message. While the message is mandatory, it is possible to omit a hint by passing in None. Both the message and the hint are supposed to be short strings with no newlines.

The obj and id parameters are optional, but I encourage their use when possible because of their utility. The obj parameter allows the message to pinpoint the issue to a single object or class, while id allows the developer to run a single search for the string to find the code that raised the message.

While it is possible to use CheckMessage directly, my understanding is that it should be considered an abstract class and that developers are expected to use the provided CheckMessage subclasses.

Our full check thus reads:

/roundtable/models.py

@register('models') def check_model_str(app_configs=None, **kwargs): problem_models = [ model for model in camelot_apps.get_models() if 'str' not in model.dict ] errors = [ Warning( "All Models must have a str method.", hint=("See https://docs.djangoproject.com/" "en/1.7/ref/models/instances/#str" " for more information."), obj=model, id='rtable.W001', ) for model in problem_models ] return errors

Below, we test our new check on roundtable and immediately discover a problem.

$ ./manage.py check rtable System check identified some issues:

WARNINGS: auth.User: (rtable.W001) All Models must have a str method. HINT: Django uses str to represent models as strings. See https://docs.djangoproject.com/en/1.7/ref/models/instances/#strfor more information. sessions.Session: (rtable.W001) All Models must have a str method. HINT: Django uses str to represent models as strings. See https://docs.djangoproject.com/en/1.7/ref/models/instances/#strfor more information.

The warnings raised above are not the problem. We don't care that the User and Session models don't define the __str__() method. The problem is that our warning is being raised in other apps, even though we limited the scope of the system check to the roundtable app when we invoked the command.

The check command accepts a list of tags and a list of apps. When a tag is specified, the system check framework will limit the run to the check functions that have that tag. However, when an app is specified, the system check framework does not limit which functions are called. The list of applications affects the contents of the app_configs argument. Every check function must thus make use of the app_configs argument to ensure that it behaves appropriately.

The app_configs argument may be in one of two states. The value of this variable may be either:

  1. None
  2. A list of AppConfig objects

The second case is easier to deal with conceptually. Django will take the list of app labels specified by the developer when check is invoked and create a list of their AppConfig instances. This list is then passed to every check function (that has the correct tags, if specified).

However, if the developer invokes check with no arguments, then the value of app_configs is None. This is counterintuitive: if app_configs is None, the check must run on all project apps.

To ensure that our check does not run indiscriminately, we must ensure it runs when:

  1. app_configs is None
  2. The roundtable AppConfig object is in app_configs

In our code below, we first ensure we are getting our model list if and only if app_configs is in the correct state. We then use this new model list in the list comprehension we had already build.

/roundtable/models.py

def check_model_str(app_configs=None, **kwargs): roundtable_app = camelot_apps.get_app_config('rtable') if (app_configs is None or roundtable_app in app_configs): evaluated_models = roundtable_app.get_models() else: evaluated_models = [] problem_models = [ model for model in evaluated_models if 'str' not in model.dict ]

We are doing quite a bit of work simply to find our problem models. We begin by asking the app registry for the app config with the label 'rtable'. If this app is in app_configs or app_configs is None, we know we can proceed and ask the roundtable AppConfig for all of the models in the roundtable app. If our conditions are not met, we create an empty list to avoid a NameError. This allows us to refer to the model list in our list comprehension, computed as before.

Modifying our Check for all apps

It is not actually in our interest to limit the check to roundtable. If we add another app to camelot, we want this check to verify the status of models there, too. However, we still want to avoid the errors from the auth and sessions contributed apps. We thus modify the check to run on all apps except those from the contributed library, which we explicitly blacklist.

/roundtable/models.py

def check_model_str(app_configs=None, **kwargs): problem_models = [ model for app in (app_configs if app_configs else camelot_apps.get_app_configs()) if not app.name.startswith('django.contrib') for model in app.get_models() if 'str' not in model.dict ]

All of our logic is now in the list comprehension. We have two loops: one for apps and one for models. If app_configs is not None, the list comprehension uses that for the list of apps. Otherwise, the list comprehension uses all of the project apps thanks to a call to apps.get_app_configs(). Once we have the list, we ensure that the name attribute of the app does not begin with django.contrib, ensuring we avoid all of the contributed apps. Observe that this is more efficient than blacklisting a tuple of app labels. Selecting the appropriate apps allows our model loop to fetch all relevant models via app.get_models() and to find the ones without a __str__() method.

Note that we can achieve the same result above using the code below. The key difference is that our new list comprehension does everything in a single loop, introspecting the objects for their AppConfig information. The code below is how most of the checks included in Django are programmed. Basic tests with timeit show that this last method is 25% faster than the method above.

/roundtable/models.py

def check_model_str(app_configs=None, **kwargs): problem_models = [ model for model in camelot_apps.get_models() if (app_configs is None or model._meta.app_config in app_configs) and not model._meta.app_config.name.startswith( 'django.contrib') and 'str' not in model.dict ]

If we create a foo app with a Bar model without the __str__() method, we are greeted by the following output:

$ ./manage.py check roundtable System check identified no issues (0 silenced). $ ./manage.py check foo System check identified some issues:

WARNINGS: foo.Bar: (rtable.W001) All Models must have a str method. HINT: See https://docs.djangoproject.com/en/1.7/ref/models/instances/#strfor more information.

System check identified 1 issue (0 silenced). $ ./manage.py check System check identified some issues:

WARNINGS: foo.Bar: (rtable.W001) All Models must have a str method. HINT: See https://docs.djangoproject.com/en/1.7/ref/models/instances/#strfor more information.

System check identified 1 issue (0 silenced).

This is exactly the behavior we desire.

Consider that /roundtable/checks.py is not the correct location for our check and that the ID rtable.W001 is misleading. We can move the existing file to /camelot/checks.py and change the ID to camelot.W001. To ensure that register() correctly adds the check to the system check framework, we can add from . import checks in the /camelot/__init__.py file (and remove the line from /roundtable/__init__.py).

We could leave it at this, but there is one last improvement we can make. Consider that our call to the register() decorator is currently:

camelot/checks.py

@register('models') def check_model_str(app_configs=None, **kwargs):

This is prone to error, as we are manually entering the string 'model' for the check tag. This is very flexible, as it allows us to create new tags on the fly: passing 'model_attr' to register() would create a new tag. However, Python and editors will not complain if we enter 'mdole' or any other possible mistakes. As we are using a default Django tag, we can import the list of default tags with from django.core.checks import Tags and replace our string with Tags.models. This is less error prone as well as more compliant with DRY (Don't Repeat Yourself) philosophy.

camelot/checks.py

from django.core.checks import register, Tags, Warning ... @register(Tags.models) def check_model_str(app_configs=None, **kwargs):

In the event that we have the desire to create our own set of Tags, we could create our own Tags class by subclassing Django's own. This allows us to create a model_attrs tag while adhering to DRY.

camelot/checks.py

from django.core.checks import register, Warning from django.core.checks import Tags as DjangoTags ... class Tags(DjangoTags): model_attrs = 'model_attrs' ... @register(Tags.model_attrs) def check_model_str(app_configs=None, **kwargs):

Our final check code reads:

camelot/checks.py

-*- coding: utf-8

from future import unicode_literals from django.apps import apps as camelot_apps from django.core.checks import register, Warning from django.core.checks import Tags as DjangoTags

class Tags(DjangoTags): model_attrs = 'model_attrs'

@register(Tags.model_attrs) def check_model_str(app_configs=None, **kwargs): problem_models = [ model for model in camelot_apps.get_models() if (app_configs is None or model._meta.app_config in app_configs) and not model._meta.app_config.name.startswith( 'django.contrib') and 'str' not in model.dict ] errors = [ Warning( "All Models must have a str method.", hint=("See https://docs.djangoproject.com/" "en/1.7/ref/models/instances/#str" " for more information."), obj=model, id='camelot.W001', ) for model in problem_models ] return errors

Creating a Model Check

Checks need not be external functions. It is possible for models to have their own checks as well. Note that while function checks may be called anything (thanks to register()), model checks must be called check() and must be class methods.

roundtable/models.py

class Knight(models.Model): ... @classmethod def check(cls, **kwargs): errors = super(Knight, cls).check(**kwargs) return errors

Just as with function checks, model checks return a list containing subclasses of CheckMessage. Below, our check returns an Error if the Knight model loses its traitor field.

roundtable/models.py

from django.core.checks import Error ... class Knight(models.Model): ... @classmethod def check(cls, **kwargs): errors = super(Knight, cls).check(**kwargs) if 'traitor' not in cls._meta.get_all_field_names(): errors.append( Error( "Knight model must have a traitor field.", obj=cls, id='RT_K.E001')) return errors

Typically, the check() method will just call other model methods, allowing for each check to be logically separate. The basic structure for this is demonstrated below, without any actual check logic.

roundtable/models.py

class Knight(models.Model): ... @classmethod def check(cls, **kwargs): errors = super(Knight, cls).check(**kwargs) errors.extend(cls._check_traitor_field(**kwargs)) return errors

@classmethod
def _check_traitor_field(cls, **kwargs):
    return []

We can re-implement our traitor field check in _check_traitor_field(), as demonstrated below. Our check() method remains very simple.

roundtable/models.py

class Knight(models.Model): ... @classmethod def check(cls, **kwargs): errors = super(Knight, cls).check(**kwargs) errors.extend(cls._check_traitor_field(**kwargs)) return errors

@classmethod
def _check_traitor_field(cls, **kwargs):
    if 'traitor' not in cls._meta.get_all_field_names():
        return [
            Error(
                "Knight model must have a traitor field.",
                obj=cls,
                id='RT_K.E001')]
    else:
        return []

System Check Framework in a Nutshell

The system check framework is a set of extensible functions that verify the state of your Django project. The framework may be directly invoked using $ ./manage.py check but will also be implicitly run during most manage.py commands.

When invoked with the purpose of checking the project, the check command accepts two argument types: tags and apps. Tags will limit which check functions are run. Apps will change the app_configs argument passed to each check function.

When creating an independent check function, the check function must make use of the app_configs argument to adhere to correct behavior and must return a list of errors, where each error is a subclass of CheckMessage (an error may also be an instance of CheckMessage itself).

A model check is similar in that it must return a list of CheckMessage subclasses but does not require an app_configs argument. Model checks must be named check and must be class methods.

The system check framework is an important addition to Django as it allows developers to add useful checks to their test suite. Used judiciously, checks are a fantastic tool to avoid errors and will be useful in the future during the upgrade process.

Additional Features

While migrations, the app registry, and the system check framework are the largest new Django features, two more are worthy of honorable mentions for their incredible utility.

Custom QuerySet Objects

For this section, let's add a new BooleanField to our Knight model.

roundtable/models.py

class Knight(models.Model): name = models.CharField(max_length=63) traitor = models.BooleanField(default=False) dances = models.BooleanField(default=True)

To make our life easier, we may wish to create a shortcut for searching for knights who are loyal and who dance. In Django 1.6, the way to do this would have been to override the model Manager and create two new methods as below. We'd also make sure to override the default manager with this new one by explicitly setting the objects attribute in our model class.

roundtable/models.py

class KnightManager(models.Manager): def dances_when_able(self): return self.get_queryset().filter(dances=True)

def loyal(self):
    return self.get_queryset().filter(traitor=False)

@python_2_unicode_compatible class Knight(models.Model): ... objects = KnightManager()

As expected, we'd then be able to easily ask for knights who are loyal and knights who dance.

from roundtable.models import Knight Knight.objects.loyal() [<Knight: Bedevere>, <Knight: Bors>, <Knight: Ector>, <Knight: Galahad>, <Knight: Gawain>, <Knight: Robin>] Knight.objects.dances_when_able() [<Knight: Bedevere>, <Knight: Galahad>, <Knight: Gawain>, <Knight: Lancelot>]

The problem with this method is that we cannot ask for both. If we attempt to chain the commands, we are greeted by an error.

Knight.objects.loyal().dances_when_able() Traceback (most recent call last): File "", line 1, in AttributeError: 'QuerySet' object has no attribute 'dances_when_able'

This occurs because Manager objects return QuerySet objects and we have not created any new methods in the latter. In Django 1.6, we would then have to override our QuerySet objects, which was trickier than expected.

Starting in Django 1.7, the recommended way of achieving our goal is not to create a custom Manager, but instead to customize a QuerySet. The QuerySet class now comes with an as_manager() method, allowing for developers to subclass QuerySet but also to provide these methods to the model's Manager.

roundtable/models.py

class KnightQuerySet(models.QuerySet): def dances_when_able(self): return self.filter(dances=True)

def loyal(self):
    return self.filter(traitor=False)

@python_2_unicode_compatible class Knight(models.Model): ... objects = KnightQuerySet.as_manager()

This method effectively allows us to chain our new methods with very little work.

Knight.objects.loyal().dances_when_able() [<Knight: Bedevere>, <Knight: Galahad>, <Knight: Gawain>]

Prefetch Objects

The code in this section is unfortunately not available in the Github repository, due to the number of changes required to make the code work.

Django supplies two ways to optimize the number of queries made to the database. The select_related() method allows for one-to-one and many-to-one (forward foreign key) relationships to be made in the same operation. The prefetch_related() method will fetch the data of two models with one-to-many (reverse foreign key) or many-to-many relationships and combine the two in Python.

The problem with prefetch_related() prior to Django 1.7 was that it was limited in its options. Starting in Django 1.7, this is no longer the case.

Imagine that our Knight models has a many-to-many relationship with a Quest model, which is turn has a one-to-many relationship with QuestGivers (multiple individuals may give a single quest). In Django 1.6, we would optimize the number of database queries made with the following code:

Knight.objects.all( ).prefetch_related( 'assigned_quests', )

The issue is that we do not have a simple way to fetch the next relation in our relationship chain.

With the Prefetch object, this is no longer true. At its most basic, the object does exactly the same as above.

Knight.objects.all( ).prefetch_related( Prefetch( 'assigned_quests', ), )

However, the Prefetch object allows for two additional parameters: a queryset and to_attr.

Setting the queryset parameter allows us, for instance, to connect our single operation to the QuestGiver model.

Knight.objects.all( ).prefetch_related( Prefetch( 'assigned_quests', queryset=Quest.objects.all( ).select_related( 'quest_giver').order_by('pk'), ), )

We can further set the to_attr parameter in our call. Without this, the results of the prefetch operation are cached in a QuerySet related to the Knight manager, which results in some overhead in Python. With the to_attr argument set to 'prefetch_quests', Django will assign the Quest objects as a list to an attribute named prefetch_quests on the Knight objects. Instead of calling {% for quest in knight.assigned_quests.all %} in the template, our code would now write {% for quest in knight.prefetch_quests %}. Using to_attr may "provide a significant speed improvement" because of this.

There is a small catch to using the to_attr parameter. Imagine that our Quest and QuestGiver models are now connected via a many-to-many relationship. We wish to fetch all of the Knight, Quest, and QuestGiver data in as efficient a manner as possible, and code the following, omitting the to_attr parameter.

Knight.objects.all( ).prefetch_related( Prefetch( 'assigned_quests', ), Prefetch( 'assigned_quests__quest_giver', ), )

The command above results in three queries, which is as good as it gets on that front. However, we also wish to avoid the Python overhead created by caching these results in QuerySet objects, and thus we add the to_attr to the code above.

Knight.objects.all( ).prefetch_related( Prefetch( 'assigned_quests', to_attr='prefetch_quests', ), Prefetch( 'assigned_quests__quest_giver', to_attr='prefetch_quest_giver', ), )

This code above results in four database queries because the Quest models are being fetched twice. The first prefetch query will fetch the Quest objects and then set them as a list in knight.prefetch_quests. When the second prefetch query occurs, it will no longer have access to the cached results of the first query and will thus fetch the data again.

We can avoid all of this by replacing assigned_quests with the to_attr argument prefetch_quests in the second Prefech object.

Knight.objects.all( ).prefetch_related( Prefetch( 'assigned_quests', to_attr='prefetch_quests', ), Prefetch( 'prefetch_quests__quest_giver', to_attr='prefetch_quest_giver', ), )

Our call now results in three database queries and minimizes memory usage in Python.

This section is but a very short treatment on what could be an entire talk/article unto itself. If you are interested in such an article, please let me know, and I will write it.

Conclusion

Django 1.7 is shaping up to be the biggest Django release since 1.0.

Migrations, App Registry, and Checks, oh my! Django 1.7 is an amazing new version. Our treatment of the five new features seen in this article is but the tip of the iceberg, with many new additions and deprecations. There are new custom lookup objects, better form error handling, and more.

We don't have time to go through and demonstrate the use of each new feature, as much fun as it would be. However, in Part IV we will discuss how to stay on top of all these changes and how to use Django and its documentation to make upgrading major releases as easy as possible.

Read Part IV: Upgrade Strategies