Welcome to my Blog!

Building a Conveyances system with Django, Django Rest Framework, and Vue Part 1

A couple of months back I came across a template containing stages for conveyance matters for a legal firm. The legal firm wanted to track progress for the matters it was handling for its clients, who are mostly banks. There are five types of matters and each has a different number of stages. Some stages are common to all while some are common to a few and others are particular to a single type of matter. A conveyance matter between a bank and an individual could have one or more matters.

Making an app for tracking these matters seemed to be a daunting task since I needed to figure out how best to deal with the different data involved. I thought to myself, that this would be the best kind of scenario to build a Django API with Django Rest Framework that would be consumed with a Vue JS frontend.

Read on to find out how I tackled this fun project.

Please note that this is a three-part series of posts. The article is divided into the following:

  • Part 1 - Backend Architecture
  • Part 2 - Frontend Architecture
  • Part 3 - Testing the Backend

Architecture

The app was developed using:

  • Python 3.9+
  • Django 4.0+
  • Django Rest Framework
  • Postgres 12
  • Vue 3
  • Node 16.15.0
  • npm 8.5.5
  • Webpack
  • Django Webpack Loader
  • Webpack Bundle Tracker

Test-Driven Development & Continuous Integration (CI)

When I started learning Python and Django, testing was difficult for me. At PyCon Namibia 2017, I ran a workshop on Testing with Django, having been introduced to it by Anna Balica in her talk at Django Under The Hood 2016. I still considered myself a beginner when it came to Python and Django, let alone testing. I missed the end of the conference but before I left, a friend of mine from Namibia, Gabriel-Tuhafeni Nhinda, looked for me to give me a book he had been asked by Harry Percival to pass on to me.

Harry had looked for me to give me a signed copy of his Test-Driven Development with Python but I had gone out. We later bumped into him and his family at the airport on our way back to Zimbabwe and I thanked him for the gift. I was so grateful and I still am very grateful, many years later. I haven’t finished the book cover to cover yet but I still refer to it a lot. Since I am reading a book he co-authored with Bob Gregory these days which also uses the TDD approach to architecture patterns with Python, it only makes sense that TDD is at the core of my latest projects.

For this project, I created the initial project and added continuous integration with GitHub Actions, CodeCov, and code formatting with black. Once everything was working, I created another branch, add-backend, which I developed the backend using TDD. After I thought I was done with the backend and all tests were passing, I created another branch, frontend, and started working on the frontend code.

As it turned out, I was wrong about the backend. I was wrong about the structure of the Stage class and BaseMatter dataclass and I was certainly wrong about the structure of ConveyanceMatter and Matter serializers. I was able to create Matter objects using stages.createcreate_conveyance_object function but because the Stage and BaseMatter classes had issues, it was a garbage-in garbage-out (GIGO) scenario.

This also affected the views, but I also managed to figure out what I needed for the frontend while working on the frontend. My frontend needs became clearer while working on the frontend code. This made me realize how important it is for backend engineers to be able to work well with frontend engineers so they can be able to meet the needs of frontend engineers. I fixed all these issues without fixing tests at first until I could consume the backend from the Vue frontend. After the frontend was working, I then went back and refactored my backend tests and code where necessary to get everything to pass again.

However, for now, TDD was used on the backend and not frontend but I do hope to add tests for the frontend later.

Backend

The backend has two apps, the accounts app for managing user authentication and the matters app for managing conveyances matters. The conveyances project also contains the project settings.py and root urls.py. Since I will need to render a Vue frontend with a Django project, I needed to create a templates folder with two files, base.html and index.html to render the app.

I am not much of a frontend person so I used Startbootstrap SB Admin 2 template for my frontend. The CSS and JS files are loaded as static files in the templates/base.html. The templates/index.html renders the Vue JS app. Django Webpack Loader and Webpack Bundle Tracker handle tracking of changes within the Vue frontend app without the need to reload the Django webserver (enabling Hot Module Replacement for Django server). The code for this file is shown below.

templates/index.html

{% extends "base.html" %}
{% load render_bundle from webpack_loader %}

{% block content %}
    <div class="container-fluid">
        <noscript>
            <strong>We're sorry but <%= htmlWebpackPlugin.options.title %> doesn't work properly without JavaScript enabled.
                Please enable it to continue.</strong>
        </noscript>

        <div id="app">
            <!-- built files will be auto injected -->
            {% render_bundle 'app' 'js' %}
        </div>
    </div>
{% endblock %}

Accounts App

For this project, not much work was done on user authentication. I decided to use the Django default authentication system. I only needed to define the User model in accounts.models.py which extends the django.contrib.auth.models.AbstractUser and inherit everything from it. I also defined a LoginForm in accounts.forms which extends django.contrib.auth.forms.AuthenticationForm.

I also only needed two URLs, for login and log out from django.contrib.auth.views. I added these to accounts/urls.py shown below. After adding urls.py, I added the app to the INSTALLED_APPS in conveyances/settings.py and included accounts.urls in conveyances/urls.py and I had a working app.

accounts/urls.py

from django.contrib.auth import views as auth_views
from django.urls import path

from .forms import LoginForm

app_name = "accounts"


urlpatterns = [
    path(
        "login/",
        auth_views.LoginView.as_view(
            template_name="accounts/login.html",
            form_class=LoginForm,
            extra_context={"title": "Login"},
        ),
        name="login",
    ),
    path("logout/", auth_views.logout_then_login, name="logout"),
]

The auth_views.logout_then_login logs out the user and redirects them to the login page again. How cool is that!

Matters App

I needed a way to be able to have a few models, one for storing Conveyance Matters which could have one or two or more matters. The matters themselves needed to be in one model despite having different fields in their data, due to having a different number of stages which are also in a different order. Creating a base class and using inheritance wouldn't have worked so I opted for Python dataclasses.dataclass (thanks to Google Tech Learning series for knowledge on how to use dataclasses for different data structures) which does not need a init() method and therefore can be created with a varying number of fields. This worked for me in matters/stages.py shown below.

matters/stages.py

from dataclasses import dataclass
from typing import Iterable, List, Text

# I left out the transfer, mortgage_bond, lost_deed_application,
# mortgage_bond_other_lawyers and mortgage_bond_cancellation
# lists used in the matters dict. You can checkout the full file in the repo.

matters = {
    "transfer": {"name": "Transfer", "stages": transfer},
    "mortgage_bond": {"name": "Mortgage Bond", "stages": mortgage_bond},
    "mortgage_bond_other_lawyers": {
        "name": "Mortgage Bond with Other Lawyers Transfering",
        "stages": mortgage_bond_other_lawyers,
    },
    "lost_deed_application": {
        "name": "Lost Deed Application",
        "stages": lost_deed_application,
    },
    " mortgage_bond_cancellation": {
        "name": "Mortgage Bond Cancellation",
        "stages": mortgage_bond_cancellation,
    },
}


class Stage:
    def __init__(self, step, stage):
        self.stage = stage
        self.step = step
        self.comment: Text = None
        self.done: bool = False


@dataclass
class BaseMatter:
    name: Text
    stages: List


def create_conveyance_object(name: str, stages: List) -> Iterable[BaseMatter]:
    matter_stages = []
    for i, stage in enumerate(stages):
        matter_stages.append(vars(Stage(i + 1, stage)))

    convenyance_matter = BaseMatter(name, matter_stages)
    return convenyance_matter

Models

For the matters app, I needed three models, Bank, Matter and ConveyanceMatter. Bank is a ForeignKey field for the ConveyanceMatter model, which makes it easy to filter for all matters for one bank (the banks are the clients for the law firm). Matter has to be a ManyToManyField since one ConveyanceMatter can be linked to more than Matter object.

I also used a uuid field for all the models as the index field so as not to expose the id of a record to the public. The matters field of the Matter model had to be a JSONField to allow partial edits to the field. This worked out later when I used nested serializers with Django Rest Framework to edit this field from the ConveyanceModelSerializer.

The complete matters/models.py is shown below:

matters/models.py

import uuid as uuid_lib
from django.db import models


class Bank(models.Model):
    uuid = models.UUIDField(db_index=True, default=uuid_lib.uuid4, editable=False)
    name = models.CharField(max_length=255)

    def __str__(self):
        return self.name


class Matter(models.Model):
    uuid = models.UUIDField(db_index=True, default=uuid_lib.uuid4, editable=False)
    name = models.CharField(max_length=255, unique=True)
    stages = models.JSONField()
    created_at = models.DateTimeField(auto_now_add=True)
    last_updated = models.DateTimeField(auto_now=True)

    def __str__(self):
        return self.name


class ConveyanceMatter(models.Model):
    uuid = models.UUIDField(db_index=True, default=uuid_lib.uuid4, editable=False)
    title = models.CharField(max_length=255)
    matters = models.ManyToManyField("Matter")
    created_at = models.DateTimeField(auto_now_add=True)
    last_updated = models.DateTimeField(auto_now=True)
    created_by = models.ForeignKey("accounts.User", on_delete=models.CASCADE)
    bank = models.ForeignKey(Bank, on_delete=models.CASCADE)
    complete = models.BooleanField(default=False)
    comment = models.TextField(blank=True, null=True)

    def __str__(self):
        return self.title

Serializers

Since matters field is a ManyToManyField for ConveyanceMatter and it is the one field that requires constant updates to the Matter object, this introduced the complexity of working with nested serializers. I needed to overwrite the create() and update() methods of the ConveyanceMatter object.

The create() method was easy enough but the update() method was complicated such that after a few trials and errors I went back to the Django Rest Framework documentation on nested serializers which recommended using the drf_writable_nested third party package, which is the route I ended up taking to make the partial edits and creating Matter objects from the ConveyanceMatterSerializer.

It also turned out that I needed a UserSerializer after all so as to be able to display the name of the currently logged in user in the frontend so I added the serializer below the matters app serializers instead of creating a serializers.py in accounts app.

matters/serializers.py

from rest_framework import serializers
from drf_writable_nested.mixins import UniqueFieldsMixin
from drf_writable_nested.serializers import WritableNestedModelSerializer

from accounts.models import User
from .models import Bank, ConveyanceMatter, Matter


class BankSerializer(serializers.ModelSerializer):
    class Meta:
        model = Bank
        fields = ("uuid", "id", "name")
        read_only_fields = ("uuid", "id")


class MatterSerializer(UniqueFieldsMixin, serializers.ModelSerializer):
    class Meta:
        model = Matter
        fields = ("uuid", "pk", "name", "stages", "created_at", "last_updated")
        read_only_fields = ("created_at", "uuid", "last_updated")
        extra_kwargs = {
            "id": {"read_only": False},
        }


class ConveyanceMatterSerializer(WritableNestedModelSerializer):
    created_by = serializers.StringRelatedField()
    bank = serializers.SlugRelatedField(
        queryset=Bank.objects.all(), read_only=False, slug_field="name"
    )
    matters = MatterSerializer(many=True)

    class Meta:
        model = ConveyanceMatter
        fields = (
            "uuid",
            "id",
            "title",
            "matters",
            "created_at",
            "last_updated",
            "created_by",
            "bank",
            "complete",
            "comment",
        )
        read_only_fields = ("created_at", "last_updated", "uuid", "pk")


class BaseMatterSerializer(serializers.Serializer):
    name = serializers.CharField(max_length=255)
    stages = serializers.JSONField()


class UserSerializer(serializers.ModelSerializer):
    class Meta:
        model = User
        queryset = User.objects.all()
        fields = ("id", "username", "first_name", "last_name")
        read_only_fields = ("id",)

Pagination

I added a little customization to the DRF pagination so I could be able to use the pagination on the frontend in matters/pagination.py as shown below:

matters/pagination.py

from collections import OrderedDict

from rest_framework import pagination
from rest_framework.response import Response


class PageNumberPagination(pagination.PageNumberPagination):

    page_size = 10

    def get_paginated_response(self, data):
        return Response(
            OrderedDict(
                [
                    ("count", self.page.paginator.count),
                    ("countItemsOnPage", self.page_size),
                    ("current", self.page.number),
                    ("next", self.get_next_link()),
                    ("previous", self.get_previous_link()),
                    ("total_pages", self.page.paginator.num_pages),
                    ("results", data),
                ]
            )
        )

Views

For the views, the CurrentUserView and BaseMatterView had to be APIViews since they are static views, one for getting the current user and the other for generating empty matter objects for the user to use in creating matter objects. The BankViewSet, MattersViewSet and ConveyanceMatterViewSet had to be viewsets so as to make the creation, update and viewing of the objects easier.

matters/views.py

from rest_framework import mixins, views, viewsets
from rest_framework.permissions import IsAuthenticated
from rest_framework.response import Response

from .models import Bank, ConveyanceMatter, Matter
from .serializers import (
    BankSerializer,
    ConveyanceMatterSerializer,
    BaseMatterSerializer,
    MatterSerializer,
    UserSerializer,
)
from .stages import create_conveyance_object, matters


class BankViewSet(
    viewsets.GenericViewSet,
    mixins.CreateModelMixin,
    mixins.ListModelMixin,
    mixins.RetrieveModelMixin,
    mixins.UpdateModelMixin,
):
    permission_classes = (IsAuthenticated,)
    queryset = Bank.objects.all().order_by("id")
    serializer_class = BankSerializer
    lookup_field = "uuid"

    def get_queryset(self):
        name = self.request.query_params.get("name")
        queryset = self.queryset

        if name:
            return queryset.filter(name__icontains=name).order_by("id")
        return queryset


class ConveyanceMatterViewSet(
    viewsets.GenericViewSet,
    mixins.CreateModelMixin,
    mixins.ListModelMixin,
    mixins.RetrieveModelMixin,
    mixins.UpdateModelMixin,
):
    queryset = ConveyanceMatter.objects.all().order_by("-created_at")
    serializer_class = ConveyanceMatterSerializer
    permission_classes = (IsAuthenticated,)
    lookup_field = "uuid"

    def get_queryset(self):
        bank = self.request.query_params.get("bank")
        title = self.request.query_params.get("title")
        queryset = self.queryset

        if bank:
            return queryset.filter(bank__name__icontains=bank).order_by("-created_at")
        if title:
            return queryset.filter(title__icontains=title).order_by("-created_at")
        return queryset

    def perform_create(self, serializer):
        serializer.save(created_by=self.request.user)


class BaseMatterView(views.APIView):
    permission_classes = (IsAuthenticated,)

    def get(self, request):
        conveyance_matters = []
        for key in matters.keys():
            conveyance_matter = create_conveyance_object(
                matters[key]["name"], matters[key]["stages"]
            )
            conveyance_matters.append(conveyance_matter)
        results = BaseMatterSerializer(conveyance_matters, many=True).data
        return Response(results)


class MatterViewSet(
    viewsets.GenericViewSet,
    mixins.CreateModelMixin,
    mixins.ListModelMixin,
    mixins.RetrieveModelMixin,
    mixins.UpdateModelMixin,
):
    queryset = Matter.objects.all().order_by("-id")
    permission_classes = (IsAuthenticated,)
    serializer_class = MatterSerializer
    lookup_field = "uuid"


class CurrentUserView(views.APIView):
    def get(self, request, format=None):
        current_user = self.request.user
        results = UserSerializer(current_user).data
        return Response(results)

URLs

The viewsets are handled by rest_framework.routers.DefaultRouter while the APIViews are handled as class based views. The matters/urls.py is shown below:

matters/urls.py

from django.urls import include, path

from rest_framework.routers import DefaultRouter

from . import views

app_name = "matters"

router = DefaultRouter()
router.register("conveyance_matters", views.ConveyanceMatterViewSet)
router.register("banks", views.BankViewSet)
router.register("matters", views.MatterViewSet)

urlpatterns = [
    path("", include(router.urls)),
    path("base_matters/", views.BaseMatterView.as_view(), name="base_matters"),
    path("current_user/", views.CurrentUserView.as_view(), name="current_user"),
]

Conclusion

This concludes the backend architecture part. Read the next post to see how I handled the frontend and integration to Django.

Read More →

Building an XML file convertor project for DMARC records using Django

A couple of months ago, we received a bounty email warning us about our Django Girls domain lacking a DMARC policy. This meant that anyone could send phishing emails using our domain which is not good. We didn’t react as quickly as we should have and only took this matter seriously when we started receiving those phishing emails from hello@djangogirls.org in our own @djangogirls.org mailboxes!

We quickly published a DMARC policy and started receiving many DMARC reports in XML format. XML is not really human-readable therefore we could not read or interpret the DMARC records being sent to us. Surfing the internet led me to an online service for analyzing DMARC records that involved signing up for an account, a cost we did not plan for as Django Girls. However, we still needed a way to read the DMARC files and I was also looking for fun projects to work on for my GitHub profile so I decided, why not kill two birds with one stone. The source code for this project is available here.

Architecture

In my spare time, I have been reading Architecture Patterns with Python by Harry J.W. Percival and Bob Gregory which has made me aware of the Domain-Driven Design (DDD) methodology and also pushed me to push myself to implement Test-Driven Development (TDD) for all my new projects. Being able to write unit tests for my code before (and sometimes) after I have written it has shown me how well I understand the code I am writing.

The app was developed using: * Python 3.9+ * Django 4.0+ * Postgres 12 * xmltodict for parsing xml.

Apps

The project is as simple as a Django project can be as it has only one app, xmlreader. It uses the Django ORM and Django’s MVC pattern. The app has been added to the settings.py file in the INSTALLED_APPS section as shown below.

Snippets of settings.py

INSTALLED_APPS += [
        "xmlreader",
]

I also changed the DATABASES setting to use Postgres engine and also provide database settings for GitHub Actions CI as shown below.

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql",
        "NAME": os.environ.get("DATABASE_NAME"),
        "USER": os.environ.get("DATABASE_USER"),
        "PASSWORD": os.environ.get("DATABASE_PASSWORD"),
        "HOST": os.environ.get("DATABASE_HOST"),
        "PORT": os.environ.get("DATABASE_PORT"),
    }
}

if os.environ.get("GITHUB_WORKFLOW"):
    DATABASES = {
        "default": {
            "ENGINE": "django.db.backends.postgresql",
            "NAME": "github_actions",
            "USER": "postgres",
            "PASSWORD": "postgres",
            "HOST": "127.0.0.1",
            "PORT": "5432",
        }
    }

And I also prefer to put my templates in one place so I also changed that setting.

TEMPLATES = [
    {
        "BACKEND": "django.template.backends.django.DjangoTemplates",
        "DIRS": [os.path.join(BASE_DIR, "templates")],
        "APP_DIRS": True,
        "OPTIONS": {
            "context_processors": [
                "django.template.context_processors.debug",
                "django.template.context_processors.request",
                "django.contrib.auth.context_processors.auth",
                "django.contrib.messages.context_processors.messages",
            ],
        },
    },
]

Then just as an example of how to configure the other environment variables, I just use the format I have adopted from working with the Django Girls repo for DEBUG mode as shown below.

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = os.environ.get("SECRET_KEY")

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = os.environ.get("DEBUG") != "FALSE"

if DEBUG:
    SECRET_KEY = "hello"

ALLOWED_HOSTS = []

if not DEBUG:
    ALLOWED_HOSTS += [os.environ.get("ALLOWED_HOSTS")]

Last but not least, I added some STATIC files settings and that’s all I needed for this to work.

STATIC_ROOT = os.path.join(BASE_DIR, "staticfiles")
STATICFILES_DIRS = [os.path.join(BASE_DIR, "static")]

Models

The app has only one model, Upload for storing the XML files. The model definition is shown below:

models.py

from django.db import models


class Upload(models.Model):
    file = models.FileField(upload_to="xml_files")
    date_uploaded = models.DateTimeField(auto_now_add=True)

    def __str__(self):
        return f"{self.file}"

Forms

The app has one form with only one field for taking the file path of the XML file, which extends the Django ModelForm. The form definition is shown below:

forms.py

from django import forms
from django.utils.translation import gettext_lazy as _

from .models import Upload


class UploadFileForm(forms.ModelForm):
    class Meta:
        model = Upload
        fields = ["file"]

    file = forms.FileField(label=_("Upload File"), allow_empty_file=True)

Views

The project has three views, an index view which renders the homepage and allows users to upload files as well as list all files uploaded, a static about which renders the page which describes the app, and a results view that reads the XML file and convert it to a dict and renders the contents of the dict.

The code for these views is shown below:

views.py

import xmltodict

from django.shortcuts import get_object_or_404, redirect, render
from django.urls import reverse

from .forms import UploadFileForm
from .models import Upload


def index(request):
    title = "Home"
    files = Upload.objects.all()
    if request.method == "POST":
        form = UploadFileForm(request.POST, request.FILES)
        if form.is_valid():
            file = form.save()
            return redirect(reverse("xmlreader:results", args=(file.pk,)))
    else:
        form = UploadFileForm()
    return render(
        request, "xmlreader/index.html", {"title": title, "form": form, "files": files}
    )


def about(request):
    title = "About"
    return render(request, "xmlreader/about.html", {"title": title})


def results(request, id):
    title = "Decoded File"
    file = get_object_or_404(Upload, id=id)

    with file.file.open(mode="rb") as xml_file:
        data_dict = xmltodict.parse(xml_file.read())

    return render(
        request, "xmlreader/results.html", {"title": title, "data_dict": data_dict}
    )

Templates

The templates/xmlreader folder has four templates, the base.html, about.html, index.html and the results.html. I will not discuss the code for the about.html and base.html files since they are both static, you can view them in the repo. I will however talk a bit about the index.html and results.html.

The index.html contains a form that has only one field, the upload file field. However, because I want to upload both the file contents and its name, I have to specify enctype="multipart/form-data" for the field to work. I also list the files which already have been uploaded with a link for each should one want to view them again below the form. The results.html presents the results of the decoded file. I had fun trying to get through the nested dictionaries.

Controllers/URLs

The xmlreader/urls.py has only three URLs and it has been included in the xml_reader/urls.py using the django.urls.include and that’s all I needed for the app to work.

Managing development environment

Pip-tools

I used pip-tools to manage the packages required by the project in the virtualenv. The requirements.in contains a list of required packages which are compiled to the requirements.txt file by running the command pip-compile. Installing the packages is done by running the standard pip install -r requirements.txt. I was introduced to pip-tools by one of our Django Girls contributors, Mark Walker, when we upgraded the Django Girls website from Django 2.0 to 2.2 and finally 3.2 and I have been hooked to it ever since.

Python-dotenv

It is best practice to have project secrets out of the settings.py. For this, I use python-dotenv to manage environment variables for my development environment and testing. pytest-dotenv makes it possible to specify the env files when running tests locally. This is another package I started working with after working with Mark Walker on upgrading the Django Girls website.

An example of a .env file is shown below:

export DJANGO_SETTINGS_MODULE="xml_reader.settings"
export DJANGO_DEBUG="FALSE"
export SECRET_KEY=""
export ALLOWED_HOSTS=[]
export COVERAGE="TRUE"
export DATABASE_NAME="dmarcs"
export DATABASE_USER=""
export DATABASE_PASSWORD=""
export DATABASE_HOST="127.0.0.1"
export DATABASE_PORT="5432"

Code formatting

I used black for code formatting instead of Flake8 since Django 4.0 + comes with all Python code already formatted by black. The other reason is having used Flake8 on Django Girls website, I noticed you can only configure your repo to give you Flake8 warnings on commit but still have to fix the Flake8 issues manually. However, black will format all your Python code automatically for you by running the command below on your local repo.

black .

Testing

I have been using pytest a lot in my project since I became so familiar working with it on the Django Girls website so it's no surprise I used pytest for unit testing this project. I coupled it with pytest-django to enable easy access to the test database, and pytest-dotenv for setting test environment variables. I also added coverage for coverage reports.

To run tests, I use the following command:

coverage run -m pytest

Continuous Integration

For the CI, I used GitHub Actions and set up three YAML files, one for Django which runs tests, and another for black which checks code formatting, and one for coverage which runs tests and upload the results to Codecov.

Final Outcome

I uploaded the XML files to the project on my localhost and managed to print PDF files of the DMARC reports we received. From the converted reports, it was clear that there were three domains that were failing DIKM and therefore would be rejected. These were gappssmtp.com (for Google, which we use for our email), mailchimp.com (our newsletter signups), and sendgrid.net (which we use to send emails from our website).

I had never configured DIKM and SPF records before so Claire reached out to her social network and we managed to get three awesome people to help us with our DMARC issues. We followed their recommendations and managed to get our automated emails working again securely and we haven’t received any more phishing emails from the hello@djangogirls.org email address!

Feel free to try and replicate this project yourself or clone my repo and play around with it.

Read More →

Building an Activity Generator with Django and Django Rest Framework

This is a REST API I developed for fun using Django and Django Rest Framework. The API allows a user to save activities they like to do when they are bored. When the user is bored, they can send a query with either name or nature in query parameters. The full source code for this project is available here.

Architecture

The REST API was developed using:

  • Python 3.9+,
  • Django 4.0+,
  • Django REST Framework, and
  • Postgres 12.

Apps

The project has two apps, an accounts app to manage user authentication and an activities app for managing activities. As is usual practice for Django, I added these apps to the INSTALLED_APPS setting in settings.py together with rest_framework and rest_framework.authtoken.

activity_generator/settings.py

INSTALLED_APPS += [
    "rest_framework",
    "rest_framework.authtoken",
    "accounts",
    "activities",
]

Project urls.py

I also included the URLs of the two apps in the project urls.py, that is in activity_generator/urls.py.

urlpatterns += [
    path("accounts/", include("accounts.urls", namespace="accounts")),
    path("", include("activities.urls", namespace="activities")),
]

Accounts App

The accounts app handles authentication.

Models

Models for accounts app are defined in accounts/models.py. It has a UserManager class which extends the django.contrib.auth.models.BaseUserManager class and has custom create_user() and create_superuser() methods which take email, first_name, last_name and password as parameters. It also has a User model that extends the django.contrib.auth.models.BaseUser and django.contrib.auth.models.PermissionsMixin classes. It also uses email as the username field thereby overriding the Django default username field, and makes first_name and last_name required fields.

accounts/models.py

from django.db import models
from django.contrib.auth.models import (
    AbstractBaseUser,
    BaseUserManager,
    PermissionsMixin,
)
from django.forms import ValidationError


class UserManager(BaseUserManager):
    """Base manager for user model."""

    def create_user(self, email, first_name, last_name, password):
        """Create a new user."""
        if not email:
            raise ValidationError("A user must have an email address.")
        email = self.normalize_email(email)
        user = self.model(email=email, first_name=first_name, last_name=last_name)
        user.set_password(password)
        user.save(using=self._db)
        return user

    def create_superuser(self, email, first_name, last_name, password):
        """Create a new superuser."""
        user = self.create_user(email, first_name, last_name, password)
        user.is_superuser = True
        user.is_staff = True
        user.save(using=self._db)
        return user


class User(AbstractBaseUser, PermissionsMixin):
    email = models.EmailField(max_length=100, unique=True)
    first_name = models.CharField(max_length=100)
    last_name = models.CharField(max_length=100)
    is_active = models.BooleanField(default=True)
    is_staff = models.BooleanField(default=False)

    objects = UserManager()

    USERNAME_FIELD = "email"
    REQUIRED_FIELDS = ["first_name", "last_name"]

    def get_full_name(self):
        return f"{self.first_name} {self.last_name}"

    def get_short_name(self):
        return self.first_name

    def __str__(self):
        return self.email

Serializers

The accounts app has two serializers, UserSerializer and AuthTokenSerializer. The UserSerializer extends rest_framework.serializers.ModelSerializer while the AuthTokenSerializer extends rest_framework.serializers.Serializer. For the UserSerializer, I override the create() method to create a new user by calling the User.objects.create_user() method. For the AuthTokenSerializer, I also override the validate() method for AuthTokenSerializer to handle user authentication and provide users with tokens.

accounts/serializers.py

from django.contrib.auth import authenticate, get_user_model
from django.utils.translation import gettext_lazy as _

from rest_framework import serializers


class UserSerializer(serializers.ModelSerializer):
    """Serializer for the user object."""

    class Meta:
        model = get_user_model()
        fields = ("email", "first_name", "last_name", "password")
        extra_kwargs = {"password": {"write_only": True, "min_length": 8}}

    def create(self, validated_data):
        """Create a new user with encrypted password and return the user."""
        return get_user_model().objects.create_user(**validated_data)


class AuthTokenSerializer(serializers.Serializer):
    """Serializer for user authentication object."""

    email = serializers.CharField()
    password = serializers.CharField(
        style={"input_type": "password"}, trim_whitespace=True
    )

    def validate(self, attrs):
        """Validate and authenticate the user."""
        email = attrs.get("email")
        password = attrs.get("password")

        user = authenticate(
            request=self.context.get("request"), username=email, password=password
        )
        if not user:
            msg = _("Unable to authenticate with provided details.")
            raise serializers.ValidationError(msg, code="authentication")
        attrs["user"] = user
        return attrs

Views

The accounts app has three views; one for creating the user, one for creating an authentication token for the user and another for managing the user.

accounts/views.py

from rest_framework import authentication, generics, permissions
from rest_framework.authtoken.views import ObtainAuthToken
from rest_framework.settings import api_settings

from .serializers import AuthTokenSerializer, UserSerializer


class CreateUserView(generics.CreateAPIView):
    """Create a new user in the system."""

    serializer_class = UserSerializer


class CreateTokenView(ObtainAuthToken):
    """Create a token for a user."""

    serializer_class = AuthTokenSerializer
    renderer_classes = api_settings.DEFAULT_RENDERER_CLASSES


class ManageUserView(generics.RetrieveUpdateAPIView):
    """Manage the authenticated user."""

    serializer_class = UserSerializer
    authentication_classes = (authentication.TokenAuthentication,)
    permission_classes = (permissions.IsAuthenticated,)

    def get_object(self):
        """Retrieve and return authenticated user."""
        return self.request.user

URLs

Wiring the views to URLS is pretty simple and straightforward. I use the .as_view() method for class based views as they extent rest_framework.generics views and that’s it.

accounts.urls.py

from django.urls import path

from . import views

app_name = "accounts"


urlpatterns = [
    path("create/", views.CreateUserView.as_view(), name="create"),
    path("token/", views.CreateTokenView.as_view(), name="token"),
    path("me/", views.ManageUserView.as_view(), name="me"),
]

The accounts app is up and running. Now, moving on to the activities app.

Activities App

The activities app has two models Tag and Activity, both of which have user as a foreign key since a user should see only their own tags and activities. A user can create tags and add them to activities when they create an activity or to an already created activity. To get activities to do when the user is bored, the user sends a GET request to “/activities/” url with a name or nature as a query parameter. For example, a GET request for activities which are outdoors would be like “/activities/?nature=outdoor”.

Models

The model definitions are shown below.

activities/models.py

from django.db import models

from accounts.models import User


class Tag(models.Model):
    """Tags to be used for an activity."""

    name = models.CharField(max_length=255)
    user = models.ForeignKey(User, on_delete=models.CASCADE)

    def __str__(self):
        return self.name


class Activity(models.Model):
    user = models.ForeignKey(User, on_delete=models.CASCADE)
    name = models.CharField(max_length=255)
    nature = models.CharField(max_length=255)
    tag = models.ForeignKey(Tag, on_delete=models.CASCADE)

    class Meta:
        verbose_name_plural = "activities"

    def __str__(self):
        return f"{self.user} - {self.name}"

Serializers

The serializers for both models extend rest_framework.serializers.ModelSerializer and are very basic, with no customization whatsoever as there was no need to override any of the base serializer methods.

activities/serializers.py

from rest_framework import serializers

from activities.models import Activity, Tag


class TagSerializer(serializers.ModelSerializer):
    """Serializer for tag objects."""

    class Meta:
        model = Tag
        fields = ("id", "name")
        read_only_fields = ("id",)


class ActivitySerializer(serializers.ModelSerializer):
    """Serializer for the activity model."""

    class Meta:
        model = Activity
        fields = ("id", "user", "name", "nature", "tag")
        read_only_fields = ("id",)

Views

For the views this time, I used rest_framework.viewsets and rest_framewok.mixins as well as rest_framework.permissions. Viewsets do not have get() and post() methods but instead have list(), create(), update(), retrieve(), delete() etc methods. The mixins allow us to specify which of the default viewset methods a user is allowed to do and permission classes allow us to specify whether authentication is required or not.

Since I am restricting users to accessing their own content, I write custom get_queryset() and perform_create() methods to override the default behaviour of those methods. The activities/views.py is shown below.

activities/views.py

from rest_framework import mixins, viewsets
from rest_framework.authentication import TokenAuthentication
from rest_framework.permissions import IsAuthenticated

from .models import Activity, Tag
from .serializers import ActivitySerializer, TagSerializer


class TagViewSet(
    viewsets.GenericViewSet, mixins.ListModelMixin, mixins.CreateModelMixin
):
    """Viewset for Tag objects."""

    authentication_classes = (TokenAuthentication,)
    permission_classes = (IsAuthenticated,)
    queryset = Tag.objects.all()
    serializer_class = TagSerializer

    def get_queryset(self):
        """Return objects for the current authenticated user only."""
        assigned_only = bool(int(self.request.query_params.get("assigned_only", 0)))
        queryset = self.queryset
        if assigned_only:
            queryset = queryset.filter(activity__isnull=False)

        return queryset.filter(user=self.request.user).order_by("-name").distinct()

    def perform_create(self, serializer):
        """Create a new tag"""
        serializer.save(user=self.request.user)


class ActivityViewSet(
    viewsets.GenericViewSet,
    mixins.ListModelMixin,
    mixins.CreateModelMixin,
    mixins.RetrieveModelMixin,
    mixins.UpdateModelMixin,
):
    """Manage activities in the database."""

    queryset = Activity.objects.all()
    serializer_class = ActivitySerializer
    authentication_classes = (TokenAuthentication,)
    permission_classes = (IsAuthenticated,)

    def get_queryset(self):
        nature = self.request.query_params.get("nature")
        name = self.request.query_params.get("name")
        queryset = self.queryset

        if nature:
            return queryset.filter(
                user=self.request.user, nature__icontains=nature
            ).order_by("-id")
        if name:
            return queryset.filter(
                user=self.request.user, name__icontains=name
            ).order_by("-id")
        return queryset.filter(user=self.request.user).order_by("-id")

    def perform_create(self, serializer):
        """Create a new activity."""
        serializer.save(user=self.request.user)

URLs

URLs for viewsets are implemented a bit differently from those of generic views using rest_framework.routers.DefaultRouter.

activities/urls.py

from django.urls import include, path

from rest_framework.routers import DefaultRouter

from . import views

app_name = "activities"

router = DefaultRouter()
router.register("tags", views.TagViewSet)
router.register("activities", views.ActivityViewSet)

urlpatterns = [
    path("", include(router.urls)),
]

Code Formatting

As in all my recent projects, I used blackfor code formatting by running the black . command in a terminal with a virtualenv enabled.

Testing

Once again for testing I used pytest since it makes it so easy to separate test data fixtures from the actual tests using the conftest.py. Especially given that the project has two apps which need to share the same authenticated and non-authenticated user clients, pytest enables me to follow the DRY principle as I just put shared fixtures in tests/unit_tests/conftest.py and then app specific fixtures, I put them in the conftest.py for each app.

I indulged myself on this project a bit by adding to the tests I usually write for models, views, forms etc, tests for serializers. For this I had to use django-factoryboy and Faker for the first time in my own app and it was fun putting together tests and fixtures for the serializers.

Testing serializers

To test serializers, I needed to create a factories.py in tests/unit_tests/test_accounts for the UserFactory and TokenFactory, and another factories.py in tests/unit_tests/test_activities for TagFactory and ActivityFactory.

tests/unit_tests/test_accounts/factories.py

import factory
from faker import Factory as FakerFactory
from pytest_factoryboy import register

from accounts.models import User

faker = FakerFactory.create()


class UserFactory(factory.django.DjangoModelFactory):
    class Meta:
        model = User

    first_name = factory.LazyAttribute(lambda x: faker.first_name())
    last_name = factory.LazyAttribute(lambda x: faker.last_name())
    email = factory.LazyAttribute(lambda x: faker.email())
    password = factory.LazyAttribute(lambda x: faker.password())


class TokenFactory(factory.Factory):
    class Meta:
        model = User

    email = factory.LazyAttribute(lambda x: faker.email())
    password = factory.LazyAttribute(lambda x: faker.password())


register(UserFactory)
register(TokenFactory)

tests/unit_tests/test_accounts/test_serializers.py

import factory
import pytest

from accounts.serializers import UserSerializer, AuthTokenSerializer
from tests.unit_tests.test_accounts.factories import UserFactory, TokenFactory


@pytest.mark.unit
def test_serialize_user_model():
    user = UserFactory()
    serializer = UserSerializer(user)

    assert serializer.data


@pytest.mark.unit
def test_serialized_data(mocker):
    valid_serialized_data = factory.build(dict, FACTORY_CLASS=UserFactory)

    serializer = UserSerializer(data=valid_serialized_data)
    assert serializer.is_valid(raise_exception=True)
    assert serializer.errors == {}


@pytest.mark.unit
def test_serialize_token():
    token = TokenFactory()
    serializer = AuthTokenSerializer(token)

    assert serializer

tests/unit_tests/test_activities/factories.py

import factory
from faker import Factory as FakerFactory
from pytest_factoryboy import register

from activities.models import Activity, Tag
from tests.unit_tests.test_accounts.factories import UserFactory

faker = FakerFactory.create()


class TagFactory(factory.django.DjangoModelFactory):
    class Meta:
        model = Tag

    name = factory.lazy_attribute(lambda x: faker.word())
    user = factory.SubFactory(UserFactory)


class ActivityFactory(factory.django.DjangoModelFactory):
    class Meta:
        model = Activity

    user = factory.SubFactory(UserFactory)
    name = factory.LazyAttribute(lambda x: faker.sentence(nb_words=3))
    nature = factory.LazyAttribute(lambda x: faker.sentence(nb_words=3))
    tag = factory.SubFactory(TagFactory)


register(TagFactory)
register(ActivityFactory)

tests/unit_tests/test_activities/test_serializers.py

import pytest

from activities.serializers import ActivitySerializer, TagSerializer
from tests.unit_tests.test_activities.factories import ActivityFactory, TagFactory


@pytest.mark.unit
def test_serialize_activity_model():
    activity = ActivityFactory()
    serializer = ActivitySerializer(activity)

    assert serializer.data


@pytest.mark.unit
def test_activity_serialized_data(user, tag):
    t = ActivityFactory.build()
    valid_serialized_data = {
        "user": user.pk,
        "name": t.name,
        "nature": t.nature,
        "tag": tag.pk,
    }
    serializer = ActivitySerializer(data=valid_serialized_data)

    assert serializer.is_valid(raise_exception=True)
    assert serializer.errors == {}


@pytest.mark.unit
def test_tag_serializer():
    tag = TagFactory()
    serializer = TagSerializer(tag)

    assert serializer.data


@pytest.mark.unit
def test_tag_serializer_data(mocker):
    t = TagFactory.build()
    valid_serializer_data = {"user": t.user.pk, "name": t.name}

    serializer = TagSerializer(data=valid_serializer_data)

    assert serializer.is_valid(raise_exception=True)
    assert serializer.errors == {}

Continuous Integration (CI)

I used GitHub Actions and defined three .yaml files for unit tests, black formatting and coverage and uploaded the coverage report to Codecov.

Conclusion

This was a fun project to build and experience TDD at work. It also made me appreciate testing event more as I had to set up factories to use for serializers. You can try and build your own version of this API to your own specification and hope you had fun following my own account of this API.

Read More →

My journey with Django Girls

This month, June 2022, marks five years for me in the position of Fundraising Coordinator for the Django Girls Foundation (DGF). The years I have been part of the DGF have been some of the amazing years of my life. I have worked with so many awesome women, including the founders, Ola and Ola and the current support team.

In this post I share with you how it all began and some of the work I have done since joining Django Girls in 2017. So how did my journey with Django Girls begin?

Starting out as an organizer

I first learned about Django Girls in December 2015, from a friend, Humphrey Butau. Humphrey wanted to organize Django Girls Harare as a build-up event to organizing the first PyCon Zimbabwe in 2016 and he needed a co-organizer for both events and I became his co-organizer.

To see how both a Django Girls event and a PyCon are run, we went on a long road trip to Windhoek, Namibia, and coached at Django Girls Windhoek as well as attended PyCon Namibia 2016 in January 2016. In April 2016 we had the first Django Girls Harare and in August 2016, the first Django Girls Masvingo. They were both lovely events and at that time I had no idea how much I would be involved with Django Girls in the future.

Meeting Ola and Ola at Django Under The Hood 2016

In November 2016, I traveled to Amsterdam, The Netherlands, for Django Under The Hood (DUTH) 2016 conference. Most of the organizers of Django Under The Hood were involved with Django Girls including the founders, Ola and Ola. It was really great meeting them in person. Apart from learning more about Django internals while I was still a beginner, I got to attend my first sprint at DUTH 2016.

I picked the Django Girls website as the project I wanted to work on during the sprints. I had trouble setting up the website on my old Windows laptop but eventually got it working. I didn’t finish the work that I had started working on with a friend I had met during the conference but I did learn a lot.

The organizers had put dinners for a limited number of conference attendees which one could sign up to attend and pick the one they wanted to attend. I got to attend the one Ola Sendecka with her husband Tomek was attending. Jeff Triplet was also part of that dinner. Though I was on a very limited budget, having attended the conference on an opportunity grant, I really enjoyed meeting other Djangonauts and learning their stories on how they started working with Django and how long they have been working with Django. My experiences at DUTH 2016 and DjangoCon Europe 2017 were a huge part of my motivation to be more involved in the Django community.

Becoming the Fundraising Coordinator

In April 2017, DGF announced that their Awesomeness Ambassador, Lucie, was leaving them and they were hiring from anywhere in the world. I looked at the role and I knew I had no issues with the fundraising bit but had zero experience on the finance and admin side of things. A friend (Humphrey) encouraged me to apply anyway. I did apply for the role and was invited to the interview.

I attended the first round of interviews and as I expected, I did answer well questions to do with fundraising and didn’t have much to say about the finance and admin side of things. I was told I would hear from them later and I did hear from them with the best news I wasn’t expecting. It turned out that their best candidate for the Awesomeness Ambassador, Claire, was very well experienced with finance and admin and not so much with fundraising so they decided to create a new position of Fundraising Coordinator, which they wanted to interview me for.

I agreed to the second interview for Fundraising Coordinator and was offered the position which I accepted and since June 2017, I have been DGF’s Fundraising Coordinator. I have enjoyed that the role is part-time and flexible and I work from anywhere and anytime as I please. It has also exposed me to many awesome people within the Django Girls and Django community. I have also learned a lot from my time with Django Girls.

Becoming the Lead Maintainer of the Django Girls Repo

From June 2017 to July 2020, I worked on the Django Girls website just to change the contents of the website as well as the look and feel. Most of the work I did was related to the content I wanted on the website for my duties as Fundraising Coordinator. When I started, Ola Sendecka would review my work until when we got to a point where I could just deploy it if it works because getting reviews was proving difficult.

The founders, trustees, and their initial group of contributors were getting burnt out and felt they needed to move on to other things so from 2018 to 2020, there was a huge maintenance gap for the website. Pull requests from volunteers were not being reviewed and issues were not being attended to. When the transition from old trustees to new trustees was officially over, I found myself being the only person on the new board of trustees who knew how the website worked so I was asked to step up and add technical support for the website to my role.

I then realized I didn’t know as much about the Django and DGF websites as I thought I did. Over the years, I have taken up some online courses on Python and Django as well as other tools to increase my knowledge and skills. I also had to manually read the code and understand it when I needed to fix a bug, add new functionality, or change the way the code has been working. It has been a challenging but rewarding and interesting journey getting to be the lead maintainer of the Django Girls Repo.

Not only have I learned more about Django, but also learned about managing an organization repo on GitHub. I have also gained confidence in pull request reviews for others and how to submit reviews that are not toxic but friendly and encouraging for newbies. I also get to merge other people’s work and deploy it as well as adding issues for volunteers to work on. I have also added the documentation on how we do things in DGF and how to use some of the new functionality we have added since the new board took over.

Some of the work I have contributed to the Django Girls repo includes:

  • Changes to the event application process.
  • Increasing test coverage for the organize app.
  • Work with volunteera to upgrade the website from Django 2.0 to 2.2 to 3.2.
  • Add robots.txt and sitemap.xml and register the Django Girls website to Google Search Console.
  • Adding event duplication, freeze, and unfreeze functionality.
  • Contributing to the Django Girls internal wiki documentation.

Important lessons from my time with Django Girls

  • Paying attention to detail is very important.
  • Sometimes code might be broken even if the tests are passing.
  • The importance of refactoring and writing clean code.
  • Performance is important when adding new features.
  • Test coverage is important, always add tests for new features.
Read More →

PyCon Zimbabwe 2016: Our Success Story

Organising the first PyCon Zimbabwe, which was held on the 24th -25th November was a long process and long journey faced with many challenges. First and formost, we had to travel more than 33hrs hours by road to Windhoek to take part in PyCon Namibia and learn and travel a similar journey back! Its not something I would enjoy talking about because it sounds so crazy and yet it was so necessary. We had to organise other events here in Zimbabwe, in preparation for PyCon Zim as well as travel to Cape Town for PyCon ZA.

We also faced financial challenges as we could not get funding from The PSF due to targeted sanctions against our country. This made it almost impossible for us to hold the event but we are so grateful to all our friends in the global Python community who supported our crowdfunding project. We are also grateful to all our sponsors and partners, The Python Software Foundation, Django Software Foundation, Siege Communications and GitHub for making our event possible. We are also grateful to all our speakers and attendees for making the event a success, we couldn't have made it without them. Special gratitude to our visitors and speakers Gabriel Nhinda (Namibia) and Petrus Janse van Rensburg (South Africa) and our keynote speakers Amanda Gelender (United States of America) and Mike Place (United States of America).

You all go down in history for making the first ever PyCon Zimbabwe a success! Thank you!

Read More →
Search
Coding Tips

Pytest assert True


Category: Django

Pytest equivalent of

unittest.TestCase.assertTrue()

is

assert some_condition.

For example

self.assertTrue(form.is_valid())

using pytest would be

assert form.is_valid().

✨Magic ✨