Stumbling to Package a Django App

Django applications can be packaged individually for reuse. Documentation on how to do so from start to finish seemed either old, sparse, or incomplete. This was my adventure and solution.

I’ve used Django to develop a few applications. For one project, I wanted to see how many active users there were interacting with the site. The visibility of the project wasn’t too big, so the solution didn’t need to scale for a large or worldwide user base. I just needed something that could handle upwards of a few hundred requests a minute.

I didn’t find anything in pip that fit the bill though. There were solutions using Redis, some that had everything in memory (which would be wiped on every release), and there were some solutions that were overly-complex for super-scale. I just needed something simple; something that could utilize my already existent PostgreSQL backend.

I decided that creating a new model and middleware using Django was the way to go. This was my first time creating a middleware, and surprisingly it was quite simple.

First I made my new application for the project:

$ python manage.py startapp online_users

Then I made the model and some helper methods to store and retrieve the data properly. All that’s stored is a reference to a user and a timestamp of their last activity (read HTTP request) on the site.

class OnlineUserActivity(models.Model):
    user = models.OneToOneField(User)
    last_activity = models.DateTimeField()

    @staticmethod
    def update_user_activity(user):
        """Updates the timestamp a user has for their last action. Uses UTC time."""
        OnlineUserActivity.objects.update_or_create(user=user, defaults={'last_activity': timezone.now()})

    @staticmethod
    def get_user_activities(time_delta=timedelta(minutes=15)):
        """
        Gathers OnlineUserActivity objects from the database representing active users.

        :param time_delta: The amount of time in the past to classify a user as "active". Default is 15 minutes.
        :return: QuerySet of active users within the time_delta
        """
        starting_time = timezone.now() - time_delta
        return OnlineUserActivity.objects.filter(last_activity__gte=starting_time).order_by('-last_activity')

Now I needed to make a middleware to extract the user and update the table upon a request.

from online_users.models import OnlineUserActivity

class OnlineNowMiddleware(object):
    """Updates the OnlineUserActivity database whenever an authenticated user makes an HTTP request."""

    @staticmethod
    def process_request(request):
        user = request.user
        if not user.is_authenticated():
            return

        OnlineUserActivity.update_user_activity(user)

Naturally, I added a few tests to ensure everything would work properly:

class OnlineUserActivityTest(TestCase):

    def setUp(self):
        self.user1 = User.objects.create(username='testUser1')
        self.user2 = User.objects.create(username='testUser2')
        self.time = timezone.now()
        self.time_five_min_ago = self.time - timedelta(minutes=5)
        self.online_user1 = OnlineUserActivity.objects.create(user=self.user1, last_activity=self.time)
        self.online_user2 = OnlineUserActivity.objects.create(user=self.user2, last_activity=self.time_five_min_ago)

    def test_update_activity_for_user(self):
        self.assertEqual(OnlineUserActivity.objects.all().count(), 2)
        OnlineUserActivity.update_user_activity(self.user1)
        self.assertEqual(OnlineUserActivity.objects.all().count(), 2)

    def test_get_active_users(self):
        online_users = OnlineUserActivity.get_user_activities(timedelta(minutes=6))
        self.assertEqual(online_users.count(), 2)

    def test_get_active_users__users_out_of_timedelta(self):
        online_users = OnlineUserActivity.get_user_activities(timedelta(minutes=1))
        self.assertEqual(online_users.count(), 1)

    def test_get_active_users_ordering(self):
        online_users = OnlineUserActivity.get_user_activities(timedelta(minutes=60))
        self.assertEqual(online_users.count(), 2)
        self.assertEqual(list(online_users), [self.online_user1, self.online_user2])

And:

class OnlineUserMiddlewareTest(TestCase):

    @staticmethod
    def get_active_user_count():
        last_hour = timedelta(minutes=60)
        return OnlineUserActivity.get_user_activities(last_hour).count()

    def create_and_login_user(self):
        password = 'test1!'
        user = User.objects.create_user(username='testUser1', email='test@test.com', password=password)
        self.client.login(username=user.username, password=password)

    def url_request(self, url):
        response = self.client.get(url, follow=True)
        self.assertEqual(response.status_code, 200)
        return response

    def test_anonymous_user_not_added(self):
        self.url_request('')
        self.assertEqual(self.get_active_user_count(), 0)

    def test_user_added_and_updated(self):
        self.create_and_login_user()
        for i in range(3):
            self.url_request('')
            self.assertEqual(self.get_active_user_count(), 1)

Everything passed! I just need to hook up the application and the middleware to my app in the settings.py file:

INSTALLED_APPS = (
   ...
    'online_users',

)
..
MIDDLEWARE_CLASSES = (
    ...
    'online_users.middleware.OnlineNowMiddleware',
)

Finally, I made the migrations and applied them to my database.

$ python manage.py makemigrations
$ python manage.py migrate

When I ran the project to see if data was showing up as expected, everything seemed to work.

Wahoo!

This gives a small view as to how many users have been “active”, or rather how many users have made at least one action or request in the requested time period.

This solution, while extremely simple, could be generalized for others to use. Why not release it to the world? There is sure to be other small projects where this would be useful.

Django’s documentation gives a pretty good account to making reusable apps. Following their instructions, I was determined to move my online_users application out of my project and into its own repository.

First thing to do was to create the new repo, django-online-users, in GitHub. I did so with the MIT license and the Python .gitignore settings. After cloning the repo to my dev box, I copied the application directory over to its new home. Now in my django-online-users directory rested online_users.

Every project needs a good README, so I updated the template GitHub generated, and I added the requirements.txt file with the Django dependency for the project. My new directory structure was starting to take shape:

screen-shot-2017-01-06-at-2-00-42-pm

To ensure the application was still working properly, I made a new virtual environment to test things out and installed the Django dependency:

$ virtualenv -p `which python3` ~/django-online-users-env
$ source ~/django-online-users-env/bin/activate
$ pip install -r requirements.txt

Now here is where I started to transition away from the Django side of things and more to the Python side. To setup a Python project, I needed to create the setup.py file and install setuptools:

$ pip install setuptools

Then I created the setup.py file. Most of this was templated from the Django documentation. The classifiers especially are guesswork, but they at least look sensible.

import os
from setuptools import find_packages, setup

with open(os.path.join(os.path.dirname(__file__), 'README.rst')) as readme:
README = readme.read()

# allow setup.py to be run from any path
os.chdir(os.path.normpath(os.path.join(os.path.abspath(__file__), os.pardir)))

setup(
    name='django-online-users',
    version='0.1',
    packages=find_packages(),
    include_package_data=True,
    license='MIT License',
    description="Tracks the time of users' last actions.",
    long_description=README,
    url='https://github.com/lawrencemq/django-online-users',
    author='Lawrence Weikum',
    author_email='lawrencemq@gmail.com',
    install_requires=open('requirements.txt').read(),
    classifiers=[
        'Environment :: Web Environment',
        'Framework :: Django',
        'Framework :: Django :: 1.10',
        'Intended Audience :: Developers',
        'License :: OSI Approved :: MIT License',
        'Operating System :: OS Independent',
        'Programming Language :: Python',
        'Programming Language :: Python :: 2',
        'Programming Language :: Python :: 2.7',
        'Programming Language :: Python :: 3',
        'Programming Language :: Python :: 3.4',
        'Programming Language :: Python :: 3.5',
        'Programming Language :: Python :: 3.6',
        'Topic :: Internet :: WWW/HTTP',
        'Topic :: Internet :: WWW/HTTP :: Dynamic Content',
    ],
)

Finally, I added the manifest:

include LICENSE
include README.rst
recursive-include online_users *.py
include requirements.txt

A funny note, I had initially written recursive-exclude by accident in this file, so when I packaged everything, everything but my code was packed. Whoops!

Speaking of packaging, it was time to test that out:

$ python setup.py sdist

It worked! Hopefully the application still worked in my initial project. Back in its directory and virtual environment, I installed the application from a local source:

$ pip install --upgrade ~/github/django-online-users/dist/django-online-users-0.1.tar.gz

Starting the application, I visited a few pages and then checked the admin portal to see that my activity was still being logged.

Success! Cool! Now, like a good programmer, I wanted to run the application’s tests again:

$ python setup.py test
...
0 tests ran...

Drat!

Admittedly, I didn’t find a ton of useful information on the web about how to link Django tests properly and run them from setup.py. Eventually I stumbled upon a solution that worked. I added these lines to my setup.py file:

setup(
    ...
    test_suite='nose.collector',
    tests_require=['nose'],
    ...
)

Now when I ran python setup.py test, things were working.

Or.. at least it was attempting to run my tests. I received many verbose errors that every test crashed and burned. Essentially, Django hadn’t been setup properly. This made sense. I no longer had my settings.py or manage.py files that described how Django should work and run.

Eventually, I found I could add a slimmed-down version of my settings to the test’s __init__.py  file to setup everything properly.

import os
import sys

from django.conf import settings
from django.conf.urls import url
from django.core.management import call_command
from django.http import HttpResponse

current_dir = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, os.path.join(current_dir, '..'))

conf_kwargs = dict(
    ALLOWED_HOSTS=('testserver', '127.0.0.1', 'localhost', '::1'),
    DATABASES={
        'default': {
            'ENGINE': 'django.db.backends.sqlite3',
            'NAME': 'test.db',
            'TEST_NAME': 'test.db'
        }
    },
    SITE_ID=1,
    MIDDLEWARE_CLASSES=(
        'django.middleware.common.CommonMiddleware',
        'django.contrib.sessions.middleware.SessionMiddleware',
        'django.contrib.auth.middleware.AuthenticationMiddleware',
        'online_users.middleware.OnlineNowMiddleware',
    ),
    INSTALLED_APPS=(
        'django.contrib.auth',
        'django.contrib.contenttypes',
        'django.contrib.sessions',
        'django.contrib.sites',
        'online_users'
    ),
)

settings.configure(**conf_kwargs)

try:
    # For django>=1.10
    from django import setup
except ImportError:
    pass
else:
    setup()

Running the tests again, things looked better. Django was being started, but my database wasn’t setup to handle the models properly – it was still blank. Adding this line to the bottom of the __init__.py seemed to solve that:

call_command('migrate')

That’s better! Now the tests against the model were running and passing, but the same couldn’t be said for the tests against the middleware. It was complaining that there were no URLs to which it could direct requests.

Alright, adding a simple URL to the settings in __init__.py seemed to help:

conf_kwargs = dict(
    ...
    ROOT_URLCONF=(
        url(r'^', lambda _: HttpResponse("
<h1>Hello World</h1>
")),
    ),
)

Haha! Finally! Success! All the tests were running, and all of the tests were passing!

Time for the final build and release.

$ python setup.py sdist bdist_wheel

This added a tarball and a wheel in a the dist folder.

All I had to do now  was add them to Pypi so others could install the app via pip. Their documentation said to use twine. So following their instructions:

$ pip install twine

Then I needed to register the project:

$ twine register dist/django-online-users-0.1.tar.gz
...
HTTPError: 410 Client Error: This API is no longer supported, instead simply upload the file.

… Well fine. I guess this is another one of those times where “the code is the documentation,” and someone forgot to update the wiki. Using Pypi’s website, I was able to register the new project by uploading the PKG-INFO directory that the build made in the django_online_users.egg_info directory.

Hoping that twine was still a good way to upload the sources, I continued to follow their documentation:

$ twine upload dist/*

Success! It uploaded the wheel and the tar.gz.

Periodically I queried pip to see if my package had been distributed across their servers yet.

$ pip search django-online-users

It took roughly an hour – not to bad in my opinion.

Back in my main project I added the new dependency in requirements.txt (django-online-users==0.1) and did a pip install. Running the project locally, everything still worked!

Horray!

The new repository may not be perfect – in fact I’m sure it’s not. Nearly all of the steps and requirements of Python distribution are new to me. At the end of the day though, this worked. Now I have a general-purpose Django application that’s freely available for anyone to use, and by using it, others can view the active user count for their projects.

See the final code on GitHub!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s