kirit.com

Created 26th April, 2005 13:12 (UTC), last edited 30th December, 2013 06:38 (UTC)

Writing about C++, Programming, Fost 4, the web, Thailand and anything else that catches my attention—with some photos thrown in

Django Async

Created 25th September, 2013 12:06 (UTC), last edited 23rd September, 2014 05:12 (UTC)

Django Async is an asynchronous execution queue for Django with proper database transaction management

Building a database backed task queue is a fairly trivial thing, but getting the database transactions exactly right is no simple matter.

Installing Django Async

Get Django Async with pip from pypi:

pip install django-async

Installation is very simple, just add the async application to your Django applications in settings.py. You also really want to use the transaction middleware (see below) and a proper transactional database (like PostgreSQL).

Using Django Async from your code

To run a job asynchronously just use the schedule function:

from async import schedule
schedule('my.function', args=(1, 2, 3), kwargs=dict(key='value'))

Tasks can be run by executing the management command flush_queue:

python manage.py flush_queue

See Using Django Async from your code for more details.

Using Django Async from the command line

Command line interaction with Django Async is through Django's management commands.

python manage.py flush_queue

For more details on options see Using Django Async from the command line.

Transaction handling

Database transactions are hard to get right, and unfortunately Django doesn't make them much easier. Firstly, you really want to be using a proper transactional database system.

Django has two major flaws when it comes to transaction handling:

  1. The Django transaction functionality fails to create composable transactions.
  2. The Django documentation makes a very poor recommendation about where to put the django.middleware.transaction.TransactionMiddleware.

The first problem is not going to get fixed in Django, but the second can be handled by putting the middleware in the right place — that is, as early as possible. The only middleware that should run before the transaction middleware is any whose functionality relies on it being first.

Within the async task execution each task is executed decorated by django.db.transaction.commit_on_success. This means that you cannot execute a task directly from within a page request if you are using the transaction middleware (this is due to problem number one above).

Pages

  1. Using Django Async from your code
  2. Using Django Async from the command line

Categories:
Posted: 23rd September, 2014 11:56 (UTC)

django-async 0.6 has been released.

  • Added group model for grouping job, slumber operation for that model.
  • Brought in time zone support in Django 1.4. Thanks to Peter Brooks for initiating the effort.
  • Added jobs limit option into flush_queue command.
  • Add field cancelled to job
  • Add a function that can be placed into the async queue for clearing out old jobs.
  • Add the ability to have multiple workers so that throughput and latency can be improved.

Detailed changes

  • Fix admin error when a job has no group
  • Fixed a bug that stopped us from running jobs in top level modules.
  • Added database indexes to speed up selection of jobs, and the progress operation and re-arranged the bookkeeping of job execution to lower transaction costs. Also added in scripts to make testing of the queue overhead simpler.
  • Fixed the generation of the progress operation URL.
  • When giving a group name to the schedule API it now does the right thing and creates a new group instance if appropriate.
  • A final job can be set on a group (using on_completion) which will be executed after all of the other jobs have been done.
  • Include the time consumed in the progress report and switch to ISO formatted dates.
  • Fix pylint 1.2.1 issues.
  • Changed the progress estimated_group_duration to estimated_total_time and added in remaining_seconds to the Slumber progress operation.
  • Modified api to accept target group parameter.
  • Make remove old job function include cancelled jobs

Fost 4 release 4.14.09.44924 now out

Posted 21st September, 2014 07:22 (UTC), last edited 21st September, 2014 08:07 (UTC)

There are a couple of new libraries that we've tried to get included previously, but are now actually fully integrated into the meta-build system that we use to test and publish the libraries — this means we won't accidentally miss them out again.

This release has not been tested on Mac — my ancient iMac has finally died, and I have no other way to properly test this platform. For now I'm leaving the Mac instructions in here, but if I don't get a new Mac or a volunteer to run the test builds then I'm going to have to drop the Mac support.

We've also got some initial experimental Docker images available to make building and deploying Fost based systems easier. Later on we hope to be able to augment these with full development environments which should make using the libraries significantly simpler.

Linux & Mac

git clone --branch=4.14.09.44924 --recursive git@github.com:KayEss/fost-hello.git
cd fost-hello
Boost/build
hello/compile
dist/bin/hello-world-d

On the Mac you will need to set DYLD_LIBRARY_PATH before running hello-world-d

export DYLD_LIBRARY_PATH=dist/lib
dist/bin/hello-world-d
Windows
git clone --branch=4.14.09.44924 --recursive git@github.com:KayEss/fost-hello.git
cd fost-hello
Boost\build
hello\compile
dist\bin\hello-world-gd

Download locations

Applications

  • beanbag — Stand alone transactional JSON database server — git@github.com:KayEss/beanbag.git
  • fost-hello — Sample seed project — git@github.com:KayEss/fost-hello.git
  • mengmon — Stand alone web server — git@github.com:KayEss/mengmom.git

Libraries

  • fost-aws — Amazon AWS and OpenStack — git@github.com:KayEss/fost-aws.git
  • fost-beanbag — Transactional JSON database — git@github.com:KayEss/fost-beanbag.git
  • fost-base — Build system and core libraries — git@github.com:KayEss/fost-base.git
  • fost-internet — Internet protocols, servers & clients — git@github.com:KayEss/fost-internet.git
  • fost-meta — All libraries in one wrapper — git@github.com:KayEss/fost-meta.git
  • fost-orm — Object/Relational mapping — git@github.com:KayEss/fost-orm.git
  • fost-postgres — PostgreSQL — git@github.com:KayEss/fost-postgres.git
  • fost-py — Python (2.x) bindings — git@github.com:KayEss/fost-py.git
  • fost-web — Web server libraries — git@github.com:KayEss/fost-web.git
  • fost-windows — Windows support — git@github.com:KayEss/fost-windows.git

Detailed change log

fost-base

  • The meta data set on the progress for a task is now made available to the meter readings.
  • There is a new fostlib::cli::monitor function that can be used to display a progress bar monitoring work towards a future.
  • Futures now return const&s instead of copies of the value when they're dereferenced.
  • Allow fostlib::future<> instances to be compared for equality.
  • The fostlib::accessors now use perfect forwarding if FOST_HAS_MOVE is defined.
  • Need to not pretty print the JSON values when showing the full settings database as we're outputting something approximating a CSV file.

fost-postgres

  • Change the test database configuration to assume ident/trust based authentication.

Categories:

Early test images

Created 12th September, 2014 04:01 (UTC), last edited 13th September, 2014 09:02 (UTC)

These are basic test images that just allow me to check that the ray tracer is outputting what it should.

These can all be built using some commands like:

git clone —recursive git@github.com:KayEss/AnimRay.git animray
cd animray
Boost/install $(lsb_release -sc)
animray/compile animray scenes release

You will find each example in the dist/bin folder and they can be run from there, e.g.:

dist/bin/affine-transformations affine.tga 1920 1080

Pictures & Photos


Categories:
Posted: 13th September, 2014 10:11 (UTC)
First put on the front page to coincide with the 0.1 release of the code.

Fost 4 release 4.14.06.44917 now out

Posted 19th July, 2014 05:36 (UTC), last edited 19th July, 2014 06:43 (UTC)

The new release was once again tagged a while ago, but I went on holiday before managing to debug the failure of the release process to push the tag to github.

Linux & Mac

git clone --branch=4.14.06.44917 --recursive git@github.com:KayEss/fost-hello.git
cd fost-hello
Boost/build
hello/compile
dist/bin/hello-world-d

On the Mac you will need to set DYLD_LIBRARY_PATH before running hello-world-d

export DYLD_LIBRARY_PATH=dist/lib
dist/bin/hello-world-d
Windows
git clone --branch=4.14.06.44917 --recursive git@github.com:KayEss/fost-hello.git
cd fost-hello
Boost\build
hello\compile
dist\bin\hello-world-gd

Download locations

  • fost-aws — Amazon AWS and OpenStack — git@github.com:KayEss/fost-aws.git
  • fost-base — Build system and core libraries — git@github.com:KayEss/fost-base.git
  • fost-hello — Sample project — git@github.com:KayEss/fost-hello.git
  • fost-internet — Internet protocols, servers & clients — git@github.com:KayEss/fost-internet.git
  • fost-meta — All libraries in one wrapper — git@github.com:KayEss/fost-meta.git
  • fost-orm — Object/Relational mapping — git@github.com:KayEss/fost-orm.git
  • fost-postgres — PostgreSQL — git@github.com:KayEss/fost-postgres.git
  • fost-py — Python (2.x) bindings — git@github.com:KayEss/fost-py.git
  • fost-windows — Windows support — git@github.com:KayEss/fost-windows.git

Detailed change log

fost-orm

  • Made the internal API for adding a transformation function to the JSON database through fostlib::jsondb::local::transformation public.

Categories:

Fost 4 release 4.14.03.44915 now out

Posted 4th April, 2014 11:16 (UTC), last edited 5th April, 2014 03:36 (UTC)

The new release was tagged on the new git repositories a couple of weeks ago. The transfer of everything to git turned out to be far more complex than I could ever have expected — I think it actually took longer and more work than converting from Visual Source Safe to Subversion, and I had to write my own software for that. The thing that had me confused for a long time was my expectation that git would allow me to handle the full history, but that doesn't seem to be the case — oh well, in any case a story for another time.

Anyway, with the move to git there's also a number of clean ups and small improvements outlined below, and I've also pushed out a meta-build system that I use to manage to the projects in their entirety at fost-dev.

The old Subversion repositories will remain, but they won't receive further updates.

Linux & Mac

git clone --branch=4.14.03.44915 --recursive git@github.com:KayEss/fost-hello.git
cd fost-hello
Boost/build
hello/compile
dist/bin/hello-world-d

On the Mac you will need to set DYLD_LIBRARY_PATH before running hello-world-d

export DYLD_LIBRARY_PATH=dist/lib
dist/bin/hello-world-d
Windows
git clone --branch=4.14.03.44915 --recursive git@github.com:KayEss/fost-hello.git
cd fost-hello
Boost\build
hello\compile
dist\bin\hello-world-gd

Download locations

  • fost-aws — Amazon AWS and OpenStack — git@github.com:KayEss/fost-aws.git
  • fost-base — Build system and core libraries — git@github.com:KayEss/fost-base.git
  • fost-hello — Sample project — git@github.com:KayEss/fost-hello.git
  • fost-internet — Internet protocols, servers & clients — git@github.com:KayEss/fost-internet.git
  • fost-meta — All libraries in one wrapper — git@github.com:KayEss/fost-meta.git
  • fost-orm — Object/Relational mapping — git@github.com:KayEss/fost-orm.git
  • fost-postgres — PostgreSQL — git@github.com:KayEss/fost-postgres.git
  • fost-py — Python (2.x) bindings — git@github.com:KayEss/fost-py.git
  • fost-windows — Windows support — git@github.com:KayEss/fost-windows.git

Detailed change log

fost-base

  • Extend the fostlib::timestamp and fostlib::date interfaces somewhat. Added a number of missing coercions and constructors to make things simpler and made sure that there is no loss in precision when converting to and from strings and JSON. Moved a number of timestamp and date members inline for speed.
  • Allow dates to have days added and subtracted from them.
  • Add today member to fostlib::date.
  • Fix some warnings for a couple of compilers.
  • Added a new split member to jcursor to allow one to be created by splitting up a single string.

fost-internet

  • Folding of long lines in MIME headers needs to be conditional on the protocol because most MIME protocols use it, but HTTP does not and some web servers don't handle folded lines.
  • The MIME header names and values need to be retrievable in a case-insensitive manner as not everybody agrees on the correct case.
  • The fostlib::url::query_string class can now also be used to parse the standard format query string on the server side. The HTTP server query string is also now always a url::query_string instance.
  • Hold the parser lock for a much shorter period when parsing a HTTP request to the web server.
  • Alter the testing HTTP server request class so it can take a query string and the headers/body are optional.

fost-py

  • The spider now tries more types of link including script and img elements.

fost-aws

  • Added support for Swift as well as S3.
  • The account to use can now be passed to s3put with -a.
  • Added s3get to fetch files and refactored the s3put command to move the logic of where the upload decision happens.

Categories: