Job Queue

glim-jobqueue - a redis jobqueue extension for glim

This is a jobqueue implementation for glim framework. It uses glim-redis. extension for redis connection. You can use job extension for async processes that consume much more time than a typical request. For instance, it can be done mail queue, push notifications, etc. as jobs.

Installation

  • First, install glim-redis extension
  • Clone the github repo, move job folder to ext folder
  • Remove .git directory.
# install glim-extensions from pip
pip install glim-extensions

Configuration

  • Append job configuration just after gredis configuration in your config file
# app/config/<env>.py
config = {
    'extensions' : {
        'gredis' : {
            'default' : {
                'host' : 'localhost',
                'port' : '6379',
                'db'   : 0
            }
        },
        'job' : {
            'default' : {
                'redis'  : 'default',
                'jobs'   : 'jobs',    # redis list name for jobs
                'failed' : 'failed',  # redis list name for failed jobs
            }
        },
    },
    # ...
}

Initializing Job Extension

To initialize job extension, type the following;

glim job:init
# this will create a job.py on your app folder

Creating Jobs

# app/jobs.py
from glim import Log
from glim_extensions.job import Job

class HelloJob(Job):
    def run(self): # self.data is registered when a Job 
        if 'author' in self.data.keys():
            Log.info('hello %s' % self.data['author'])
        else:
            Log.info('hello glim!')

Producing Jobs

from glim_extensions.job import JobQueue
data = {
    'author' : 'Aras Can Akin'
}
JobQueue.push(HelloJob(data)) # returns True or False
JobQueue.push(HelloJob)

Consuming Jobs

glim job:consume --name jobs
# output
# hello Aras Can Akin
# hello

Failed Jobs

In jobqueue extension, the failed jobs can be moved to an another redis list by raising exceptions. The $ glim job:consume command is looking for jobs to throw exception if a failed job occurs.

# app/jobs.py
from glim import Log
from glim_extensions.job import FailedJobError
from glim_extensions.job import Job

class HelloJob(Job):
    def run(self):
        if 'author' in self.data.keys():
            Log.info('hello %s' % self.data['author'])
        else:
            raise FailedJobError()

In this example, when you push a job without author, the job will go to failed list in redis

Roadmap

  • the job system should work for many other message queue services like AWSQ, RabbitMQ, IronMQ, etc.
  • command for flushing job queue
  • add feature to push/pop scheduled jobs