Job Queue

glim-jobqueue - a redis jobqueue extension for glim

This is a jobqueue implementation for glim framework. It uses glim-redis. extension for redis connection. You can use job extension for async processes that consume much more time than a typical request. For instance, it can be done mail queue, push notifications, etc. as jobs.

Installation

• First, install glim-redis extension
• Clone the github repo, move job folder to ext folder
• Remove .git directory.

Shell

# install glim-extensions from pip
pip install glim-extensions

Configuration

• Append job configuration just after gredis configuration in your config file

Python

# app/config/<env>.py
config = {
    'extensions' : {
        'gredis' : {
            'default' : {
                'host' : 'localhost',
                'port' : '6379',
                'db'   : 0
            }
        },
        'job' : {
            'default' : {
                'redis'  : 'default',
                'jobs'   : 'jobs',    # redis list name for jobs
                'failed' : 'failed',  # redis list name for failed jobs
            }
        },
    },
    # ...
}

Initializing Job Extension

To initialize job extension, type the following;

Shell

glim job:init
# this will create a job.py on your app folder

Creating Jobs

Python

# app/jobs.py
from glim import Log
from glim_extensions.job import Job

class HelloJob(Job):
    def run(self): # self.data is registered when a Job 
        if 'author' in self.data.keys():
            Log.info('hello %s' % self.data['author'])
        else:
            Log.info('hello glim!')

Producing Jobs

Python

from glim_extensions.job import JobQueue
data = {
    'author' : 'Aras Can Akin'
}
JobQueue.push(HelloJob(data)) # returns True or False
JobQueue.push(HelloJob)

Consuming Jobs

Shell

glim job:consume --name jobs
# output
# hello Aras Can Akin
# hello

Failed Jobs

In jobqueue extension, the failed jobs can be moved to an another redis list by raising exceptions. The $ glim job:consume command is looking for jobs to throw exception if a failed job occurs.

Python

# app/jobs.py
from glim import Log
from glim_extensions.job import FailedJobError
from glim_extensions.job import Job

class HelloJob(Job):
    def run(self):
        if 'author' in self.data.keys():
            Log.info('hello %s' % self.data['author'])
        else:
            raise FailedJobError()

In this example, when you push a job without author, the job will go to failed list in redis

Roadmap

• the job system should work for many other message queue services like AWSQ, RabbitMQ, IronMQ, etc.
• command for flushing job queue
• add feature to push/pop scheduled jobs