Hangfire: Task Scheduler for .NET

Hangfire is a multi-threaded and scalable task scheduler built on client-server architecture on .NET stack (Task Parallel Library and Reflection) with the intermediate storage in a database. There is a free LGPL v3 version with open source. In this article, we are going to explore how to use Hangfire.

Picture from hangfire.io

Contents:

• Operation peculiarities
• Hangfire server features
• Recurrent and delayed jobs
• Brief review of Quartz.NET
• Load testing
• Conclusions

 Operation peculiarities

As you can see on the image, a client adds a job to a database, while a server fetches the database and perform jobs in background.

Keep in mind the following information:

• both a client and a server have access to the shared database and assemblies where job classes are declared.
• There is load scalability – you may increase the number of servers.
• Hangfire cannot work without databases. By default, it supports SQL Server, and there are extensions for popular DBMS. The commercial edition supports Redis.
• You may use Hangfire in ASP.NET applications, Windows Services, console applications, as well as in Azure Worker Role.

From the client’s perspective, working with a task is based on «fire-and-forget» principle. There is nothing happening on the client side other than saving the task to a database.

For example, we want to execute MethodToRun in a separate process:

BackgroundJob.Enqueue(() => MethodToRun(42, "foo"));

Hangfire serializes this job with input values and stores it in the database:

{
    "Type": "HangClient.BackgroundJobClient_Tests, HangClient, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null",
    "Method": "MethodToRun",
    "ParameterTypes": "(\"System.Int32, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089\",\"System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089\")",
    "Arguments": "(\"42\",\"\\\"foo\\\"\")"
}

This information is sufficient to call the Methodtorun method in a separate process through reflection, provided you have access to the Hangclient assembly in which it is declared.

Surely, it is not necessary to keep the code for the background processing in the assembly with the client. The dependency diagram is as follows:

The client and server should have access to the shared assembly, while the embedded interface does not require it. It is possible to replace the implementation of the stored job by replacing the assembly, which the server application refers to.

It is convenient for recurring jobs if MethodToRun matches both in old and new assemblies. The only restriction is to have a public modifier.

If you need to create an object and call its method, Hangfire does it for you:

BackgroundJob.Enqueue<EmailSender>(x => x.Send(13, "Hello!"));

It may even retrieve the EmailSender instance using a DI-container.

Also, it simplifies the server deployment, for example, in a separate Windows Service:

public partial class Service1 : ServiceBase
{
    private BackgroundJobServer _server;

    public Service1()
    {
        InitializeComponent();
        GlobalConfiguration.Configuration.UseSqlServerStorage("connection_string");
    }

    protected override void OnStart(string() args)
    {
        _server = new BackgroundJobServer();
    }

    protected override void OnStop()
    {
        _server.Dispose();
    }
}

After the service start-up, the Hangfire server will start retrieving jobs from databases and execute them.

To manage the job processing, you may use the embedded web dashboard:

Hangfire server features

The server contains its custom thread pool implemented using Task Parallel Library based on Task.WaitAll. For more information, refer to BackgroundProcessingServer.

It supports Web Farm and Web Garden:

You don’t want to consume additional Thread Pool threads with background processing – Hangfire Server uses custom, separate and limited thread pool.
You are using Web Farm or Web Garden and don’t want to face with synchronization issues – Hangfire Server is Web Garden/Web Farm friendly by default.

We can create any amount of Hangfire servers and do not care of their synchronization. Hangfire ensures that one server will execute only one job. You can check it using sp_getapplock (Refer to the SqlServerDistributedLock class).

As it has been discussed, the Hangfire server does not require any particular host and can be deployed from Console App to Azure Web Site. When hosting in ASP.NET, you need to take into account such general features of IIS as process recycling, auto-start  (startMode=«AlwaysRunning»), etc. However, a scheduler documentation provides meaningful information in this case.

In addition, the quality of the documentation is perfect. The source code of Hangfire is public and of high quality, as well as there are no barriers to deploy a local server and debug a code.

Recurrent and delayed jobs

Hangfire allows users to create recurring jobs with the minimum interval per minute:

RecurringJob.AddOrUpdate(() => MethodToRun(42, "foo"), Cron.Minutely);

It is possible to run the job manually or delete it:

RecurringJob.Trigger("task-id");
RecurringJob.RemoveIfExists("task-id");

In addition, you may delay the job execution:

BackgroundJob.Schedule(() => MethodToRun(42, "foo"), TimeSpan.FromDays(7));

To do this, use CRON expressions (the NCrontab project supports it). For example, the following job will be performed at 2:15 am every day:

RecurringJob.AddOrUpdate("task-id", () => MethodToRun(42, "foo"), "15 2 * * *");

Brief review of Quartz.NET

As an alternative to Hangfire, in .NET framework, we can use Quartz.NET, a port of Java task scheduler – Quartz. Quartz.NET solves similar job, as well as supports any amount of clients (creating jobs) and servers (executing jobs) that use a shared database. However, they differ in their execution process.

I must admit that my first experience with Quartz.NET was not successful. The code from GitHub had not been compiled until I manually fixed references to several missed files and assemblies. There is no splitting into server and client parts in the project. Quartz.NET is applied as a separate DDL. So, if you want the particular instance to add jobs, rather than to execute them, you need to set up the instance.

Quartz.NET is free and allows you to store jobs using in-memory, and popular databases (SQL Server, Oracle, MySQL, SQLite, etc.). The in-memory storage keeps data in the memory of one server process that performs jobs. It is possible to implement several server processes only when storing jobs in the database.

For a synchronization, Quartz.NET uses one common algorithm. For example, by signing up in the QRTZ_LOCKS table, you can simultaneously perform only one scheduler process with a unique ID. A simple status modification in the QRTZ_TRIGGERS table will return the job to be performed.

The job-class in Quartz.NET must implement the IJob interface:

public interface IJob
{
    void Execute(IJobExecutionContext context);
}

With a similar restriction, it is easy to serialize the job: a database stores a full class name, which is enough to get a type of the job class using Type.GetType(name). The JobDataMap class allows passing parameters in the job. In addition, you can modify parameters of the stored job.

As for multithreading, Quartz.NET uses classes of System.Threading: new Thread() (refer to QuartzThread),  custom thread pools, and synchronization using Monitor.Wait/Monitor.PulseAll.

The official documentation does not provide much information on this topic. Have a look at this part on clustering: Lesson 11: Advanced (Enterprise) Features. On the StackOverflow site, a user recommended referring to Guides on Quartz.  Since Java and .NET developers do not support similar API, Quartz.NET does not have many releases and updates.

An example of the client API: registration of the HelloJob recurring job

IScheduler scheduler = GetSqlServerScheduler();
scheduler.Start();

IJobDetail job = JobBuilder.Create<HelloJob>()
    .Build();

ITrigger trigger = TriggerBuilder.Create()
    .StartNow()
    .WithSimpleSchedule(x => x
    .WithIntervalInSeconds(10)
    .RepeatForever())
    .Build();

scheduler.ScheduleJob(job, trigger);

You can find the main features of these schedulers in the table below:

[table id=13 /]

Load testing

To check how Hangfire will cope with a huge number of jobs, I created a simple client that creates jobs per 0.2 s. Each job writes a row with the debugging data in a database. By restricting a client to 100K jobs, I implemented two client instances and one server with dotMemory. Six hours later, I got 200K successful jobs in Hangfire and 200K added rows in the database. The screenshot below shows the profiling service results – two snapshots of memory: before and after their execution:

At the next stages, there were already 20 client processes and 20 server processes. The job execution time was increased and became a random value. However, these processes did not impact Hangfire:

Conclusions

In my opinion, Hangfire is better. It is a free and public service, which cuts costs on development and maintenance of distributed systems. Do you use anything like that? I suggest sharing your thoughts in comments.