We have been using Beanstalkd as our preferred choice for queues to handle requests and processes then asynchronously. Our workers were all written in Php.

Though this worked well for us we would seldom get some issues which would get fixed only after restarting the server. The issue with this approach is that the data stored in the queue would be lost forever as Beanstalkd does not support persistent storage.

Also in the longer run, we needed a solution that supported clustering. So with all this in perspective, we decided to look for alternatives to Beanstalkd.

IronMQ and Amazon SQS

The ideal choice was to go with a cloud service provider. So we decided to explore Amazon SQS and IronMQ.

A rough calculation made us realize that AmazonSQS was not the right choice for us. It had a free tier that did not meet our expectations and the paid tier was way beyond our budget.

IronMQ seemed to be the second choice but they had frequent outages and this was a huge issue for us as we could not compromise on reliability.

Choosing RabbitMQ

With the cloud options being ruled out we turned our attention to the open source community for solutions.

After a few reading and asking our friends and colleagues we narrowed down on RabbitMQ as it solved all our problems at hand.

It supported clustering and persistent storage. This was a home run for us as we were mainly looking to solve this problem.

Without further delay, we modified the config parameters in the env file and made sure our APIs interacted with the RabbitMQ cluster.

To make the infrastructure more resilient we also created separate worker servers and decoupled them from the main application.

Conclusion

It's been a few months of running RabbitMQ and we are satisfied with the overall performance.

RabbitMQ comes with a built-in interface to manage and monitor queues. We loved it. This was something that we always wanted in Beanstalkd.