Dragisa Krsmanovic

We have a system that uses Apache Camel and ActiveMQ. It handles periodic bursts of 20,000 messages. The end consumer is slow and it takes about an hour to process all the messages. On their route, messages are passed from one queue to another. On our production system we stumbled onto an unexpected problem. After finishing few thousand messages the whole system would freeze.

The problem was with ActiveMQ flow control and transacted routes in Camel.

When you have slow consumers, to prevent queues from growing infinitely, ActiveMQ has a limit of how many messages you can put in a queue. When limit is reached, producer is, by default, forced to wait until the resources free. You can set these limits on system level or per queue level. Problem arises when you have transacted Camel route and system limit is reached.

Route that moves messages from queue A to queue B is inside a JMS transaction – you can’t remove message from queue A until the message is successfully placed on queue B.

If system limit is reached, no new messages can be sent to any queue. So, producers are forced to wait, transaction doesn’t complete, messages can’t be taken off queue A and no resource gets freed. The  whole processing freezes.

There are numerous ways you can work around this problem. You can turn off flow control and potentially let queues grow indefinitely.

In our case, solution was to set per-queue limits so that system limits can never be reached. Sum of limits for all queues needs to be less than the system limit. That way, as consumer takes messages from queue B, new messages can come in, transactions can complete and messages can be taken off queue A. Messages are consumed from queues A and B at the same pace and the whole system works fine.

In our case, I’ve set memoryLimit to 10m for our 13 queues and system memoryUsage to 180m.

For details see :

• ActiveMQ Producer Flow Control
• ActiveMQ Slow Consumer Handling