A Look into Message Brokers: Apache Kafka and RabbitMQ

With devices getting interconnected now, more than ever before, the communication infrastructure needs transformational shifts. The web of connected applications and devices can no longer operate at the backdrop of a traditional communication model.

Now, to tackle the new-age communication challenges among connected devices, we have something called ‘message middleware’. There are forms of middleware communication, and one of the popular ones that we are going to discuss here is a message broker.

What Is a Message Broker?

A message broker is a server-based communication model and is often referred to as an interface engine or integration broker. It acts as a bridge between the production-side and receiver-side, allowing smooth flow of messages from one end to another.

The movement of messages is based on queuing one after another, forming a queue-cluster. Now, when the production-side sends a message, it gets grouped into the queue. The messages are rolled out when the receiver is ready to accept. This ensures that even in the temporary absence of the receiver, there are no messages lost or downtime due to communication lag.

There are various forms of message broker available out there, the most common being Apache Kafka, Rabbit MQ, Amazon MQ, Cloverleaf, IBM Mq, and many more. Here, we will focus on the first two forms of a message broker.

An Insight into RabbitMQ and Apache Kafka

What is RabbitMQ?

RabbitMQ message broker works on the principle of AMQP or Advanced Message Queuing Protocol. It is very different from a normal message broker as it employs an additional element call exchange to simplify the complex flow message between producer and to many receivers i.e one too many.

How Does RabbitMQ Function?

As mentioned, RabbitMQ involves an additional component called exchange in its architecture, which to a considerable extent defines its working. So, in normal cases when there is a single receiver accepting messages, the flow of communication is quite straightforward. But things get trickier when we include more than one receiver.

Now, in situations where there are multiple receivers, an exchange component is added to receive messages and then send them to the queue. Here, the queue component is attached to the receiver with bindings that contain specific binding keys.

When a message is transmitted to the exchange, it has a routing key. Now, how the exchange deals with a particular message depends on the type of exchange. So, let us see the types of exchange under RabbitMQ.

Types of Exchanges

1. Fanout Exchange: In case of fanout, the exchange sends the messages through the queues to the final receiver without paying heed to the routing key.
2. Direct Exchange: This type of exchange is unidimensional, as the exchanges send the message to the queues, and has a similar binding key to the routing key. For instance, if the routing key is ‘Q’, the message will be strictly sent through the binding key ‘Q.
3. Topic Exchange: Topic Exchange is unique in the way keys are appropriated for both the routing and binding structure. Here, the exchange operates on a partial match of words or numbers or phrases. So, if a routing key says R-‘Red, Green’ and the binding key has either a red or green key i.e R-‘ Red’ or R-‘ Green’ message will move through that particular queue.
4. Header Exchange: In the case of header exchange, the transmission of messages through exchanges to queues depends on the message header rather than the routing key carrying that particular message.

There is a special type of exchange in RabbitMQ, called Nameless Exchange. Here, the routing key is mapped to Queues name or key and not the binding key.

Pros

• Easy to set up and work with
• Highly scalable communication solution, as it can handle 1mn messages per second
• Has a simple and intuitive UI
• Operates on standard protocols
• Complete capture the different messaging patterns

Cons

• AMQP model of communication is complex for beginners
• Written in erlang, which increases the complexity as the source is tough to read due to its unfamiliar syntax

What is Apache Kafka?

Apache Kafka is more just a message broker, it is an enterprise transport mechanism that moves data quickly from the source to the target system. Initially, when the data needs to be transmitted from five source systems to six target systems, a total of thirty integration needs to be created, which itself is a challenge.

So, using Apache Kafka is a complete no-brainer, when you are working with multiple sources & target systems. Let us see some use cases, alongside the pros and cons of this modern message broker, originating for the family of Linkedin.

Use Cases

• Activity tracking 
• Messaging system 
• Accumulating applications logs
• Gathering crucial metrics form different operational processes
• Decoupling of system dependencies. 

Pros

1. Has a low latency rate of 10ms, which results from the decoupling of messages from the source
2. Low latency results in high throughput for Apache Kafka, which also  indicates that kafka can handle high volume messages flow with high velocity
3. Is resistant to the failure in the messaging nodes or system, and that is why it is attributed to be fault tolerant
4. Data stored in the kafka can be easily accessed by anyone with permission clearance
5. Iis highly scalable as it can handle large chunks of data transmission simultaneously

Cons

1. Has performance lag as the number of queues increases
2. Lacks diversity monitoring and data management tools
3. During message alteration, the performance of Apache Kafka takes a dip

Conclusion

Companies dealing in various domains have completely changed their data stream model with the arrival of message broker software.  For instance, big players like Netflix, and Uber use Apache Kafka. On the other side, some companies are left behind in the technology adoption race.

Here, carrying the brand of leading IT consulting firms, we, at Brainvire feel the responsibility to introduce these new technologies to smaller and mid-sized companies. We provide a wide array of IT solutions from cloud service, to digital marketing of top app development.