There are a few common challenges involved in designing integrations, and many of them are presented as functional requirements or non-functional requirements. Today we'll see how patterns like publish/subscribe, guaranteed delivery, dead lettering, and time to live - alongside a MuleSoft implementation and services - can help us overcome these challenges.

First Things First

These implementations are for fictional scenarios created solely to give developers a hands-on way to practice solving common challenges. Please note that real projects may have other requirements or restrictions that can change the best way to address the challenge. That said, it will be useful to demonstrate how to read through the requirements, recognize which pattern to apply, and understand how to implement your solution.

Practice Scenario Context

All projects are an implementation of the proposed design using MuleSoft and Anypoint MQ. Consider that this project is receiving the input request from an HTTP source with an entity Person defined in the BODY, being POSTed to a Modern API.  Also, the Anypoint MQ global config is already set up on the Global Elements with the credentials and the destination settings pointing to some exchange with bounded queues hosted in a MuleSoft partner's organization within Anypoint Platform. Any specific setup needed will be mentioned in the particular pattern section.

1. Publish/Subscribe Pattern

This pattern enables a Message producer to deliver one single "Message A" to every queue enlisted to the exchange specified by the source. In business terms, it's most commonly known as broadcasting.

Proposed Solution Drawing

The drawing below was created to help us understand how the requirements are met:

This drawing shows a "Message A" being delivered to a specific message broker, which is addressed to a fanout type exchange. Then one single copy of "Message A" is delivered to Queue 1, one copy is delivered to Queue 2, and one copy is delivered to every queue bound to the exchange.

Proposed Design Implementation

Here we have a screenshot of the flows handling the message according to the pattern:

Apart from logging, we have accomplished the task of sending one message with one event (Publish) to an exchange and broadcasting one copy of the message to all queues, which are being watched by Subscribers 1 and 2. Right after the message lands in the respective queues, the messages are consumed and made available to the flow; and in this case, there are logging events showing the message received with the information of the person to whom it was sent.

The reason that the published message becomes available on both queues (and would become available on as many queues as you set) is that on Anypoint MQ, the exchanges are by default configured with this FANOUT required setting. When using other message brokers, be sure to check whether they support this implementation of exchanges, and, if they do, how to set it up.

Business Outcomes

If you make good use of this pattern, enterprise outcomes include: 

1. A high level of system synchronism where one source continuously sends data to many targets;
2. High performance through parallelism, allowing each subscriber to process your message with its own available resources; and
3. Higher data resilience since persisting your messages in a queue can avoid data loss in some exceptions during the processing.

2. Guaranteed Delivery Pattern

This pattern ensures that a message, once received, won't get lost in the void. In business terms, it's most commonly known as data persistence.

Proposed Solution Drawing

The drawing below was created to help us understand how the requirements are met:

In this drawing, we again see "Message A" being delivered to the message broker and the queue receiving the message through the exchange.  When it is consumed by "Consumer" and assigned to a flow, sometimes an exception may be thrown during processing. The Consumer is notified of the unhandled exception and puts the message back in the queue in the first position rather than at the end of the line. This behaviour is commonly referred to as "redelivery."

Proposed Design Implementation

Here we have a screenshot of the flows handling the message according to the pattern:

The guaranteed delivery pattern is hidden between the Publish and the Subscriber, because what really matters for this pattern is 1) keeping the message in the queue until a proper acknowledgement is given, and 2) keeping the message persisted to allow it to be retrieved even after the server crashes or restarts. 

Within Anypoint MQ, all regular queues are by default hosted across multiple availability zones inside a region and persisted in a durable storage to retain any messages that were already stored. So if one availability zone goes down, the service still operates normally in that region. Proper acknowledgement is also given behind the scenes in this case.

In a MuleSoft flow, when you set the subscriber acknowledge mode to AUTO ACK, it will ack any given message after the end of the listener's flow, and if any unhandled exception is thrown, it will send the reject command.

Feel free to reference this project's source on GitHub for more details.

Business Outcomes

If you make good use of this pattern, enterprise outcomes include: 

1. Higher data resilience since sending your messages to a persisted queue can prevent data loss in unhandled errors during the processing and in cases of server crashes/restarts.
2. Asynchronous integrations: for the message being sent to the message broker, the API will be able to provide a response to the client and release the client's connection while still handling messages through other flows.

Dead Lettering Pattern

This pattern was developed to handle messages that receive any command other than an ACK, as well as messages that don't receive a command at all. 

Proposed Solution Drawing

The drawing below was created to help us understand how the requirements are met:

The consumer is retrieving a message (Message A) and starting its processing. In any case where this message is rejected with no acknowledgement, a copy will be addressed to a Dead Letter Exchange (if there is one set up in the queue).

Proposed Design Implementation

Here we have a screenshot of the flows handling the message according to the pattern:

This flow may look far more complicated than our previous flows. But the difference that is relevant for us here is the acknowledgement mode and redelivery policy set to the Subscriber. When using the MANUAL ack mode, no acknowledge command is sent to the message broker, unless it is explicitly written in a flow. MANUAL ack also implies that it is necessary to consider how to handle a message that may happen to stay in flight due to lack of acknowledgement, since it will consume resources to keep the message waiting for acknowledgement forever.

When implemented with Anypoint MQ, due to the chosen distributed architecture to provide the messaging feature as a service, with everything being handled through HTTP requests to the Anypoint Platform, you can always set a default acknowledgement timeout for every message in every queue, making the message available to other consumers (redelivery) if it exceeds the specified value. If properly set up, the redelivery policy will generate a different exception when the reattempts exceed the threshold. So the developer would use the reject command to trigger the dead lettering, sending the message to the queue assigned as a DLQ to the current source queue.

For more details, check this project's source on GitHub.

Business Outcomes

If you make good use of this pattern, enterprise outcomes include the ability to:

1. Store messages for subsequent analysis that are filtered due to malformation or any invalid content (usually permanent errors) according to an expected input.
2. Set a second flow for messages that do not meet the highest criterion but can still match a lower criterion.
3. Remove a message from a queue for a specified period and then put it back into the original queue to retry (this requires additional implementations).

Time to Live Pattern

This pattern helps us dispose of messages after a certain period of time. In business terms, it's most commonly known as Message Expiration.

Proposed Solution Drawing

The drawing below was created to help us understand how the requirements are met. There is a fictional timeline in these four images, starting with T=0; for every image, you can consider that now T is T+1.

In the first image, at T=0, we can see "Message A" being posted to the message broker and stored in "Queue 1."

In the second image, at T=1, Message A is consumed by the Consumer listening to Queue 1, and in parallel, there is a Message B being posted to the same Queue 1.

In the third image, at T=2, Message B is still in the Queue because message A is still being processed. Meanwhile, there is a new message (Message C) being posted to Queue 1.

In the last image, at T=3, the Consumer is still busy with message A. But due to a time to live set for all messages in Queue 1, message B has expired and is rejected within the message Broker.

Proposed Design Implementation

Here we have a screenshot of the flows handling the message according to the pattern:

In the implementation shown above, the project was set up using a different way of retrieving messages from a queue. It is implemented using the Consumer component instead of the Subscriber component. Note that a Consumer cannot be placed as a source to a flow, so when a flow invokes it, it triggers a start pooling command that will fetch any message available in a given period. If no message is fetched in this period, it throws an ANYPOINT-MQ:TIMEOUT exception. Also, there is a new operation implemented that has a sleep command to delay the response on purpose in order to simulate a scenario in which the message exceeds the time to live timeout.

The time to live pattern is not visible on the implementation because it is also set on the message broker. When you create any queue on Anypoint MQ, one of its mandatory settings is to set the Message TTL, with a minimum value of 1 minute and a maximum value of 14 days. If a message stays in the queue for a longer period of time than the value specified in your settings, it will expire. Note that other message brokers might allow queues without TTL, making queues store messages for as long as is needed.

For more details, check this project's source on GitHub.

Business Outcomes

If you make good use of this pattern, you'll enable systems to distinguish when a message has lost its meaning or utility based on time.

Conclusion

There are many other patterns that you can utilize in your design in order to address both functional and non-functional requirements. I selected these four patterns based on my experience working with MuleSoft. This Snippet was inspired by the content created for the MuleSoft Official SP Community Meetup, which can be found below:

See Messaging Patterns with MuleSoft - Portuguese Session at MuleSoft Meetups São Paulo

Have you ever used one of these patterns to meet a requirement in a solution you designed? Comment below to tell me which patterns you've used most frequently in real-world scenarios!

References

MuleSoft sources:

MuleSoft Documentation | MuleSoft Documentation

Anypoint MQ | MuleSoft Documentation

EIP sources:

Enterprise Integration Patterns - Messaging Patterns Overview

Enterprise Integration Patterns - Publish-Subscribe Channel

Enterprise Integration Patterns - Guaranteed Delivery

Enterprise Integration Patterns - Dead Letter Channel

Enterprise Integration Patterns - Message Expiration