MuleSoft Connected with Oracle Streaming Service using the Kafka Connector
Apache Kafka is a very popular streaming service, and I could say that it is the leader in the market. Is part of the Apache Initiative, but vendors like Confluent offer a managed service that can be very powerful for organizations who are looking to start using the platform, without the need to control/manage the underlined infrastructure.
But there are other options for streaming services that can be also useful for organizations who don’t want to get too much into the Kafka complexity. And in my case, since I am constantly working with Oracle Cloud Infrastructure, I’ve found a very interesting option for Kafka in Oracle Streaming Service.
Oracle Streaming Service is a managed service running on top of Oracle Cloud. Is a serverless service, that users just need to provision and start creating streams/partitions that can be used for multiple purposes.
With MuleSoft, it is common to have the need to connect to a Kafka Topic to consume a stream of messages. I’ve found that a very common scenario, for example in Retail, where they need to constantly send information from their Point-of-Sale to a central location. And if you think of a retailer who has multiple branches all over the country, then a stream makes a lot of sense. And for integration purposes, to have MuleSoft in the middle to enrich, transform and ultimately push the information to a system-of-records, then this mix of platforms offers you a very strong alternative.
In this article we will show how to connect MuleSoft with Oracle Streaming Service, using the Kafka compatibility mode that this Oracle service offers.
The first thing we need to do is to create a stream in Oracle Cloud:
That is a very straight-forward step, we just need to click on that blue button and fill in these parameters:
The name for your stream
Select the compartment where you want to create the stream
Create a New Stream Pool
Define the number of partitions
And that’s it. You don’t need to worry about anything extra.
Now, let’s move to MuleSoft.
Imagine we need to create a MuleSoft application that consumes the stream that we have just created.
MuleSoft does not have a connector to OCI Streams, but it does have a connector for KAFKA. Let’s look to a very simple application:
As you can see, we are using the KAFKA connector. We have the Message listener processor to listen into the stream and a simple Logger processor to print in the log file the payload/message.
Now let’s see what the connector connection is going to ask us:
The first thing to highlight is the Connection type. It offers different options:
a) Consumer Plaintext Connection
b) Consumer Kerberos Connection
c) Consumer SASL/PLAIN Connection
d) Consumer SASL/SCRAM Connection
e) Consumer SASL/TOKEN Connection
The one we need to use is option c.
Also notice that we need to configure the bootstrap server, and as you can see, we’ve already filled in that information. We will show you where we get that info in the next paragraphs.
After that, we need to define our Group ID. That is something you need to define, in my case: test-consumer-group.