Buildly Core with Kafka

Getting real-time actionable insights from processed data is an important pattern for modern data analytics. Constant information investigation helps in making significant decisions at the right time. Following the same idea in the previous posts, in this article, we are providing a tutorial of how to connect Buildly Core to an Apache Kafka cluster. Apache Kafka is used to create real-time streaming data analytics, reduce the complexity of data pipelines and make the communication between systems simpler and manageable.

Kafka

“Apache Kafka is an open-source stream-processing software platform developed by LinkedIn and donated to the Apache Software Foundation, and is written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.”[1]

“Kafka® is primarily used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies”.[2]

To learn more about each piece of Apache Kafka, check the intro section of their website. And for a complete guide, the Kafka documentation also does an excellent job.

Before Starting

First of all, we need to have a Buildly Core instance up and running, so we can connect the service we are going to create to publish data to and consume data from a Kafka cluster. If you haven’t used it before, here is the Quickstart documentation to get a local instance of the Buildly Core up and running.

Running Kafka and Zookeeper

In our example, we run everything as a docker container and this is the Apache Kafka image we are using. If you need more information about how to run the Apache Kafka image, you can access the GitHub repository and go through one of the examples.

To run our Kafka Docker container, we also need to have an Apache Zookeeper instance and this is the Zookeeper image we are using.

“ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.”[3]

If you are using Docker Compose and want to skip all this process of deploying and configuring Kafka and Zookeeper, you can use the following docker-compose file that is already configured to run them with the images mentioned above:

Connecting Buildly Core to Kafka

We have created a service called Kafka Service and published it to Buildly Marketplace. This service is responsible for connecting to a Kafka cluster, publishing data that it receives via HTTP requests, and consumes data from a previously specified topic via HTTP requests.

Let’s configure our Kafka service before connecting it to Buildly Core. We have previously created a topic inside our Kafka cluster called Topic1, where we’re going to publish and consume data. The service has only three environment variables to configure the connection to Kafka as we can see below:

After configuring and running our service, we need to connect Buildly Core to it. For that, we have to send the following POST request to Buildly Core with our Admin credentials, so the service is registered.

Let’s check the Buildly Core Swagger specification, so we can see if the IoT service is accessible via Buildly Core.

Publishing and Consuming Data

Now, we can publish data to our Kafka cluster. So, let’s do it!

We are going to send the following data to our Kafka cluster, and for that, we need to send a POST request to the endpoint kafka/data/ as described below.

We sent our first data to our Kafka service with success. To verify if the message was successfully published to the Kafka cluster, we are going to consume it. We will send the following GET request to the endpoint kafka/data/ as described below to retrieve the published data.

And this is the response:

Conclusion

This was a simple and short example of how we can connect a Kafka cluster to Buildly Core using a service to handle the connection. We can do much more and even integrate with the Data Mesh, so data from different sources can be aggregated.

References

  1. Apache Kafka – https://en.wikipedia.org/wiki/Apache_Kafka
  2. Apache Kafka – https://kafka.apache.org/
  3. Apache Zookeeper – https://zookeeper.apache.org/