Hazelcast Jet is a distributed, high-performance stream processing engine built on top of the Hazelcast in-memory data grid. It provides an easy-to-use API for processing large volumes of data in real-time. In this blog post, we will explore how to work with Hazelcast Jet sources and sinks in Java using the Hazelcast Jet library.

What are Sources and Sinks?

In Hazelcast Jet, a source is a component that generates data and feeds it into the Jet streaming engine. It could be a Kafka topic, a Hazelcast map, or any other data source. On the other hand, a sink is a component that consumes the output data produced by Jet. It could be a Hazelcast map, a database table, or any other data sink.

Working with Sources

Hazelcast Jet provides various built-in sources that you can use out of the box. To create a source, you will typically use a builder pattern and specify the necessary configuration options. Let’s take a look at an example of creating a Kafka source:

import com.hazelcast.jet.pipeline.Pipeline;
import com.hazelcast.jet.pipeline.Sources;
import org.apache.kafka.common.serialization.StringDeserializer;

// Create a Kafka source
Pipeline p = Pipeline.create();
p.readFrom(Sources.kafka("topic-name")
    .properties(properties)
    .keyDeserializer(StringDeserializer.class)
    .valueDeserializer(StringDeserializer.class))
    .map(record -> record.value())
    .writeTo(Sinks.logger());

In this example, we create a Kafka source by calling Sources.kafka("topic-name") and then configure the necessary properties such as the key and value deserializers. We then use the source in the pipeline by calling readFrom(source). After processing the data, we can write the output to a sink using writeTo(sink).

Working with Sinks

Similar to sources, Hazelcast Jet provides various built-in sinks that can be easily integrated into your pipeline. Let’s take an example of writing data to a Hazelcast map:

import com.hazelcast.jet.pipeline.Pipeline;
import com.hazelcast.jet.pipeline.Sinks;

// Create a Hazelcast map sink
Pipeline p = Pipeline.create();
p.readFrom(Sources.list("input-list"))
    .map(word -> word.toUpperCase())
    .writeTo(Sinks.map("output-map"));

In this example, we start the pipeline by reading data from a source, in this case, a list. We then apply a transformation to convert each word to uppercase using the map operation. Finally, we write the transformed data to a Hazelcast map sink by calling writeTo(Sinks.map("output-map")).

Conclusion

In this blog post, we explored how to work with Hazelcast Jet sources and sinks in Java using the Hazelcast Jet library. Sources are used to fetch data from various external systems, while sinks are used to write the output data produced by Jet. Hazelcast Jet provides a rich set of built-in sources and sinks, making it easy to integrate with different data sources and sinks in your applications. By leveraging the power of Hazelcast Jet, you can easily process and analyze large volumes of streaming data in real-time.

#JetStreaming #HazelcastJet