Java demo repository | Difficulty: Intermediate Java | Format: Seven chapters with code and exercises Build a Java assistant one capability at a time, following the rounds of Codepocalypse Now: LangChain4j vs. JetBrains Koog. Each chapter explains the API that introduces the next capability, gives you an experiment to run, and identifies a pattern you can use in your own application. The examples use LangChain4j builders and Java interfaces directly. ...
GitHub Repository | Duration: 2 hours | Difficulty: Beginner to Advanced Overview This hands-on workshop walks you through building a complete real-time data pipeline using Confluent Cloud. You’ll stream live cryptocurrency price data from the CoinGecko API, process it with Apache Flink SQL, and materialize it as Apache Iceberg tables using Tableflow — all queryable through DuckDB. What You’ll Learn Set up and configure a Kafka cluster on Confluent Cloud Create topics and ingest live data using HTTP Source Connector Materialize Kafka topics as Iceberg tables with Tableflow Write Flink SQL for real-time stream processing Query real-time and historical data with DuckDB ...
Building an API is just the first step. You also need to deploy that API and help your customers onboard to drive API consumption. Design APIs in Insomnia, generate configurations for Kong Gateway and publish REST, GraphQL, and gRPC services to the Kong Dev Portal to enable your audience. In this workshop, we will go through all steps of the API Management cycle - from designing API specifications to publishing APIs for public consumption. ...
How can I implement an average aggregation that implements incremental functions, namely count and sum? Kafka Streams natively supports incremental aggregation functions, in which the aggregation result is updated based on the values captured by each window. Incremental functions include count, sum, min, and max. An average aggregation cannot be computed incrementally. However, as this tutorial shows, it can be implemented by composing incremental functions, namely count and sum. Consider a topic with events that represent ratings of movies. In this tutorial, we’ll write a program that calculates and maintains a running average rating for each movie. ...
This is a brief manual for a workshop that taken place online at https://youtube.com/confluent Workshop prerequisites and setup Prerequisites Ensure you install the following toolset on your computer: Confluent Cloud You should have your login and password information handy after you sign up for Confluent Cloud. The ccloud init script will ask you for your login information. Confluent Cloud CLI Docker (We use Docker to build images locally) install Docker Desktop for MacOS Git Java 11 (or later) Your favorite IDE or text editor Personally, I recommend IntelliJ IDEA. kubectl k3d to run a local Kubernetes cluster jq for fancy json manipulation skaffold to build, run and deploy images k9s a fancy console GUI for Kubernetes Before you proceed, be sure to complete the following steps: Getting code git clone https://github.com/confluentinc/demo-scene (1) cd streaming-movies-workshop (2) 1 Clone the repository 2 Change directory of the workshop folder Getting only what you need If you follow steps below, you should check out only directory that has source code relevant to this post. mkdir ~/temp/demo-scene cd ~/temp/demo-scene git init . git config core.sparsecheckout true git remote add origin -f https://github.com/confluentinc/demo-scene/ echo "streaming-movies-workshop/*" >> .git/info/sparse-checkout git pull --depth=1 origin master cd streaming-movies-workshop ls -lh If you are on Mac, you can use brew to install all dependencies by running make install-deps. ...
This is a playbook of the workshop on running Event-driven microservices on PKS (together with Pivotal) The code of the example application is available on Github. Table 1. Revisions history Version Date Comments v1.1 02/11/2020 Confluent + Pivotal Workshop in NYC 2020 v1.0 10/7/2019 Initial revision for Spring One Platform 2019 Prerequisites Java 8+ Confluent Platform 5.4 or newer Optional: Confluent Cloud account ...
This is a playbook of the workshop on running Event-driven microservices on PKS (together with Pivotal) The code of the example application is available on Github. Table 1. Revisions history Version Date Comments v1.2 02/18/2020 Confluent + Pivotal Workshop in Chicago 2020 v1.1 02/11/2020 Confluent + Pivotal Workshop in New York 2020 v1.0 10/7/2019 Initial revision for Spring One Platform 2019 Prerequisites Java 8+ Confluent Platform 5.4 or newer Optional componenrs: Confluent CLI for local testing. Confluent Cloud account for cloud testing ...
This is a playbook of the workshop delivered at SpringOne Platform 2019 in Austin, TX. The code of the example application is available on Github. Prerequisites Java 8+ Confluent Platform 5.3 or newer Optional: Confluent Cloud account Let’s start building the app As always, we’ll begin by generating a project starter. In this starter, you should enable "Spring for Apache Kafka", "Spring for Apache Kafka Streams", "Cloud Stream" and "Spring Web Starter." Next, update your pom.xml to add support for Schema Registry and Avro. Confluent provides Avro serializers that integrate with Schema Registry. <project> <dependencies> <!-- other dependencies --> <dependency> <groupId>io.confluent</groupId> <artifactId>kafka-schema-registry-client</artifactId> (1) <version>5.3.0</version> </dependency> <dependency> <groupId>org.apache.avro</groupId> <artifactId>avro</artifactId> (2) <version>1.8.2</version> </dependency> <dependency> <groupId>io.confluent</groupId> <artifactId>kafka-avro-serializer</artifactId> (3) <version>5.2.1</version> </dependency> <dependency> <groupId>io.confluent</groupId> <artifactId>kafka-streams-avro-serde</artifactId> <version>5.3.0</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> </exclusions> </dependency> </dependencies> <repositories> <!-- other maven repositories the project --> <repository> <id>confluent</id> <url>https://packages.confluent.io/maven/</url> (4) </repository> </repositories> <plugins> <!-- other maven plugins in the project --> <plugin> <groupId>org.apache.avro</groupId> <artifactId>avro-maven-plugin</artifactId> <version>1.8.2</version> <executions> <execution> <phase>generate-sources</phase> <goals> <goal>schema</goal> </goals> <configuration> <sourceDirectory>src/main/resources/avro</sourceDirectory> (5) <outputDirectory>${project.build.directory}/generated-sources</outputDirectory> <stringType>String</stringType> </configuration> </execution> </executions> </plugin> </plugins> </project> 1 Confluent Schema Registry client 2 Avro dependency 3 Avro SerDes 4 Confluent Maven repository 5 Source directory where you put your Avro files and store generated Java POJOs The architecture of a Spring Boot application Your application will include the following components: use.avsc: an Avro file where we define a schema for our domain model. SpringAvroApplication.java: the starting point of your application. This class also includes configuration for the new topic that your application is using. Producer.java: a component that encapsulates the Kafka producer Consumer.java: a listener of messages from the Kafka topic KafkaController.java: a RESTful controller that accepts HTTP commands in order to publish a message in the Kafka topic Creating a user Avro file { "namespace": "io.confluent.developer", (1) "type": "record", "name": "User", "fields": [ (2) { "name": "name", "type": "string", "avro.java.string": "String" }, { "name": "age", "type": "int" } ] } 1 An avro-maven-plugin will generate the User POJO in the io.confluent.developer package. 2 This POJO has name and age properties. Creating a Spring Boot application class @SpringBootApplication public class SpringAvroApplication { @Value("${topic.name}") (1) private String topicName; @Value("${topic.partitions-num}") private Integer partitions; @Value("${topic.replication-factor}") private short replicationFactor; @Bean NewTopic newTopic() { (2) return new NewTopic(topicName, partitions, replicationFactor); } public static void main(String[] args) { SpringApplication.run(SpringAvroApplication.class, args); } } 1 These are the topic parameters injected by Spring from application.yaml file. 2 Spring Boot creates a new Kafka topic based on the provided configurations. As an application developer, you’re responsible for creating your topic instead of relying on auto-topic creation, which should be false in production environments. Creating a producer component @Service @CommonsLog(topic = "Producer Logger") public class Producer { @Value("${topic.name}") (1) private String TOPIC; private final KafkaTemplate<String, User> kafkaTemplate; @Autowired public Producer(KafkaTemplate<String, User> kafkaTemplate) { (2) this.kafkaTemplate = kafkaTemplate; } void sendMessage(User user) { this.kafkaTemplate.send(this.TOPIC, user.getName(), user); (3) log.info(String.format("Produced user -> %s", user)); } } 1 A topic name will be injected from application.yaml. 2 Spring will initialize KafkaTemplate with properties provided in application.yaml. 3 We will send messages to the topic using user’s name as the key. Spring instantiates all these components during the application startup, and the application becomes ready to receive messages via the REST endpoint. The default HTTP port is 9080 and can be changed in the application.yaml configuration file. Creating a consumer component @Service @CommonsLog(topic = "Consumer Logger") public class Consumer { @Value("${topic.name}") (1) private String topicName; @KafkaListener(topics = "users", groupId = "group_id") (2) public void consume(ConsumerRecord<String, User> record) { log.info(String.format("Consumed message -> %s", record.value())); } } 1 The topic name will be injected from the application.yaml. 2 With the @KafkaListener annotation, a new consumer will be instantiated by the spring-kafka framework. Creating the KafkaController component @RestController @RequestMapping(value = "/user") (1) public class KafkaController { private final Producer producer; @Autowired KafkaController(Producer producer) { (2) this.producer = producer; } @PostMapping(value = "/publish") public void sendMessageToKafkaTopic(@RequestParam("name") String name, @RequestParam("age") Integer age) { this.producer.sendMessage(new User(name, age)); (3) } } 1 KafkaController is mapped to the /user HTTP endpoint. 2 Spring injects the producer component. <3 >When a new request comes to the /user/publish endpoint, the producer sends it to Kafka. Running the example Prerequisites In this guide, I assume that you have the Java Development Kit (JDK) installed. If you don’t, I highly recommend using SDKMAN! to install it. You’ll also need Confluent Platform 5.3 or newer installed locally. If you don’t already have it, follow the Confluent Platform Quick Start. Be sure to install the Confluent CLI as well (see step 4 in this section of the quick start). Start Kafka and Schema Registry confluent local start schema-registry The Confluent CLI provides local mode for managing your local Confluent Platform installation. The Confluent CLI starts each component in the correct order. TBD sample output Building and running your Spring Boot application In the examples directory, run ./mvnw clean install -DskipTests=true to compile and produce a runnable JAR. After that, you can run the following command: java -jar target/kafka-avro-0.0.1-SNAPSHOT.jar Testing the producer/consumer REST service For simplicity, I like to use the curl command, but you can use any REST client (like Postman or the REST client in IntelliJ IDEA to): curl -X POST -d 'name=vik&age=33' http://localhost:9080/user/publish 2019-06-06 22:52:59.485 INFO 28910 --- [nio-9080-exec-1] Producer Logger : Produced user -> {"name": "vik", "age": 33} 2019-06-06 22:52:59.559 INFO 28910 --- [ntainer#0-0-C-1] Consumer Logger : Consumed message -> {"name": "vik", "age": 33} Running the application using Confluent Cloud To use this demo application with Confluent Cloud, you are going to need the endpoint of your managed Schema Registry and an API key/secret. Both can be easily retrieved from the Confluent Cloud UI once you select an environment. At least one Kafka cluster must be created to access your managed Schema Registry. Once you select the Schema Registry option, you can retrieve the endpoint and create a new API/secret. An example Confluent Cloud configuration can find in application-cloud.yaml: topic: name: users partitions-num: 6 replication-factor: 3 server: port: 9080 spring: kafka: bootstrap-servers: - mybootstrap.confluent.cloud:9092 (1) properties: # CCloud broker connection parameters ssl.endpoint.identification.algorithm: https sasl.mechanism: PLAIN request.timeout.ms: 20000 retry.backoff.ms: 500 sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username="ccloud_key" password="ccloud_secret"; (2) security.protocol: SASL_SSL # CCloud Schema Registry Connection parameter schema.registry.url: https://schema-registry.aws.confluent.cloud (3) basic.auth.credentials.source: USER_INFO (4) schema.registry.basic.auth.user.info: sr_ccloud_key:sr_ccloud_key (5) consumer: group-id: group_id auto-offset-reset: earliest key-deserializer: org.apache.kafka.common.serialization.StringDeserializer value-deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer producer: key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer template: default-topic: logging: level: root: info 1 Cloud bootstrap server 2 Broker key and secret 3 Confluent Cloud Schema Registry URL 4 Schema Registry authentication configuration 5 Cloud Schema Registry key and secret To run this application in cloud mode, activate the cloud Spring profile. In this case, Spring Boot will pick up application-cloud.yaml configuration file that contains the connection to data in Confluent Cloud. java -jar -Dspring.profiles.active=cloud target/kafka-avro-0.0.1-SNAPSHOT.jar ...
Developing Event-driven Microservices with Spring Boot, Confluent Cloud, Kotlin, and Java. Workshop prerequisites and setup Prerequisites Ensure you install the following toolset on your computer: Confluent Cloud You should have your login and password information handy after you sign up for Confluent Cloud. The ccloud init script will ask you for your login information. Confluent Cloud CLI Docker install Docker Desktop for MacOS Docker Compose (installed with Docker Desktop) Git Java 11 (or later) Your favorite Java IDE or even text editor Personally, I recommend IntelliJ IDEA. Setup Before you proceed, be sure to complete the following steps: Getting code git clone https://github.com/gAmUssA/springone2020-workshop (1) cd springone2020-workshop (2) 1 Clone the repository 2 Change directory of the workshop folder ...