This is a brief manual for a workshop that taken place online at https://youtube.com/confluent

Workshop prerequisites and setup

Prerequisites

Ensure you install the following toolset on your computer:

Before you proceed, be sure to complete the following steps:

Getting code

git clone https://github.com/confluentinc/demo-scene            (1)
cd streaming-movies-workshop                                    (2)
1Clone the repository
2Change directory of the workshop folder
Getting only what you need

If you follow steps below, you should check out only directory that has source code relevant to this post.

mkdir ~/temp/demo-scene
cd ~/temp/demo-scene
git init .
git config core.sparsecheckout true
git remote add origin -f https://github.com/confluentinc/demo-scene/
echo "streaming-movies-workshop/*" >> .git/info/sparse-checkout
git pull --depth=1 origin master
cd streaming-movies-workshop
ls -lh
If you are on Mac, you can use brew to install all dependencies by running make install-deps.

☸️ Kubernetes

You can try to deploy apps to local Kubernetes clusters. There are plenty of options available - minikube, k3d, Docker for Desktop. Frankly, I was having hard times to use those. You can try local minikube cluster with make create-local-minikube-cluster. In this tutorial, I will use Google Kubernetes Service to run my test apps. If you want to follow same route you need to install Google Cloud SDK tools. You can create GKE Kubernetes cluster by calling make create-gke-cluster command. You can destroy GKE Kubernetes cluster after that by calling make destroy-gke-cluster.

0️⃣ Provisioning Confluent Cloud cluster

$ ccloud login --save       (1)
$ make create-ccloud-cluster  (2)
1Login to your Confluent Cloud account.
2The CCloud Stack script will ask you to log in to your CCloud account.

It will automatically provision Kafka and ksqlDB cluster.

1️⃣ Build and Deploy apps

This workshop includes two apps - the microservices developed with Spring Boot. - movies-generator - loads movie data to Kafka cluster, and randomly generates new ratings. - ratings-processor - processes new ratings, and constantly recalculates new rating for given movie.

Build and Smoke test
./gradlew test  (1)
1This command will download gradle wrapper (if it wasn’t previously installed )

Explore applications

  • Model generation from AVRO Schema (Movie, Rating)

  • Producer application using KafkaTemplate

  • Ratings Processor App

deploy to Kubernetes
skaffold run (1)
1This command will build images for

2️⃣ Create Materialized view

Connect to ksqlDB with CLI

In this exercise, we’re going to use ksqlDB Cloud UI. But you also can run CLI using docker.

docker run -it confluentinc/ksqldb-cli:0.13.0 ksql -u $KSQL_API_KEY -p $KSQL_API_SECRET $KSQLDB_ENDPOINT
Materialized view
CREATE STREAM RATED_MOVIES_STREAM WITH (
    kafka_topic = 'RATED_MOVIES',
    value_format = 'avro'
);

CREATE TABLE RATED_MOVIES_VIEW AS SELECT
  RATED_MOVIES_STREAM.TITLE as TITLE,
  LATEST_BY_OFFSET(RELEASE_YEAR) as RELEASE_YEAR,
  LATEST_BY_OFFSET(MOVIE_ID) as MOVIE_ID,
  LATEST_BY_OFFSET(RATING) as CURRENT_RATING
FROM RATED_MOVIES_STREAM
GROUP BY TITLE
EMIT CHANGES;

If you are getting error about accessing the movies and ratings topic you need grant access to Kafka topic movies and ratings to ksqlDB server with command

CCLOUD_KSQL_ID=`ccloud ksql app list -ojson | jq -r '.[0].id'`
CCLOUD_KAFKA_ID=`ccloud kafka cluster list -ojson | jq -r '.[0].id'`
ccloud ksql app configure-acls ${CCLOUD_KSQL_ID} "*" --cluster ${CCLOUD_KAFKA_ID}

# or
ccloud ksql app configure-acls `ccloud ksql app list -ojson | jq -r '.[0].id'` "*" --cluster `ccloud kafka cluster list -ojson | jq -r '.[0].id'`

where ccloud ksql app list -ojson | jq -r '.[0].id' gives your id of ksqlDB cluster and ccloud kafka cluster list -ojson | jq -r '.[0].id' gives you id of Kafka cluster