Oracle Database
ORDERS
Oracle
Database
5
DVX_ORDERS
Oracle Database
ORDERS
Oracle
Database
6
µ Oracle Application
DVX_ORDERS
Container Cloud
Oracle Database
ORDERS
Oracle
Locally running
Node application
µ Database
ORDERS
Oracle
Locally running
Node application
µ Database
9
What does the Twitter for System Driven Event Interaction
look like?
Producers
Producers
Topic
KAFKA TERMINOLOGY
Message
• Topic Key
Value
• Message Time
• Broker Broker
• Producer Topic
• Consumer
Key
Value Consumers
Time
Producers
Broker
Topic
CONSUMING
• Messages are available to consumers only when they have been committed
• Kafka does not push
• Unlike JMS
• Read does not destroy
• Unlike JMS Topic
• (some) History available
• Offline consumers can catch up
• Consumers can re-consume from the past
• Delivery Guarantees
• Ordering maintained
• At-least-once (per consumer) by default; at-most-once and exactly-once can be
implemented
Key
Value Consumers
Time
Producers
Broker
Topic
Consumers
tcp
Producers
tcp
Broker
Topic
WHAT’S SO SPECIAL?
• Durable
• Scalable
• High volume
• High speed
• Available
• Distributed
• Open
• Quick start
• Free (no license costs)
• “Self Fulfilling Prophecy” (positive feedback loop)
CONFLUENT == ENTERPRISE KAFKA
• Freemium model
• Support
• Training
• Confluent Cloud
• Platform and Tools
<CTRL F5>
F5
F5 Application
F5
CTRLF5
Server
CTRLF5
CTRLF5
F5
F5 Application
F5
CTRLF5
Server
CTRLF5
CTRLF5
F5
F5 Application
F5
CTRLF5
Server
CTRLF5
CTRLF5
FAST DATA AND ACTIVE UI
• Handle influx
• Publish findings instantaneously
• Update UI & notify end user immediately
• Analyze in real time
• Decoupled components
• No data loss when a component is temporarily down
• Scalable with volume of events and of number of clients
THE CASE AT HAND
Tweets on
#ukoug17
#ukoug_tech17
#ukoug_apps17
#ukoug_jde17
Show live
tweet feed
for Allow users
conferences like tweets
to
–and show Show live
Client live list of tweet
liked tweets
aggregates
Client per
conference Show a live
list of top 3
Client liked tweets
Client per
conference
DEMO - REAL TIME, CROSS CLOUD, CROSS
TECHNOLOGY PUSH
Tweets on
#ukoug17
you #ukoug_tech17 Client
#ukoug_apps17
#ukoug_jde17
Client
Client
Client
THE CASE AT HAND – STEP ONE
Tweets on
#ukoug17 #ukoug_tech17
#ukoug_apps17
#ukoug_jde17
Tweets
Topic
Show live
tweet feed
for
Client conferences
Client
Client
Client
THE CASE AT HAND – STEP ONE AND TWO
Tweets on
#ukoug17 #ukoug_tech17
#ukoug_apps17
#ukoug_jde17
Tweets
Topic
Show live
tweet feed
for
Client conferences
Client
Client
Client
KAFKA CONSUMER IN NODE
GET EVENTS PUSHED INTO APPLICATION
THE CASE AT HAND
SERVER SENT EVENTS FOR PUSH BACK
Tweets on
#ukoug17 #ukoug_tech17
#ukoug_apps17
#ukoug_jde17
Tweets
Topic
Client
Server Sent
Event
Client
Client
SERVER SENT EVENT – SERVER SIDE
Client
Client
Server Sent
Event
Client
Client
SERVER SENT EVENT – CLIENT SIDE
Client
Client
Server Sent
Event
Client
Client
LIVE TWEET STREAM
Server Sent
Event
THE CASE AT HAND
TWEET LIKES – CLIENT TO SERVER TO ALL CLIENTS
Tweets on
#ukoug17 #ukoug_tech17
#ukoug_apps17
#ukoug_jde17
Tweets
Topic
Show live
Allow users
tweet feed
for to like tweets
–and show
conferences
live list of
Client liked tweets
Client
SS
Client E
Client
THE CASE AT HAND
WEB SOCKETS – FOR BI DIRECTIONAL PUSH
Tweets on
#ukoug17 #ukoug_tech17
#ukoug_apps17
#ukoug_jde17
Tweets
Topic
Show live
Allow users
tweet feed
for to like tweets
–and show
conferences
live list of
Client liked tweets
Client WebSockets
SSE
Client
Client
TWEET LIKES BROADCASTING
WebSockets
WebSockets
THE CASE AT HAND
STREAMING ANALYSIS OF TWEET EVENTS
Tweets on
#ukoug17 #ukoug_tech17
#ukoug_apps17
#ukoug_jde17
Tweets
Show live Topic
Allow users
tweet feed
Show live
to like tweets
for
–and showtweet
conferencesaggregates
live list of
liked tweetsper
Client conference
Client WebSockets
SSE
Client
Client
THE CASE AT HAND - STREAMING ANALYSIS OF TWEETS
Tweets on
#ukoug17 #ukoug_tech17
#ukoug_apps17
#ukoug_jde17
Tweets tweetAnalytics
Show live Topic Topic
Allow users
tweet feed
Show live
to like tweets
for
–and showtweet
conferencesaggregates
live list of
liked tweetsper
Streaming
Tweets
µ
Client conference Aggregation
Client WebSockets
Client SSE
Client
KAFKA STREAMS
Aggregate
Topic
Join
Map
(Xform)
Topic
Publish
EXAMPLE OF KAFKA STREAMS
Topic Set Conference as key
groupBy
Sum/Avg/Top3 by key
TweetMessage (==conference)
Conference Aggregate
Text and possibly per time
Author window
Hashtag Topic
Latest Conference
Join
Details
As JSON
Publish
Topic: CountTweetsPerConference
KAFKA STREAMS –
RUNNING COUNT TWEETS PER CONFERENCE
STREAMING TWEET ANALYTICS
PUSHED TO CLIENTS
Server Sent
Event
THE CASE AT HAND - STREAMING ANALYSIS
OF TWEET LIKES
Tweets on
#ukoug17 #ukoug_tech17
#ukoug_apps17
#ukoug_jde17
Client WebSockets
Client SSE
Client
KSQL FOR DECLARATIVE STREAM ANALYTICS
THROUGH CONTINUOUS QUERIES
Client SSE
Client
WEBSOCKETS – SERVER SIDE
RUNNING TOP 3 OF
BEST LIKED TWEETS PER CONFERENCE
Server Sent
Event
END TO END FLOW CLOUD ENABLED
Cache
API
EventHub CS
API
µ Tweets Likes
Tweet
Count
Likes
Top3
UI µ
Client
Chrome
Tweets µ Likes µ
Client Aggregation Aggregation
Firefox
Key aspects of this demo – What Kafka can do for you
• Open
• Scalable
• Reliable & Available
• Fast
• Complete historical record
• => == =>
• : lucas-jellema
• : www.amis.nl, info@amis.nl