12,573,413 members 31,278 online Sign in

articles quick answers discussions community help Searchforarticles,questions,tips

Articles Development Lifecycle Installation General

Running Apache Kafka on Windows without Cygwin

GopalTiwari, 10 Jan 2016 CPOL Rate this:
2.54 6 votes

It will give information about Installing and Configuring Apache Kafka & Zookeeper for Windows Operating System

Introduction
This tutorial aims to provide a step by step guide to run Apache Kafka on a windows OS. This guide will also provide instructions to setup Java & zookeeper. Apache
kafka is a fast & scalable messaging queue, capable of handeling real heavy loads in context of read & write. You can find more about kafka onhttp://kafka.apache.org/.
Apache kafka requires a zookeeper instance running which is used for reliable distributed cordination. Please find more about Zookeeper
onhttps://zookeeper.apache.org/.

Check this video for Kafka setup on windows.

 Running Apache Kafka on Windows without Cygwin

Downloading the Required Files

1. Download Server JRE according to your OS & CPU architecture from http://www.oracle.com/technetwork/java/javase/downloads/jre8downloads2133155.html
2. Download and install 7zip from http://www.7zip.org/download.html
3. Download & extract Zookeeper using 7zip from http://zookeeper.apache.org/releases.html
4. Download & extract Kafka using 7zip from http://kafka.apache.org/downloads.html

Installation
a. Install & Setup Java Runtime Environment JRE

1. Start JRE installation and hit the check box Change destination folder, then click install.
1. Change the installation directory to any path without spaces in folder name. Ex C:\Java\jre1.8.0_xx\. By default it will be C:\Program Files\Java\jre1.8.0_xx then
click Next.
2. Now open system environment variables dialogue by opening Control Panel > System > Advanced system settings > Environment Variables
3. Hit New button in User variables section then type JAVA_HOME in Variable name & give your jre path in Variable value. It should look like:
Java path and version may change according to the version of Kafka you are using

1. Now click ok.

2. Search for Path variable in the System Variable section in Environment Variables dialogue box you just opened.
3. Edit the path and type ;%JAVA_HOME%\bin at the end of the text already written there just like image below
 1. To confirm java installation just open cmd and type java version, you should be able to see version of the java you just installed.

If your command prompt some what looks like the image above, you are good to go. Else you need to recheck whether your setup version is matching with OS
architecture x86, x64 or environment variables path is correct in or not.

b. Installing & Running Zookeeper

1. Goto your zookeeper config directory. For me its C:\zookeeper3.4.7\conf

2. Rename file zoo_sample.cfg to zoo.cfg
3. Open zoo.cfg in any text editor like notepad but Ill prefer notepad++.
4. Find & edit dataDir=/tmp/zookeeper to dataDir=C:\zookeeper3.4.7\data
5. Add entry in System Environment Variables as we did for java

a. Add in System Variables ZOOKEEPER_HOME = C:\zookeeper3.4.7

b. Edit System Variable named Path add ;%ZOOKEEPER_HOME%\bin;

6. You can change the default zookeeper port in zoo.cfg file Default port 2181.
7. Run zookeeper by opening a new cmd & type zkserver.
8. You will see the command prompt with some details like image below.
Congratulations your zookeeper is up & running on port 2181.

c. Setting Up Kafka

1. Go to your Kafka config directory. For me its C:\kafka_2.110.9.0.0\config

2. Edit file server.properties
3. Find & edit line log.dirs=/tmp/kafkalogs to log.dir= C:\kafka_2.110.9.0.0\kafkalogs.
4. If your zookeeper is running on some other machine or cluster you can edit zookeeper.connect=localhost:2181 to your custom IP & port. For this demo we are
using same machine so no need to change. Also Kafka port & broker.id are configurable in this file. Leave other settings as it is.
5. Your Kafka will run on default port 9092 & connect to zookeepers default port which is 2181.

d.Running Kafka Server

Note:

Please ensure that your zookeeper is up & running before starting Kafka server.
 1. Go to your kafka installation directory C:\kafka_2.110.9.0.0\
2. Open a command prompt here by pressing Shift + right click and choose Open command window here option
3. Now type .\bin\windows\kafkaserverstart.bat .\config\server.properties & press enter.

1. If everything went fine your command prompt will look like this one

1. Now your Kafka is up & running, so now you can create topics & produce or consume data from java/scala code & command prompt also.

e. Creating Kafka Topic

1. Now create a topic with name test & replication factor 1 as we have only one Kafka server running. If you have a cluster with more than 1 Kafka servers
running, you can increase the replicationfactor accordingly which will increase the data availability & act like a faulttolerant system.
2. Open a new command prompt in the location C:\kafka_2.110.9.0.0\bin\windows
3. Type following command and hit enter
kafkatopics.bat create zookeeper localhost:2181 replicationfactor 1 partitions 1 topic test

f.Creating a producer & consumer to test server

1. Open a new command prompt in the location C:\kafka_2.110.9.0.0\bin\windows

2. To start a producer type command kafkaconsoleproducer.bat brokerlist localhost:9092 topic test.
3. Again open a new command prompt in the same location as C:\kafka_2.110.9.0.0\bin\windows
4. Now start a consumer by typing command kafkaconsoleconsumer.bat zookeeper localhost:2181 topic test.
5. Now you will have two command prompts like image below

1. Now type anything in the producer command prompt & press enter and you should be able to see the message in the other consumer command prompt.
 1. If you are able to push & see your messages in consumer side, you are done with Kafka setup.

Some Other Useful Commands

1. List Topics: kafkatopics.bat list zookeeper localhost:2181
2. Describe Topic: kafkatopics.bat describe zookeeper localhost:2181 topic [Topic Name]
3. Read messages from beginning: kafkaconsoleconsumer.bat zookeeper localhost:2181 topic [Topic Name] frombeginning
4. Delete Topic: kafkarunclass.bat kafka.admin.TopicCommand delete topic [topic_to_delete] zookeeper localhost:2181

Reference Links
1. http://kafka.apache.org/documentation.html
2. https://zookeeper.apache.org/doc/trunk/zookeeperStarted.html

Conclusion
I am sure, this article will be helpful for Beginners & Intermediary Programmers to Start a kafka Server on their Windows Servers or Desktops.
Yours Friendly

Gopal Tiwari

License
This article, along with any associated source code and files, is licensed under The Code Project Open License
CPOL

Share
EMAIL TWITTER

About the Author

You may also be interested in...

Pro How to Build and Scale with Microservices Pro The Importance of Monitoring Containers

Apache Hadoop for Windows Platform Smarter Security Camera: A POC Using the
Intel IoT Gateway
 Apache Kafka 0.9 Scala Producer/Consumer Connecting to Microsoft Azure with Node
With Some RxScala Pixie Dust RED

Comments and Discussions

You must Sign In to use this message board.

Search Comments Go

Spacing Tight Layout Normal Per page 25 Update

First Prev Next

Error Brokerlist NecroJohnson 7Sep16 20:26

I just made use of this Sacha Barber 15Mar16 23:32

hi Member 11547475 23Feb16 23:43

Re: hi GopalTiwari 24Feb16 22:50

Re: hi Member 11547475 25Feb16 3:07

Re: hi GopalTiwari 25Feb16 5:30

Re: hi Member 11547475 26Feb16 9:49

Re: hi Member 11547475 26Feb16 11:58

Message Removed GopalTiwari 27Feb16 9:05

 Re: hi Member 11547475 28Feb16 0:58

Re: hi GopalTiwari 28Feb16 1:08

Re: hi Member 11547475 2Mar16 9:35

Re: hi Member 11547475 14Mar16 14:03

Java edition? Member 12332048 17Feb16 11:02

kafkapython on Windows? Member 12332048 16Feb16 14:09

This is great Sacha Barber 11Feb16 20:01

Re: This is great GopalTiwari 27Feb16 7:07

kafka in windows bnagarajan 13Jan16 0:50

Re: kafka in windows GopalTiwari 27Feb16 7:11

Uber for cleaning Member 12257127 11Jan16 2:56

Last Visit: 31Dec99 18:00 Last Update: 5Nov16 17:43 Refresh 1

General News Suggestion Question Bug Answer Joke Praise Rant Admin

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

Permalink | Advertise | Privacy | Terms of Use | Mobile Layout: fixed | Article Copyright 2016 by GopalTiwari
Select Language
Web02 | 2.8.161101.1 | Last Updated 11 Jan 2016 fluid Everything else Copyright CodeProject, 19992016