IT · Linux · Uncategorized

Kafka on CentOS Google CLoud

  1. Install Kafka

  2. Open firewall

firewall-cmd  --permanent --zone=public --add-port=2181/tcp

firewall-cmd --permanent --zone=public --add-port=9092/tcp

firewall-cmd --reload

firewall-cmd --info-zone=public

3. Create firewall rule on Google cloud:

B1: Google Cloud PlatForm –> VPC Network –> Firewall Rule

B2: click Create Firewall Rule

B3: Fill information

  • Name
  • Description
  • Target : All instances in the network
  • Protocol and ports: check tcp, and fill port

B4: Create Rule

4. Creating Systemd Unit Files án Starting the Kafka Server 

Link Source: https://www.digitalocean.com/community/tutorials/how-to-install-apache-kafka-on-centos-7

In this section, we will create systemd unit files for the Kafka service. This will help us perform common service actions such as starting, stopping, and restarting Kafka in a manner consistent with other Linux services.

Zookeeper is a service that Kafka uses to manage its cluster state and configurations. It is commonly used in many distributed systems as an integral component. If you would like to know more about it, visit the official Zookeeper docs.

Create the unit file for zookeeper:


sudo vi /etc/systemd/system/zookeeper.service

Enter the following unit definition into the file:


/etc/systemd/system/zookeeper.service
[Unit]
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
User=kafka
ExecStart=/home/kafka/kafka/bin/zookeeper-server-start.sh /home/kafka/kafka/config/zookeeper.properties
ExecStop=/home/kafka/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

The [Unit] section specifies that Zookeeper requires networking and the filesystem to be ready before it can start.

The [Service] section specifies that systemd should use the zookeeper-server-start.sh and zookeeper-server-stop.sh shell files for starting and stopping the service. It also specifies that Zookeeper should be restarted automatically if it exits abnormally.

Next, create the systemd service file for kafka:


sudo vi /etc/systemd/system/kafka.service

Enter the following unit definition into the file:

/etc/systemd/system/kafka.service
[Unit]
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
User=kafka
ExecStart=/bin/sh -c '/home/kafka/kafka/bin/kafka-server-start.sh /home/kafka/kafka/config/server.properties > /home/kafka/kafka/kafka.log 2>&1'
ExecStop=/home/kafka/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

The [Unit] section specifies that this unit file depends on zookeeper.service. This will ensure that zookeeper gets started automatically when the kafa service starts.

The [Service] section specifies that systemd should use the kafka-server-start.sh and kafka-server-stop.sh shell files for starting and stopping the service. It also specifies that Kafka should be restarted automatically if it exits abnormally.

Now that the units have been defined, start Kafka with the following command:

sudo systemctl start kafka

To ensure that the server has started successfully, check the journal logs for the kafka unit:

journalctl -u kafka

You should see output similar to the following:

Output
Jul 17 18:38:59 kafka-centos systemd[1]: Started kafka.service.

You now have a Kafka server listening on port 9092.

While we have started the kafka service, if we were to reboot our server, it would not be started automatically. To enable kafka on server boot, run:
sudo systemctl enable kafka

Now that we’ve started and enabled the services, let’s check the installation.

Step 5: Set up Producer send message to remote server

Worked with

from server.properties Uncomment

listeners=PLAINTEXT://:9092

And

advertised.listeners=PLAINTEXT://:9092

Replace  with actual IP.

6. Command Line Kafka Popular

  • View detail consumer in consumer group: consumer – topic – partition-offset
bin/kafka-consumer-groups.sh  --bootstrap-server localhost:9092,localhost:9093,localhost:9094  --describe --group GroupConsumer 
  • View list consumer and consumer group
bin/kafka-consumer-groups.sh  --bootstrap-server localhost:9092,localhost:9093,localhost:9094  --list
  • View detail Topic
bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-topic 
  • View List Topic
bin/kafka-topics.sh --list --zookeeper localhost:2181
  • View Data in Topic
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic twitterstream --from-beginning
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s