MonHun
/
Obsidian
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Obsidian/Recognition/Programing/Kafka(AMQP)/Kafka 설치.md

5.0 KiB
Raw Blame History

Windows

  • 다운로드

https://kafka.apache.org/downloads Scala 2.13에 해당하는 .tgz파일 다운받아서 임의경로에 압축해제

  • Zookeeper 실행
cd C:\Dev\kafka_2.13-3.2.1\bin\windows &
.\zookeeper-server-start.bat ..\..\config\zookeeper.properties
  • Kafka 서버 실행
(관리자권한으로 실행)

cd C:\Dev\kafka_2.13-3.2.1\bin\windows &
.\kafka-server-start.bat ..\..\config\server.properties
  • 실행확인
    • Kafka 서버 : 9092
    • zookeeper : 2181
클러스터 구성
  1. 각 서버의 config/server.propertise내용 수정

각 서버 구성에 맞게 IP, Port수정, broker.id 중복 안되도록 설정

broker.id=1
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://192.168.12.21:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600

log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1

offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

zookeeper.connect=192.168.12.21:2181, 192.168.12.168:2181
zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0
  1. config/zookeeper.properties 내용 수정
dataDir=/tmp/zookeeper
clientPort=2181
maxClientCnxns=0
admin.enableServer=false

initLimit=5
syncLimit=2
server.1=192.168.12.21:2888:3888
server.2=192.168.12.168:2888:3888

  1. 위 dataDir 경로에 myid파일생성(확장자 없음)후 broker.id 입력

  2. logs폴더의 meta.properties파일을 삭제 후 kafka 서버 기동

WSL

## 다운로드
wget http://downloads.apache.org/kafka/3.2.1/kafka_2.13-3.2.1.tgz

# 압축해제 및 경로이동
tar -zxvf kafka_2.13-3.2.1.tgz $ cd kafka_2.13-3.2.1

# 주키퍼 설정파일 백업
cp kafka_2.13-3.2.1/config/zookeeper.properties kafka_2.13-3.2.1/config/zookeeper_origin.properties



# 앙상블구성 준비 ####################################################
## Zookeeper설정
vi config/zookeeper.properties

	initLimit=5

	syncLimit=2
	
	server.1=192.168.12.21:2888:3888
	
	server.2=192.168.12.168:2888:3888

## myid 설정 (각각의 서버에서 설정)
	#server-(1) (192.168.0.202)
	mkdir /tmp/zookeeper
	echo 1 > /tmp/zookeeper/myid
	
	
	#server-(2) (192.168.0.193)
	mkdir /tmp/zookeeper
	echo 2 > /tmp/zookeeper/myid
	
	
	#server-(3) (192.168.0.208)
	mkdir /tmp/zookeeper
	echo 3 > /tmp/zookeeper/myid

## 카프카 설정(각각의 서버에서 설정)
vi config/server.properties
	#192.168.0.202 개발서버 1번 IP
	#192.168.0.193 개발서버 2번 IP
	#192.168.0.208 개발서버 3번 IP
	
	##########################
	#서버 1 
	##########################
	broker.id=1
	listeners=PLAINTEXT://:9092
	advertised.listeners=PLAINTEXT://192.168.0.202:9092
	
	#서버1호스트명:서버1포트,서버2호스트명:서버2포트,서버3호스트명:서버2포트/주기퍼노드명
	zookeeper.connect=192.168.0.202:2181, 192.168.0.193:2181, 192.168.0.208:2181
	
	##########################
	#서버 2
	##########################
	broker.id=2
	listeners=PLAINTEXT://:9092
	advertised.listeners=PLAINTEXT://192.168.0.193:9092
	
	#서버1호스트명:서버1포트,서버2호스트명:서버2포트,서버3호스트명:서버2포트/주기퍼노드명
	zookeeper.connect=192.168.0.202:2181, 192.168.0.193:2181, 192.168.0.208:2181
	
	##########################
	#서버 3
	##########################
	broker.id=3
	listeners=PLAINTEXT://:9092
	advertised.listeners=PLAINTEXT://192.168.0.208:9092
	
	#서버1호스트명:서버1포트,서버2호스트명:서버2포트,서버3호스트명:서버2포트/주기퍼노드명
	zookeeper.connect=192.168.0.202:2181, 192.168.0.193:2181, 192.168.0.208:2181

# 앙상블구성 준비 완료 ####################################################
 [기타 설정 참고](https://kafka.apache.org/documentation/#brokerconfigs)
 

# Zookeeper, Kafka실행
bin/zookeeper-server-start.sh config/zookeeper.properties 
bin/kafka-server-start.sh config/server.properties
  • 토픽생성
.\kafka-topics.bat --create --bootstrap-server localhost:9092 --topic oingdaddy

## 옵션 ##
–bootstrap-server : 브로커 리스트
–replication-factor : 리플리케이션 팩터 수
–partitions : 파티션 수
–topic : 토픽 이름을 입력
  • 토픽에 메시지 보내기
kafka-console-producer.bat  --broker-list  localhost:9092 --topic oingdaddy
  • 토픽에서 메시지 가져오기
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic simple-push-topic --from-beginning
  • 컨슈머그룹 LAG확인
kafka-consumer-groups.bat --bootstrap-server localhost:9092 --describe --group simple-push-topic

  • 토픽목록 확인 kafka-topics.bat --list --bootstrap-server 192.168.12.21:9092

  • 토픽 삭제 kafka-topics --delete --bootstrap-server localhost:2181 --topic sample_topic01

  • commend Shell zookeeper-shell 192.168.12.21:2181,192.168.12.168:2181