Obsidian/Recognition/Programing/Kafka(AMQP)/Kafka 설치.md

199 lines
5.0 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters!

This file contains ambiguous Unicode characters that may be confused with others in your current locale. If your use case is intentional and legitimate, you can safely ignore this warning. Use the Escape button to highlight these characters.

#### Windows
- 다운로드
> https://kafka.apache.org/downloads
> Scala 2.13에 해당하는 .tgz파일 다운받아서 임의경로에 압축해제
- Zookeeper 실행
``` CMD
cd C:\Dev\kafka_2.13-3.2.1\bin\windows &
.\zookeeper-server-start.bat ..\..\config\zookeeper.properties
```
- Kafka 서버 실행
```CMD
(관리자권한으로 실행)
cd C:\Dev\kafka_2.13-3.2.1\bin\windows &
.\kafka-server-start.bat ..\..\config\server.properties
```
- 실행확인
- Kafka 서버 : 9092
- zookeeper : 2181
##### 클러스터 구성
1. 각 서버의 config/server.propertise내용 수정
> 각 서버 구성에 맞게 IP, Port수정, broker.id 중복 안되도록 설정
``` Properties
broker.id=1
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://192.168.12.21:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.12.21:2181, 192.168.12.168:2181
zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0
```
2. config/zookeeper.properties 내용 수정
```Properties
dataDir=/tmp/zookeeper
clientPort=2181
maxClientCnxns=0
admin.enableServer=false
initLimit=5
syncLimit=2
server.1=192.168.12.21:2888:3888
server.2=192.168.12.168:2888:3888
```
3. 위 dataDir 경로에 myid파일생성(확장자 없음)후 broker.id 입력
4. logs폴더의 meta.properties파일을 삭제 후 kafka 서버 기동
---
#### WSL
```Shell
## 다운로드
wget http://downloads.apache.org/kafka/3.2.1/kafka_2.13-3.2.1.tgz
# 압축해제 및 경로이동
tar -zxvf kafka_2.13-3.2.1.tgz $ cd kafka_2.13-3.2.1
# 주키퍼 설정파일 백업
cp kafka_2.13-3.2.1/config/zookeeper.properties kafka_2.13-3.2.1/config/zookeeper_origin.properties
# 앙상블구성 준비 ####################################################
## Zookeeper설정
vi config/zookeeper.properties
initLimit=5
syncLimit=2
server.1=192.168.12.21:2888:3888
server.2=192.168.12.168:2888:3888
## myid 설정 (각각의 서버에서 설정)
#server-(1) (192.168.0.202)
mkdir /tmp/zookeeper
echo 1 > /tmp/zookeeper/myid
#server-(2) (192.168.0.193)
mkdir /tmp/zookeeper
echo 2 > /tmp/zookeeper/myid
#server-(3) (192.168.0.208)
mkdir /tmp/zookeeper
echo 3 > /tmp/zookeeper/myid
## 카프카 설정(각각의 서버에서 설정)
vi config/server.properties
#192.168.0.202 개발서버 1번 IP
#192.168.0.193 개발서버 2번 IP
#192.168.0.208 개발서버 3번 IP
##########################
#서버 1
##########################
broker.id=1
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://192.168.0.202:9092
#서버1호스트명:서버1포트,서버2호스트명:서버2포트,서버3호스트명:서버2포트/주기퍼노드명
zookeeper.connect=192.168.0.202:2181, 192.168.0.193:2181, 192.168.0.208:2181
##########################
#서버 2
##########################
broker.id=2
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://192.168.0.193:9092
#서버1호스트명:서버1포트,서버2호스트명:서버2포트,서버3호스트명:서버2포트/주기퍼노드명
zookeeper.connect=192.168.0.202:2181, 192.168.0.193:2181, 192.168.0.208:2181
##########################
#서버 3
##########################
broker.id=3
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://192.168.0.208:9092
#서버1호스트명:서버1포트,서버2호스트명:서버2포트,서버3호스트명:서버2포트/주기퍼노드명
zookeeper.connect=192.168.0.202:2181, 192.168.0.193:2181, 192.168.0.208:2181
# 앙상블구성 준비 완료 ####################################################
[기타 설정 참고](https://kafka.apache.org/documentation/#brokerconfigs)
# Zookeeper, Kafka실행
bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties
```
---
- 토픽생성
```CMD
.\kafka-topics.bat --create --bootstrap-server localhost:9092 --topic oingdaddy
## 옵션 ##
bootstrap-server : 브로커 리스트
replication-factor : 리플리케이션 팩터 수
partitions : 파티션 수
topic : 토픽 이름을 입력
```
- 토픽에 메시지 보내기
``` CMD
kafka-console-producer.bat --broker-list localhost:9092 --topic oingdaddy
```
- 토픽에서 메시지 가져오기
```CMD
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic simple-push-topic --from-beginning
```
- 컨슈머그룹 LAG확인
```CMD
kafka-consumer-groups.bat --bootstrap-server localhost:9092 --describe --group simple-push-topic
```
- 토픽목록 확인
kafka-topics.bat --list --bootstrap-server 192.168.12.21:9092
- 토픽 삭제
kafka-topics --delete --bootstrap-server localhost:2181 --topic sample_topic01
- commend Shell
zookeeper-shell 192.168.12.21:2181,192.168.12.168:2181