Obsidian/Recognition/Programing/Kafka(AMQP)/Kafka 설치.md

199 lines
5.0 KiB
Markdown
Raw Normal View History

2023-08-14 16:19:25 +00:00
#### Windows
- 다운로드
> https://kafka.apache.org/downloads
> Scala 2.13에 해당하는 .tgz파일 다운받아서 임의경로에 압축해제
- Zookeeper 실행
``` CMD
cd C:\Dev\kafka_2.13-3.2.1\bin\windows &
.\zookeeper-server-start.bat ..\..\config\zookeeper.properties
```
- Kafka 서버 실행
```CMD
(관리자권한으로 실행)
cd C:\Dev\kafka_2.13-3.2.1\bin\windows &
.\kafka-server-start.bat ..\..\config\server.properties
```
- 실행확인
- Kafka 서버 : 9092
- zookeeper : 2181
##### 클러스터 구성
1. 각 서버의 config/server.propertise내용 수정
> 각 서버 구성에 맞게 IP, Port수정, broker.id 중복 안되도록 설정
``` Properties
broker.id=1
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://192.168.12.21:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.12.21:2181, 192.168.12.168:2181
zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0
```
2. config/zookeeper.properties 내용 수정
```Properties
dataDir=/tmp/zookeeper
clientPort=2181
maxClientCnxns=0
admin.enableServer=false
initLimit=5
syncLimit=2
server.1=192.168.12.21:2888:3888
server.2=192.168.12.168:2888:3888
```
3. 위 dataDir 경로에 myid파일생성(확장자 없음)후 broker.id 입력
4. logs폴더의 meta.properties파일을 삭제 후 kafka 서버 기동
---
#### WSL
```Shell
## 다운로드
wget http://downloads.apache.org/kafka/3.2.1/kafka_2.13-3.2.1.tgz
# 압축해제 및 경로이동
tar -zxvf kafka_2.13-3.2.1.tgz $ cd kafka_2.13-3.2.1
# 주키퍼 설정파일 백업
cp kafka_2.13-3.2.1/config/zookeeper.properties kafka_2.13-3.2.1/config/zookeeper_origin.properties
# 앙상블구성 준비 ####################################################
## Zookeeper설정
vi config/zookeeper.properties
initLimit=5
syncLimit=2
server.1=192.168.12.21:2888:3888
server.2=192.168.12.168:2888:3888
## myid 설정 (각각의 서버에서 설정)
#server-(1) (192.168.0.202)
mkdir /tmp/zookeeper
echo 1 > /tmp/zookeeper/myid
#server-(2) (192.168.0.193)
mkdir /tmp/zookeeper
echo 2 > /tmp/zookeeper/myid
#server-(3) (192.168.0.208)
mkdir /tmp/zookeeper
echo 3 > /tmp/zookeeper/myid
## 카프카 설정(각각의 서버에서 설정)
vi config/server.properties
#192.168.0.202 개발서버 1번 IP
#192.168.0.193 개발서버 2번 IP
#192.168.0.208 개발서버 3번 IP
##########################
#서버 1
##########################
broker.id=1
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://192.168.0.202:9092
#서버1호스트명:서버1포트,서버2호스트명:서버2포트,서버3호스트명:서버2포트/주기퍼노드명
zookeeper.connect=192.168.0.202:2181, 192.168.0.193:2181, 192.168.0.208:2181
##########################
#서버 2
##########################
broker.id=2
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://192.168.0.193:9092
#서버1호스트명:서버1포트,서버2호스트명:서버2포트,서버3호스트명:서버2포트/주기퍼노드명
zookeeper.connect=192.168.0.202:2181, 192.168.0.193:2181, 192.168.0.208:2181
##########################
#서버 3
##########################
broker.id=3
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://192.168.0.208:9092
#서버1호스트명:서버1포트,서버2호스트명:서버2포트,서버3호스트명:서버2포트/주기퍼노드명
zookeeper.connect=192.168.0.202:2181, 192.168.0.193:2181, 192.168.0.208:2181
# 앙상블구성 준비 완료 ####################################################
[기타 설정 참고](https://kafka.apache.org/documentation/#brokerconfigs)
# Zookeeper, Kafka실행
bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties
```
---
- 토픽생성
```CMD
.\kafka-topics.bat --create --bootstrap-server localhost:9092 --topic oingdaddy
## 옵션 ##
bootstrap-server : 브로커 리스트
replication-factor : 리플리케이션 팩터 수
partitions : 파티션 수
topic : 토픽 이름을 입력
```
- 토픽에 메시지 보내기
``` CMD
kafka-console-producer.bat --broker-list localhost:9092 --topic oingdaddy
```
- 토픽에서 메시지 가져오기
```CMD
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic simple-push-topic --from-beginning
```
- 컨슈머그룹 LAG확인
```CMD
kafka-consumer-groups.bat --bootstrap-server localhost:9092 --describe --group simple-push-topic
```
- 토픽목록 확인
kafka-topics.bat --list --bootstrap-server 192.168.12.21:9092
- 토픽 삭제
kafka-topics --delete --bootstrap-server localhost:2181 --topic sample_topic01
- commend Shell
zookeeper-shell 192.168.12.21:2181,192.168.12.168:2181