Minio Bucket 通知試用&&說明

kafka 安裝

docker run -p 2181:2181 -p 9092:9092 \
      --name kafka --rm \
      --env ADVERTISED_HOST=HOSTIP \
      --env ADVERTISED_PORT=9092 \
      spotify/kafka

minio (mac local)

wget https://dl.minio.io/server/minio/release/darwin-amd64/minio 
chmod +x minio 
mkdir data
./minio server ./data

config minio notify

路徑 ~/.minio/config.json

"kafka": {
    "1": {
        "enable": true,
        "brokers": ["localhost:9092"],
        "topic": "bucketevents"
       }
    }

下載client 並配置minio server

wget https://dl.minio.io/client/mc/release/darwin-amd64/mc 
chmod +x mc 
./mc --help
./mc config host add myminio http://localhost:9000 GT481SA7MEF5D2G29N0L QqaFn67VTbko3FRE0IfZVTYpztb3UpjKW0RloHSt

創建bucket 並啟用evnet

注意配置之前，如果沒有配置kafka 通知，配置完成之后，minio server 需要重啟，不然會報錯

./mc mb myminio/images
./mc events add myminio/images arn:minio:sqs:us-east-1:1:kafka --suffix .png

測試

• 下載 kafkacat

  kafka 信息查看(mac)

brew install kafkacat

• 上傳圖片測試

./mc cp app.png myminio/images

• 效果
  

說明

我們可以添加多種通知，實現一個pipeline 的文件處理服務，還是比較方便的

• 添加另外的一個通知

./mc events add myminio/images arn:minio:sqs:us-east-1:1:kafka --suffix .txt

• 效果
  
• 查看配置的通知

 ./mc events list myminio/images --json
{"status":"success","id":"","events":["s3:ObjectCreated:*","s3:ObjectRemoved:*","s3:ObjectAccessed:*"],"prefix":"","suffix":".png","arn":"arn:minio:sqs::1:kafka"}
{"status":"success","id":"","events":["s3:ObjectCreated:*","s3:ObjectRemoved:*","s3:ObjectAccessed:*"],"prefix":"","suffix":".jpg","arn":"arn:minio:sqs::1:kafka"}
{"status":"success","id":"","events":["s3:ObjectCreated:*","s3:ObjectRemoved:*","s3:ObjectAccessed:*"],"prefix":"","suffix":".txt","arn":"arn:minio:sqs::1:kafka"}

信息存儲

實際上默認的配置都是在一個.minio.sys 文件夾中

• 參考圖
  
• 內容

<NotificationConfiguration>
<QueueConfiguration>
<Id></Id>
<Filter>
<S3Key>
<FilterRule>
<Name>suffix</Name>
<Value>.png</Value>
</FilterRule>
</S3Key>
</Filter>
<Event>s3:ObjectCreated:*</Event>
<Event>s3:ObjectRemoved:*</Event>
<Event>s3:ObjectAccessed:*</Event>
<Queue>arn:minio:sqs::1:kafka</Queue>
</QueueConfiguration>
<QueueConfiguration>
<Id></Id>
<Filter>
<S3Key>
<FilterRule>
<Name>suffix</Name>
<Value>.jpg</Value>
</FilterRule>
</S3Key>
</Filter>
<Event>s3:ObjectCreated:*</Event>
<Event>s3:ObjectRemoved:*</Event>
<Event>s3:ObjectAccessed:*</Event>
<Queue>arn:minio:sqs::1:kafka</Queue>
</QueueConfiguration>
<QueueConfiguration>
<Id></Id>
<Filter>
<S3Key>
<FilterRule>
<Name>suffix</Name>
<Value>.txt</Value>
</FilterRule>
</S3Key>
</Filter>
<Event>s3:ObjectCreated:*</Event>
<Event>s3:ObjectRemoved:*</Event>
<Event>s3:ObjectAccessed:*</Event>
<Queue>arn:minio:sqs::1:kafka</Queue>
</QueueConfiguration>
</NotificationConfiguration>

參考資料

https://minio.io/downloads.html
https://github.com/edenhill/kafkacat
https://hub.docker.com/r/spotify/kafka/
https://docs.minio.io/docs/minio-bucket-notification-guide
https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html