排查logstash2.4升级到5.0版本后kafka不兼容问题-凯发app官方网站

参考文档：
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5/changelog.md
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5/developer.md
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5/readme.md.md
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5/lib/logstash/inputs/kafka.rb
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-kafka-5.0.4/lib/logstash/outputs/kafka.rb




缘由：
之前对elkb环境从2.4版本升级到最新的5.0稳定版本，主要升级步骤可以参考http://blog.chinaunix.net/uid-25057421-id-5754668.html，后来发现kafka集群运行报错，现在把排查过程记录如下，仅供参考


之前环境：
logstash2.4
logstash-input-kafka-2.0.9
logstash-output-kafka-2.0.5
kafka_2.10-0.8.2.2.tgz




升级后环境：
logstash5.0
logstash-input-kafka-2.0.9
logstash-output-kafka-2.0.5


报错信息：
[2016-11-16t14:35:44,739][error][logstash.inputs.kafka    ] unknown setting 'zk_connect' for kafka
[2016-11-16t14:35:44,741][error][logstash.inputs.kafka    ] unknown setting 'topic_id' for kafka
[2016-11-16t14:35:44,741][error][logstash.inputs.kafka    ] unknown setting 'reset_beginning' for kafka


实施步骤：
1，根据错误查看程序哪里报错
grep  "unknown setting" /usr/share/logstash/ -r
/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:          self.logger.error("unknown setting '#{name}' for #{@plugin_name}")


2，查看程序相关代码，发现需要查看plugins的config定义文件等
    def validate_check_invalid_parameter_names(params)
      invalid_params = params.keys
      # filter out parameters that match regexp keys.
      # these are defined in plugins like this:
      #   config /foo.*/ => ...
      @config.each_key do |config_key|
        if config_key.is_a?(regexp)
          invalid_params.reject! { |k| k =~ config_key }
        elsif config_key.is_a?(string)
          invalid_params.reject! { |k| k == config_key }
        end
      end


      if invalid_params.size > 0
        invalid_params.each do |name|
          self.logger.error("unknown setting '#{name}' for #{@plugin_name}")
        end
        return false
      end # if invalid_params.size > 0
      return true
    end # def validate_check_invalid_parameter_names


3，进入插件总目录查看具体信息
cd  /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5
发现重点查看如下文件
grep config ./* -r |awk '{print $1}' |uniq 
./changelog.md:
./developer.md:see
./lib/logstash/inputs/kafka.rb:#
./lib/logstash/inputs/kafka.rb:
./readme.md:-
binary


1）首先看changelog.md，就有发现logstash-input-3.0.0.beta1开始就不在向后兼容，且剔除了jruby-kafka，注意这里有个坑2）会讲到，4.0.0版本说开始支持kafka 0.9，5.0.0又说开始


支持0.10切不向后兼容，这破坏性更新也是够了。看来问题找到了我的kafka版本是kafka_2.10-0.8.2.2.tgz，kafka版本不兼容导致的。  
changelog.md部分文档如下：
## 5.0.4
  - update to kafka version 0.10.0.1 for bug fixes
## 5.0.0
  - support for kafka 0.10 which is not backward compatible with 0.9 broker.
## 4.0.0
  - republish all the gems under jruby.
  - update the plugin to the version 2.0 of the plugin api, this change is required for logstash 5.0 compatibility. see
  - support for kafka 0.9 for ls 5.x
## 3.0.0.beta1
 - refactor to use new java based consumer, bypassing jruby-kafka
 - breaking: change configuration to match kafka's configuration. this version is not backward compatible


2）之前我看developer.md文档时,看配置语法都正确，还以为是却少依赖关系jruby-kafka library呢，这个再logstash2.x是在用的(另外对比logstash5.x发现5版本少了不少插件。另外


kafka版本写的是0.8.1.1，看来这个developer.md没有及时更新（与后面kafka.rb文件不一致），谁要是看到了麻烦及时更新啊，虽是小问题但是也可能误导我等屁民。当然也有可能是我没


有全面看文档导致的。
developer.md文档结尾如下：
dependencies
====================
* apache kafka version 0.8.1.1
* jruby-kafka library


3）开始看readme.md文档，特意看了下kafka的兼容性 看来logstas-input-kafka5.0.5和logstash-output-kafka5.0.4只能用kafka0.10了。如果你想用kafka0.9还想用logstash5.0,你的


logstash-input-kafka和logstash-output-kafka只能降级版本到4.0.0了，这里都说他是中间过渡版本了，所以还是随大流吧。
## kafka compatibility
here's a table that describes the compatibility matrix for kafka broker support. please remember that it is good advice to upgrade brokers before consumers/producers 


since brokers target backwards compatibility. the 0.9 broker will work with both the 0.8 consumer and 0.9 consumer apis but not the other way around.


| kafka broker version | logstash version | input plugin | output plugin | why? |
|:---------------:|:------------------:|:--------------:|:---------------:|:------|
| 0.8           | 2.0 - 2.x   | < 3.0.0 | <3.0.0 | legacy, 0.8 is still popular |
| 0.9           | 2.0 - 2.3.x   |   3.0.0 | 3.0.0  | intermediate release before 0.10 that works with old ruby event api `[]`  |
| 0.9          | 2.4, 5.0           |   4.0.0 | 4.0.0  | intermediate release before 0.10 with new get/set api |
| 0.10         | 2.4, 5.0           |   5.0.0 | 5.0.0  | track latest kafka release. not compatible with 0.9 broker |


4）现在看来只能升级kafka版本了。最后我看了下jar-dependencies发现了kafka-clients-0.10.0.1.jar
ls /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5/vendor/jar-dependencies/runtime-jars/
kafka-clients-0.10.0.1.jar  log4j-1.2.17.jar  lz4-1.3.0.jar  slf4j-api-1.7.21.jar  slf4j-log4j12-1.7.21.jar  snappy-java-1.1.2.6.jar


5）还有一个文件没有看，怀着好奇心我看了一眼，发现之前都白费力气了，这里才是最有参考价值的的主参考文档啊，是捷径啊，隐藏的够深的，差点错过了，汗！
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5/lib/logstash/inputs/kafka.rb
kafka.rb部分文档如下：
# this input will read events from a kafka topic. it uses the the newly designed
# 0.10 version of consumer api provided by kafka to read messages from the broker.
#
# here's a compatibility matrix that shows the kafka client versions that are compatible with each combination
# of logstash and the kafka input plugin: 
# 
# [options="header"]
# |==========================================================
# |kafka client version |logstash version |plugin version |security features |why?
# |0.8       |2.0.0 - 2.x.x   |<3.0.0 | |legacy, 0.8 is still popular 
# |0.9       |2.0.0 - 2.3.x   | 3.x.x |basic auth, ssl |works with the old ruby event api (`event['product']['price'] = 10`)  
# |0.9       |2.4.0 - 5.0.x   | 4.x.x |basic auth, ssl |works with the new getter/setter apis (`event.set('[product][price]', 10)`)
# |0.10      |2.4.0 - 5.0.x   | 5.x.x |basic auth, ssl |not compatible with the 0.9 broker 
# |==========================================================
# 
# note: we recommended that you use matching kafka client and broker versions. during upgrades, you should
# upgrade brokers before clients because brokers target backwards compatibility. for example, the 0.9 broker
# is compatible with both the 0.8 consumer and 0.9 consumer apis, but not the other way around.


6）升级kafka_2.10-0.8.2.2.tgz为kafka_2.11-0.10.0.1.tgz （我看kafka-clients-0.10.0.1.jar，所以没有用最新的kafka_2.11-0.10.1.0.tgz）
大概步骤
关闭老kafka
/usr/local/kafka/bin/kafka-server-stop.sh /usr/local/kafka/config/server.properties
备份老配置文件
server.properties和zookeeper.properties
删除kafka
rm -rf /usr/local/kafka/
rm -rf /data/kafkalogs/*
安装配置新kafka
wget
tar zxvf kafka_2.11-0.10.0.1.tgz -c /usr/local/
ln -s /usr/local/kafka_2.11-0.10.0.1 /usr/local/kafka
diff了下server.properties和zookeeper.properties变动不大可以直接使用
启动新kafka
/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties &


7)注意几个关键配置需要修改
config :bootstrap_servers, :validate => :string, :default => "localhost:9092"
config :group_id, :validate => :string, :default => "logstash"
config :topics, :validate => :array, :default => ["logstash"]
config :consumer_threads, :validate => :number, :default => 1
除了上面的几个关键配置外，kafka的topic分片信息需要重新create一份，否则kafkamonitor监控不出active topic consumer图形，但实际是在工作中。