Logstash 7.7 x+導出官網 java 插件為離線包，重安裝失敗bug：ERROR: Something went wrong when installing file:///xxx, message: Illformed requirement ["=java_xxx"]

本文轉載自查看原文 2020-12-21 17:44 1190 logstash.org/ logstash/ java/ logstash demo/ ruby

標題中bug：在最后按照離線包處說明，給出解決方式

公司需要使用最新版logstash來開發，我們自己開放性框架

github下載對應源碼：Logstash 7.10.1 github 地址

解壓windows 直接解壓/linux unzip xxxx.zip
生成java插件所需api 核心 jar 包文件：進入項目根目錄執行-windows:gradlew.bat assemble linux：./gradlew assemble

java 示列插件git地址：logstash-input-java_input_example

plugin demo 根目錄，增加java.properties 配置文件 gradle.properties，添加值：LOGSTASH_CORE_PATH=/data/logstashcpl/logstash-7.10.1/logstash-core 該核心路徑指向上一步中核心jar包路徑
打包插件：./gradlew gem

在引入 Logstash 官網下載地址下載了 logstash-7.10.1

安裝上面打包的插件：bin/logstash-plugin install --no-verify --local 你打包出來的插件

bin/logstash-plugin install --no-verify --local /data/plugins/logstash-input-java_input_example-master/logstash-input-java_input_example-1.0.1.gem

運行該插件：

bin/logstash -e "input {java_input_example {}} output {stdout { codec => rubydebug }}"

查看該插件在logstash中的名稱

[root@VM_centos logstash-7.10.1]# bin/logstash-plugin list
Using JAVA_HOME defined java: /data/java/jdk1.8.0_271/
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
....
logstash-input-imap
logstash-input-java_input_example
logstash-input-jm
.....

#logstash-input-java_input_example 即為插件名

導出該插件用於離線環境安裝（我司內網，沒辦法在內網編譯安裝）bin/logstash-plugin prepare-offline-pack --overwrite --output [包名] 插件名，上一步獲取的插件名即可

#打離線包
[root@VM_centos logstash-7.10.1]# bin/logstash-plugin prepare-offline-pack --overwrite --output /data/java_input.zip logstash-input-java_input_example
Using JAVA_HOME defined java: /data/java/jdk1.8.0_271/
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
Offline package created at: /data/java_input.zip

You can install it with this command `bin/logstash-plugin install file:///data/java_input.zip`

安裝該離線包：bin/logstash-plugin install file:///data/java_input.zip

我在執行上述步驟是遇到問題如下：

demo 插件問題：該錯誤來源：

No signature of method: org.gradle.api.internal.tasks.DefaultTaskDependency $TaskDependencySet.getAt() is applicable for argument types: (ArrayList) values:

dependsOn [downloadAndInstallJRuby, removeObsoleteJars, vendor, generateRubySupportFiles] 修改如下

dependsOn([downloadAndInstallJRuby, removeObsoleteJars, vendor, generateRubySupportFiles])

demo插件問題：

/data/plugins/logstash-input-java_input_example-master/src/main/java/org/logstashplugins/JavaInputExample.java:3: error: package co.elastic.logstash.api does not exist
import co.elastic.logstash.api.Configuration;

　　該錯誤原因為build.gradle 去匹配核心jar包時未匹配到，因為demo中

implementation fileTree(dir: LOGSTASH_CORE_PATH, include: "**/logstash-core-?.?.?.jar")
#logstash-core-?.?.?.jar 中 ? 匹配一個字符，一旦 x.y.z.jar包中版本超過9，達到兩個字符就會匹配不到
#修改為：
implementation fileTree(dir: LOGSTASH_CORE_PATH, include: "**/logstash-core-?.??.?.jar") #或指定你生成的jar包版本

安裝離線包：

ERROR: Something went wrong when installing file:///data//plugins/java-input.zip, message: Illformed requirement ["=java_input_example"]

　　該錯誤是由於默認命名插件時以 java開頭導致：build.gradle 文件中

pluginInfo.pluginType      = "input"
pluginInfo.pluginClass     = "JavaInputExample"
pluginInfo.pluginName      = "java_input_example" //此處命名java開頭導致

分析安裝離線包導致的該問題：

根目錄搜索該報錯信息：

[root@VM_centos logstash-7.10.1]# grep Illformed -nr ./*
./vendor/jruby/lib/ruby/stdlib/rubygems/requirement.rb:110:      raise BadRequirementError, "Illformed requirement [#{obj.inspect}]"
[root@VM_63_147_centos logstash-7.10.1]# vim ./vendor/jruby/lib/ruby/stdlib/rubygems/requirement.rb +110

輸出堆棧信息：puts caller

  def self.parse(obj)
    return ["=", obj] if Gem::Version === obj

    unless PATTERN =~ obj.to_s
      puts caller
      raise BadRequirementError, "Illformed requirement [#{obj.inspect}]"
    end

    if $1 == ">=" && $2 == "0"
      DefaultRequirement
    else
      [$1 || "=", Gem::Version.new($2)]
    end
  end

　　堆棧信息如下

Installing file: /data/plugins/java-input.zip
/data/logstash/logstash-7.10.1/vendor/jruby/lib/ruby/stdlib/rubygems/requirement.rb:140:in `block in initialize'
/data/logstash/logstash-7.10.1/vendor/jruby/lib/ruby/stdlib/rubygems/requirement.rb:140:in `map!'
/data/logstash/logstash-7.10.1/vendor/jruby/lib/ruby/stdlib/rubygems/requirement.rb:140:in `initialize'
/data/logstash/logstash-7.10.1/vendor/jruby/lib/ruby/stdlib/rubygems/requirement.rb:65:in `create'
/data/logstash/logstash-7.10.1/vendor/jruby/lib/ruby/stdlib/rubygems/dependency.rb:60:in `initialize'
/data/logstash/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/bundler-1.17.3/lib/bundler/dependency.rb:82:in `initialize'
/data/logstash/logstash-7.10.1/lib/pluginmanager/bundler/logstash_injector.rb:65:in `dependency'
/data/logstash/logstash-7.10.1/lib/pluginmanager/bundler/logstash_injector.rb:52:in `collect'
/data/logstash/logstash-7.10.1/lib/pluginmanager/bundler/logstash_injector.rb:52:in `inject!'
/data/logstash/logstash-7.10.1/lib/pluginmanager/pack_installer/local.rb:58:in `execute'
/data/logstash/logstash-7.10.1/lib/pluginmanager/install.rb:47:in `execute'
/data/logstash/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:68:in `run'
/data/logstash/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/subcommand/execution.rb:11:in `execute'
/data/logstash/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:68:in `run'
/data/logstash/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:133:in `run'
/data/logstash/logstash-7.10.1/lib/pluginmanager/main.rb:64:in `<main>'
ERROR: Something went wrong when installing file:///data/plugins/java-input.zip, message: Illformed requirement ["=java_input_example"]

　通過上述信息定位到：

vim ./lib/pluginmanager/pack_installer/pack.rb +65

  class Pack
    class GemInformation
      EXTENSION = ".gem"
      SPLIT_CHAR = "-"
      JAVA_PLATFORM_RE = /-java/
      DEPENDENCIES_DIR_RE = /dependencies/

      attr_reader :file, :name, :version, :platform

      def initialize(gem)
        @file = gem
        extracts_information
      end

      def dependency?
        @dependency
      end

      def plugin?
        !dependency?
      end

      private
      def extracts_information
        basename = ::File.basename(file, EXTENSION)
        parts = basename.split(SPLIT_CHAR)
        @dependency = ::File.dirname(file) =~ DEPENDENCIES_DIR_RE
        puts "basename: #{basename}"
        puts "parts: #{parts}"
        if basename.match(JAVA_PLATFORM_RE)
          @platform = parts.pop
          @version = parts.pop
          @name = parts.join(SPLIT_CHAR)
        else
          @platform = nil
          @version = parts.pop
          @name = parts.join(SPLIT_CHAR)
        end
      end
    end

　輸出：basename、parts 可以分析出，因為打包名為 logstash-input-java_input_example-1.0.1 而 logstash 安裝源碼中，使用了

JAVA_PLATFORM_RE = /-java/

if basename.match(JAVA_PLATFORM_RE) 此處匹配到了，/-java/ 認為打包平台為 java 不是 ruby 導致

[root@VM_centos logstash-7.10.1]# bin/logstash-plugin install file:///data/plugins/java-input.zip
...
arguments: ["install", "file:///data/plugins/java-input.zip"]
subcmd_class: LogStash::PluginManager::Install
arguments: ["file:///data/plugins/java-input.zip"]
Installing file: /data/plugins/java-input.zip
basename: logstash-input-java_input_example-1.0.1

解決方案就比較多了，當然不建議修改源碼，

直接修改插件名，別用 java 開頭就好了

多說一點。

樓主排查的時候懷疑是不是打包就有問題，少打了 -java 參數到包名中去，捋了好久，具體過程就不說了，看結果吧

使用 bin/logstash-plugin prepare-offline-pack 去打包的話

[root@VM_centos logstash-7.10.1]# grep -rn "my collect:" ./
./vendor/bundle/jruby/2.5.0/gems/paquet-0.2.1/lib/paquet/gem.rb:32:      puts "my collect: #{collect_required_gems}"
[root@VM_centos logstash-7.10.1]# vim ./vendor/bundle/jruby/2.5.0/gems/paquet-0.2.1/lib/paquet/gem.rb +32

module Paquet
  class Gem
    RUBYGEMS_URI = "https://rubygems.org/downloads"

    attr_reader :gems, :ignores

    def initialize(target_path, cache = nil)
      @target_path = target_path
      @gems = []
      @ignores = []
      @cache = cache
    end

    def add(name)
      @gems << name
    end

    def ignore(name)
      @ignores << name
    end

    def pack
      Paquet::ui.info("Cleaning existing target path: #{@target_path}")

      FileUtils.rm_rf(@target_path)
      FileUtils.mkdir_p(@target_path)
      puts "sq collect: #{collect_required_gems}"
      package_gems(collect_required_gems)
    end

輸出打包命令去收集所需gems時參數都為：

[root@VM_centos logstash-7.10.1]# bin/logstash-plugin prepare-offline-pack --overwrite --output /data/plugins/java-input.zip logstash-input-java_input_example
Using JAVA_HOME defined java: /data/java/jdk1.8.0_271/
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
arguments: ["prepare-offline-pack", "--overwrite", "--output", "/data/plugins/java-input.zip", "logstash-input-java_input_example"]
subcmd_class: LogStash::PluginManager::PrepareOfflinePack
arguments: ["--overwrite", "--output", "/data/plugins/java-input.zip", "logstash-input-java_input_example"]
my test: ["logstash-input-java_input_example"]
my collect: [#<Paquet::Dependency:0x51d76ad3 @version=#<Gem::Version "1.0.1">, @name="logstash-input-java_input_example", @platform="ruby">]

可以發現上面最后一行 collect 收集參數中 platform 就是ruby 不是 java
至此樓主也懶得研究了，研究這幾天炸了。
反正 logstash 官網這插件 demo 質量太差，然后處理打包上，源碼感覺也不夠嚴謹，導致了這樣的bug，分割字符串，正則匹配太SX了

最后樓主將修改后打出的包放到內網成功如下：

SIS3.0.45.0 /data/logstash-7.10.1 # bin/logstash-plugin install file:///data/java-input.zip
Using JAVA_HOME defined java: /data/share/java/jdk
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK

Installing file: /data/java-input.zip
Install successful
SIS3.0.45.0 /home/fantom/party/logstash-7.10.1 #
SIS3.0.45.0 /data/logstash-7.10.1 # bin/logstash -e "input {input_example_java {}} output {stdout { codec => rubydebug }}"
Using JAVA_HOME defined java: /data/java/jdk
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK

Sending Logstash logs to /data/logstash-7.10.1/logs which is now configured via log4j2.properties
[2020-12-21T21:14:01,451][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.10.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 25.131-b11 on 1.8.0_131-b11 +indy +jit [linux-x86_64]"}
[2020-12-21T21:14:01,858][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/data/logstash-7.10.1/data/queue"}
[2020-12-21T21:14:01,878][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/home/fantom/party/logstash-7.10.1/data/dead_letter_queue"}
[2020-12-21T21:14:02,485][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-12-21T21:14:02,529][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"a33c9423-70fa-4794-9995-f02bc72e2192", :path=>"/home/fantom/party/logstash-7.10.1/data/uuid"}
[2020-12-21T21:14:05,000][INFO ][org.reflections.Reflections] Reflections took 88 ms to scan 1 urls, producing 23 keys and 47 values
[2020-12-21T21:14:08,510][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["config string"], :thread=>"#<Thread:0x21cc9c5e run>"}
[2020-12-21T21:14:11,199][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>2.66}
[2020-12-21T21:14:11,252][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-12-21T21:14:11,432][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
{
    "@timestamp" => 2020-12-21T13:14:11.308Z,
       "message" => "message        1 of 3       ",
      "@version" => "1"
}
{
    "@timestamp" => 2020-12-21T13:14:11.319Z,
       "message" => "message        2 of 3       ",
      "@version" => "1"
}
{
    "@timestamp" => 2020-12-21T13:14:11.319Z,
       "message" => "message        3 of 3       ",
      "@version" => "1"
}
[2020-12-21T21:14:12,511][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2020-12-21T21:14:13,048][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
[2020-12-21T21:14:13,060][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2020-12-21T21:14:13,450][INFO ][logstash.runner          ] Logstash shut down.

免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。

猜您在找 Error:(x,x) java: 程序包com.xxx.xxx不存在 Error:(x,x) java: 程序包com.xxx.xxx不存在問題的解決方法 Wrong FS: hdfs://xxx/xxx expected: file:/// Error:(4, 25) java: 找不到符號符號:類 xxx位置:程序包 xxx.xxx Error:(4, 25) java: 找不到符號符號:類 xxx位置:程序包 xxx.xxx Whoops, looks like something went wrong 項目報錯 exception 'RedisException' with message 'Redis server went away' in XXX windows7安裝docker異常：looks like something went wrong in step ‘looking for vboxmanage.exe’ idea Error:(4, 25) java: 找不到符號符號:類 xxx位置:程序包 xxx.xxx程序包xxx不存在 MySQL | Java.sql.SQLException: null, message from server: "Host 'xxx.xxx.xxx.xxx' is not allowed to connect 異常解決