Kettle中如何正確存放插件和調試插件

本文轉載自查看原文 2022-04-12 12:29 1412 18Java&Kettle

調試遇到的問題

按照給定的源碼，外部插件是沒有加加載的。因為外部插件有兩種加載方式。一個是以jar包方式，一個是plugin.xml方式加裝。調試發現如果以jar形式加載，需要將jar包放到/ui/plugins目錄下（而不是源代碼目錄/plugins目錄）而這種方式導致有些插件pdi-xml-plugin插件加載會報錯Unexpected error loading class: General - org/pentaho/metaverse/api/analyzer/kettle/step/IStepExternalResourceConsumer
，分析發現pdi-xml-plugin的pom.xml文件發現依賴pentaho-metaverse-api包，但是看了下Eclipse 中Maven Dependencies有這個包。我猜可能這種加載jar包找不到這個包（我也驗證過，將maven倉庫中的這個包拷貝到這個插件libs目錄下確實不報沒有這個錯，但是報其他包的錯。感覺拷貝maven包的方式肯定不行，而且其他插件也需要這個包也要拷貝，造成包重復，也違背了maven倉庫管理原則），如是要考慮其他方法。但是看了經過mvn命令行編譯好的.源代碼目錄pentaho-kettle-9.2.0.3-R\assemblies\client\target下的pdi-ce-9.2.0.3-477.zip解壓后，我們看到如下目錄
lib:所有maven倉庫的包
libswt:不同平台下的swt包(linux,osx64,win64)
plugins:所有外部插件
ui:都是些*.xul文件
launcher：里含有launcher.jar文件和launcher.properties文件而launcher.properties中指定了lib，libswt路徑


main=org.pentaho.di.ui.spoon.Spoon
libraries=../test:../lib:../libswt
classpath=../classes:../:../ui:../ui/images:../lib
system-property.pentaho.installed.licenses.file=${PENTAHO_INSTALLED_LICENSE_PATH}

在main方法中 KettleEnvironment.init()之前添加了

 System.setProperty("KETTLE_PLUGIN_BASE_FOLDERS","E:\\DevOps\\ETL\\pentaho-kettle-9.2.0.3-R\\plugins");

重新cd到源碼目錄，用如下命令編譯項目

mvn clean package

運行調試都報錯

Caused by: java.lang.NoClassDefFoundError: com/google/api/client/json/JsonFactory
	at org.pentaho.googledrive.lifecycle.GoogleDrivePluginLifecycleListener.onEnvironmentInit(GoogleDrivePluginLifecycleListener.java:49)
	at org.pentaho.di.core.lifecycle.KettleLifecycleSupport.onEnvironmentInit(KettleLifecycleSupport.java:116)
	at org.pentaho.di.core.lifecycle.KettleLifecycleSupport.onEnvironmentInit(KettleLifecycleSupport.java:107)
	at org.pentaho.di.core.KettleEnvironment.initLifecycleListeners(KettleEnvironment.java:184)
	at org.pentaho.di.core.KettleEnvironment.init(KettleEnvironment.java:157)

調試分析異常

經驗證去掉System.setProperty("KETTLE_PLUGIN_BASE_FOLDERS","E:\DevOps\ETL\pentaho-kettle-9.2.0.3-R\plugins");這句話打包程序運行就不會報錯。
如果不加入這句話，需要將plugins拷貝到pentaho-kettle-9.2.0.3-R\ui下，否則jar包就不會加載，本來將編譯好的plugins目錄及文件拷貝到pentaho-kettle-9.2.0.3-R\ui下調試，發現kettile-json-plugin-core插件報錯

General - ERROR (version Unknown) : Unexpected error loading class for plugin JsonOutputAnalyzerPlugin
General - ERROR (version Unknown) : org.pentaho.di.core.exception.KettlePluginException: 
General - Unexpected error loading class:
General - org/pentaho/metaverse/api/analyzer/kettle/step/IStepExternalResourceConsumer

猜測缺少包，但maven倉庫中有這個包，只是這個插件下lib下沒有這個包（而且這個不像內部插件調試環境有Maven Dependciies庫，外部插件雖然也有Maven Dependciies庫，但是以jar包形式加載這個時候還沒有進入調試環境，所有即使有Maven Dependciies庫也沒用，還必須各插件下lib目錄有所有的依賴包）
Spoon.bat指定了jre,launcher.jar
因為lib所有jar包都是公用的，所以可以加載外部的jar包插件不報錯.

遺留問題1
這里需要理解的是有些插件下面lib目錄有些依賴的jar包，有些沒有，有些不完整。

從Spoon.java啟動調試的不好地方

1.調試插件有些插件缺依賴的jar包，需要拷貝依賴包到編譯的插件下的lib目錄下
2.system下的karaf（osgi插件），比如編譯號的data-integration\system\karaf\system\org\pentaho\di\plugins下的file-open-save-plugin插件加載不了，導致內部插件Text file input插件中的【Browse】打不開對話框，經調試無法osgi的file-open-save-plugin插件
可以從/kettle-ui-swt/src/main/java/org/pentaho/di/ui/core/events/dialog/SelectionAdapterFileDialog.java中的widgetSelectedHelper方法中的

 extensionPointWrapper.callExtensionPoint( log, 
KettleExtensionPoint.SpoonOpenSaveNew.id, fileDialogOperation );

目前最佳的做法

將整個程序編譯打包后的代碼pentaho-kettle-9.2.0.3-R\assemblies\client\target下的pdi-ce-9.2.0.3-477.zip解壓，將plugins整個目錄拷貝到pentaho-kettle-9.2.0.3-R\ui目錄下，雖然有些插件報錯，但是插件都加載出來，而且可以調試，后續就是解決這些報錯問題。

public class PluginRegistryPluginType extends BasePluginType implements PluginTypeInterface {

  private static PluginRegistryPluginType INSTANCE = new PluginRegistryPluginType();

  public PluginRegistryPluginType() {
    super( RegistryPlugin.class, "Plugin Extensions", "Plugin Registry Extension Types" );
    populateFolders( "pluginRegistry" );
  }

  @Override
  protected String getXmlPluginFile() {
    // This property is set by the import-export.bat/sh command line script to disable this native resource file from
    // processing.
    if ( !Utils.isEmpty( getProperty( "pentaho.disable.karaf", "" ) ) ) {
      return "KettleFileDisabled.xmldisabledxyzabc";  //we return a file that does not exist to suppress exceptions.
    }
    return Const.XML_FILE_KETTLE_REGISTRY_EXTENSIONS;
  }

XML_FILE_KETTLE_REGISTRY_EXTENSIONS = "kettle-registry-extensions.xml"內容如下

<registry-extensions>

  <registry-extension id="PdiOsgiBridge">

    <description>PDI-OSGI-Bridge Extension</description>

    <tooltip/>    
<classname>org.pentaho.di.osgi.registryExtension.OSGIPluginRegistryExtension</classname>

    <meta-classname/>

    <version-browser-classname/>

  </registry-extension>
</registry-extensions>

注意
PluginRegistryExtension
ExtensionPointPluginType

項目中有兩種實現了PluginRegistryExtension接口類的

org.pentaho.di.osgi.registryExtension.OSGIPluginRegistryExtension
org.pentaho.di.bigdata.ShimDependentPluginRegistryPlugin
這里主要說明OSGIPluginRegistryExtension

PluginRegistry.java中registerType方法中

 for ( PluginRegistryExtension ext : extensions ) {

      ext.searchForType( pluginType );

    }

PluginRegistryExtension接口代碼如下

public interface PluginRegistryExtension {

  void init( PluginRegistry registry );
  void searchForType( PluginTypeInterface pluginType );
  String getPluginId( Class<? extends PluginTypeInterface> pluginType, Object 
pluginClass );

}

OSGI需另行研究
org.pentaho.di.osgi.OSGIPluginType
通過Everything工具高級搜索OSGIPluginRegistryExtension，找到了pdi-osgi-bridge-core-9.2.0.3-477.jar包，同時又去maven倉庫中找到了對應的源代碼包pdi-osgi-bridge-core-9.2.0.3-477-sources.jar解壓就可以看到OSGIPluginRegistryExtension.java代碼實現,同時包含了OsgPlugin及OsgPluginType實現

@RegistryPlugin(id = "OSGIRegistryPlugin", name = "OSGI")
public class OSGIPluginRegistryExtension implements PluginRegistryExtension {
  private static OSGIPluginRegistryExtension INSTANCE;
  private OSGIPluginTracker tracker = OSGIPluginTracker.getInstance();
  private Logger logger = LoggerFactory.getLogger( getClass() );
  private KarafBoot boot = new KarafBoot();
  private StatusGetter<Boolean> kettleClientEnvironmentInitialized = new StatusGetter<Boolean>() {
    @Override public Boolean get() {
      return KettleClientEnvironment.isInitialized();
    }
  };
  private AtomicBoolean initializedKaraf = new AtomicBoolean( false );

  public OSGIPluginRegistryExtension() {
    INSTANCE = this;
  }

  public static OSGIPluginRegistryExtension getInstance() {
    if ( INSTANCE == null ) {
      throw new IllegalStateException( "Kettle is supposed to construct this first" );
    }
    return INSTANCE;
  }

  // FOR UNIT TEST ONLY
  protected static void setInstance( OSGIPluginRegistryExtension instance ) {
    INSTANCE = instance;
  }

  // FOR UNIT TEST ONLY
  protected void setTracker( OSGIPluginTracker tracker ) {
    this.tracker = tracker;
  }

  // FOR UNIT TEST ONLY
  protected void setLogger( Logger logger ) {
    this.logger = logger;
  }


  // FOR UNIT TEST ONLY
  protected void setKettleClientEnvironmentInitialized( StatusGetter<Boolean> kettleClientEnvironmentInitialized ) {
    this.kettleClientEnvironmentInitialized = kettleClientEnvironmentInitialized;
  }

  @VisibleForTesting
  void setKarafBoot( KarafBoot boot ){
    this.boot = boot;
  }

  public KarafBoot getKarafBoot(){
    return boot;
  }

  public synchronized void init( final PluginRegistry registry ) {
    if ( PentahoSystem.getInitializedStatus() != PentahoSystem.SYSTEM_INITIALIZED_OK && !initializedKaraf.getAndSet( true )) {
      String userDir = System.getProperty("pentaho.user.dir", ".");
      IApplicationContext context = new StandaloneApplicationContext( userDir, userDir );
      PentahoSystem.init(context);
      boot.startup( null );
    }
    PluginRegistry.addPluginType( OSGIPluginType.getInstance() );
    tracker.registerPluginClass( PluginInterface.class );

    tracker.addPluginLifecycleListener( PluginInterface.class,
      new PluginRegistryOSGIServiceLifecycleListener( registry ) );
  }

  @Override
  public void searchForType( PluginTypeInterface pluginType ) {
    tracker.registerPluginClass( pluginType.getClass() );
  }

  @Override
  public String getPluginId( Class<? extends PluginTypeInterface> pluginType, Object pluginClass ) {
    try {
      return (String) tracker.getBeanPluginProperty( pluginType, pluginClass, "ID" );
    } catch ( Exception e ) {
      logger.error( e.getMessage(), e );
    }
    return null;
  }
}