SpringBoot整合WebMagic一次踩坑記錄：java.lang.IncompatibleClassChangeError: null

今天第一次做Springboot項目整合webMagic項目時。剛剛創建的一個新的項目啟動時突然發現報錯了！！！

報錯如下：

java.lang.IncompatibleClassChangeError: null
at us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler.push(DuplicateRemovedScheduler.java:35) ~[classes/:na]
at us.codecraft.webmagic.Spider.setScheduler(Spider.java:197) ~[classes/:na]
at com.itheima.utils.JobProcessor.startSpider(JobProcessor.java:106) ~[classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_131]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_131]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) ~[spring-context-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_131]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_131]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_131]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]

我瘋狂百度呀，找資料呀也找不出個所以然來。

異步任務的代碼：

package com.wangjie.crawlerjob.task;

import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
import us.codecraft.webmagic.Page;
import us.codecraft.webmagic.Site;
import us.codecraft.webmagic.Spider;
import us.codecraft.webmagic.processor.PageProcessor;
import us.codecraft.webmagic.scheduler.BloomFilterDuplicateRemover;
import us.codecraft.webmagic.scheduler.QueueScheduler;
import us.codecraft.webmagic.selector.Html;

/**
 * @author 落魄書生
 * @data 2020/9/23 - 10:50
 */
@Component
public class JobProcessor implements PageProcessor {

    private String url = "https://search.51job.com/list/000000,000000,0000,01%252c32,9,99,java,2,1.html?lang=c&postchannel=0000&workyear=99&cotype=99&degreefrom=99&jobterm=99&companysize=99&ord_field=0&dibiaoid=0&line=&welfare=";


    @Override
    public void process(Page page) {
        Html html = page.getHtml();
        System.out.println(html.toString());

    }


    private Site site = Site.me()
            .setCharset("gbk") // 設置編碼
            .setTimeOut(10000) // 設置超時時間
            .setRetrySleepTime(3000) // 設置 重試的間隔時間
            .setRetryTimes(3); // 設置重試的次數
    
    @Override
    public Site getSite() {
        return site;
    }

    /**
     * initialDelay: 當項目啟動之后 隔多久運行這個定時任務
     * fixedDelay：  每隔多久執行這個方法
     */
    @Scheduled(initialDelay = 1000, fixedDelay = 100 * 1000)
    public void process() {

        Spider.create(new JobProcessor())
                .addUrl(url)
                .setScheduler(new QueueScheduler().setDuplicateRemover(new BloomFilterDuplicateRemover(100000)))
                .thread(10)
                .run();


    }


}

 

后來我就試試換了一個SpringBoot的版本。和把一些亂七八糟的依賴刪掉。把版本降低到2.0.2版本。然后抱着試試的心態竟然成功了！神奇

我又嘗試把版本換回2.3.4版本。發現它又行了。然后就解決。問題是啥我一臉懵逼，解決后我還是一臉懵。自己技術還是太菜了，努力學習吧，騷年~