webmagic icon indicating copy to clipboard operation
webmagic copied to clipboard

动态调整线程数量会报reject错误

Open yuweiming2016 opened this issue 4 years ago • 0 comments

有个需求,需要动态调整爬虫的线程池数量,因为爬虫在运行中,我只能下如下的代码去修改:

//当前运行的线程数是10,希望通过以下代码调整为5 if (spider.getStatus() == Status.Running) { spider.stop(); spider.thread(5); spider.start(); }

但偶尔会得到以下错误,请帮忙看看@code4craft

Exception in thread "Thread-13" java.util.concurrent.RejectedExecutionException: Task us.codecraft.webmagic.thread.CountableThreadPool$1@3ea050c0 rejected from java.util.concurrent.ThreadPoolExecutor@4d86ad21[Shutting down, pool size = 8, active threads = 8, queued tasks = 0, completed tasks = 458] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at us.codecraft.webmagic.thread.CountableThreadPool.execute(CountableThreadPool.java:70) at us.codecraft.webmagic.Spider.run(Spider.java:316) at java.lang.Thread.run(Thread.java:748) 2021-03-13 18:09:07.757 [pool-8-thread-1][] DEBUG org.

yuweiming2016 avatar Mar 13 '21 10:03 yuweiming2016