No running queues Or no feed

[MickeyAl]

Hi all,

I have installed ail using the docker image by following the instructions. The docker image deployment went Successfully but the is no activity on the dashboard.

Any ideas?

thanks in advance.

[mokaddem]

Hi, What is your data source? Are you using pystemon to feed the plateform or are you using the import_dir.py file?

[MickeyAl]

Hi, I'm using pystemon and I edited the ail config file accordingly. Script seems to be running well, but no feeding (same for the pystemon-feeder script)

[mokaddem]

Hi, Sorry for the late response...

Could you share the output of pystemon-feeder.py as well as your configuration file? A misconfiguration could prevent one of the script to pop pastes.

[Asunhack]

Hi, I've the same problem, all working good and I reboot my server and this "error" appear.. You want a screen ?

[mokaddem]

Sure!

[Asunhack]

[mokaddem]

Do you have the latest version of AIL? We added a backoff time (see #169) From what I can see, pystemon is not pushing pastes to redis, so pystemon-feeder.py has nothing to pop. Can you check that pystemon is really fetching pastes? (by checking output logs of pystemon.py or by checking inside the redis server)

[Asunhack]

He really fetching paste, I've recolted paste last week and friday I reboot all and the error appear.. However I've change nothing in the pystemon-feeder.py script... One output of pystemon

So I guess it works good ..

[Asunhack]

I don't know if that can be related but I've this output sometimes from Flask_server.py

[Asunhack]

I did the modification of #169 but nothing change...

[mokaddem]

This error of Flask_server.py is a known 'bug' and is not related to your issue. So, pystemon seems to be downloading. Can you check that the option redis->queue is sets to yes in pystemon.yaml? If it isn't, can you set it and try again?

[Asunhack]

Ok. Yes the option redis->queue is set to yes..

[Asunhack]

I check yhe output of :

(1) Queue_AIL 
(2) Logging_AIL
(3) LevelDB_AIL
(4) Redis_AIL

They were normal

But the outuput of Script_AIL was only : spleeping (exept for Mixer.py)

I don't know if that can be related ?

[mokaddem]

The sleeping output is normal, it just means that there is nothing to process; which is normal as AIL is not receiving pastes. Anyway, it seems that the problem is between pystemon and AIL since pystemon-feeder.py is not receiving data. Can you manually check that pystemon is writing pastes on disk and that it is pushing paths into redis (by doing a monitor or checking inside the key).

[Asunhack]

I think it pystemon didn't writing pastes because in logs/Script_info_-2018-02-13.log I have only that :

How can I check that it is pushinh paths into redis ?

[mokaddem]

The chain is as follow: pystemon -> pystemon-feeder -> mixer -> global -> ...

Does pystemon writes pastes to disk?

To check in redis (with pystemon running):

• connect to the redis server with the redis client: redis-cli -p 6379 -n 10
• execute the monitor command: monitor
• check for the LPUSH instructions

[Asunhack]

In pystemon.yaml I set save-all : yes into archive: And now i'ts working !

Thanks you very much for your help and your time !

[mokaddem]

Good! @MickeyAl does this solution solves your issue?

[cybertschunk]

I've unfortunately the same problem. I'm also using pystemon to feed the framework, but pystemon-feeder isn't giving any output. On the website it says "No running queues or no feed". I'm using an up to date version and followed the setup instructions

[mokaddem]

Can you check your pystemon configuration? Does save-all : yes and queue: yes apply? What is the output of pystemon? Is it downloading pastes?

[Asunhack]

And don't you miss to start pytemon-feeder.py ? It's located in /bin/feeder

[elbae]

Hi, I'm facing the same problem. The scripts pystemon.py and pystemon-feeder.py are running. Redis->queue is enabled in pystemon.yaml, with db 10 and save-all is marked yes. redis-cli -p 6379 -n 10 shows proper activities. Data is save under pystemon dir (ie /opt/pystemon/archive/slexy.org/2019/07/16/s2ZHVerp6i.gz). Do I miss something?

[elbae]

Ok, I was missing this:

# 101 pastes processed feed
# 102 raw pastes feed
topic = '102'

instead of topic = '101'

[DeepCodeSec]

In my case, pystemon was failing because the path defined by archive > dir-all in pystemon.yaml did not exist. After creating the directory archive, pastes were downloaded and fed into AIL.

[albatroaz94]

Hi, I have the same "No running queues" problem. I have followed the instructions and pystemon is downloading the pastes but the problem is as soon as i run pystemon-feeder.py it gives the following error.

root@miyuru-VirtualBox:/home/miyuru/AIL-framework/bin/feeder# ./pystemon-feeder.py Traceback (most recent call last): File "./pystemon-feeder.py", line 19, in import zmq ModuleNotFoundError: No module named 'zmq'

i have applied "save-all : yes" and "queue: yes" and "redis-cli -p 6379 -n 10" is prompting activities too.

Any help would be greatly appriciated. Thank you.

[Asunhack]

Hi, It's the first time that you run AIL ? Do you have the latest version of AIL ? Did you used pystemon.yaml ?

Can you check that redis -> queue is sets to yes in pystemon.yaml ?

And did you check if pystemon if really fetching pastes ..?

[albatroaz94]

Hi, sorry for the late reply. Yes this is my first time and thank you for pointing out that i was using an older version of AIL and i reinstalled the latest version. by the way, the above mentioned error has solved and pystemon-feeder.py showing some activities.

(AILENV) root@miyuru-VirtualBox:~/AIL-framework/bin/feeder# ./pystemon-feeder.py archive/pastebin.com/2019/11/21/3A5mx7nk.gz IOError: Increasing sleep time archive/pastebin.com/2019/11/21/70PAeHKT.gz IOError: Increasing sleep time archive/pastebin.com/2019/11/21/G08XGkJy.gz IOError: Increasing sleep time archive/pastebin.com/2019/11/21/NQk2sNjX.gz IOError: Increasing sleep time archive/pastebin.com/2019/11/21/xqEy29Xi.gz IOError: Increasing sleep time archive/pastebin.com/2019/11/21/3gh1Aps5.gz IOError: Increasing sleep time archive/pastebin.com/2019/11/21/5hQ0kCw0.gz IOError: Increasing sleep time archive/pastebin.com/2019/11/21/QrppxkCg.gz IOError: Increasing sleep time archive/pastebin.com/2019/11/21/z2kcjkVa.gz IOError: Increasing sleep time archive/pastebin.com/2019/11/21/9RWhRadU.gz IOError: Increasing sleep time

Is this normal? redis -> queue is set to yes in pystemon.yaml and i think pystemon is fetching pastes. pastes are saved under pystemon/archive/ directory. The "No running queues Or no feed" remains the same. Any solution?

[mokaddem]

Hey @albatroaz94, Could you check that your pystemon path is set correctly in /configs/core.cfg under the [Directories] section pystemonpath variable. Basically, the pystemonpath should be set so that

pystemonpath + 'archive/pastebin.com/2019/11/21/z2kcjkVa.gz'

is the absolute path.

[rc042]

Hi, I don't know if it's better to open a new thread... I try here and apologies if I digging up this topic.

I have the same issue and the web interface "No running queue or no feed".

I use pystemon from https://github.com/cvandeplas/pystemon because I have some issue with the one provided by CIRCL.

• pystemon is started and file are downloaded in gz
• pystemon-feeder.py is launched and archives seems to be processed [1]
• Redis seems to be working fine, I have the LPUSH [2]
• I checked the different screens but I didn't see something relevant [3]

I probably forget something but I don't know where to search.

Thanks for your feedbacks :)

[1]

archive/gist.github.com/2020/05/07/polRk_772cbed408a6d387ae99ea0ef90d5305.gz
archive/ideone.com/2020/05/07/FGKKy0.gz
archive/ideone.com/2020/05/07/GlVgVc.gz
archive/ideone.com/2020/05/07/Ej9A5o.gz

[2]

1588856982.151460 [0 10.211.55.7:49111] "SELECT" "10"
1588856982.151696 [10 10.211.55.7:49111] "LPUSH" "pastes" "archive/gist.github.com/2020/05/07/polRk_675f464b32540b3e99dbbcfc018f590d.gz"
1588856982.153305 [10 127.0.0.1:37198] "LPOP" "pastes"

[3]

Redis_AIL
ARDB_AIL
Logging_AIL
Queue_AIL
Script_AIL
Flask_AIL

[Gr3gbug]

Hi Guys, I'm in the same situation of @reedcrif . I try to do all the suggested steps, and it seems that all works fine, but i can't see anything on the AIL framework. Some suggestions about this issue? Below you can see all configurations file and some screenshot, in the following order: AIL framework, core.cfg file, pystemon yaml file, pystemon_feeder.py output, pystemon output, redis output. AIL SCREEN core.cfg file of ail Pystemon Yaml file Pystemon feeder output Pystemon Output Redis output