IOError: No "ipcontroller-engine.json"

[jedie]

Unser Linux:

(DrQueue_env)jens@vm:~/DrQueue_env$ source set_settings 
++ DRQUEUE_POOL='DrQueue pool 1'
++ export DRQUEUE_POOL
++ DRQUEUE_ROOT=/home/jens/drqueue
++ export DRQUEUE_ROOT
++ IPYTHON_DIR=/home/jens/drqueue/ipython
++ export IPYTHON_DIR

(DrQueue_env)jens@vm:~/DrQueue_env$ drqueue --verbose --no-ssh master
Running DrQueue master on 127.0.1.1 with PID 20648.
MongoDB started on 127.0.0.1 with PID 20654. Logging to /home/jens/drqueue/logs/mongodb.log.
Waiting for MongoDB to start up . . . 
IPython controller started with PID 20669. Logging to /home/jens/drqueue/logs/ipcontroller.log.

(DrQueue_env)jens@vm:~/DrQueue_env$ drqueue --verbose --no-ssh slave
+ drqueue --verbose --no-ssh slave
Traceback (most recent call last):
  File "/home/jens/DrQueue_env/bin/drqueue", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/home/jens/DrQueue_env/src/drqueueipython/bin/drqueue", line 1024, in <module>
    main()
  File "/home/jens/DrQueue_env/src/drqueueipython/bin/drqueue", line 212, in main
    for line in open(eo_file).readlines():
IOError: [Errno 2] No such file or directory: '/home/jens/drqueue/ipython/profile_default/security/ipcontroller-engine.json'

Under windows the same, stating the slave:

IOError: [Errno 2] No such file or directory: 'C:\\Users\\jens\\AppData\\Roaming\\DrQueueIPython\\drqueue\\ipython\\profile_default/security/ipcontroller-engine.json'

What's this ipcontroller-engine.json ???

I found nothing here: https://ssl.drqueue.org/redmine/projects/drqueueipython/search?utf8=%E2%9C%93&q=ipcontroller-engine.json&scope=&all_words=&all_words=1&titles_only=&wiki_pages=1&submit=OK

[kaazoo]

The ipcontroller command should automatically create those json files:

ipcontroller
2015-07-25 02:12:28.694 [IPControllerApp] Using existing profile dir: u'/usr/local/drqueue/ipython/profile_default'
2015-07-25 02:12:28.712 [IPControllerApp] Hub listening on tcp://127.0.0.1:51247 for registration.
2015-07-25 02:12:28.715 [IPControllerApp] Hub using DB backend: 'NoDB'
2015-07-25 02:12:28.970 [IPControllerApp] hub::created hub
2015-07-25 02:12:28.971 [IPControllerApp] writing connection info to /usr/local/drqueue/ipython/profile_default/security/ipcontroller-client.json
2015-07-25 02:12:28.972 [IPControllerApp] writing connection info to /usr/local/drqueue/ipython/profile_default/security/ipcontroller-engine.json
2015-07-25 02:12:28.973 [IPControllerApp] task::using Python leastload Task scheduler
2015-07-25 02:12:28.973 [IPControllerApp] Heartmonitor started
2015-07-25 02:12:28.993 [IPControllerApp] Creating pid file: /usr/local/drqueue/ipython/profile_default/pid/ipcontroller.pid
2015-07-25 02:12:29.008 [scheduler] Scheduler started [leastload]
2015-07-25 02:12:29.011 [IPControllerApp] client::client '\x00\xfe1\xff\xb9' requested 'connection_request'
2015-07-25 02:12:29.012 [IPControllerApp] client::client ['\x00\xfe1\xff\xb9'] connected
^C

The json files are deleted when ipcontroller stops.

Was the slave started while the master was still running? Were they both started on the same machine?

[jedie]

Hm. So i must start the "ipcontroller" and then the "master" process?

From the Wiki i found only on https://ssl.drqueue.org/redmine/projects/drqueueipython/wiki/Wiki#Design-changes this info:

• Master runs IPController
• Slave runs IPEngine

But nothing about how to start them...

[kaazoo]

The ipcontoller process is started by the master program. You'll see it in the terminal output. First it starts MongoDB and after that it starts ipcontroller. Once ipcontroller is up, ipengine instances (wrapped by slave program) can connect to ipcontroller. They use the information stored in ipcontroller-engine.json. Clients (control programs for manipulating tasks/jobs) use he information stored in pcontroller-client.json.

[jedie]

The ipcontoller process is started by the master program. ... Clients (control programs for manipulating tasks/jobs) use he information stored in pcontroller-client.json.

My setup will look like this:

• master run on linux
• slave run on windows

So the slave must have access to the ipcontroller-engine.json created on the linux master ?!? e.g.: create the "DrQueue directories" on a SMB share ?!?

[kaazoo]

Yes, you can run a Samba server on the master. This can be mounted as a network drive on Windows. In the job template the path needs to be replaced with the name of the network drive. An alternative is to use SSH. On Windows you could use for example https://code.google.com/p/win-sshfs/ .

[jedie]

OK, i'm working on a linux+samba / windows setup...

I create a samba share with: https://github.com/jedie/DrQueueHelperFiles/blob/master/linux/setup_samba_share.sh

But the direcory /ipython/profile_default/security/ and the files in there has restricted access permissions:

• 0700 the dir /ipython/profile_default/security/
• 0600 the *.json files

I add a work-a-round with: https://github.com/jedie/DrQueueHelperFiles/commit/e22166391585522f492ee93fbe83ae58bb1d0829

But this is not really a good idea, isn't it?!?

[kaazoo]

As far as I understand, IPython locks the permissions to the user running the ip_controller (DrQueue master). So it would work without problems if you have the same user (e.g. 'drqueue' on the linux server runnning the master and also for connecting via Samba. The Samba user of course has to map to the correct local user.

The permission fix would have to be executed every time you restart the master. So, it's not rellly elegant. drqueue securitytries to do the same (see https://github.com/kaazoo/DrQueueIPython/blob/master/bin/drqueue_security).