benschneider
Get a variable from gnuplot
Sometimes it is required to get the values of variables from gnuplot. For example after:
gnuplot> f(x) = m*x + b
gnuplot> fit f(x) 'calibration.data' using 1:2 via m, b
iter chisq delta/lim lambda m b
0 1.5671569869e+02 0.00e+00 4.44e+00 1.000000e+00 1.000000e+00
1 1.6202293571e+00 -9.57e+07 4.44e-01 4.193938e-01 8.336373e-01
2 2.1262262773e-02 -7.52e+07 4.44e-02 4.900177e-01 1.305259e-01
3 1.2002761375e-02 -7.71e+05 4.44e-03 4.994356e-01 6.485437e-02
4 1.2002753199e-02 -6.81e-01 4.44e-04 4.994444e-01 6.479260e-02
iter chisq delta/lim lambda m b
After 4 iterations the fit converged.
final sum of squares of residuals : 0.0120028
rel. change during last iteration : -6.81204e-07
degrees of freedom (FIT_NDF) : 8
rms of residuals (FIT_STDFIT) = sqrt(WSSR/ndf) : 0.0387343
variance of residuals (reduced chisquare) = WSSR/ndf : 0.00150034
Final set of parameters Asymptotic Standard Error
======================= ==========================
m = 0.499444 +/- 0.004265 (0.8538%)
b = 0.0647926 +/- 0.02646 (40.84%)
correlation matrix of the fit parameters:
m b
m 1.000
b -0.886 1.000
gnuplot> print m_err
0.00426450345234437
gnuplot> print b_err
0.0264605480528866
gnuplot>
Now from Python you can access these variables.
Thank you very much for the contribution! This is indeed quite good and would give access to the superior Gnuplot fit function.
The problem comes with the "Popen.communicate()" function, as it terminates the pipe after usage: https://stackoverflow.com/questions/25754045/python-subprocess-i-o-operation-on-closed-file So I am looking for a way to avoid it, and allow multiple readouts.
Thank you very much for the contribution! This is indeed quite good and would give access to the superior Gnuplot fit function.
The problem comes with the "Popen.communicate()" function, as it terminates the pipe after usage: https://stackoverflow.com/questions/25754045/python-subprocess-i-o-operation-on-closed-file So I am looking for a way to avoid it.
Oh, I'm sorry, I hurried. What if you rewrite it like this: Library
import sys
from subprocess import PIPE, Popen
from threading import Thread
try:
from queue import Queue, Empty
except ImportError:
from Queue import Queue, Empty # python 2.x
ON_POSIX = 'posix' in sys.builtin_module_names
def enqueue_std(out, queue):
for line in iter(out.readline, ''):
queue.put(line)
out.close()
class Gnuplot(object):
"""docstring for Gnuplot"""
def __init__(self):
self.p = Popen(['gnuplot', '-p'], stdin=PIPE, stderr=PIPE, stdout=PIPE, bufsize=1,
close_fds=ON_POSIX, shell=False, universal_newlines=True)
self.q_err = Queue()
self.t_err = Thread(target=enqueue_std, args=(self.p.stderr, self.q_err))
self.t_err.daemon = True # thread dies with the program
self.t_err.start()
self.q_out = Queue()
self.t_out = Thread(target=enqueue_std, args=(self.p.stdout, self.q_out))
self.t_out.daemon = True # thread dies with the program
self.t_out.start()
def c(self, command):
self.p.stdin.write(command + '\n') # \n 'send return in python 2.7'
self.p.stdin.flush() # send the command in python 3.4+
def get(self, variable, timeout=0.1):
self.p.stdin.write('print ' + variable + '\n')
# read line without blocking
try:
line = self.q_err.get(timeout=timeout) # or .get_nowait()
return line
except Empty:
return None
Use library:
#!/usr/bin/env python3
from gnuplot import Gnuplot as gp
figure1 = gp()
figure1.c('set terminal postscript eps color enhanced "Helvetica" 30')
figure1.c('set output "%s"' % 'result.ps')
figure1.c('set size 1.25,1.33')
figure1.c('set pointsize 2')
figure1.c('set nokey')
figure1.c('set linestyle 1 linetype -1 linewidth 1')
figure1.c('set xlabel "Detector"')
figure1.c('set ylabel "Calibration Quantity"')
figure1.c('set title "Title"')
figure1.c('plot sin(x)')
my_pi = figure1.get('pi')
print(my_pi)
And:
./test.py
3.14159265358979
ls
gnuplot.py __pycache__ test.py result.ps
Thanks again,
This is a good find, and seems to work.
In fact, the communicate function also seems use a thread for the readout https://github.com/python/cpython/blob/master/Lib/subprocess.py
I will experiment with the options a little and see what works best.
I will use your suggested code since it is very clean and works very well! Thank you very much Dmitry, excellent work!
I made small edits. When you request the value of a variable, there may be data in the buffer. They need to reset. Also, after sending the command to gnuplot, you may need to read the errors or the log.
import sys
from subprocess import PIPE, Popen
from threading import Thread
try:
from queue import Queue, Empty
except ImportError:
from Queue import Queue, Empty # python 2.x
ON_POSIX = 'posix' in sys.builtin_module_names
class Gnuplot(object):
"""docstring for Gnuplot"""
def __init__(self):
self.p = Popen(['gnuplot', '-p'], stdin=PIPE, stderr=PIPE, stdout=PIPE, bufsize=1,
close_fds=ON_POSIX, shell=False, universal_newlines=True)
self.q_err = Queue()
self.t_err = Thread(target=self.enqueue_std, args=(self.p.stderr, self.q_err))
self.t_err.daemon = True # thread dies with the program
self.t_err.start()
self.q_out = Queue()
self.t_out = Thread(target=self.enqueue_std, args=(self.p.stdout, self.q_out))
self.t_out.daemon = True # thread dies with the program
self.t_out.start()
def enqueue_std(self, out, queue):
for line in iter(out.readline, ''):
queue.put(line)
out.close()
def c(self, command):
self.p.stdin.write(command + '\n') # \n 'send return in python 2.7'
self.p.stdin.flush() # send the command in python 3.4+
def get_c(self, vtype=str, timeout=0.1):
# read line without blocking
lines = []
while True:
try:
line = self.q_err.get(timeout=timeout) # or .get_nowait()
lines.append(vtype(line.strip()))
except Empty:
break
return lines
def get(self, variable, vtype=float, timeout=0.1):
# clean buffer before put command
self.get_c()
self.p.stdin.write('print ' + variable + '\n')
# read line without blocking
try:
line = self.q_err.get(timeout=timeout) # or .get_nowait()
return vtype(line.strip())
except Empty:
return None
use:
#!/usr/bin/env python3
from gnuplot import Gnuplot as gp
figure1 = gp()
figure1.c('set terminal postscript eps color enhanced "Helvetica" 30')
figure1.c('set output "%s"' % 'result.ps')
figure1.c('set size 1.25,1.33')
figure1.c('set pointsize 2')
figure1.c('set nokey')
figure1.c('set linestyle 1 linetype -1 linewidth 1')
figure1.c('set xlabel "Detector"')
figure1.c('set ylabel "Calibration Quantity"')
figure1.c('set title "Title"')
figure1.c('set fit logfile "/dev/null"')
figure1.c('set fit quiet')
figure1.c('FIT_LIMIT = 1e-6')
figure1.c('f(x) = m*x**2 + b')
figure1.c('fit f(x) "calibration.data" using 1:2 via m, b')
figure1.c('plot "calibration.data" using 1:2, f(x) w l')
# !!! wrong command !!!
figure1.c('wrong command')
log = figure1.get_c()
print("\n".join(log))
m = figure1.get('m')
b = figure1.get('b')
m_err = figure1.get('m_err')
b_err = figure1.get('b_err')
print("m = {0}".format(m))
print("b = {0}".format(b))
print("m_err = {0}".format(m_err))
print("b_err = {0}".format(b_err))
./regression.py
gnuplot> wrong command
^
line 0: invalid command
m = 0.04317673009566
b = 1.14943292650526
m_err = 0.00344378125924026
b_err = 0.173332048570009
I do not know how to check the result of the execution of each command. The timeout is now used, since the execution of a command in gnuplot may take some time. If you check after each command, it will be very slow.
I also found it very strange: gnuplot does not use stdout. It sends the entire message (data output or error messages) to strerr! It is very uncomfortable. Cannot separate messages and errors.
Wow! Just need to read the documentation.
gnuplot> help set print
The `set print` command redirects the output of the `print` command to a file.
Syntax:
set print
set print "-"
set print "<filename>" [append]
set print "|<shell_command>"
set print $datablock [append]
`set print` with no parameters restores output to <STDERR>. The <filename>
"-" means <STDOUT>. The `append` flag causes the file to be opened in append
mode. A <filename> starting with "|" is opened as a pipe to the
<shell_command> on platforms that support piping.
The destination for `print` commands can also be a named data block. Data
block names start with '$', see also `inline data`.
To separate messages into errors and data, you need to say in the object's constructor:
self.c('set print "-"')
Thats actually explains a lot, also why my earlier attempts to just read stdout failed.