marrlab
latex code generation for sampled hyperparameters of diva fbopt
If we run any benchmark configuration file, we get a hyperparameters.csv file where each line represent a hyperparameter configuration for a particular algorithm, now we want to show that in a paper, which we need to format into latex. We need a script to transform the csv file into latex for display in a paper.
Below is an example of the benchmark folder where one could find hyperparameters.csv
├── commit.txt ├── config.txt ├── graphics │ ├── diva_fbopt_full │ ├── radar_dist.png │ ├── radar.png │ ├── scatterpl │ ├── sp_matrix_dist.png │ ├── sp_matrix_dist_reg.png │ ├── sp_matrix.png │ ├── sp_matrix_reg.png │ └── variational_plots ├── hyperparameters.csv ├── results.csv └── rule_results ├── 0.csv ├── 1.csv ├── 2.csv ├── 3.csv ├── 4.csv ├── 5.csv ├── 6.csv └── 7.csv
In the results plot, we have
We need to let the reader know what each index correspond to which hyperparameter using the hyperparameter.csv:
cat zoutput/benchmarks/pacs_benchmark_resnet50_model_only/hyperparameter
s.csv
,method,model,params
0,erm,erm,{}
1,diva,diva,"{'gamma_y': 6173000, 'gamma_d': 17000, 'zx_dim': 64, 'zy_dim': 64, 'zd_dim': 32}"
2,diva,diva,"{'gamma_y': 563000, 'gamma_d': 36906000, 'zx_dim': 96, 'zy_dim': 64, 'zd_dim': 96}"
3,diva,diva,"{'gamma_y': 1871000, 'gamma_d': 50395000, 'zx_dim': 0, 'zy_dim': 32, 'zd_dim': 32}"
4,diva,diva,"{'gamma_y': 1568000, 'gamma_d': 7257000, 'zx_dim': 64, 'zy_dim': 64, 'zd_dim': 64}"
5,diva,diva,"{'gamma_y': 114000, 'gamma_d': 12501000, 'zx_dim': 32, 'zy_dim': 64, 'zd_dim': 32}"
6,diva,diva,"{'gamma_y': 2808000, 'gamma_d': 2936000, 'zx_dim': 96, 'zy_dim': 64, 'zd_dim': 64}"
7,diva,diva,"{'gamma_y': 183000, 'gamma_d': 285000, 'zx_dim': 64, 'zy_dim': 64, 'zd_dim': 96}"
8,diva,diva,"{'gamma_y': 12961000, 'gamma_d': 30203000, 'zx_dim': 96, 'zy_dim': 96, 'zd_dim': 64}"
9,hduva,hduva,"{'gamma_y': 183000, 'zy_dim': 64, 'zd_dim': 96}"
10,hduva,hduva,"{'gamma_y': 2808000, 'zy_dim': 64, 'zd_dim': 64}"
11,hduva,hduva,"{'gamma_y': 6173000, 'zy_dim': 64, 'zd_dim': 32}"
12,hduva,hduva,"{'gamma_y': 563000, 'zy_dim': 64, 'zd_dim': 96}"
13,hduva,hduva,"{'gamma_y': 1568000, 'zy_dim': 64, 'zd_dim': 64}"
14,hduva,hduva,"{'gamma_y': 114000, 'zy_dim': 64, 'zd_dim': 32}"
15,hduva,hduva,"{'gamma_y': 1871000, 'zy_dim': 32, 'zd_dim': 32}"
16,hduva,hduva,"{'gamma_y': 12961000, 'zy_dim': 96, 'zd_dim': 64}"
17,dann,dann,{'gamma_reg': 0.46}
18,dann,dann,{'gamma_reg': 0.03}
19,dann,dann,{'gamma_reg': 0.56}
20,dann,dann,{'gamma_reg': 21.400000000000002}
21,dann,dann,{'gamma_reg': 1.3}
22,dann,dann,{'gamma_reg': 0.030000000000000002}
23,dann,dann,{'gamma_reg': 2.9499999999999997}
24,dann,dann,{'gamma_reg': 3.83}
in fbopt branch,
./run_benchmark_standalone https://github.com/marrlab/DomainLab/blob/fbopt/examples/benchmark/pacs_diva_fbopt_and_baselines.yaml
or ./run_benchmark_slurm https://github.com/marrlab/DomainLab/blob/fbopt/examples/benchmark/pacs_diva_fbopt_and_baselines.yaml
snakemake will first sample the hyperparameters, before other jobs being runed, so hyperparameters.csv will be generated, we need a way to put that into latex.
code
import pandas as pd
import json
df = pd.read_csv(f)
df = df.merge(
pd.json_normalize(df['params'].apply(lambda x: json.loads(x.replace("'", '"')))),
left_index=True, right_index=True
).rename(columns={
'Unnamed: 0': 'Configuration',
'method': 'Method', 'model': 'Model',
#'k_i_gain_ratio':
'mu_clip': '$\mu_{clip}$',
'gamma_d': '$\gamma_d$',
'gamma_y': '$\gamma_y$',
'gamma_reg': '$\gamma_{reg}$',
'zx_dim': 'dim($z_x$)',
'zy_dim': 'dim($z_y$)',
'zd_dim': 'dim($z_d$)',
}).drop(columns=['params'])
print(df.to_latex(na_rep='--', index=False, escape=False))
result with the csv above:
\begin{tabular}{rllrrrrrr}
\toprule
Configuration & Method & Model & $\gamma_y$ & $\gamma_d$ & dim($z_x$) & dim($z_y$) & dim($z_d$) & $\gamma_{reg}$ \\
\midrule
0 & erm & erm & -- & -- & -- & -- & -- & -- \\
1 & diva & diva & 6173000.0 & 17000.0 & 64.0 & 64.0 & 32.0 & -- \\
2 & diva & diva & 563000.0 & 36906000.0 & 96.0 & 64.0 & 96.0 & -- \\
3 & diva & diva & 1871000.0 & 50395000.0 & 0.0 & 32.0 & 32.0 & -- \\
4 & diva & diva & 1568000.0 & 7257000.0 & 64.0 & 64.0 & 64.0 & -- \\
5 & diva & diva & 114000.0 & 12501000.0 & 32.0 & 64.0 & 32.0 & -- \\
6 & diva & diva & 2808000.0 & 2936000.0 & 96.0 & 64.0 & 64.0 & -- \\
7 & diva & diva & 183000.0 & 285000.0 & 64.0 & 64.0 & 96.0 & -- \\
8 & diva & diva & 12961000.0 & 30203000.0 & 96.0 & 96.0 & 64.0 & -- \\
9 & hduva & hduva & 183000.0 & -- & -- & 64.0 & 96.0 & -- \\
10 & hduva & hduva & 2808000.0 & -- & -- & 64.0 & 64.0 & -- \\
11 & hduva & hduva & 6173000.0 & -- & -- & 64.0 & 32.0 & -- \\
12 & hduva & hduva & 563000.0 & -- & -- & 64.0 & 96.0 & -- \\
13 & hduva & hduva & 1568000.0 & -- & -- & 64.0 & 64.0 & -- \\
14 & hduva & hduva & 114000.0 & -- & -- & 64.0 & 32.0 & -- \\
15 & hduva & hduva & 1871000.0 & -- & -- & 32.0 & 32.0 & -- \\
16 & hduva & hduva & 12961000.0 & -- & -- & 96.0 & 64.0 & -- \\
17 & dann & dann & -- & -- & -- & -- & -- & 0.46 \\
18 & dann & dann & -- & -- & -- & -- & -- & 0.03 \\
19 & dann & dann & -- & -- & -- & -- & -- & 0.56 \\
20 & dann & dann & -- & -- & -- & -- & -- & 21.40 \\
21 & dann & dann & -- & -- & -- & -- & -- & 1.30 \\
22 & dann & dann & -- & -- & -- & -- & -- & 0.03 \\
23 & dann & dann & -- & -- & -- & -- & -- & 2.95 \\
24 & dann & dann & -- & -- & -- & -- & -- & 3.83 \\
\bottomrule
\end{tabular}
