Reproducibility of the training

[ginnocen]

Maja spotted an issue in the reproducibility of the training results. This is a known issue in the DNN training.
I would consider the following options to try to cure it (as suggested in https://keras.io/getting_started/faq/#how-can-i-obtain-reproducible-results-using-keras-during-development)

-try to run the script with this method
CUDA_VISIBLE_DEVICES="" PYTHONHASHSEED=0 python steer_analysis.py
This reduces the randomness in the parallelization. It also implies a very slow training, so use something like 200-300 events for the training only and just 10 epochs.

• Set the random seed in every code you use np and at the very beginning of the macro, before you initialize or load any keras, or TF

import os
import sys
import random
from array import array
import numpy as np
np.random.seed(1001)
import matplotlib

Also make sure that the same seed is not initialized anywhere else in the code. Be aware that initializing np.random.seed()
doesn't initialize e.g. the "random" object in case it is used. What i find cleaner is to use the same object for the randomization everywhere

• Use the option shuffle=False in the function:

model.fit(shuffle=False)

I managed to get consistent and stable results for this version of the code 33d7945 with the modifications described above.

[dsekihat]

Ciao,
I also found that tensorflow-determinism may help reproducibility.
Maybe, you have already found it, too. Just in case,
https://github.com/NVIDIA/framework-determinism
cheers, Daiki

[saganatt]

Ciao,
Thanks @dsekihat for the link - it helped to get more determinism.
My final solution that I use for checking is almost deterministic - I can have at most 2 different results over 5, 10 runs. Enough to detect potential algorithm changes.

1. Replace the beginning of dnn_optimiser.py with:

import os
os.environ['PYTHONHASHSEED'] = "123"
os.environ["CUDA_VISIBLE_DEVICES"] = ""
os.environ['TF_DETERMINISTIC_OPS'] = "1"

import random
random.seed(12334)
import numpy as np
np.random.seed(1001)
import tensorflow as tf
tf.random.set_seed(123)

# Make TF using 1 thread on CPU
tf.config.threading.set_intra_op_parallelism_threads(1)
tf.config.threading.set_inter_op_parallelism_threads(1)

Make sure there are no other seed() calls in the code.

2. Add shuffle=False to the fit() call.

3. Then you can invoke this with just python steer_analysis.py. I also set in *.yml 200 train events, 100 test, 200 apply.