Not quite matching Mogrifier evaluation results #6
Description
Hi,
I tried training and evaluating the Mogrifier LSTM on the standard PTB data using the commands described here:
https://github.com/deepmind/lamb/tree/master/lamb/experiment/mogrifier
The output contains:
I0427 15:09:37.012961 140302239332160 evaluation.py:73] final valid_det_t0.8800000000000001 xe: 3.96009 (73760.0), batches: 1054
I0427 20:18:13.086667 139838128674624 evaluation.py:255] final valid_mca_d0.6_t0.92 xe: 3.94770 (73760.0), batches: 1054 (ns=200)
Converting the cross-entropy values to perplexities I get:
e^3.96009 = 51.8
e^3.94770 = 52.5
The paper reports 51.4 and 52.1 respectively. At 0.4, the difference could just be due to some uncontrolled source of randomness, but I wanted to check if I was missing anything.
I set up the code as described in the repository. To provide full information about what I have installed, I've included the results from conda list below. I'm running with a single Titan X, which has 12 Gb of RAM (I did not need to change max_time_steps for it to run).
Thanks!
# packages in environment at /home/jkummerf/anaconda3/envs/tfp3.7:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main
_tflow_select 2.1.0 gpu
absl-py 0.9.0 py37_0
astor 0.8.0 py37_0
blas 1.0 mkl
c-ares 1.15.0 h7b6447c_1001
ca-certificates 2020.1.1 0
certifi 2020.4.5.1 py37_0
contextlib2 0.6.0.post1 pypi_0 pypi
cudatoolkit 10.0.130 0
cudnn 7.6.5 cuda10.0_0
cupti 10.0.130 0
dm-sonnet 1.36 pypi_0 pypi
dm-tree 0.1.4 pypi_0 pypi
gast 0.2.2 py37_0
google-pasta 0.2.0 py_0
grpcio 1.27.2 py37hf8bcb03_0
h5py 2.10.0 py37h7918eee_0
hdf5 1.10.4 hb1b8bf9_0
intel-openmp 2020.0 166
keras-applications 1.0.8 py_0
keras-preprocessing 1.1.0 py_1
lamb 1.0 dev_0 <develop>
ld_impl_linux-64 2.33.1 h53a641e_7
libedit 3.1.20181209 hc058e9b_0
libffi 3.2.1 hd88cf55_4
libgcc-ng 9.1.0 hdf63c60_0
libgfortran-ng 7.3.0 hdf63c60_0
libprotobuf 3.11.4 hd408876_0
libstdcxx-ng 9.1.0 hdf63c60_0
markdown 3.1.1 py37_0
mkl 2020.0 166
mkl-service 2.3.0 py37he904b0f_0
mkl_fft 1.0.15 py37ha843d7b_0
mkl_random 1.1.0 py37hd6b4f25_0
ncurses 6.2 he6710b0_0
numpy 1.18.1 py37h4f9e942_0
numpy-base 1.18.1 py37hde5b4d6_1
openssl 1.1.1g h7b6447c_0
opt_einsum 3.1.0 py_0
pip 20.0.2 py37_1
protobuf 3.11.4 py37he6710b0_0
python 3.7.7 hcf32534_0_cpython
readline 8.0 h7b6447c_0
scipy 1.4.1 py37h0b6359f_0
semantic-version 2.8.4 pypi_0 pypi
setuptools 46.1.3 py37_0
six 1.14.0 py37_0
sqlite 3.31.1 h62c20be_1
tabulate 0.8.7 pypi_0 pypi
tensorboard 1.15.0 pyhb230dea_0
tensorflow 1.15.0 gpu_py37h0f0df58_0
tensorflow-base 1.15.0 gpu_py37h9dcbed7_0
tensorflow-estimator 1.15.1 pyh2649769_0
tensorflow-gpu 1.15.0 h0d30ee6_0
termcolor 1.1.0 py37_1
tk 8.6.8 hbc83047_0
webencodings 0.5.1 py37_1
werkzeug 0.16.1 py_0
wheel 0.34.2 py37_0
wrapt 1.12.1 py37h7b6447c_1
xz 5.2.5 h7b6447c_0
zlib 1.2.11 h7b6447c_3