Question About Reproduction Experiment A-GEM

[youichigouda]

Dear author and everyone,

I got the A-GEM software from here github and tried the reproduction experiment described in the paper, but I could not reproduce it well.

I have tried the following steps.

environment:
-OS: ubuntu18.04
-GPU: RTX2080ti
-Anaconda ver4.8.3 (* by "conda -V")

steps:
mkdir anaconda3/work/zwk_agem
cd anaconda3/work/zwk_agem
git clone https://github.com/facebookresearch/agem
cd agem
conda create -n agem27 python=2.7
conda activate agem27
conda install numpy -y
conda install tensorflow-gpu==1.9.0 -y
conda install opencv -y
conda install matplotlib -y
conda install ipython -y

./replicate_results_iclr19.sh MNIST 3

I ran the above and got the pickle file (PERMUTE_MNIST_HERDING_FC-S_True_A-GEM_...._.pickle).
Then I inputed the pickle file into agem_plots.ipynb.
The resulting accuracy was different from the value stated in the paper.
(Thesis: 89.1%, my result: 65.3%)

Could you please tell me how can I get the same results as yours?
Please let me know if there is any information missing.
Thank you very much.

Sincerely,
Yoichi Goda
e-mail: youichi.gouda@gmail.com

[arslan-chaudhry]

Try setting the --examples-per-task flag to full training dataset size (i.e.) 50000. For ICLR'19 work, we used all the examples from the training dataset in MNIST experiments. For 'Tiny Memory in Continual Learning', we used 1000 examples per task. I should correct this in the replicate_results_iclr19.sh.

[youichigouda]

Dear Arslan-Chaudhry,
Thank you very much for your reply.
I'll try your advice, thank you!

[houshangwu]

@youichigouda Did you end up with 89% results as suggested by the authors?

One more thing, I found that the resnet-18-pretrained-imagenet download link has expired, but the author could not be contacted. Could you please help me provide this part of the data. thank you very much.？
Thank you very much.

Sincerely,
houshangwu
e-mail: houshangwu@126.com

[arslan-chaudhry]

To unblock yourself, you can download the pre-trained ResNet-18 from here:

https://www.dropbox.com/s/vs50vpm8c5l7hc6/model.ckpt.data-00000-of-00001?dl=0
https://www.dropbox.com/s/sl6hbk2ajs1b1ae/model.ckpt.index?dl=0
https://www.dropbox.com/s/su0352nuco3dbb8/model.ckpt.meta?dl=0

Replace the lines 31-33 in download_cub_awa.sh with

wget -O model.ckpt.data-00000-of-00001 https://www.dropbox.com/s/vs50vpm8c5l7hc6/model.ckpt.data-00000-of-00001?dl=0
wget -O model.ckpt.index https://www.dropbox.com/s/sl6hbk2ajs1b1ae/model.ckpt.index?dl=0
wget -O model.ckpt.meta https://www.dropbox.com/s/su0352nuco3dbb8/model.ckpt.meta?dl=0

I should send a fix to the original script.

[houshangwu]

@arslan-chaudhry Dear Arslan-Chaudhry,
Thank you very much for your reply.