Unable to recreate paper results

[tagoyal]

Hi,
I am trying to train a new model with the ldc2017 data. I took the config parameters from the gold config files. However, I get a very low blue score on the test data (around 7 bleu).

Could you point me to the correct training procedure and parameters you used for training?

Thanks,
Tanya

[freesunshine0316]

Hi,
How did you preprocess the data? You need to simplify the AMR side and tokenize the English side with a PTB_tokenizer.

[tagoyal]

Hi,
I simplify the amr using the code that you have provided, then create the jsons. I then on to the vocabulary extraction part, with code in the data sub-directory.

When exactly should the tokenization happen?

[tagoyal]

Could the difference be because of the change in dataset? I am running it on the LDC2017T10 dataset? Although the difference between the accuracy I get and the paper accuracy is pretty huge.

Could you share the dataset used in the paper?
Thanks,
Tanya Goyal

[freesunshine0316]

I think the simplifier does tokenization for you, and you could use your own tokenizer too.
If you directly use my released model for decoding, then you have to use the released word embeddings also. Otherwise it's wrong.
I don't think the dataset is the problem, as someone else told me that my model generalizes well on them.

[tagoyal]

I don't use the released model, I use the code to train my own model.
This is the config file I use:

{
"train_path": "data/amr2.0_gold/training.json",
"finetune_path": "",
"test_path": "data/amr2.0_gold/dev.json",
"word_vec_path": "data/vectors_amr2.0.txt.st",
"suffix": " gold3",
"model_dir": "logs_g2s",
"isLower": true,

"pointer_gen": true,
"use_coverage": true,
"attention_vec_size": 300,
"batch_size": 20,
"beam_size": 5,

"num_syntax_match_layer": 9,
"max_node_num": 200,
"max_in_neigh_num": 2,
"max_out_neigh_num": 10,
"min_answer_len": 0,
"max_answer_len": 100,
"learning_rate": 1e-3,
"lambda_l2": 1e-3,
"dropout_rate": 0.1,
"cov_loss_wt": 0.1,
"max_epochs": 10,
"optimize_type": "adam",

"with_highway": true,
"highway_layer_num": 1,

"with_char": true,
"char_dim": 50,
"char_lstm_dim": 100,
"max_char_per_word": 20,

"attention_type": "hidden_embed",
"way_init_decoder": "all",
"edgelabel_dim": 50,
"neighbor_vector_dim": 300,
"fix_word_vec": true,
"compress_input": true,
"compress_input_dim": 300,

"gen_hidden_size": 300,
"num_softmax_samples": 100,
"mode": "ce_train",
"CE_loss":false,
"reward_type":"bleu",

"config_path": "config.json",
"generate_config": false

}

[freesunshine0316]

Can you share some of your processed data? Also I use pretrained embeddings from Glove 840B.

[tagoyal]

These are some of the samples from the training data
sample.txt

I also use the same pre-trained embeddings from Glove.

[freesunshine0316]

It's obvious that you didn't tokenize your sentences. You can try using https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer_PTB.perl, and then you have to re-do everyting, such as extracting vocabulary, extracting embedding, making JSON files for training, dev and test ...

[freesunshine0316]

Hope that can help.

[tagoyal]

Um, do you mean tokenize the "sent" part of the input file?

[freesunshine0316]

Yes, and tokenization is pretty standard for text generation and other natural language processing tasks....

[tagoyal]

Sure, I'll so that then .
Thanks!

[tagoyal]

Can you share the LDC2015E86 dataset so I can compare results against my other models ?

[freesunshine0316]

may I ask for your email address? I assume your institute has the license as you already have the LDC17 dataset.

[tagoyal]

It's tanyagoyal@utexas.edu
Thanks much!