Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ The framework is still largely a work in progress. I'd like to add more examples
To train an LSTM language model on the Mikolov data with the specified hyperparameters in `lm.py`, first specify the framework directory and run the example script with:

export SEQ2SEQROOT=/directory/of/repository
bash runs/example/lm_ptb_train.py
bash runs/examples/lm_ptb_train.py

The hyperparmeters specify a very large language model (512 input dim, 1024 hidden dim, 3 layer LSTM)! I trained using a Titan X (12GB), and tuned the memory accordingly. With it, training was possible at 383s per epoch, and after epoch 31 the validation shows:

Expand All @@ -21,9 +21,9 @@ The hyperparmeters specify a very large language model (512 input dim, 1024 hidd
Done. Validation loss: 338222.015259. Average Token loss: 4.585439. Perplexity: 98.046266.
Monitored quantity improved.

The validation results and checkpointed models should be saved in `runs/example/` (specified in the script). Test the checkpointed model with:
The validation results and checkpointed models should be saved in `runs/examples/` (specified in the script). Test the checkpointed model with:

bash runs/example/lm_ptb_test.sh
bash runs/examples/lm_ptb_test.sh

I was able to achieve these results on the test set:

Expand All @@ -42,7 +42,7 @@ In this example, we will use the silver parse trees in `data/parse/` generated f

To train a seq2seq parser with the specified hyperparameters in `seq2seq.py`, run:

bash runs/example/seq2seq_ptb_train.sh
bash runs/examples/seq2seq_ptb_train.sh

The hyperparmeters specify a very large seq2seq model (512 input dim, 512 hidden dim, 2 layer encoder/decoder)! I trained using a Titan X (12GB), and tuned the memory accordingly. With it, training was possible at 349s per epoch, and after epoch 45 the validation shows:

Expand Down
2 changes: 1 addition & 1 deletion runs/examples/lm/lm_ptb_test.sh
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
python $SEQ2SEQROOT/test.py \
--run runs/example/ \
--run runs/examples/ \
--model LSTMLanguageModel \
--populate lm.model \
--test data/lm/test \
Expand Down
2 changes: 1 addition & 1 deletion runs/examples/lm/lm_ptb_train.sh
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
python $SEQ2SEQROOT/main.py \
--run runs/example/ \
--run runs/examples/ \
--model LSTMLanguageModel \
--train data/lm/train \
--dev data/lm/valid \
Expand Down
2 changes: 1 addition & 1 deletion runs/examples/seq2seq/seq2seq_ptb_test.sh
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
python $SEQ2SEQROOT/test.py \
--run runs/example/ \
--run runs/examples/ \
--model LSTMLanguageModel \
--populate lm.model \
--test data/lm/test \
Expand Down