This project is continued in seq2seq models.
The objective of this project is to build a model that can take a sentence with spelling mistakes as input, and output the same sentence, but with the mistakes corrected.
See article that explains how to create the input data (the sentences with spelling mistakes) for this model.
Do not be afraid of changing or adding/removing in the code as you run the notebook. The notebook is an excellent tool for experimenting.
New in TF1.4
If books do not contain all the letters in "letters = ['a','b','c',...]", there will be errors. TF1.4 have Swedish letters, remove these if running English texts.
Running big datasets, e.g. wikipedia, and long sentences will take months on cpu only. Start very small! (In tf2.5 (very small model): very powerful cpu: 478ms/step, very powerful gpu 29ms/step)
TF 1.9 and books2
This notebook has been run in Anaconda3 with Tensorflow 1.9. My environment did not support running on the GPU, which would have run about 100 times faster (even on a 2060).
Books 2 are Swedish government texts which are free to distribute. Use any texts you like.
Note, that the notebook saves checkpoints. It should be relatively easy to add code for reading any such checkpoint, to be inserted before the training code. Use the code from the inferencing part for example. In TF 2.0 it is possible to save weights, and reload and run specific sets of weights any number of times. An other improvement could be to save the model, and reload it. Although, since the version 2.x of TF is very different, it may be more useful to work on a newer codebase.
LSTM TF 2.0
Same use of books folder, but different neural network. One LSTM, bidirectional (and other stuff). This notebook has been run on books2 from scratch. It quickly overfits on such a small sample of text. Allowing caps and adding more texts should improve results, but adding computing time, and the need for memory.
The model seems to work also on TF 2.1 and TF 2.5. At some point it may only work on TF 2.5, because of using new functions.
Errors
Lots of errors are caused by large work units on small gpus. You may need to test smaller batch sizes if gpu memory is limited. Try 32 - 16 - 8 - 4 - 2 - 1, in descending order, to see when the error goes away.
There seems to be a rounding error in TF regarding batches or validation. Exchange the following two lines, with their equivalents in the code.
encoder_inputs = Input(batch_input_shape=(None, max_encoder_seq_length, num_encoder_tokens))
model.fit(
[encoder_input_data, decoder_input_data],
decoder_target_data,
batch_size=batch_size,
epochs=epochs)
If you are running TF 2.4 you may get an error related to Numpy. Follow these suggested edits on array_ops.py: tensorflow/models#9706 My file to edit was in: /home/user/anaconda3/envs/tf24/lib/python3.8/site-packages/tensorflow/python/ops
You might get strange errors about dnn not found or similar. This code can help (insert after import tensorflow as tf).
gpus = tf.config.experimental.list_physical_devices('GPU')
if not gpus:
print("No GPU available")
try:
# Currently, memory growth needs to be the same across GPUs
for gpu in gpus:
tf.config.experimental.set_memory_growth(gpu, True)
logical_gpus = tf.config.experimental.list_logical_devices('GPU')
print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
except RuntimeError as e:
print(e)
Create environment to run the Jupyter notebook in linux
# Works on Ubuntu 20.04
conda create -n esri
# choose any name you like
conda activate esri
conda install -c esri tensorflow-gpu
# *https://askubuntu.com/questions/1338314/setting-up-tensorflow-gpu-conda-environment-with-cuda-11-2-and-cudnn-8-1-8-2-cu
conda install -c anaconda pandas
conda install -c anaconda scikit-learn
conda install anaconda-navigator
anaconda-navigator
[if you ever have to revert back to an earlier version of a package (se versions with: conda list --revisions) and get an error with (replace X with a revision number): conda install --rev X, use: conda install -c esri --rev X]
# Solve the numpy error, if it comes up, as explained above.
I may have forgotten something. Something may have changed since. Please study the conda documentation if you have problems.
The original file has been changed to run on TF 1.4.
In a Terminal paste:
conda create -n tensorflow1.4 python=3.5
(For TF 2.4.1: conda create -n tf24 tensorflow-gpu=2.4.1 python=3.8)
conda activate tensorflow1.4
(For TF 2.4.1: conda activate tf24)
conda install tensorflow=1.4 or pip install tensorflow==1.4
(For TF 2.4.1: skip above step)
conda install -c anaconda pandas
conda install -c anaconda scikit-learn
conda install anaconda-navigator
Run Anaconda:
anaconda-navigator
Choose Jupyter notebook (you may need to install it first, by selecting install) and navigate to the .ipynb file. You need to have a ./books subfolder, below where the file is, with the books in.
In Windows
Install Anaconda and clone the base installation. Install above apps by opening a terminal in the cloned instance. Possibly you will only be able to install only a few TF versions.
When running and you encounter crashes, remove all graphics applications in the Tray, e.g. [manufacturer] settings and [manufacturer] experience.