This repo contains pytorch implementations reproducing the results from the paper:
Petra Bosilj, Erchan Aptoula, Tom Duckett, and Grzegorz Cielniak: “Transfer learning between crop types for semantic segmentation of crops versus weeds in precision agriculture”, Journal of Field Robotics (2019)
It relies on the original data used in the paper, which can be found here. If you find this data or code helpful in your research, you can cite our paper (bibtex entry can be found at the bottom of this file).
The list of required python packages can be found in requirements.txt.
The implementation of both the training protocol is in train.py and inference+evaluation in test.py.
To train a model, run the train.py. The command allows the following options:
> python ./train.py --help
usage: train.py [-h] -d PATH -gt {full,partial} [-s HEIGHT WIDTH] [-spi COL_SAMPLES ROW_SAMPLES] [-e EPOCHS] [-pi PRINT_ITERATION] [-se SAVE_EPOCH] -n FILENAME_PREFIX [-m PATH] [-t {continue,finetune}]
optional arguments:
-h, --help show this help message and exit
-d PATH, --data-folder PATH
Path to the dataset.
-gt {full,partial}, --ground-truth {full,partial}
Whether full or partial annotations are used in training. This is important for data normalisation (always done from the test set).
-s HEIGHT WIDTH, --sample-size HEIGHT WIDTH
Input images are divided into samples of dimensions HEIGHTxWIDTH
-spi COL_SAMPLES ROW_SAMPLES, --samples-per-image COL_SAMPLES ROW_SAMPLES
Take at most COL_SAMPLESxROW_SAMPLES from the image
-e EPOCHS, --epochs EPOCHS
Number of training epochs.
-pi PRINT_ITERATION, --print-iteration PRINT_ITERATION
How often the current loss is displayed (in iterations).
-se SAVE_EPOCH, --save-epoch SAVE_EPOCH
How often the current model is saved (in epochs).
-n FILENAME_PREFIX, --net-out-name FILENAME_PREFIX
Prefix of the saved model filenames.
-m PATH, --model PATH
Path to the pre-trained model file.
-t {continue,finetune}, --training_mode {continue,finetune}
Mode in which to continue training. `continue` loads the optimiser and continues training with saved settings. `finetune` loads the model only and reinitialises the optimiser with
finetuning settings.
Example:
This example will run the training on carrots for 100 epochs, displaying the loss every 10 iterations and saving the model every epoch into files with a prefix SegNetBasic_CA17_FS.
> python ./train.py -d /path/to/carrots -gt full -e 100 -pi 10 -se 1 -n SegNetBasic_CA17_FS
To continue training the last model produced by the above command, use (same freqency of displaying loss/saving model):
> python ./train.py -d /path/to/carrots -gt full -e 200 -pi 10 -se 1 -n SegNetBasic_CA17_FS -m SegNetBasic_CA17_FS_e100.pt -t continue
Assuming the best performing model was in epoch 137 and saved in SegNetBasic_CA17_FS_e_137.pt, following will finetune this network for onions using partial annotations only and save the resulting models with a prefix SegNet_Basic_ON17_partial_FT_CA17:
> python ./train.py -d /path/to/onions -gt partial -e 50 -pi 10 -se 1 -n SegNetBasic_ON17_partial_FT_CA17 -m SegNetBasic_CA17_FS_e137.pt -t finetune
Resuming training of a network which was fine-tuned from pre-initialised weights is the same as continuing training of a randomly initialised model (example above):
> python ./train.py -d /path/to/onions -gt partial -e 100 -pi 10 -se 1 -n SegNetBasic_ON17_partial_FT_CA17 -m SegNetBasic_ON17_partial_FT_CA17_e50.pt -t continue
To perform inference and evaluation on the test data use test.py. It allows testing one or multiple models at once, optionally saving the output images predicted by the network, and/or saving the best performing model under a new file name. The command allows the following options:
> python ./test.py --help
usage: test.py [-h] -d PATH -gt {full,partial} [-s HEIGHT WIDTH] [-spi COL_SAMPLES ROW_SAMPLES] -m PATH [PATH ...] [-v] [-si] [-cb FILENAME]
optional arguments:
-h, --help show this help message and exit
-d PATH, --data-folder PATH
Path to the dataset.
-gt {full,partial}, --ground_truth {full,partial}
Whether full or partial annotations are used in training. This is important for data normalisation (always done from the test set).
-s HEIGHT WIDTH, --sample-size HEIGHT WIDTH
Input images are divided into samples of dimensions HEIGHTxWIDTH
-spi COL_SAMPLES ROW_SAMPLES, --samples-per-image COL_SAMPLES ROW_SAMPLES
Take at most COL_SAMPLESxROW_SAMPLES from the image
-m PATH [PATH ...], --model PATH [PATH ...]
Path to the pre-trained model file(s).
-v, --verbose Print per-image accuracy and kappa scores.
-si, --save-images Save output and ground truth images for visual inspection.
-cb FILENAME, --copy-best FILENAME
Copy the best model to FILENAME.
Example:
Test the model saved in the file SegNetBasic_CA17_FS_e120.pt (model after epoch 120), with per-image output and saving the images for visual inspection.
> python ./test.py -d /path/to/carrots -gt full -m SegNetBasic_CA17_FS_e120.pt -v -si
When testing a model trained with partial ground truth it is important to specify this for testing, as training set stats are used for data normalisation before inference.
> python ./test.py -d /path/to/onions -gt partial -m SegNetBasic_ON17_partial_FT_CA17_e75.pt
Test multiple models at once and save a copy of the best model in SegNetBasic_CA17_FS_best.pt.
> python ./test.py -d /path/to/carrots -gt full -m SegNetBasic_CA17_FS_*.pt -cb SegNetBasic_CA17_FS_best.pt
The dataloader is implemented in dataset/crop_datasets.py and uses transformations defined in dataset/transforms.py. It assumes a 80-20 data split (train: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16] test: [17, 18, 19, 20]) which can be changed by including train.txt or text.txt file in the data folder containing folder names for training or testing (other set determined automatically).
Two dataset specific parameters are:
sample_size Optional[Tuple[int, int]]: input images will be cut into tiled samples of this size. Default value(384x512)matches the input size of SegNetBasic (originalcaffeimplementation)samples_per_image: Optional[Tuple[int, int]]: number of samples to take from each image in x- and y-direction (if different from the maximum allowed bysample_size)partial_truth: bool: load partial grount truth, with certain pixels unlabelled, for training (ignored while testing)
Examples of using this dataloader can be seen by running the file such as:
> python crop_dataset.py
and can also be found in dataloader_demo.ipynb. They also show how to access class probabilities calculated from the test data (which is used to set the weighting for the loss function). Unlabelled data is ignored while calculating class probabilities.
Note: the dataloader has now been tested for loading the partial ground truth, but the training configuration does not process it yet. Need to include ignore_index in the optimizer.
The network is implemented in segnet/segnet_basic.py. However, for proper behavour when training (from scratch) it is important to initialise the bias learning rates to x2 of other weight parameters learning rates. It is also important to apply weight decay only to the weights and not biases if used. For proper fine-tuning behaviour, conv-classifier learning rates should additionally be increased by x10. These details were encoded in the SegNetBasic architecture in caffe but have to be set up after net initialisation in pytorch. They are encapsulated in segnet.utils.init_SGD_optimizer().
In addition to the standard parameters in_channels and num_classes, the implementation allows adjusting the depth of the encoder-decoder through depth parameter (default 4 like in the original caffe implementation) and adding an additional batch normalisation layer following the classification layer with final_batch_norm param (not used in the original implementation).
Step-by-step explanation of how the learning rate and optimiser are through segnet.utils.init_SGD_optimizer() can be found in set_segnet_lr.ipynb.
@article{bosilj2019transfer,
author = {Bosilj, Petra and Aptoula, Erchan and Duckett, Tom and Cielniak, Grzegorz},
title = {Transfer learning between crop types for semantic segmentation of crops versus weeds in precision agriculture},
journal = {Journal of Field Robotics},
volume={37},
number={1},
pages={7--19},
year={2020},
publisher={Wiley}
}