conclusions.tex

\chapter{Contributions and Implications}
\label{cha:conclusions}

In this closing chapter, we provide an overview of the whole thesis. In section \ref{sec:rq} we take another look at the research questions and the manner in which they were approached in each part of the thesis. The achievements made through the course of this study are laid out in Section \ref{sec:achievements}. Finally, a brief summary of the thesis is provided, as well as some future directions.

\section{Research Questions}
\label{sec:rq}
In this section, we revisit the research questions and briefly discuss the ways they have been approached in this work.

\textbf{RQ-A}: \textit{What are the available supervised and unsupervised STS methods, and how do they perform in multilingual and multi-domain settings?}

The first part of the thesis focussed on various supervised and unsupervised STS methods. Embedding aggregation-based STS methods were explored, along with sentence encoders, Siamese neural networks and transformers in STS. All the methods were evaluated using three popular English STS datasets. Furthermore, for each STS method, we analysed the ability of the method to perform in a multilingual and multi-domain setting by evaluating them in Arabic, Spanish and bio-medical STS datasets. The results indicated that the best supervised STS method is the one based on transformers, and that sentence encoders are the best unsupervised STS method.

\textbf{RQ-B}: \textit{Can the neural STS methods be applied in TMs? How efficient and effective are they compared to the real-world TM tools?}

The second part of the thesis focussed on applying the developed neural STS methods in TMs. Considering the accuracy and efficiency, we picked three sentence encoders; Infersent, Universal Sentence Encoder and SBERT. We designed a TM matching algorithm based on these. We then evaluated the proposed algorithm using a real-world TM; DGT-TM. The results from each of the sentence encoders were compared with the results from Okapi, which uses edit distance to acquire the best match from the translation memory. The results showed that our approach returns better matches than Okapi in many cases. This was confirmed with an automatic evaluation as well as with a human evaluation. Furthermore, the proposed algorithm is very fast and can be used in real-world applications.

Our conclusion is that the STS methods have progressed to a point where they can improve TM matching and retrieval. Unlike those of the early days, the current neural models are optimised for real-world applications. Therefore, we recommend that these methods should be employed widely in TM matching and retrieval.

\textbf{RQ-C}: \textit{Can the state-of-the-art STS methods be adapted to the QE task? Can these simple STS architectures outperform current complex QE methods?}

The third part of the thesis focussed on applying the neural STS methods developed here to translation quality estimation. The current state-of-the-art in QE is neural models such as OpenKiwi \autocite{kepler-etal-2019-openkiwi} and DeepQuest \autocite{ive-etal-2018-deepquest}. However, these neural QE architectures are complex and need a lot of computing resources to train a QE model. We proposed redefining the QE task as a cross-lingual STS task and applying the STS architectures we experimented with in Part I of the thesis to QE to address this limitation. We applied the state-of-the-art, transformer-based STS methods to sentence-level QE by changing the input embeddings to be cross-lingual. This approach was very simple compared to the existing QE solutions. Our approach outperformed existing QE methods in 15 language pairs providing state-of-the-art results. We extended the architecture to word-level QE and evaluated the proposed architecture in eight language pairs. Our simple word-level architecture outperformed other complex neural models in all the language pairs providing state-of-the-art results. 

Our conclusion is that the state-of-the-art STS methods can easily be adapted to the QE task by changing the embeddings to be cross-lingual. These architectures are simple compared to the existing neural architectures in OpenKiwi and DeepQuest, yet the proposed architectures in this study outperformed them in all the language pairs. 

\section{Achievements}
\label{sec:achievements}
This study was conceived in order to apply STS methods in the applications of translation technology. First, we studied and employed different STS methods and then applied them in translation memories and translation quality estimation.

For STS in Part I of the thesis, we started with unsupervised STS methods. In Chapter \ref{cha:sts_state_of_the_art_methods}, we employed vector aggregation-based unsupervised STS methods. For the first time, we integrated contextual word embedding models such as BERT, ELMo and Flair in vector-aggregation-based unsupervised STS methods. We showed that vector-aggregation-based STS methods could be improved with contextual embeddings. Our method is the current state of the art in vector-aggregation-based STS methods in several datasets \autocite{ranasinghe-etal-2019-enhancing}. In Chapter \ref{cha:sts_siamese_neural_networks}, we employed Siamese neural networks in the STS task. While in 2018, an LSTM-based Siamese neural network was the state of the art in STS, we improved this architecture by including simple GRUs in the network. Our GRU-based Siamese neural network outperformed the LSTM-based Siamese neural network architecture in multiple datasets, providing state-of-the-art results before the transformer era. This GRU based Siamese neural network stands as one of the best STS methods based on traditional word embeddings for multiple STS datasets \autocite{ranasinghe-etal-2019-semantic}.

For TMs in Part II of the thesis, we started by analysing existing TM systems. The majority of these systems are based on edit distance. One of the major weaknesses of edit-distance-based TM systems is that they cannot retrieve semantically similar segments from the TM. As a result, third-generation TM systems have been proposed. Even though these third-generation TM systems have addressed the limitations of the edit-distance-based TM systems, they are not popular in the community since they are largely inefficient, and there is not much performance gain in using them. Addressing this gap, in Chapter \ref{cha:tm_sentence_encoders}, we proposed to use deep-learning-based STS methods in a TM retrieval task. Considering the efficiency, we proposed a novel TM retrieval method based on sentence encoders and compared its performance against a popular TM system, Okapi. We showed that in most cases the method we proposed can retrieve better matches than Okapi. Furthermore, the proposed method is fast enough to be used in real-world applications. This is the first study that employs neural sentence encoders in a TM retrieval task. We believe our contribution will pave the way for a new direction in the development of third-generation TM systems \autocite{ranasinghe-etal-2020-intelligent, ranasinghe:2021}.

For QE in Part III of the thesis, we started with the analysis of the existing QE systems. The majority of these QE systems are based on complex neural network architectures and need a lot of computing resources to train a QE model, which we have identified as a major limitation. To address this weakness, we propose to redefine the QE task as a cross-lingual STS task and apply the STS architectures we experimented with in Part I of the thesis to QE, which are considerably simpler than the existing QE models. First, we proposed two STS architectures based on transformers for the sentence-level QE task. Then we extended the best sentence-level architecture for word-level QE by changing the output layer. We evaluated the proposed algorithms in 15 different language pairs. We showed that our simple method outperforms the current QE systems in all the languages at both the sentence-level and the word-level, providing state-of-the-art results. The approaches that we proposed for QE are rapidly gaining popularity due to their superior performance, and we believe that our contributions have already revolutionised the QE field \autocite{ranasinghe-etal-2020-transquest, ranasinghe-etal-2020-transquest-wmt2020}.  Furthermore, the multilingual QE approaches we proposed in Chapter \ref{cha:qe_multilingual} would be beneficial for many low-resource languages for which training data is difficult to find \autocite{ranasinghe-etal-2021-exploratory}.

The key points of strength in this work can be summarised as follows:


\paragraph{The use of state-of-the-art methods} - In this work, we employed state-of-the-art methodologies and kept improving the architectures of the models in newer experiments as the field of NLP was concurrently progressing. For instance, we adapted state-of-the-art transformers in our study. Even for the unsupervised STS approaches, we utilised contextual embeddings. Furthermore, we employed architectures and embeddings that have proven stable and reliable in a variety of tasks. For example, the cross-lingual embedding model we used in Part III of the thesis has been evaluated on various cross-lingual benchmarks, proving this approach's reliability.

\paragraph{Multilingualism} - A common theme in our study was that we investigated models that are not dependent on a language and do not need language specific processing. Almost all the experiments that were conducted in the course of this research are language-independent by employing embeddings. To demonstrate that, we constantly evaluated our models in a variety of languages. For example, the QE method we proposed in Part III of the thesis was evaluated in 15 language pairs, including low-resource languages such as Sinhala and Nepalese. We believe that the multilingual nature of our experiments would benefit many languages. 

\paragraph{Open access nature of the research: replicability and creation of new resources in the form of code} - For almost all the experiments that were conducted in the course of this research, the code and the machine learning models have been made available to let other researchers replicate or experiment further in the same direction. In most cases, we have used established and standard datasets to compare the proposed methods with the available methods. Out of the codebases released in the study, two projects have been extremely popular in the community. The transformer-based STS method in Chapter \ref{cha:sts_transformers} was released as a Python library named \textit{ststransformers} and has more than 3,000 downloads from the community\footnote{The latest download statistics for \textit{ststransformers} is available on \url{https://pepy.tech/project/ststransformers}}. The QE framework, \textit{TransQuest}, which was released for Part III of the thesis, has more than 10,000 downloads. Furthermore, the various pre-trained models released for \textit{TransQuest} have more than 12,000 downloads\footnote{The latest download statistics for \textit{TransQuest} framework is available on \url{https://pepy.tech/project/transquest}. The number of downloads for each pre-trained \textit{TransQuest} model is available on \url{https://huggingface.co/TransQuest}}. With these statistics, we believe that the software released in this study has been highly useful and has created a significant impact on the NLP community.

\paragraph{Competitiveness} - In this study, we aimed to build the best approach possible for a given task. We always compared our method to the best systems available in each dataset. For instance, we evaluated the QE method we developed on the WMT 2020 QE shared task. Our approach won first place in all the language pairs, outperforming all other participants across the world, proving our study's competitiveness.



\section{Summary and Future Directions}
To conclude, this thesis focused on employing deep-learning-based STS methods in the applications of translation technology. We developed language-independent STS methods and utilised them in two applications in translation technology, translation memory systems and translation quality estimation. We made original contributions to each of the applications. We provided the best existing algorithm to perform translation memory matching and retrieval. The QE method we proposed in this study using STS architectures also provided state-of-the-art results outperforming other QE systems and winning the WMT 2020 QE shared task, outperforming the competition of all leading research labs across the globe. Our models employed sound and reliable machine learning techniques and proved remarkably effective across multiple languages.

As future work, we would like to expand the experiments we conducted in this study to different languages and domains, focussing more on low-resource languages. Furthermore, the ML community is moving towards explainable ML solutions. In this vein, we would also like to explore the explainability of our proposed ML models in future work.