Clifford Algebra Multi-Vector Transformer

NN - Bruno Francesco Nocera, Leonardo Colosi

Introduction

The task chosen is a NLP binary classification task about sentiment analysis of reviews. Given in input a text review the model has to output if it is a good review or a bad one.In order to solve the sentimental analysis task we have decided to implement a classical transformer model with a custom attention function based on geometric algebra operations (see Geometric Algebra). Our approach involves building an encoder model capable of producing meaningful embeddings with lower dimensionality

How to run

Load the MultiVectorTransformer.ipynb in colab and press Run All button. In the global config cell there are some parameters that could be change to make experiments.

TODO:

• MultiVector embedding
• Positional encoding
• SterableGeometricProduct product attenction function
  • (n,d) (d,n) --> (n,n) multivectros
  • FC for each grade (?)
  • Normalization
  • Optimization (alternative to test)
    • Remove learnable gm
    • Parallelize subspaces projection
    • Use samaller algebra dimention
    • Change geometric product logic
• Test against dot product transformer
• Plot losses
• Metrics