Validate embedding and wic caching

[Garrafao]

@sachinn12 reported inconsistenties across model run repetitions. I added a first test for cache validation:

LSCDBenchmark/tests/integration/test_cache.py

Line 19 in d99728c

def test_bert_embedding_apd_change_graded_ger(self) -> None:

The test runs through for me 1. when disabling embedding caching, 2. when disabling wic caching, and 3. when disabling both. I was thus not able to find caching inconsistencies yet. But, these could be related to specific datasests, or models (e.g. DM). We should add more tests.
For now, caching wic and embedding caching was disabled here:

LSCDBenchmark/src/wic/model.py

Line 77 in d99728c

#self._save_cache() # temporarily disabled

LSCDBenchmark/src/wic/contextual_embedder.py

Line 466 in d99728c

#if self.embedding_cache is not None:

Uncomment to enable caching again.

We should check whether caching is dataset-specific or not.

#82 may be related.

[Garrafao]

Embedding cache validated and enabled with Commit 831f940.

Current embedding cache with example:

     dataset.name dataset.preprocessing dataset.spelling_normalization._'s dataset.spelling_normalization._n't dataset.spelling_normalization._'ve dataset.spelling_normalization._'d dataset.spelling_normalization._'re dataset.spelling_normalization._'ll  contextual_embedder.pre_target_tokens contextual_embedder.ckpt                                    id target
0  testwug_en_111                   raw                                 's                                 n't                                 've                                 'd                                 're                                 'll                                    0.5          bert-base-cased  036590a8-ec21-43b0-8b86-78d1646a5ee5    arm
/mount/projekte50/projekte/cik/users/dominik/LSCDBenchmark/results/.bert/036590a8-ec21-43b0-8b86-78d1646a5ee5.pkl

Cache keys are defined in variable self.metadata with is set in a config yaml:

LSCDBenchmark/conf/task/wic/contextual_embedder.yaml

Line 34 in 69a566f

embedding_cache:

Here, keys for layers, etc. should be added.

[Garrafao]

For WiC cache we should add metadata variable in config:

LSCDBenchmark/conf/task/wic.yaml

Line 3 in 69a566f

And call it here:

LSCDBenchmark/src/wic/model.py

Line 51 in 69a566f

self._cache = self.as_df()

[Garrafao]

Implement discowug embedding cache test because of problems observed in #94 .

[Garrafao]

Commit bc2ad42 fixes and validates the wic cache. WiC caching is defined seperately in the configs for embedders and dm. By default it is disabled for embedders but enabled for dm. It can be enabled from cmdlne by override (see tests). We have integration tests demonstrating the usage. It is much more parallel to the embedding cache now, but not entirely. It stores all predictions in same file (inefficient) and does not have the target word as key (thus relies on unique identifiers within datasets across words).