Attentive interface by varisd · Pull Request #802 · ufal/neuralmonkey

varisd · 2019-03-15T14:16:09Z

Moved attention-related attributes/methods to a separate class Attentive. Every decoder that requires computing attention against the encoders should inherit this class.

jindrahelcl

Jak se k tomu má encoders/attentive.py ? Jedno z toho bych přejmenoval (spíš ten enkodér) a taky bych tomu enkodéru oddědil věci, který jsou tady vyčleněný, pokud to dává smysl.

jlibovicky · 2019-03-15T15:22:12Z

A rekuretní dekodér není attentive?

varisd · 2019-03-15T15:24:56Z

Rekurentni je attentive, ale ten tedka predelavam v dalsim PR, tak aby pouzival stejnou implementaci enkoder attentionu jako transformer.
V budoucnu to bude chtit doresit attention_histories a nebo to zpetne narobovat na ty attention objekty.

varisd · 2019-03-15T15:58:49Z

Jak se k tomu má encoders/attentive.py ? Jedno z toho bych přejmenoval (spíš ten enkodér) a taky bych tomu enkodéru oddědil věci, který jsou tady vyčleněný, pokud to dává smysl.

Vubec nesouvisi. Ono je to tezky: Ondrovo attention_heads jsou neco jineho nez heads v multi_head_attentionu. Nejbliz by se to dalo popsat jako: Ondra udela klasik self-attention (kde queries je prazdny; nedela scaled_dot_product, ale projekce, tak ze to jde), pak dane vektory secte v case a nakonec reshapne z shape(t) na shape(t/h, h).

Tezky je na tom to, ze bychom museli zmenit jeho terminy, aby to nevnaselo zmatek, ale pak by to tolik nesedelo na terminy v tom jeho clanku.

Nechal bych to jako TODO pro refactor attentionu / attention objektu.

jlibovicky · 2019-03-15T16:05:53Z

IMHO stačí přejmenovat ten Ondrovo soubor nebo třídu. Něco jako self-attentive temporal encoder nebo tak něco.

varisd · 2019-03-15T16:08:11Z

Jo, neco vymyslim.

Edit: Prejmenovano na Structured

jindrahelcl

Napřed prosim vyřešit vztah k #786 plus poznámky, který jsem dal k tomu, rebasnout a pak na to mrknu ještě jednou

jlibovicky · 2019-03-20T22:40:56Z

Nejvdřív je potřeba pořešit #796, na kterým je tohleto naroubovaný.

jindrahelcl · 2019-03-20T22:45:57Z

Takže #796 je na #786 a tohle je na obojím? :-)

jlibovicky · 2019-03-21T12:08:09Z

#796 je nezávsilý. #786 z hodo jde vyrebasovat pryč jednoduše.

Autoregressive decoders refactor

Unify RNN encoders

varisd · 2019-05-02T13:38:13Z

ping
opravil jsem tests/classifier, tak ted uz by to snad melo byt vse, ne?

Updates for new version of mypy

jindrahelcl · 2019-05-09T12:49:43Z

 # pylint: enable=too-few-public-methods


+def _bucket_boundaries(max_length, min_length=8, length_bucket_step=1.1):


jindrahelcl · 2019-05-09T12:51:48Z

@@ -0,0 +1,167 @@
+"""TODO."""


jindrahelcl · 2019-05-09T12:52:24Z

+                the attention.
+
+        TODO:
+            Generalize the attention.


co je tohle todo?

jindrahelcl · 2019-05-09T12:55:31Z

+                queries, enc_states, enc_masks, self.n_heads_enc,
+                self.n_heads_hier, attn_dropout_cbs, dropout_cb)
+
+        # TODO: remove this - this is already checked in the constructor


odstranit? nebo smazat todo?

jindrahelcl · 2019-05-09T12:59:38Z

        """
        check_argument_types()
-        BaseRunner[BeamSearchDecoder].__init__(self, output_series, decoder)
+        super().__init__(output_series, decoder)


tyhle změny už jsem dělal a jsou zamergovaný, proč to tu je? je to narebasovaný správně?

jindrahelcl · 2019-05-09T13:00:14Z

 rouge==0.2.1
 typeguard
-sacrebleu
+sacrebleu==1.3.1


to tu nemá bejt. nová verze sacrebleu už je kompatibilní - byl to jejich bug

jindrahelcl · 2019-05-09T13:00:20Z

 rouge==0.2.1
 typeguard
-sacrebleu
+sacrebleu==1.3.1


jindrahelcl · 2019-05-09T13:00:33Z

@@ -1 +1 @@
-mypy
+mypy==0.660


jindrahelcl · 2019-05-09T13:03:24Z

Vypadá to, že je to špatně narebasovaný, protože se jako součást ukazuje tvuj PR (#818) řešící to, co jsem udělal v #819

Fixed Dataset.subset when buffer_size is None

…encoder attention

varisd · 2019-05-09T15:22:33Z

Rebase opraven.

jindrahelcl · 2019-05-09T17:20:05Z

ten pull request je nastavenej na mergování do branche autoregressive_refactor. Nechceš mergovat do mastera?

jindrahelcl reviewed Mar 15, 2019

View reviewed changes

varisd requested a review from jindrahelcl March 18, 2019 16:47

varisd force-pushed the attentive_interface branch from 806f6a2 to d67d5c1 Compare March 18, 2019 16:49

jlibovicky force-pushed the autoregressive_refactor branch from dee6855 to 60164f9 Compare March 18, 2019 19:41

jindrahelcl requested changes Mar 20, 2019

View reviewed changes

varisd force-pushed the autoregressive_refactor branch from 3495b1d to cd1e15a Compare March 21, 2019 14:34

jlibovicky force-pushed the autoregressive_refactor branch 2 times, most recently from 3e5ab5a to 89eb9c2 Compare March 26, 2019 09:36

varisd force-pushed the autoregressive_refactor branch 2 times, most recently from 6ed29a5 to c446c9c Compare April 1, 2019 15:13

jlibovicky and others added 7 commits April 3, 2019 10:13

Merge pull request #796 from ufal/autoregressive_refactor

cb4531c

Autoregressive decoders refactor

refactor rnn encoder

3d34228

adjust tests and bugfix

229de60

pylint fix

96f99ef

get rid of raw rnn encoder

68b4fb2

address review

2088ca2

Merge pull request #809 from ufal/rnn_enc_refactor

056df1d

Unify RNN encoders

jindrahelcl and others added 4 commits May 3, 2019 15:28

fix mypy for new version

98accc6

Merge pull request #819 from ufal/mypy_fix

8bcf6f8

Updates for new version of mypy

Fixed Dataset.subset when buffer_size is None

0542e00

froze the required versions of mypy and sacrebleu

109f9b6

varisd requested a review from jindrahelcl May 9, 2019 12:38

varisd force-pushed the attentive_interface branch from 8c1c458 to 3fbbcf7 Compare May 9, 2019 12:41

jindrahelcl requested changes May 9, 2019

View reviewed changes

jindrahelcl mentioned this pull request May 9, 2019

T2T batching #786

Open

Fixed Dataset.subset when buffer_size is None

02071ec

varisd force-pushed the attentive_interface branch from 3fbbcf7 to 209b68b Compare May 9, 2019 13:35

varisd added 6 commits May 9, 2019 15:37

Merge pull request #816 from ufal/dataset_subset_fix

1bb61f6

Fixed Dataset.subset when buffer_size is None

created inheritable decoders.Attentive class providing interface for …

dbe69e0

…encoder attention

bs_decoder: checking correct object for Attentive

2057e0f

renamed attentive encoder to structured encoder

36e9e1b

fixing tests

cfdeb42

fixed tests/classifier.ini

38c0bf8

varisd force-pushed the attentive_interface branch from 209b68b to 38c0bf8 Compare May 9, 2019 13:39

		# pylint: enable=too-few-public-methods


		def _bucket_boundaries(max_length, min_length=8, length_bucket_step=1.1):

		@@ -1 +1 @@
		mypy
		mypy==0.660

Uh oh!

Conversation

varisd commented Mar 15, 2019

Uh oh!

jindrahelcl left a comment

Choose a reason for hiding this comment

Uh oh!

jlibovicky commented Mar 15, 2019

Uh oh!

varisd commented Mar 15, 2019

Uh oh!

varisd commented Mar 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jlibovicky commented Mar 15, 2019

Uh oh!

varisd commented Mar 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jindrahelcl left a comment

Choose a reason for hiding this comment

Uh oh!

jlibovicky commented Mar 20, 2019

Uh oh!

jindrahelcl commented Mar 20, 2019

Uh oh!

jlibovicky commented Mar 21, 2019

Uh oh!

varisd commented May 2, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jindrahelcl commented May 9, 2019

Uh oh!

varisd commented May 9, 2019

Uh oh!

jindrahelcl commented May 9, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

varisd commented Mar 15, 2019 •

edited

Loading

varisd commented Mar 15, 2019 •

edited

Loading