get puzzled in bag

[915288938lx]

agetnts/dtqn.py
in observe() function, we tiled self.context.obs, self.context.action, self.bag.obss and self.bag.actions, and pass them into the self.policy_network, then get the possible_bag_actions, possible_bag_obss, finally update the obss of the bag with the optimal obss, but we only used the updated bag in get_action() function , why ?

in replay_buffer.py
in sample_with_bag() function, we just randomly sampled the previous record before current step , why these randomly sampled data can be used to train "persistent memory" in the bag_attention network, i got so much puzzle here, hope your reply. THKS

[kevslinger]

Hi there, and thank you for your interest in DTQN! A few things to unpack here. First things first, if you're looking for the version of DTQN most closely resembling the paper, please use the paper branch. The main branch contains the bag, which is still experimental.

Regarding your first question, the point of doing the tiling in observe is to find the optimal bag when we have a full bag and want to put a new observation there. After we have a full context of observations, we begin putting the oldest observations into the bag. Once that is full, we need to figure out which observations should be put into the bag, and which can be discarded. We compare all possible bag combinations, and select the bag which resulted in the maximum Q-value. This is still an experimental idea, so your feedback is appreciated!

For the replay buffer sampling, we need to be able to train the agent with a context and a bag, to match the types of inputs we're giving it during evaluation. At first, since the agent has no idea what a good context/bag looks like, we need to randomly sample a bag for the agent to use. This will expose the agent to more types of observations and, hopefully, improve its ability to select a good bag in the future. This, again, is experimental, so feedback welcome! Thanks for your questions

[915288938lx]

Hi there, and thank you for your interest in DTQN! A few things to unpack here. First things first, if you're looking for the version of DTQN most closely resembling the paper, please use the paper branch. The main branch contains the bag, which is still experimental.

Regarding your first question, the point of doing the tiling in observe is to find the optimal bag when we have a full bag and want to put a new observation there. After we have a full context of observations, we begin putting the oldest observations into the bag. Once that is full, we need to figure out which observations should be put into the bag, and which can be discarded. We compare all possible bag combinations, and select the bag which resulted in the maximum Q-value. This is still an experimental idea, so your feedback is appreciated!

For the replay buffer sampling, we need to be able to train the agent with a context and a bag, to match the types of inputs we're giving it during evaluation. At first, since the agent has no idea what a good context/bag looks like, we need to randomly sample a bag for the agent to use. This will expose the agent to more types of observations and, hopefully, improve its ability to select a good bag in the future. This, again, is experimental, so feedback welcome! Thanks for your questions

Thanks for your reply, I 've been reading the paper brunch and also doing some experiment on it, the training curve (gv memery-7x7) fluctuates a lot in the head 2/3 part (total steps: 2000000), but finally convergence to an average success rate by 0.8, it is still an impressive result ! I have added some DQN enhancement on it, such as PER, Noisy net, n-step td, hoping these work

[kevslinger]

Awesome, let me know how those go! I hope the DQN enhancements are helpful in your experiments :)

[915288938lx]

Awesome, let me know how those go! I hope the DQN enhancements are helpful in your experiments :)

[915288938lx]

Bug in utils/context.py , line 46:

 self.obs = np.full([self.max_length, self.env_obs_length], self.obs_mask), 

self.obs[0]=obs

in carflag env, self. obs_mask is int -5, so the dtype of self.obs will be inffered as np.int32 , when float array obs comes, say [0.123,0.0,0.0], when stored to self.obs, it will become [0,0,0], all decimals will lose precision due to rounding, we should explicitly specify the type of self.obs

Question in mask stuff. In utils/context.py, in reset() function, the mask pairs:
self.obs_mask=obs_mask (here i understood. mask value should be set to value that will not happend during real experiment)
self.reward_mask=0.0
self.done_mask=True
why use 0.0 as reward_mask, True as done_mask, and why not use an action mask if we use previous ations as part of our obs, (self.action = RNG.rng.integers(self.num_actions, size=(self.max_length, 1))? when we introduce previous actions as observation in train course, if we randomly init the self.action, then the tail part of unfilled action context will become unstable when used in transformer's decoder self-attention part, because there is no mask.

[kevslinger]

I think that is a good point about the carflag env. The bag implementation is definitely experimental, and I mostly test with the gridverse environments, which do not use floating point observation spaces. You may have just caught a bug with the bag!

Basically, the intuition for masking with done = True and reward = 0 is that we don't want the model to learn that there is any reward to be gained from that context. Using an action mask is tricky because you either have to provide the same valid action for each timestep in the mask, OR you have to add an additional valid action (a no-op) to your environment, which your model then needs to learn to ignore. Instead, I wanted to train the model that in the mask observations, no matter what action you take, you cannot get a reward