Add a function to obtain transition probabilities in a given state

[kasanari]

If we want to be able to obtain optimal policies (for baselines or validation) through value iteration we need a function that can return the state transition probability for a given state and action (or all probabilities in a given state). In our case the only uncertainty comes from the attacker behaviour. If the attacker action is assumed to be known (because the attacker acts first) then one can iterate over all possible defender actions to get the possible next states.

This does not fit into the Gym/PettingZoo API so we are free to do implement this as we please. Here is one way of doing it:

https://github.com/alessiodm/drl-zh/blob/main/01_MDPs.

[kasanari]

If we were to assume the attacker did not act first, then we would need agent classes to provide a probability distribution over their actions.

[mrkickling]

Is this still relevant?

[kasanari]

Not for me at the moment. @sandorstormen might.

[sandorstormen]

It would be very nice to have a function which returns a distribution (conditioned on actions) over next possible states. But I don't have an immediate use case for this.

[mrkickling]

I have no clue how this would be implemented, so if this is useful I need some more explanation. Otherwise just keeping it in the backlog and if it comes up again we can have some discussions about implementation.

[sandorstormen]

I have no clue how this would be implemented, so if this is useful I need some more explanation.

Simple. Let's say we have one agent with a state class. Given some attack step which is going to be executed, what are all the possible states that the agent can have after the action. What are the probabilities of each of those next states?

[mrkickling]

This requires that we know all other agents policies, right? Or how else would the probabilities be calculated?

[kasanari]

Yes, the probability numbers will be affected by the policies. I think a first implementation would be to simply list all the possible next states (and their rewards) given all possible combinations of attacker and defender actions. Then one can attach numbers to each one of those based on the respective policies.

[kasanari]

Now that I think about it, we might be able to factorize it such that you get the probability for each attack/defense step, rather than the monolithic state. e.g. if we fix the defender action to host1:notPresent and the attacker is random then the probs might be

{
host1:notPresent: 1.0, # 1.0 because defenses are always activated
host1:connect: 0.0, # 0.0 because defended attack steps are never taken
host2:connect: 0.5, # assume attacker policy would randomly pick between host1 host2 and host 3.
host3:connect: 0.5, # assume attacker policy would randomly pick between host1 host2 and host 3.
}

[mrkickling]

This works looking at the vejde-malsim where observations are listed in this way for both attackers and defenders.
In the mal-simulator, there is much more to state (action surface, etc) and the format differ between attackers and defenders (attackers don't store performed nodes of the defender but the opposite is true).
For defenders we have the observed_nodes (which would also include false positives and negatives), but attackers don't have the same thing.

So just having a dictionary like this is not the same as the state from mal-simulator perspective imo.