policies¶

categorical¶

class xuance.torch.policies.categorical.ActorCriticPolicy(*args: Any, **kwargs: Any)[source]¶

Bases: Module

Actor-Critic for stochastic policy with categorical distributions. (Discrete action space)

Parameters:

action_space (Discrete) – The discrete action space.
representation (Module) – The representation module.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

forward(observation: torch.Tensor | dict)[source]¶

Returns the hidden states, action distribution, and values.

Parameters:: observation – The original observation of agent.
Returns:: The outputs of representation. a_dist: The distribution of actions output by actor. value: The state values output by critic.
Return type:: outputs

class xuance.torch.policies.categorical.ActorPolicy(*args: Any, **kwargs: Any)[source]¶

Bases: Module

Actor for stochastic policy with categorical distributions. (Discrete action space)

Parameters:

action_space (Discrete) – The discrete action space.
representation (Module) – The representation module.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

forward(observation: torch.Tensor | dict)[source]¶

Returns the hidden states, action distribution.

Parameters:: observation – The original observation of agent.
Returns:: The outputs of representation. a_dist: The distribution of actions output by actor.
Return type:: outputs

class xuance.torch.policies.categorical.PPGActorCritic(*args: Any, **kwargs: Any)[source]¶

Bases: Module

Actor-Critic for PPG with categorical distributions. (Discrete action space)

Parameters:

action_space (Discrete) – The discrete action space.
representation (Module) – The representation module.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

forward(observation: torch.Tensor | dict)[source]¶

Returns the actors representation output, action distribution, values, and auxiliary values.

Parameters:: observation – The original observation of agent.
Returns:: The outputs of actor representation. a_dist: The distribution of actions output by actor. value: The state values output by critic. aux_value: The auxiliary values output by aux_critic.
Return type:: policy_outputs

class xuance.torch.policies.categorical.SACDISPolicy(*args: Any, **kwargs: Any)[source]¶

Bases: Module

Actor-Critic for SAC with categorical distributions. (Discrete action space)

Parameters:

action_space (Discrete) – The discrete action space.
representation (Module) – The representation module.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

Qaction(observation: torch.Tensor | dict)[source]¶

Returns the evaluated Q-values for current observations.

Parameters:: observation – The original observation.
Returns:: The Q-value calculated by the first critic network. q_2: The Q-value calculated by the other critic network.
Return type:: q_1

Qpolicy(observation: torch.Tensor | dict)[source]¶

Feedforward and calculate the action probabilities, log of action probabilities, and Q-values.

Parameters:: observation – The original observation of an agent.
Returns:: The probabilities of actions. log_action_prob: The log of action probabilities. q_1: The Q-value calculated by the first critic network. q_2: The Q-value calculated by the other critic network.
Return type:: act_prob

Qtarget(observation: torch.Tensor | dict)[source]¶

Calculate the action probabilities, log of action probabilities, and Q-values with target networks.

Parameters:: observation – The original observation of an agent.
Returns:: The probabilities of actions. log_action_prob: The log of action probabilities. target_q: The minimum of Q-values calculated by the target critic networks.
Return type:: new_act_prob

forward(observation: torch.Tensor | dict)[source]¶

Returns the output of actor representation and samples of actions.

Parameters:: observation – The original observation of an agent.
Returns:: The outputs of the actor representation. act_sample: The sampled actions from the distribution output by the actor.
Return type:: outputs

soft_update(tau=0.005)[source]¶

categorical_marl¶

class xuance.torch.policies.categorical_marl.Basic_ISAC_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: Module

Basic_ISAC_Policy: The basic policy for independent soft actor-critic.

Parameters:

action_space (Box) – The continuous action space.
n_agents (int) – The number of agents.
actor_representation (ModuleDict) – A dict of representation modules for each agent’s actor.
critic_representation (ModuleDict) – A dict of representation modules for each agent’s critic.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – Other arguments.

Qaction(observation: torch.Tensor | dict, agent_ids: torch.Tensor, agent_key: str = None, rnn_hidden_critic_1: Dict[str, List[torch.Tensor]] | None = None, rnn_hidden_critic_2: Dict[str, List[torch.Tensor]] | None = None)[source]¶

Returns the evaluated Q-values for current observation-action pairs.

Parameters:

observation (Union[Tensor, dict]) – The original observation.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden_critic_1 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_1 representation.
rnn_hidden_critic_2 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_2 representation.

Returns:

The updated rnn states for critic_1_representation. rnn_hidden_critic_new_2: The updated rnn states for critic_2_representation. q_1: The Q-value calculated by the first critic network. q_2: The Q-value calculated by the other critic network.

Return type:

rnn_hidden_critic_new_1

Qpolicy(observation: Dict[str, torch.Tensor], agent_ids: torch.Tensor = None, avail_actions: Dict[str, torch.Tensor] = None, agent_key: str = None, rnn_hidden_actor: Dict[str, List[torch.Tensor]] | None = None, rnn_hidden_critic_1: Dict[str, List[torch.Tensor]] | None = None, rnn_hidden_critic_2: Dict[str, List[torch.Tensor]] | None = None)[source]¶

Returns Q^policy of current observations and actions pairs.

Parameters:

observation (Dict[Tensor]) – The observations.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.
agent_key (str) – Calculate actions for specified agent.
rnn_hidden_actor (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for actor representation.
rnn_hidden_critic_1 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_1 representation.
rnn_hidden_critic_2 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_2 representation.

Returns:

The updated rnn states for actor_representation. rnn_hidden_critic_new_1: The updated rnn states for critic_1_representation. rnn_hidden_critic_new_2: The updated rnn states for critic_2_representation. act_prob_dict: The probabilities of actions. log_action_prob_dict: The log of action probabilities. q_1: The evaluation of Q values with critic 1. q_2: The evaluation of Q values with critic 2.

Return type:

rnn_hidden_actor_new

Qtarget(next_observation: Dict[str, torch.Tensor], agent_ids: torch.Tensor = None, avail_actions: Dict[str, torch.Tensor] = None, agent_key: str = None, rnn_hidden_actor: Dict[str, List[torch.Tensor]] | None = None, rnn_hidden_critic_1: Dict[str, List[torch.Tensor]] | None = None, rnn_hidden_critic_2: Dict[str, List[torch.Tensor]] | None = None)[source]¶

Returns the Q^target of next observations and actions pairs.

Parameters:

next_observation (Dict[Tensor]) – The observations of next step.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.
agent_key (str) – Calculate actions for specified agent.
rnn_hidden_actor (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for actor representation.
rnn_hidden_critic_1 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_1 representation.
rnn_hidden_critic_2 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_2 representation.

Returns:

Return type:

rnn_hidden_actor

forward(observation: Dict[str, torch.Tensor], agent_ids: torch.Tensor = None, avail_actions: Dict[str, torch.Tensor] = None, agent_key: str = None, rnn_hidden: Dict[str, List[torch.Tensor]] | None = None)[source]¶

Returns actions of the policy.

Parameters:

observation (Dict[Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. actions (Dict[Tensor]): The actions output by the policies.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

property parameters_actor¶

property parameters_critic¶

soft_update(tau=0.005)[source]¶

class xuance.torch.policies.categorical_marl.COMA_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: Module

COMA_Policy: Counterfactual Multi-Agent Actor-Critic Policy with categorical distributions.

Parameters:

action_space (Optional[Dict[str, Discrete]]) – The discrete action space.
n_agents (int) – The number of agents.
representation_actor (ModuleDict) – A dict of representation modules for each agent’s actor.
representation_critic (ModuleDict) – A dict of representation modules for each agent’s critic.
mixer (Module) – The mixer module that mix together the individual values to the total value.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – The other args.

copy_target()[source]¶

forward(observation: Dict[str, torch.Tensor], agent_ids: torch.Tensor | None = None, avail_actions: Dict[str, torch.Tensor] = None, agent_key: str = None, rnn_hidden: Dict[str, List[torch.Tensor]] | None = None, epsilon=0.0, test_mode=False)[source]¶

Returns actions of the policy.

Parameters:

observation (Dict[str, Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states of actor representation.
epsilon – The epsilon.

Returns:

The new RNN hidden states of actor representation. act_probs (dict): The probabilities of the actions.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

get_values(state: torch.Tensor, observation: Dict[str, torch.Tensor], actions: Dict[str, torch.Tensor], agent_ids: torch.Tensor = None, rnn_hidden: Dict[str, List[torch.Tensor]] | None = None, target=False)[source]¶

Get evaluated critic values.

Parameters:

state – Tensor: The global state.
observation (Dict[str, Tensor]) – The input observations for the policies.
actions (Dict[str, Tensor]) – The input actions.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states of critic representation.
target – If to use target critic network to calculate the critic values.

Returns:

The new RNN hidden states of critic representation. values (dict): The evaluated critic values.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

property parameters_actor¶

property parameters_critic¶

class xuance.torch.policies.categorical_marl.CommNet_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: MAAC_Policy

Returns actions of the policy.

Parameters:

observation (Dict[str, Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states of actor representation.

Returns:

The new RNN hidden states of actor representation. pi_dists (dict): The stochastic policy distributions.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

property parameters_model¶

class xuance.torch.policies.categorical_marl.IC3Net_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: CommNet_Policy

Returns actions of the policy.

Parameters:

observation (Dict[str, Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states of actor representation.

Returns:

The new RNN hidden states of actor representation. pi_dists (dict): The stochastic policy distributions.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

class xuance.torch.policies.categorical_marl.MAAC_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: Module

MAAC_Policy: Multi-Agent Actor-Critic Policy with categorical policies.

Parameters:

action_space (Optional[Dict[str, Discrete]]) – The discrete action space.
n_agents (int) – The number of agents.
representation_actor (ModuleDict) – A dict of representation modules for each agent’s actor.
representation_critic (ModuleDict) – A dict of representation modules for each agent’s critic.
mixer (Module) – The mixer module that mix together the individual values to the total value.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – The other args.

Returns actions of the policy.

Parameters:

observation (Dict[str, Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states of actor representation.

Returns:

The new RNN hidden states of actor representation. pi_dists (dict): The stochastic policy distributions.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

get_values(observation: Dict[str, torch.Tensor], agent_ids: torch.Tensor = None, agent_key: str = None, rnn_hidden: Dict[str, List[torch.Tensor]] | None = None)[source]¶

Get critic values via critic networks.

Parameters:

observation (Dict[str, Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states of critic representation.

Returns:

The new RNN hidden states of critic representation. values (dict): The evaluated critic values.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

property parameters_model¶

value_tot(values_n: torch.Tensor, global_state=None)[source]¶

class xuance.torch.policies.categorical_marl.MAAC_Policy_Share(*args: Any, **kwargs: Any)[source]¶

Bases: MAAC_Policy

MAAC_Policy_Share: Multi-agent actor-critic Policy with categorical policies and shared representations.

Parameters:

action_space (Optional[Dict[str, Discrete]]) – The discrete action space.
n_agents (int) – The number of agents.
representation (ModuleDict) – A dict of representation modules.
mixer (Module) – The mixer module that mix together the individual values to the total value.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – The other args.

Returns actions of the policy.

Parameters:

observation (Dict[str, Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states of actor representation.

Returns:

The new RNN hidden states of actor representation. pi_dists (dict): The stochastic policy distributions.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

get_values(observation: Dict[str, torch.Tensor], agent_ids: torch.Tensor = None, agent_key: str = None, rnn_hidden: Dict[str, List[torch.Tensor]] | None = None)[source]¶

Get critic values via critic networks.

Parameters:

observation (Dict[str, Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states of critic representation.

Returns:

The new RNN hidden states of critic representation. values (dict): The evaluated critic values.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

property parameters_model¶

class xuance.torch.policies.categorical_marl.MASAC_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: Basic_ISAC_Policy

Basic_ISAC_Policy: The basic policy for independent soft actor-critic.

Parameters:

action_space (Box) – The continuous action space.
n_agents (int) – The number of agents.
actor_representation (ModuleDict) – A dict of representation modules for each agent’s actor.
critic_representation (ModuleDict) – A dict of representation modules for each agent’s critic.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – Other arguments.

Qaction(joint_observation: torch.Tensor | None = None, joint_actions: torch.Tensor | None = None, agent_ids: torch.Tensor | None = None, agent_key: str = None, rnn_hidden_critic_1: Dict[str, List[torch.Tensor]] | None = None, rnn_hidden_critic_2: Dict[str, List[torch.Tensor]] | None = None)[source]¶

Returns the evaluated Q-values for current observation-action pairs.

Parameters:

joint_observation (Optional[Tensor]) – The joint observations of the team.
joint_actions (Tensor) – The joint actions of the team.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden_critic_1 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_1 representation.
rnn_hidden_critic_2 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_2 representation.

Returns:

Return type:

rnn_hidden_critic_new_1

Qpolicy(joint_observation: torch.Tensor | None = None, joint_actions: torch.Tensor | None = None, agent_ids: torch.Tensor = None, agent_key: str = None, rnn_hidden_critic_1: Dict[str, List[torch.Tensor]] | None = None, rnn_hidden_critic_2: Dict[str, List[torch.Tensor]] | None = None)[source]¶

Returns Q^policy of current observations and actions pairs.

Parameters:

joint_observation (Optional[Tensor]) – The joint observations of the team.
joint_actions (Optional[Tensor]) – The joint actions of the team.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden_critic_1 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_1 representation.
rnn_hidden_critic_2 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_2 representation.

Returns:

The updated rnn states for critic_1_representation. rnn_hidden_critic_new_2: The updated rnn states for critic_2_representation. q_1: The evaluations of Q^policy with critic 1. q_2: The evaluations of Q^policy with critic 2.

Return type:

rnn_hidden_critic_new_1

Qtarget(joint_observation: torch.Tensor | None = None, joint_actions: torch.Tensor | None = None, agent_ids: torch.Tensor = None, agent_key: str = None, rnn_hidden_critic_1: Dict[str, List[torch.Tensor]] | None = None, rnn_hidden_critic_2: Dict[str, List[torch.Tensor]] | None = None)[source]¶

Returns the Q^target of next observations and actions pairs.

Parameters:

joint_observation (Optional[Tensor]) – The joint observations of the team.
joint_actions (Optional[Tensor]) – The joint actions of the team.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden_critic_1 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_1 representation.
rnn_hidden_critic_2 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_2 representation.

Returns:

The updated rnn states for critic_1_representation. rnn_hidden_critic_new_2: The updated rnn states for critic_2_representation. q_target: The evaluations of Q^target.

Return type:

rnn_hidden_critic_new_1

class xuance.torch.policies.categorical_marl.MeanFieldActorCriticPolicy(*args: Any, **kwargs: Any)[source]¶

Bases: Module

Mean-field actor-critic policy.

This policy maintains separate actor and critic networks for each agent type (model key), embeds the mean action of neighboring agents, and produces Boltzmann policies.

Parameters:

action_space (Discrete) – A mapping from model keys to discrete action spaces.
n_agents (int) – Total number of agents in the environment.
representation_actor (ModuleDict) – Actor state encoder modules for each model key.
representation_critic (ModuleDict) – Critic state encoder modules for each model key.
actor_hidden_size (Sequence[int], optional) – Hidden layer sizes for actor networks.
critic_hidden_size (Sequence[int], optional) – Hidden layer sizes for critic networks.
normalize (ModuleType, optional) – Normalization layer to apply after each hidden layer.
initialize (Callable[..., Tensor], optional) – Weight initialization function.
activation (ModuleType, optional) – Activation function class for hidden layers.
device (str|int|torch.device, optional) – Device identifier for module placement.
use_distributed_training (bool) – If True, wrap components in DistributedDataParallel.
**kwargs – Additional keyword arguments: use_parameter_sharing (bool): Whether to share parameters across agent types. model_keys (List[str]): Keys identifying different agent types. rnn (str): RNN type, e.g., “LSTM” or “GRU”. use_rnn (bool): Flag indicating whether to include RNN layers. action_embedding_hidden_size (Sequence[int]): Hidden sizes for action mean embedding. temperature (float): Temperature parameter for Boltzmann policy.

Returns actions of the policy.

Parameters:

observation (Dict[str, Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states of actor representation.

Returns:

The new RNN hidden states of actor representation. pi_dists (dict): The stochastic policy distributions.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

get_boltzmann_policy(q)[source]¶

Convert Q-values to a Boltzmann (softmax) policy distribution.

Parameters:: q (Tensor) – Q-value tensor of shape […, n_actions].
Returns:: Probability distribution over actions with same shape as q.
Return type:: Tensor

get_mean_actions(actions: Dict[str, torch.Tensor], agent_mask_tensor: torch.Tensor, batch_size: int)[source]¶

Compute mean one-hot action vectors of each agent’s neighbors.

For each batch and agent, exclude the agent’s own action and average the one-hot action encodings of its alive neighbors.

Parameters:

actions (Dict[str, Tensor]) – Mapping from model keys to chosen action indices of shape [batch_size * n_agents].
agent_mask_tensor (Tensor) – Binary mask of shape [batch_size, n_agents] indicating alive (1) or dead (0) agents.
batch_size (int) – Number of samples in the batch.

Returns:

Mean one-hot action tensor of shape [batch_size, n_agents, n_actions_max].

Return type:

Tensor

get_values(observation: Dict[str, torch.Tensor], actions_mean: Dict[str, torch.Tensor] = None, agent_ids: torch.Tensor = None, agent_key: str = None, rnn_hidden: Dict[str, List[torch.Tensor]] | None = None)[source]¶

Get critic values via critic networks.

Parameters:

observation (Dict[str, Tensor]) – The input observations for the policies.
actions_mean (Dict[str, Tensor]) – The mean actions of each agent’s neighbors.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states of critic representation.

Returns:

The new RNN hidden states of critic representation. values (dict): The evaluated critic values.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

property parameters_model¶

class xuance.torch.policies.categorical_marl.TarMAC_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: IC3Net_Policy

Returns actions of the policy.

Parameters:

observation (Dict[str, Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states of actor representation.

Returns:

The new RNN hidden states of actor representation. pi_dists (dict): The stochastic policy distributions.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

coordination_graph¶

class xuance.torch.policies.coordination_graph.Coordination_Graph(n_vertexes: int, graph_type: str = 'FULL', device: str | int | torch.device | None = None)[source]¶

Bases: object

Construct a deep coordination graph.

Parameters:

n_vertexes (int) – The number of vertexes in the graph.
graph_type (str) – The type of graph, default is “FULL”.

set_coordination_graph()[source]¶: Reset the coordination graph.

class xuance.torch.policies.coordination_graph.DCG_payoff(*args: Any, **kwargs: Any)[source]¶

Bases: DCG_utility

The payoff module for deep coordination graph.

Parameters:

dim_input (int) – The dimension of input for the payoff module.
dim_hidden (int) – The dimension of hidden layer for the payoff module.
dim_act (int) – The dimension of actions.
low_rank_payoff (int) – The low rank payoff.
payoff_rank (int) – The rank of payoff.
device (Optional[Union[str, int, torch.device]]) – The device for running the model, default is None.

forward(hidden_states_n: torch.Tensor, edges_from: torch.Tensor = None, edges_to: torch.Tensor = None)[source]¶

Calculate the payoff values for the graph constructed by multiple agents.

Parameters:

hidden_states_n – The hidden states for the representations of n agents.
edges_from – The edges from others to self, default is None.
edges_to – The edges from self to others, default is None.

Returns: Mean of payoff values for edge_from and edge_to.

class xuance.torch.policies.coordination_graph.DCG_utility(*args: Any, **kwargs: Any)[source]¶

Bases: Module

The utility module for deep coordination graph.

Parameters:

dim_input (int) – The dimension of input for the utility module.
dim_hidden (int) – The dimension of hidden layer for the utility module.
dim_output (int) – The dimension of output for the utility module.
device (Optional[Union[str, int, torch.device]]) – The device for running the model, default is None.

device¶: All utilities share the same parameters

forward(hidden_states_n: torch.Tensor)[source]¶

Calculate the utility values for multiple agents.

Parameters:: hidden_states_n (Tensor) – The hidden states for the representations of n agents.

Returns: The utility values for multiple agents.

core¶

class xuance.torch.policies.core.ActorNet(*args: Any, **kwargs: Any)[source]¶

Bases: Module

The actor network for deterministic policy, which outputs activated continuous actions directly.

Parameters:

state_dim (int) – The input state dimension.
action_dim (int) – The dimension of continuous action space.
hidden_sizes (Sequence[int]) – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
device (Optional[Union[str, int, torch.device]]) – The calculating device.

forward(x: torch.Tensor, avail_actions: torch.Tensor | None = None)[source]¶: Returns the output of the actor. :param x: The input tensor. :type x: Tensor :param avail_actions: The actions mask values when use actions mask, default is None. :type avail_actions: Optional[Tensor]

class xuance.torch.policies.core.BasicQhead(*args: Any, **kwargs: Any)[source]¶

Bases: Module

A base class to build Q network and calculate the Q values.

Parameters:

state_dim (int) – The input state dimension.
n_actions (int) – The number of discrete actions.
hidden_sizes – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
device (Optional[Union[str, int, torch.device]]) – The calculating device.

forward(x: torch.Tensor)[source]¶: Returns the output of the Q network. :param x: The input tensor. :type x: Tensor

class xuance.torch.policies.core.BasicRecurrent(*args: Any, **kwargs: Any)[source]¶

Bases: Module

Build recurrent neural network to calculate Q values.

forward(x: torch.Tensor, h: torch.Tensor, c: torch.Tensor = None)[source]¶: Returns the rnn hidden and Q-values via RNN networks.

class xuance.torch.policies.core.C51Qhead(*args: Any, **kwargs: Any)[source]¶

Bases: Module

A base class to build Q network and calculate the distributional Q values.

Parameters:

state_dim (int) – The input state dimension.
n_actions (int) – The number of discrete actions.
atom_num (int) – The number of atoms.
hidden_sizes – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
device (Optional[Union[str, int, torch.device]]) – The calculating device.

forward(x: torch.Tensor)[source]¶

Returns the discrete action distributions. :param x: The input tensor. :type x: Tensor

Returns:: The probability distribution of the discrete actions.
Return type:: dist_probs

class xuance.torch.policies.core.CategoricalActorNet(*args: Any, **kwargs: Any)[source]¶

Bases: Module

The actor network for categorical policy, which outputs a distribution over all discrete actions.

Parameters:

state_dim (int) – The input state dimension.
action_dim (int) – The dimension of continuous action space.
hidden_sizes (Sequence[int]) – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
device (Optional[Union[str, int, torch.device]]) – The calculating device.

forward(x: torch.Tensor, avail_actions: torch.Tensor | None = None)[source]¶

Returns the stochastic distribution over all discrete actions. :param x: The input tensor. :type x: Tensor :param avail_actions: The actions mask values when use actions mask, default is None. :type avail_actions: Optional[Tensor]

Returns:: CategoricalDistribution(action_dim), a distribution over all discrete actions.
Return type:: self.dist

class xuance.torch.policies.core.CategoricalActorNet_SAC(*args: Any, **kwargs: Any)[source]¶

Bases: CategoricalActorNet

The actor network for categorical policy in SAC-DIS, which outputs a distribution over all discrete actions.

Parameters:

state_dim (int) – The input state dimension.
action_dim (int) – The dimension of continuous action space.
hidden_sizes (Sequence[int]) – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
device (Optional[Union[str, int, torch.device]]) – The calculating device.

forward(x: torch.Tensor, avail_actions: torch.Tensor | None = None)[source]¶

Returns:: CategoricalDistribution(action_dim), a distribution over all discrete actions.
Return type:: self.dist

class xuance.torch.policies.core.CriticNet(*args: Any, **kwargs: Any)[source]¶

Bases: Module

The critic network that outputs the evaluated values for states (State-Value) or state-action pairs (Q-value).

Parameters:

input_dim (int) – The input dimension (dim_state or dim_state + dim_action).
hidden_sizes (Sequence[int]) – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
device (Optional[Union[str, int, torch.device]]) – The calculating device.

forward(x: torch.Tensor)[source]¶: Returns the output of the Q network. :param x: The input tensor. :type x: Tensor

class xuance.torch.policies.core.DuelQhead(*args: Any, **kwargs: Any)[source]¶

Bases: Module

A base class to build Q network and calculate the dueling Q values.

Parameters:

state_dim (int) – The input state dimension.
n_actions (int) – The number of discrete actions.
hidden_sizes – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
device (Optional[Union[str, int, torch.device]]) – The calculating device.

forward(x: torch.Tensor)[source]¶

Returns the dueling Q-values. :param x: The input tensor. :type x: Tensor

Returns:: The dueling Q-values.
Return type:: q

class xuance.torch.policies.core.GaussianActorNet(*args: Any, **kwargs: Any)[source]¶

Bases: Module

The actor network for Gaussian policy, which outputs a distribution over the continuous action space.

Parameters:

state_dim (int) – The input state dimension.
action_dim (int) – The dimension of continuous action space.
hidden_sizes (Sequence[int]) – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
device (Optional[Union[str, int, torch.device]]) – The calculating device.

forward(x: torch.Tensor)[source]¶

Returns the stochastic distribution over the continuous action space. :param x: The input tensor. :type x: Tensor

Returns:: A distribution over the continuous action space.
Return type:: self.dist

class xuance.torch.policies.core.GaussianActorNet_SAC(*args: Any, **kwargs: Any)[source]¶

Bases: Module

The actor network for Gaussian policy in SAC, which outputs a distribution over the continuous action space.

Parameters:

state_dim (int) – The input state dimension.
action_dim (int) – The dimension of continuous action space.
hidden_sizes (Sequence[int]) – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
device (Optional[Union[str, int, torch.device]]) – The calculating device.

forward(x: torch.Tensor)[source]¶

Returns the stochastic distribution over the continuous action space. :param x: The input tensor. :type x: Tensor

Returns:: A distribution over the continuous action space.
Return type:: self.dist

class xuance.torch.policies.core.QMIX_FF_mixer(*args: Any, **kwargs: Any)[source]¶

Bases: Module

The feedforward mixer without the constraints of monotonicity.

forward(values_n, states=None)[source]¶

Returns the feedforward total Q-values.

Parameters:

values_n – The individual Q-values.
states – The global states.

class xuance.torch.policies.core.QMIX_mixer(*args: Any, **kwargs: Any)[source]¶

Bases: Module

The QMIX mixer. (Monotonicity)

Parameters:

dim_state (int) – The dimension of global state.
dim_hidden (int) – The size of rach hidden layer.
dim_hypernet_hidden (int) – The size of rach hidden layer for hyper network.
n_agents (int) – The number of agents.
device (Optional[Union[str, int, torch.device]]) – The calculating device.

forward(values_n, states)[source]¶

Returns the total Q-values for multi-agent team.

Parameters:

values_n – The individual values for agents in team.
states – The global states.

Returns:

The total Q-values for the multi-agent team.

Return type:

q_tot

class xuance.torch.policies.core.QRDQNhead(*args: Any, **kwargs: Any)[source]¶

Bases: Module

A base class to build Q networks for QRDQN policy.

Parameters:

state_dim (int) – The input state dimension.
n_actions (int) – The number of discrete actions.
atom_num (int) – The number of atoms.
hidden_sizes – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
device (Optional[Union[str, int, torch.device]]) – The calculating device.

forward(x: torch.Tensor)[source]¶

Returns the quantiles of the distribution. :param x: The input tensor. :type x: Tensor

Returns:: The quantiles of the action distribution.
Return type:: quantiles

class xuance.torch.policies.core.QTRAN_alt(*args: Any, **kwargs: Any)[source]¶

Bases: Module

The basic QTRAN module.

Parameters:

dim_state (int) – The dimension of the global state.
action_space (Dict[str, Discrete]) – The action space for all agents.
dim_hidden (int) – The dimension of the hidden layers.
n_agents (int) – The number of agents.
dim_utility_hidden (int) – The dimension of the utility hidden states.
use_parameter_sharing (bool) – Whether to use parameters sharing trick.
device – Optional[Union[str, int, torch.device]]: The calculating device.

forward(states: torch.Tensor, hidden_state_inputs: torch.Tensor, actions_onehot: torch.Tensor)[source]¶

Calculating the joint Q and V values.

Parameters:

states (Tensor) – The global states.
hidden_state_inputs (Tensor) – The joint hidden states inputs for QTRAN network.
actions_onehot (Tensor) – The joint onehot actions for QTRAN network.

Returns:

The evaluated joint Q values. v_jt (Tensor): The evaluated joint V values.

Return type:

q_jt (Tensor)

class xuance.torch.policies.core.QTRAN_base(*args: Any, **kwargs: Any)[source]¶

Bases: Module

The basic QTRAN module.

Parameters:

dim_state (int) – The dimension of the global state.
action_space (Dict[str, Discrete]) – The action space for all agents.
dim_hidden (int) – The dimension of the hidden layers.
n_agents (int) – The number of agents.
dim_utility_hidden (int) – The dimension of the utility hidden states.
use_parameter_sharing (bool) – Whether to use parameters sharing trick.
device – Optional[Union[str, int, torch.device]]: The calculating device.

forward(states: torch.Tensor, hidden_state_inputs: torch.Tensor, actions_onehot: torch.Tensor)[source]¶

Calculating the joint Q and V values.

Parameters:

states (Tensor) – The global states.
hidden_state_inputs (Tensor) – The joint hidden states inputs for QTRAN network.
actions_onehot (Tensor) – The joint onehot actions for QTRAN network.

Returns:

The evaluated joint Q values. v_jt (Tensor): The evaluated joint V values.

Return type:

q_jt (Tensor)

class xuance.torch.policies.core.VDN_mixer(*args: Any, **kwargs: Any)[source]¶

Bases: Module

The value decomposition networks mixer. (Additivity)

forward(values_n, states=None)[source]¶

deterministic¶

class xuance.torch.policies.deterministic.BasicQnetwork(*args: Any, **kwargs: Any)[source]¶

Bases: Module

The base class to implement DQN based policy

Parameters:

action_space (Discrete) – The action space, which type is gym.spaces.Discrete.
representation (Module) – The representation module.
hidden_size (Sequence[int]) – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

copy_target()[source]¶

forward(observation: torch.Tensor | dict) → tuple[source]¶

Returns the output of the representation, greedy actions, and the evaluated Q-values.

Parameters:: observation – The original observation input.
Returns:: The hidden state output by the representation. argmax_action: The greedy actions. evalQ: The evaluated Q-values.
Return type:: outputs

target(observation: torch.Tensor | dict) → tuple[source]¶

Returns the output of the representation, greedy actions, and the evaluated Q-values via target networks.

Parameters:: observation – The original observation input.
Returns:: The hidden state output by the representation. argmax_action: The greedy actions from target networks. targetQ: The evaluated Q-values output by target Q-network.
Return type:: outputs_target

class xuance.torch.policies.deterministic.C51Qnetwork(*args: Any, **kwargs: Any)[source]¶

Bases: Module

The policy for C51 distributional deep Q-networks.

Parameters:

action_space (Discrete) – The action space, which type is gym.spaces.Discrete.
atom_num (int) – The number of atoms.
v_min (float) – The lower bound of value distribution.
v_max (float) – The upper bound of value distribution.
representation (Module) – The representation module.
hidden_size – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

copy_target()[source]¶

forward(observation: torch.Tensor | dict)[source]¶

Returns the output of the representation, greedy actions, and the evaluated Z-values.

Parameters:: observation – The original observation input.
Returns:: The hidden state output by the representation. argmax_action: The greedy actions. eval_Z: The evaluated Z-values.
Return type:: outputs

target(observation: torch.Tensor | dict)[source]¶

Returns the output of the representation, greedy actions, and the evaluated Z-values via target networks.

Parameters:: observation – The original observation input.
Returns:: The hidden state output by the representation. argmax_action: The greedy actions from target networks. target_Z: The evaluated Z-values output by target Z-network.
Return type:: outputs_target

class xuance.torch.policies.deterministic.DDPGPolicy(*args: Any, **kwargs: Any)[source]¶

Bases: Module

The policy of deep deterministic policy gradient.

Parameters:

action_space (Space) – The action space.
representation (Module) – The representation module.
actor_hidden_size (Sequence[int]) – List of hidden units for actor network.
critic_hidden_size (Sequence[int]) – List of hidden units for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

Qaction(observation: torch.Tensor | dict, action: torch.Tensor)[source]¶: Returns the evaluated Q-values of state-action pairs.

Qpolicy(observation: torch.Tensor | dict)[source]¶: Returns the evaluated Q-values by calculating actions via actor networks.

Qtarget(observation: torch.Tensor | dict)[source]¶: Returns the evaluated Q-values via target networks.

forward(observation: torch.Tensor | dict)[source]¶

Returns the output of the actor representations, and the actions.

Parameters:: observation – The original observation input.
Returns:: The output of the actor representations. act: The actions calculated by the actor.
Return type:: outputs

soft_update(tau=0.005)[source]¶

class xuance.torch.policies.deterministic.DRQNPolicy(*args: Any, **kwargs: Any)[source]¶

Bases: Module

The policy of deep recurrent Q-networks.

Parameters:

action_space (Discrete) – The action space.
representation (Module) – The representation module.
**kwargs – The other arguments.

copy_target()[source]¶

forward(observation: torch.Tensor | dict, *rnn_hidden: torch.Tensor)[source]¶

Returns the output of the representation, greedy actions, the evaluated Q-values and the RNN hidden states.

Parameters:

observation – The original observation input.
rnn_hidden – The RNN hidden state.

Returns:

The hidden state output by the representation. argmax_action: The greedy actions. evalQ: The evaluated Q-values. (hidden_states, cell_states): The updated RNN hidden states.

Return type:

outputs

init_hidden(batch)[source]¶

init_hidden_item(rnn_hidden, i)[source]¶

target(observation: torch.Tensor | dict, *rnn_hidden: torch.Tensor)[source]¶

class xuance.torch.policies.deterministic.DuelQnetwork(*args: Any, **kwargs: Any)[source]¶

Bases: Module

The policy for deep dueling Q-networks.

Parameters:

action_space (Discrete) – The action space, which type is gym.spaces.Discrete.
representation (Module) – The representation module.
hidden_size – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

copy_target()[source]¶

forward(observation: torch.Tensor | dict)[source]¶

Returns the output of the representation, greedy actions, and the evaluated Q-values.

Parameters:: observation – The original observation input.
Returns:: The hidden state output by the representation. argmax_action: The greedy actions. evalQ: The evaluated Q-values.
Return type:: outputs

target(observation: torch.Tensor | dict)[source]¶

Returns the output of the representation, greedy actions, and the evaluated Q-values via target networks.

Parameters:: observation – The original observation input.
Returns:: The hidden state output by the representation. argmax_action: The greedy actions from target networks. targetQ: The evaluated Q-values output by target Q-network.
Return type:: outputs_target

class xuance.torch.policies.deterministic.MPDQNPolicy(*args: Any, **kwargs: Any)[source]¶

Bases: PDQNPolicy

The policy of multi-pass parameterised deep Q network.

Parameters:

observation_space – The observation spaces.
action_space – The action spaces.
representation (Module) – The representation module.
conactor_hidden_size (Sequence[int]) – List of hidden units for actor network.
qnetwork_hidden_size (Sequence[int]) – List of hidden units for q network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

Qeval(state, action)[source]¶

Qpolicy(state)[source]¶

Qtarget(state, action)[source]¶

class xuance.torch.policies.deterministic.NoisyQnetwork(*args: Any, **kwargs: Any)[source]¶

Bases: Module

The policy for noisy deep Q-networks.

Parameters:

action_space (Discrete) – The action space, which type is gym.spaces.Discrete.
representation (Module) – The representation module.
hidden_size – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

copy_target()[source]¶

forward(observation: torch.Tensor | dict)[source]¶

Returns the output of the representation, greedy actions, and the evaluated Q-values.

Parameters:: observation – The original observation input.
Returns:: The hidden state output by the representation. argmax_action: The greedy actions. evalQ: The evaluated Q-values.
Return type:: outputs

target(observation: torch.Tensor | dict)[source]¶

Returns the output of the representation, greedy actions, and the evaluated Q-values via target networks.

Parameters:: observation – The original observation input.
Returns:: The hidden state output by the representation. argmax_action: The greedy actions from target networks. targetQ: The evaluated Q-values output by target Q-network.
Return type:: outputs_target

update_noise(noisy_bound: float = 0.0)[source]¶: Updates the noises for network parameters.

class xuance.torch.policies.deterministic.PDQNPolicy(*args: Any, **kwargs: Any)[source]¶

Bases: Module

The policy of parameterised deep Q network.

Parameters:

observation_space – The observation spaces.
action_space – The action spaces.
representation (Module) – The representation module.
conactor_hidden_size (Sequence[int]) – List of hidden units for actor network.
qnetwork_hidden_size (Sequence[int]) – List of hidden units for q network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

Atarget(state)[source]¶

Qeval(state, action)[source]¶

Qpolicy(state)[source]¶

Qtarget(state, action)[source]¶

con_action(state)[source]¶

soft_update(tau=0.005)[source]¶

class xuance.torch.policies.deterministic.QRDQN_Network(*args: Any, **kwargs: Any)[source]¶

Bases: Module

The policy for quantile regression deep Q-networks.

Parameters:

action_space (Discrete) – The action space, which type is gym.spaces.Discrete.
quantile_num (int) – The number of quantiles.
representation (Module) – The representation module.
hidden_size – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

copy_target()[source]¶

forward(observation: torch.Tensor | dict)[source]¶

Returns the output of the representation, greedy actions, and the evaluated Z-values.

Parameters:: observation – The original observation input.
Returns:: The hidden state output by the representation. argmax_action: The greedy actions. eval_Z: The evaluated Z-values.
Return type:: outputs

target(observation: torch.Tensor | dict)[source]¶

Returns the output of the representation, greedy actions, and the evaluated Z-values via target networks.

Parameters:: observation – The original observation input.
Returns:: The hidden state output by the representation. argmax_action: The greedy actions from target networks. target_Z: The evaluated Z-values output by target Z-network.
Return type:: outputs_target

class xuance.torch.policies.deterministic.SPDQNPolicy(*args: Any, **kwargs: Any)[source]¶

Bases: PDQNPolicy

The policy of split parameterised deep Q network.

Parameters:

observation_space – The observation spaces.
action_space – The action spaces.
representation (Module) – The representation module.
conactor_hidden_size (Sequence[int]) – List of hidden units for actor network.
qnetwork_hidden_size (Sequence[int]) – List of hidden units for q network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

Qeval(state, action)[source]¶

Qpolicy(state)[source]¶

Qtarget(state, action)[source]¶

class xuance.torch.policies.deterministic.TD3Policy(*args: Any, **kwargs: Any)[source]¶

Bases: Module

The policy of twin delayed deep deterministic policy gradient.

Parameters:

action_space (Space) – The action space.
representation (Module) – The representation module.
actor_hidden_size (Sequence[int]) – List of hidden units for actor network.
critic_hidden_size (Sequence[int]) – List of hidden units for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

Qaction(observation: torch.Tensor | dict, action: torch.Tensor)[source]¶: Returns the evaluated Q-values of state-action pairs.

Qpolicy(observation: torch.Tensor | dict)[source]¶: Returns the evaluated Q-values by calculating actions via actor networks.

Qtarget(observation: torch.Tensor | dict)[source]¶: Returns the evaluated Q-values via target networks.

forward(observation: torch.Tensor | dict)[source]¶

Returns the output of the actor representations, and the actions.

Parameters:: observation – The original observation input.
Returns:: The output of the actor representations. act: The actions calculated by the actor.
Return type:: outputs

soft_update(tau=0.005)[source]¶

deterministic_marl¶

class xuance.torch.policies.deterministic_marl.BasicQnetwork(*args: Any, **kwargs: Any)[source]¶

Bases: Module

The base class to implement DQN based policy

Parameters:

action_space (Optional[Dict[str, Discrete]]) – The action space, which type is gym.spaces.Discrete.
n_agents (int) – The number of agents.
representation (ModuleDict) – A dict of the representation module for all agents.
hidden_size (Sequence[int]) – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters’ initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – Other arguments.

Qtarget(observation: Dict[str, torch.Tensor], agent_ids: Dict[str, torch.Tensor], agent_key: str = None, rnn_hidden: Dict[str, List[torch.Tensor]] | None = None)[source]¶

Returns the Q^target of next observations and actions pairs.

Parameters:

observation (Dict[Tensor]) – The observations.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. q_target: The evaluations of Q^target.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

copy_target()[source]¶

Returns actions of the policy.

Parameters:

observation (Dict[Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. argmax_action (Dict[str, Tensor]): The actions output by the policies. evalQ (Dict[str, Tensor])： The evaluations of observation-action pairs.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

property parameters_model¶

class xuance.torch.policies.deterministic_marl.DCG_policy(*args: Any, **kwargs: Any)[source]¶

Bases: Module

The deep coordination graph policy.

Parameters:

action_space (Optional[Dict[str, Discrete]]) – The action space, which type is gym.spaces.Discrete.
n_agents (int) – The number of agents.
representation (ModuleDict) – A dict of the representation module for all agents.
utility (Module) – The utility module that outputs an agent’s utility value.
payoffs (Module) – The payoff module that outputs two agents’ payoff value.
dcgraph (Module) – The deep coordination graph module.
hidden_size_bias (Sequence[int]) – List of hidden units for fully connect layers of bias net.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – Other arguments.

copy_target()[source]¶

get_hidden_states(batch_size: int, observation: Dict[str, torch.Tensor], rnn_hidden: Dict[str, List[torch.Tensor]] | None = None, use_target_net=False)[source]¶

Get the hidden states of the representations for all agents.

Parameters:

batch_size (int) – The batch size.
observation (Dict[Tensor]) – The input observations for the policies.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.
use_target_net (bool) – Whether to use a target network or not.

Returns:

The RNN hidden states for next step calculating. hidden_states_n: The hidden states of the representations that what we want.

Return type:

rnn_hidden

property parameters_model¶

class xuance.torch.policies.deterministic_marl.Independent_DDPG_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: Module

The policy of deep deterministic policy gradient.

Parameters:

action_space (Optional[Dict[str, Box]]) – The action space.
n_agents (int) – The number of agents.
actor_representation (Optional[ModuleDict]) – The representation module for actor network.
critic_representation (Optional[ModuleDict]) – The representation module for critic network.
actor_hidden_size (Sequence[int]) – List of hidden units for actor network.
critic_hidden_size (Sequence[int]) – List of hidden units for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – Other arguments.

Atarget(next_observation: Dict[str, torch.Tensor], agent_ids: torch.Tensor = None, agent_key: str = None, rnn_hidden: Dict[str, List[torch.Tensor]] | None = None)[source]¶

Returns the next actions by target policies.

Parameters:

next_observation (Dict[Tensor]) – The observations of next step.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. next_actions (Dict[Tensor]): The next actions.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

Qpolicy(observation: Dict[str, torch.Tensor], actions: Dict[str, torch.Tensor], agent_ids: torch.Tensor = None, agent_key: str = None, rnn_hidden: Dict[str, List[torch.Tensor]] | None = None)[source]¶

Returns Q^policy of current observations and actions pairs.

Parameters:

observation (Dict[Tensor]) – The observations.
actions (Dict[Tensor]) – The actions.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. q_eval: The evaluations of Q^policy.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

Qtarget(next_observation: Dict[str, torch.Tensor], next_actions: Dict[str, torch.Tensor], agent_ids: torch.Tensor = None, agent_key: str = None, rnn_hidden: Dict[str, List[torch.Tensor]] | None = None)[source]¶

Returns the Q^target of next observations and actions pairs.

Parameters:

next_observation (Dict[Tensor]) – The observations of next step.
next_actions (Dict[Tensor]) – The actions of next step.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. q_target: The evaluations of Q^target.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

forward(observation: Dict[str, torch.Tensor], agent_ids: torch.Tensor = None, agent_key: str = None, rnn_hidden: Dict[str, List[torch.Tensor]] | None = None)[source]¶

Returns actions of the policy.

Parameters:

observation (Dict[Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. actions (Dict[Tensor]): The actions output by the policies.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

property parameters_actor¶

property parameters_critic¶

soft_update(tau=0.005)[source]¶

class xuance.torch.policies.deterministic_marl.Independent_TD3_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: Independent_DDPG_Policy, Module

The policy of deep deterministic policy gradient.

Parameters:

action_space (Optional[Dict[str, Box]]) – The action space.
n_agents (int) – The number of agents.
actor_representation (Module) – The representation module for actor network.
critic_representation (Module) – The representation module for critic network.
actor_hidden_size (Sequence[int]) – List of hidden units for actor network.
critic_hidden_size (Sequence[int]) – List of hidden units for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – Other arguments.

Returns Q^policy of current observations and actions pairs.

Parameters:

observation (Dict[Tensor]) – The observations.
actions (Dict[Tensor]) – The actions.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. q_eval: The evaluations of Q^policy.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

Returns the Q^target of next observations and actions pairs.

Parameters:

next_observation (Dict[Tensor]) – The observations of next step.
next_actions (Dict[Tensor]) – The actions of next step.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. q_target: The evaluations of Q^target.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

property parameters_critic¶

soft_update(tau=0.005)[source]¶

class xuance.torch.policies.deterministic_marl.MADDPG_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: Independent_DDPG_Policy

The policy of deep deterministic policy gradient.

Parameters:

action_space (Optional[Dict[str, Box]]) – The action space.
n_agents (int) – The number of agents.
actor_representation (Module) – The representation module for actor network.
critic_representation (Module) – The representation module for critic network.
actor_hidden_size (Sequence[int]) – List of hidden units for actor network.
critic_hidden_size (Sequence[int]) – List of hidden units for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – Other arguments.

Qpolicy(joint_observation: torch.Tensor, joint_actions: torch.Tensor, agent_ids: torch.Tensor = None, agent_key: str = None, rnn_hidden: Dict[str, List[torch.Tensor]] | None = None)[source]¶

Returns Q^policy of current observations and actions pairs.

Parameters:

joint_observation (Tensor) – The joint observations of the team.
joint_actions (Tensor) – The joint actions of the team.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. q_eval: The evaluations of Q^policy.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

Qtarget(joint_observation: torch.Tensor, joint_actions: torch.Tensor, agent_ids: torch.Tensor = None, agent_key: str = None, rnn_hidden: Dict[str, List[torch.Tensor]] | None = None)[source]¶

Returns the Q^target of next observations and actions pairs.

Parameters:

joint_observation (Tensor) – The joint observations of the team.
joint_actions (Tensor) – The joint actions of the team.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. q_target: The evaluations of Q^target.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

class xuance.torch.policies.deterministic_marl.MATD3_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: MADDPG_Policy, Module

The policy of deep deterministic policy gradient.

Parameters:

action_space (Optional[Dict[str, Box]]) – The action space.
n_agents (int) – The number of agents.
actor_representation (Module) – The representation module for actor network.
critic_representation (Module) – The representation module for critic network.
actor_hidden_size (Sequence[int]) – List of hidden units for actor network.
critic_hidden_size (Sequence[int]) – List of hidden units for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – Other arguments.

Returns Q^policy of current observations and actions pairs.

Parameters:

joint_observation (Tensor) – The joint observations of the team.
joint_actions (Tensor) – The joint actions of the team.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The evaluations of Q^policy calculated by critic A. q_eval_B (Dict[Tensor]): The evaluations of Q^policy calculated by critic B. q_eval (Dict[Tensor]): The evaluations of Q^policy averaged by critic A and Critic B.

Return type:

q_eval_A (Dict[Tensor])

Returns the Q^target of next observations and actions pairs.

Parameters:

joint_observation (Tensor) – The joint observations of the team.
joint_actions (Tensor) – The joint actions of the team.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The evaluations of Q^target.

Return type:

q_target (Dict[Tensor])

property parameters_critic¶

soft_update(tau=0.005)[source]¶

class xuance.torch.policies.deterministic_marl.MFQnetwork(*args: Any, **kwargs: Any)[source]¶

Bases: Module

The base class to implement Mean Field Reinforcement Learning - MFQ.

Parameters:

action_space (Optional[Dict[str, Discrete]]) – The action space, which type is gym.spaces.Discrete.
n_agents (int) – The number of agents.
representation (ModuleDict) – A dict of the representation module for all agents.
hidden_size (Sequence[int]) – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters’ initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – Other arguments.

Qtarget(observation: Dict[str, torch.Tensor], actions_mean: Dict[str, torch.Tensor], agent_ids: Dict[str, torch.Tensor], agent_key: str = None, rnn_hidden: Dict[str, List[torch.Tensor]] | None = None)[source]¶

Returns the Q^target of next observations and actions pairs.

Parameters:

observation (Dict[Tensor]) – The observations.
actions_mean (Dict[str, Tensor]) – The mean of each agent’s neighbors.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. q_target: The evaluations of Q^target.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

copy_target()[source]¶

forward(observation: Dict[str, torch.Tensor], agent_ids: torch.Tensor = None, actions_mean: Dict[str, torch.Tensor] = None, avail_actions: Dict[str, torch.Tensor] = None, agent_key: str = None, rnn_hidden: Dict[str, List[torch.Tensor]] | None = None)[source]¶

Returns actions of the policy.

Parameters:

observation (Dict[Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
actions_mean (Dict[str, Tensor]) – The mean actions of each agent’s neighbors.
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. argmax_action (Dict[str, Tensor]): The actions output by the policies. evalQ (Dict[str, Tensor])： The evaluations of observation-action pairs.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

get_boltzmann_policy(q)[source]¶

get_mean_actions(actions: Dict[str, torch.Tensor], agent_mask_tensor: torch.Tensor, batch_size: int)[source]¶

property parameters_model¶

class xuance.torch.policies.deterministic_marl.MixingQnetwork(*args: Any, **kwargs: Any)[source]¶

Bases: BasicQnetwork

The base class to implement value-decomposition based policy.

Parameters:

action_space (Optional[Dict[str, Discrete]]) – The action space, which type is gym.spaces.Discrete.
n_agents (int) – The number of agents.
representation (ModuleDict) – A dict of the representation module for all agents.
mixer (Module) – The mixer module that mix together the individual values to the total value.
hidden_size (Sequence[int]) – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – Other arguments.

Q_tot(individual_values: Dict[str, torch.Tensor], states: torch.Tensor | None = None)[source]¶

Returns the total Q values.

Parameters:

individual_values (Dict[str, Tensor]) – The individual Q values of all agents.
states (Optional[Tensor]) – The global states if necessary, default is None.

Returns:

The evaluated total Q values for the multi-agent team.

Return type:

evalQ_tot (Tensor)

Qtarget_tot(individual_values: Dict[str, torch.Tensor], states: torch.Tensor | None = None)[source]¶

Returns the total Q values with target networks.

Parameters:

individual_values (Dict[str, Tensor]) – The individual Q values of all agents.
states (Optional[Tensor]) – The global states if necessary, default is None. (Shape: batch * dim_state)

Returns:

The evaluated total Q values calculated by target networks.

Return type:

q_target_tot (Tensor)

copy_target()[source]¶

property parameters_model¶

class xuance.torch.policies.deterministic_marl.Qtran_MixingQnetwork(*args: Any, **kwargs: Any)[source]¶

Bases: BasicQnetwork

The base class to implement weighted value-decomposition based policy.

Parameters:

action_space (Optional[Dict[str, Discrete]]) – The action space, which type is gym.spaces.Discrete.
n_agents (int) – The number of agents.
representation (ModuleDict) – A dict of the representation module for all agents.
mixer (Module) – The mixer module that mix together the individual values to the total value.
qtran_mixer (Module) – The feedforward mixer module that mix together the individual values to the total value.
hidden_size (Sequence[int]) – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – Other arguments.

Q_tot(individual_values: Dict[str, torch.Tensor], states: torch.Tensor | None = None)[source]¶

Returns the total Q values.

Parameters:

individual_values (Dict[str, Tensor]) – The individual Q values of all agents.
states (Optional[Tensor]) – The global states if necessary, default is None.

Returns:

The evaluated total Q values for the multi-agent team.

Return type:

evalQ_tot (Tensor)

Q_tran(states: torch.Tensor, hidden_states: Dict[str, torch.Tensor], actions: Dict[str, torch.Tensor], agent_mask: Dict[str, torch.Tensor] = None, avail_actions: Dict[str, torch.Tensor] = None)[source]¶

Returns the total Q values.

Parameters:

states (Tensor) – The global states.
hidden_states (Dict[str, Tensor]) – The hidden states.
actions (Dict[str, Tensor]) – The executed actions.
agent_mask (Dict[str, Tensor]) – Agent mask values, default is None.
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.

Returns:

The evaluated joint Q values. v_jt (Tensor): The evaluated joint V values.

Return type:

q_jt (Tensor)

Q_tran_target(states: torch.Tensor, hidden_states: Dict[str, torch.Tensor], actions: Dict[str, torch.Tensor], agent_mask: Dict[str, torch.Tensor] = None, avail_actions: Dict[str, torch.Tensor] = None)[source]¶

Returns the total Q values.

Parameters:

states (Tensor) – The global states.
hidden_states (Dict[str, Tensor]) – The hidden states.
actions (Dict[str, Tensor]) – The executed actions.
agent_mask (Dict[str, Tensor]) – Agent mask values, default is None.
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.

Returns:

The evaluated joint Q values. v_jt (Tensor): The evaluated joint V values.

Return type:

q_jt (Tensor)

Qtarget(observation: Dict[str, torch.Tensor], agent_ids: Dict[str, torch.Tensor], agent_key: str = None, rnn_hidden: Dict[str, List[torch.Tensor]] | None = None)[source]¶

Returns the Q^target of next observations and actions pairs.

Parameters:

observation (Dict[Tensor]) – The observations.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. rep_hidden_state (Dict[str, Tensor]): The hidden states. q_target: The evaluations of Q^target.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

copy_target()[source]¶

Returns actions of the policy.

Parameters:

observation (Dict[Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. rep_hidden_state (Dict[str, Tensor]): The hidden states. argmax_action (Dict[str, Tensor]): The actions output by the policies. evalQ (Dict[str, Tensor])： The evaluations of observation-action pairs.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

property parameters_model¶

class xuance.torch.policies.deterministic_marl.Weighted_MixingQnetwork(*args: Any, **kwargs: Any)[source]¶

Bases: MixingQnetwork

The base class to implement weighted value-decomposition based policy.

Parameters:

action_space (Optional[Dict[str, Discrete]]) – The action space, which type is gym.spaces.Discrete.
n_agents (int) – The number of agents.
representation (ModuleDict) – A dict of the representation module for all agents.
mixer (Module) – The mixer module that mix together the individual values to the total value.
ff_mixer (Module) – The feedforward mixer module that mix together the individual values to the total value.
hidden_size (Sequence[int]) – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – Other arguments.

copy_target()[source]¶

property parameters_model¶

q_centralized(observation: Dict[str, torch.Tensor], agent_ids: Dict[str, torch.Tensor], agent_key: str = None, rnn_hidden: Dict[str, List[torch.Tensor]] | None = None)[source]¶

Returns the centralised Q value.

Parameters:

observation (Dict[Tensor]) – The observations.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. evalQ_cent (Tensor): The evaluated centralised Q values.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

q_feedforward(individual_values: Dict[str, torch.Tensor], states: torch.Tensor | None = None)[source]¶

Returns the total Q values with feedforward mixer networks.

Parameters:

individual_values (Dict[str, Tensor]) – The individual Q values of all agents.
states (Optional[Tensor]) – The global states if necessary, default is None.

Returns:

The evaluated total Q values for the multi-agent team.

Return type:

evalQ_tot (Tensor)

target_q_centralized(observation: Dict[str, torch.Tensor], agent_ids: Dict[str, torch.Tensor], agent_key: str = None, rnn_hidden: Dict[str, List[torch.Tensor]] | None = None)[source]¶

Returns the centralised Q value with target networks.

Parameters:

observation (Dict[Tensor]) – The observations.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. q_target_cent (Tensor): The evaluated centralised Q values with target networks.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

target_q_feedforward(individual_values: Dict[str, torch.Tensor], states: torch.Tensor | None = None)[source]¶

Returns the total Q values with target feedforward mixer networks.

Parameters:

individual_values (Dict[str, Tensor]) – The individual Q values of all agents.
states (Optional[Tensor]) – The global states if necessary, default is None.

Returns:

The evaluated total Q values for the multi-agent team.

Return type:

q_target_tot (Tensor)

dreamer¶

class xuance.torch.policies.dreamer.DreamerV2Policy(*args: Any, **kwargs: Any)[source]¶

Bases: Module

actor_critic_forward(posteriors: torch.Tensor, recurrent_states: torch.Tensor, terms: torch.Tensor) → Dict[str, List[Any]][source]¶

static compute_lambda_values(rewards: torch.Tensor, values: torch.Tensor, continues: torch.Tensor, bootstrap: torch.Tensor = None, horizon: int = 15, lmbda: float = 0.95) → torch.Tensor[source]¶

hard_update()[source]¶

model_forward(obs: torch.Tensor, acts: torch.Tensor, is_first: torch.Tensor) → Tuple[torch.distributions.Independent, torch.distributions.Independent, torch.distributions.Independent, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor][source]¶

class xuance.torch.policies.dreamer.DreamerV3Policy(*args: Any, **kwargs: Any)[source]¶

Bases: Module

actor_critic_forward(posteriors: torch.Tensor, recurrent_states: torch.Tensor, terms: torch.Tensor) → Dict[str, List[Any]][source]¶

model_forward(obs: torch.Tensor, acts: torch.Tensor, is_first: torch.Tensor) → Tuple[SymLogDistribution, TwoHotEncodingDistribution, torch.distributions.Independent, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor][source]¶

soft_update(tau=0.02)[source]¶

gaussian¶

class xuance.torch.policies.gaussian.ActorCriticPolicy(*args: Any, **kwargs: Any)[source]¶

Bases: Module

Actor-Critic for stochastic policy with Gaussian distributions. (Continuous action space)

Parameters:

action_space (Box) – The continuous action space.
representation (Module) – The representation module.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

forward(observation: torch.Tensor | dict)[source]¶

Returns the hidden states, action distribution, and values.

Parameters:: observation – The original observation of agent.
Returns:: The outputs of representation. a_dist: The distribution of actions output by actor. value: The state values output by critic.
Return type:: outputs

class xuance.torch.policies.gaussian.ActorPolicy(*args: Any, **kwargs: Any)[source]¶

Bases: Module

Actor for stochastic policy with Gaussian distributions. (Continuous action space)

Parameters:

action_space (Box) – The continuous action space.
representation (Module) – The representation module.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

forward(observation: torch.Tensor | dict)[source]¶

Returns the hidden states, action distribution.

Parameters:: observation – The original observation of agent.
Returns:: The outputs of representation. a_dist: The distribution of actions output by actor.
Return type:: outputs

class xuance.torch.policies.gaussian.PPGActorCritic(*args: Any, **kwargs: Any)[source]¶

Bases: Module

Actor-Critic for PPG with Gaussian distributions. (Continuous action space)

Parameters:

action_space (Box) – The continuous action space.
representation (Module) – The representation module.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

forward(observation: torch.Tensor | dict)[source]¶

Returns the actors representation output, action distribution, values, and auxiliary values.

Parameters:: observation – The original observation of agent.
Returns:: The outputs of actor representation. a_dist: The distribution of actions output by actor. value: The state values output by critic. aux_value: The auxiliary values output by aux_critic.
Return type:: policy_outputs

class xuance.torch.policies.gaussian.SACPolicy(*args: Any, **kwargs: Any)[source]¶

Bases: Module

Actor-Critic for SAC with Gaussian distributions. (Continuous action space)

Parameters:

action_space (Box) – The continuous action space.
representation (Module) – The representation module.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

Qaction(observation: torch.Tensor | dict, action: torch.Tensor)[source]¶

Returns the evaluated Q-values for current observation-action pairs.

Parameters:

observation – The original observation.
action – The selected actions.

Returns:

The Q-value calculated by the first critic network. q_2: The Q-value calculated by the other critic network.

Return type:

q_1

Qpolicy(observation: torch.Tensor | dict)[source]¶

Feedforward and calculate the log of action probabilities, and Q-values.

Parameters:: observation – The original observation of an agent.
Returns:: The log of action probabilities. q_1: The Q-value calculated by the first critic network. q_2: The Q-value calculated by the other critic network.
Return type:: log_action_prob

Qtarget(observation: torch.Tensor | dict)[source]¶

Calculate the log of action probabilities and Q-values with target networks.

Parameters:: observation – The original observation of an agent.
Returns:: The log of action probabilities. target_q: The minimum of Q-values calculated by the target critic networks.
Return type:: log_action_prob

forward(observation: torch.Tensor | dict)[source]¶

Returns the output of actor representation and samples of actions.

Parameters:: observation – The original observation of an agent.
Returns:: The outputs of the actor representation. act_sample: The sampled actions from the distribution output by the actor.
Return type:: outputs

soft_update(tau=0.005)[source]¶

gaussian_marl¶

class xuance.torch.policies.gaussian_marl.Basic_ISAC_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: Module

Basic_ISAC_Policy: The basic policy for independent soft actor-critic.

Parameters:

action_space (Optional[Dict[str, Box]]) – The continuous action space.
n_agents (int) – The number of agents.
actor_representation (ModuleDict) – A dict of representation modules for each agent’s actor.
critic_representation (ModuleDict) – A dict of representation modules for each agent’s critic.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – Other arguments.

Qpolicy(observation: Dict[str, torch.Tensor], actions: Dict[str, torch.Tensor], agent_ids: torch.Tensor = None, agent_key: str = None, rnn_hidden_critic_1: Dict[str, List[torch.Tensor]] | None = None, rnn_hidden_critic_2: Dict[str, List[torch.Tensor]] | None = None)[source]¶

Returns Q^policy of current observations and actions pairs.

Parameters:

observation (Dict[Tensor]) – The observations.
actions (Dict[Tensor]) – The actions.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden_critic_1 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_1 representation.
rnn_hidden_critic_2 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_2 representation.

Returns:

The updated rnn states for critic_1_representation. rnn_hidden_critic_new_2: The updated rnn states for critic_2_representation. q_1: The evaluation of Q values with critic 1. q_2: The evaluation of Q values with critic 2.

Return type:

rnn_hidden_critic_new_1

Qtarget(next_observation: Dict[str, torch.Tensor], next_actions: Dict[str, torch.Tensor], agent_ids: torch.Tensor = None, agent_key: str = None, rnn_hidden_critic_1: Dict[str, List[torch.Tensor]] | None = None, rnn_hidden_critic_2: Dict[str, List[torch.Tensor]] | None = None)[source]¶

Returns the Q^target of next observations and actions pairs.

Parameters:

next_observation (Dict[Tensor]) – The observations of next step.
next_actions (Dict[Tensor]) – The actions of next step.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden_critic_1 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_1 representation.
rnn_hidden_critic_2 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_2 representation.

Returns:

The updated rnn states for critic_1_representation. rnn_hidden_critic_new_2: The updated rnn states for critic_2_representation. q_target: The evaluations of Q^target.

Return type:

rnn_hidden_critic_new_1

Returns actions of the policy.

Parameters:

observation (Dict[Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. actions (Dict[Tensor]): The actions output by the policies.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

property parameters_actor¶

property parameters_critic¶

soft_update(tau=0.005)[source]¶

class xuance.torch.policies.gaussian_marl.MAAC_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: Module

MAAC_Policy: Multi-Agent Actor-Critic Policy with Gaussian distributions.

Parameters:

action_space (Box) – The continuous action space.
n_agents (int) – The number of agents.
representation_actor (ModuleDict) – A dict of representation modules for each agent’s actor.
representation_critic (ModuleDict) – A dict of representation modules for each agent’s critic.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – Other arguments.

Returns actions of the policy.

Parameters:

observation (Dict[str, Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states of actor representation.

Returns:

The new RNN hidden states of actor representation. pi_dists (dict): The stochastic policy distributions.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

get_values(observation: Dict[str, torch.Tensor], agent_ids: torch.Tensor = None, agent_key: str = None, rnn_hidden: Dict[str, List[torch.Tensor]] | None = None)[source]¶

Get critic values via critic networks.

Parameters:

observation (Dict[str, Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states of critic representation.

Returns:

The new RNN hidden states of critic representation. values (dict): The evaluated critic values.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

property parameters_model¶

value_tot(values_n: torch.Tensor, global_state=None)[source]¶

class xuance.torch.policies.gaussian_marl.MASAC_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: Basic_ISAC_Policy

Basic_ISAC_Policy: The basic policy for independent soft actor-critic.

Parameters:

action_space (Box) – The continuous action space.
n_agents (int) – The number of agents.
actor_representation (ModuleDict) – A dict of representation modules for each agent’s actor.
critic_representation (ModuleDict) – A dict of representation modules for each agent’s critic.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
device (Optional[Union[str, int, torch.device]]) – The calculating device.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – Other arguments.

Returns Q^policy of current observations and actions pairs.

Parameters:

joint_observation (Optional[Tensor]) – The joint observations of the team.
joint_actions (Optional[Tensor]) – The joint actions of the team.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden_critic_1 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_1 representation.
rnn_hidden_critic_2 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_2 representation.

Returns:

Return type:

rnn_hidden_critic_new_1

Returns the Q^target of next observations and actions pairs.

Parameters:

joint_observation (Optional[Tensor]) – The joint observations of the team.
joint_actions (Optional[Tensor]) – The joint actions of the team.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden_critic_1 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_1 representation.
rnn_hidden_critic_2 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_2 representation.

Returns:

The updated rnn states for critic_1_representation. rnn_hidden_critic_new_2: The updated rnn states for critic_2_representation. q_target: The evaluations of Q^target.

Return type:

rnn_hidden_critic_new_1