policies¶

categorical¶

class xuance.mindspore.policies.categorical.ActorCriticPolicy(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

Actor-Critic for stochastic policy with categorical distributions. (Discrete action space)

Parameters:

action_space (Discrete) – The discrete action space.
representation (Module) – The representation module.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

construct(observation: mindspore.Tensor)[source]¶

Returns the hidden states, action distribution, and values.

Parameters:: observation – The original observation of agent.
Returns:: The outputs of representation. logits: The categorical logits for the actor distribution. value: The state values output by critic.
Return type:: outputs

class xuance.mindspore.policies.categorical.ActorPolicy(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

Actor for stochastic policy with categorical distributions. (Discrete action space)

Parameters:

action_space (Discrete) – The discrete action space.
representation (Module) – The representation module.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

construct(observation: mindspore.Tensor)[source]¶

Returns the hidden states, action distribution.

Parameters:: observation – The original observation of agent.
Returns:: The outputs of representation. logits: The categorical logits for the actor distribution.
Return type:: outputs

class xuance.mindspore.policies.categorical.PPGActorCritic(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

Actor-Critic for PPG with categorical distributions. (Discrete action space)

Parameters:

action_space (Discrete) – The discrete action space.
representation (Module) – The representation module.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

construct(observation: mindspore.Tensor)[source]¶

Returns the actors representation output, action distribution, values, and auxiliary values.

Parameters:: observation – The original observation of agent.
Returns:: The outputs of actor representation. a_logits: The categorical logits for the actor distribution. value: The state values output by critic. aux_value: The auxiliary values output by aux_critic.
Return type:: policy_outputs

class xuance.mindspore.policies.categorical.SACDISPolicy(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

Actor-Critic for SAC with categorical distributions. (Discrete action space)

Parameters:

action_space (Discrete) – The discrete action space.
representation (Module) – The representation module.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

Qaction(observation: mindspore.Tensor | dict)[source]¶

Returns the evaluated Q-values for current observations.

Parameters:: observation – The original observation.
Returns:: The Q-value calculated by the first critic network. q_2: The Q-value calculated by the other critic network.
Return type:: q_1

Qpolicy(observation: mindspore.Tensor | dict)[source]¶

Feedforward and calculate the action probabilities, log of action probabilities, and Q-values.

Parameters:: observation – The original observation of an agent.
Returns:: The probabilities of actions. log_action_prob: The log of action probabilities. q_1: The Q-value calculated by the first critic network. q_2: The Q-value calculated by the other critic network.
Return type:: act_prob

Qtarget(observation: mindspore.Tensor | dict)[source]¶

Calculate the action probabilities, log of action probabilities, and Q-values with target networks.

Parameters:: observation – The original observation of an agent.
Returns:: The probabilities of actions. log_action_prob: The log of action probabilities. target_q: The minimum of Q-values calculated by the target critic networks.
Return type:: new_act_prob

construct(observation: mindspore.Tensor)[source]¶

Returns the output of actor representation and samples of actions.

Parameters:: observation – The original observation of an agent.
Returns:: The outputs of the actor representation. logits: The distribution of actions output by actor.
Return type:: outputs

soft_update(tau=0.005)[source]¶

categorical_marl¶

class xuance.mindspore.policies.categorical_marl.Basic_ISAC_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

Basic_ISAC_Policy: The basic policy for independent soft actor-critic.

Parameters:

action_space (Box) – The continuous action space.
n_agents (int) – The number of agents.
actor_representation (ModuleDict) – A dict of representation modules for each agent’s actor.
critic_representation (ModuleDict) – A dict of representation modules for each agent’s critic.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – Other arguments.

class xuance.mindspore.policies.categorical_marl.COMA_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

COMA_Policy: Counterfactual Multi-Agent Actor-Critic Policy with categorical distributions.

Parameters:

action_space (Optional[Dict[str, Discrete]]) – The discrete action space.
n_agents (int) – The number of agents.
representation_actor (ModuleDict) – A dict of representation modules for each agent’s actor.
representation_critic (ModuleDict) – A dict of representation modules for each agent’s critic.
mixer (Module) – The mixer module that mix together the individual values to the total value.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – The other args.

construct(observation: Dict[str, mindspore.Tensor], agent_ids: mindspore.Tensor | None = None, avail_actions: Dict[str, mindspore.Tensor] = None, agent_key: str = None, rnn_hidden: Dict[str, List[mindspore.Tensor]] | None = None, **kwargs)[source]¶

Returns actions of the policy.

Parameters:

observation (Dict[str, Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states of actor representation.

Returns:

The new RNN hidden states of actor representation. pi_logits (dict): The output of the actors.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

copy_target()[source]¶

get_values(state: mindspore.Tensor, observation: Dict[str, mindspore.Tensor], actions: Dict[str, mindspore.Tensor], agent_ids: mindspore.Tensor = None, rnn_hidden: Dict[str, List[mindspore.Tensor]] | None = None, target=False)[source]¶

Get evaluated critic values.

Parameters:

state – Tensor: The global state.
observation (Dict[str, Tensor]) – The input observations for the policies.
actions (Dict[str, Tensor]) – The input actions.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states of critic representation.
target – If to use target critic network to calculate the critic values.

Returns:

The new RNN hidden states of critic representation. values (dict): The evaluated critic values.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

property parameters_actor¶

property parameters_critic¶

class xuance.mindspore.policies.categorical_marl.CommNet_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: MAAC_Policy

forward(observation: Dict[str, mindspore.Tensor], agent_ids: mindspore.Tensor | None = None, avail_actions: Dict[str, mindspore.Tensor] = None, agent_key: str = None, rnn_hidden: Dict[str, List[mindspore.Tensor]] | None = None, alive_ally: dict | None = None)[source]¶

get_values(observation: Dict[str, mindspore.Tensor], agent_ids: mindspore.Tensor = None, agent_key: str = None, rnn_hidden: Dict[str, List[mindspore.Tensor]] | None = None, alive_ally: dict | None = None)[source]¶

Get critic values via critic networks.

Parameters:

observation (Dict[str, Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states of critic representation.

Returns:

The new RNN hidden states of critic representation. values (dict): The evaluated critic values.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

property parameters_model¶

class xuance.mindspore.policies.categorical_marl.IC3Net_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: CommNet_Policy

forward(observation: Dict[str, mindspore.Tensor], agent_ids: mindspore.Tensor | None = None, avail_actions: Dict[str, mindspore.Tensor] = None, agent_key: str = None, rnn_hidden: Dict[str, List[mindspore.Tensor]] | None = None, alive_ally: dict | None = None)[source]¶

Get critic values via critic networks.

Parameters:

observation (Dict[str, Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states of critic representation.

Returns:

The new RNN hidden states of critic representation. values (dict): The evaluated critic values.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

class xuance.mindspore.policies.categorical_marl.MAAC_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

MAAC_Policy: Multi-Agent Actor-Critic Policy with categorical policies.

Parameters:

action_space (Optional[Dict[str, Discrete]]) – The discrete action space.
n_agents (int) – The number of agents.
representation_actor (dict) – A dict of representation modules for each agent’s actor.
representation_critic (dict) – A dict of representation modules for each agent’s critic.
mixer (Module) – The mixer module that mix together the individual values to the total value.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initializer (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – The other args.

construct(observation: Dict[str, mindspore.Tensor], agent_ids: mindspore.Tensor | None = None, avail_actions: Dict[str, mindspore.Tensor] = None, agent_key: str = None, rnn_hidden: Dict[str, List[mindspore.Tensor]] | None = None)[source]¶

Returns actions of the policy.

Parameters:

observation (Dict[str, Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states of actor representation.

Returns:

The new RNN hidden states of actor representation. pi_dists (dict): The stochastic policy distributions.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

get_values(observation: Dict[str, mindspore.Tensor], agent_ids: mindspore.Tensor = None, agent_key: str = None, rnn_hidden: Dict[str, List[mindspore.Tensor]] | None = None)[source]¶

Get critic values via critic networks.

Parameters:

observation (Dict[str, Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states of critic representation.

Returns:

The new RNN hidden states of critic representation. values (dict): The evaluated critic values.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

property parameters_model¶

value_tot(values_n: mindspore.Tensor, global_state=None)[source]¶

class xuance.mindspore.policies.categorical_marl.MAAC_Policy_Share(*args: Any, **kwargs: Any)[source]¶

Bases: MAAC_Policy

MAAC_Policy_Share: Multi-agent actor-critic Policy with categorical policies and shared representations.

Parameters:

action_space (Optional[Dict[str, Discrete]]) – The discrete action space.
n_agents (int) – The number of agents.
representation (ModuleDict) – A dict of representation modules.
mixer (Module) – The mixer module that mix together the individual values to the total value.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – The other args.

construct(inputs: mindspore.Tensor | dict, *rnn_hidden, **kwargs)[source]¶

Returns actions of the policy.

Parameters:

observation (Dict[str, Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states of actor representation.

Returns:

The new RNN hidden states of actor representation. pi_dists (dict): The stochastic policy distributions.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

trainable_param()[source]¶

value_tot(values_n: mindspore.Tensor, global_state=None)[source]¶

class xuance.mindspore.policies.categorical_marl.MASAC_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: Basic_ISAC_Policy

Basic_ISAC_Policy: The basic policy for independent soft actor-critic.

Parameters:

action_space (Box) – The continuous action space.
n_agents (int) – The number of agents.
actor_representation (ModuleDict) – A dict of representation modules for each agent’s actor.
critic_representation (ModuleDict) – A dict of representation modules for each agent’s critic.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – Other arguments.

class xuance.mindspore.policies.categorical_marl.MeanFieldActorCriticPolicy(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

Mean-field actor-critic policy.

This policy maintains separate actor and critic networks for each agent type (model key), embeds the mean action of neighboring agents, and produces Boltzmann policies.

Parameters:

action_space (Discrete) – A mapping from model keys to discrete action spaces.
n_agents (int) – Total number of agents in the environment.
representation_actor (Optional[Dict[str, Module]]) – Actor state encoder modules for each model key.
representation_critic (Optional[Dict[str, Module]]) – Critic state encoder modules for each model key.
actor_hidden_size (Sequence[int], optional) – Hidden layer sizes for actor networks.
critic_hidden_size (Sequence[int], optional) – Hidden layer sizes for critic networks.
normalize (Optional[tk.layers.Layer]) – Normalization layer to apply after each hidden layer.
initialize (Optional[tk.initializers.Initializer]) – Weight initialization function.
activation (Optional[tk.layers.Layer]) – Activation function class for hidden layers.
use_distributed_training (bool) – If True, wrap components in DistributedDataParallel.
**kwargs – Additional keyword arguments: use_parameter_sharing (bool): Whether to share parameters across agent types. model_keys (List[str]): Keys identifying different agent types. rnn (str): RNN type, e.g., “LSTM” or “GRU”. use_rnn (bool): Flag indicating whether to include RNN layers. action_embedding_hidden_size (Sequence[int]): Hidden sizes for action mean embedding. temperature (float): Temperature parameter for Boltzmann policy.

Returns actions of the policy.

Parameters:

observation (Dict[str, Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states of actor representation.

Returns:

The new RNN hidden states of actor representation. pi_dists (dict): The stochastic policy distributions.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

get_mean_actions(actions: Dict[str, mindspore.Tensor], agent_mask_tensor: mindspore.Tensor, batch_size: int)[source]¶

Compute mean one-hot action vectors of each agent’s neighbors.

For each batch and agent, exclude the agent’s own action and average the one-hot action encodings of its alive neighbors.

Parameters:

actions (Dict[str, Tensor]) – Mapping from model keys to chosen action indices of shape [batch_size * n_agents].
agent_mask_tensor (Tensor) – Binary mask of shape [batch_size, n_agents] indicating alive (1) or dead (0) agents.
batch_size (int) – Number of samples in the batch.

Returns:

Mean one-hot action tensor of shape [batch_size, n_agents, n_actions_max].

Return type:

Tensor

get_values(observation: Dict[str, mindspore.Tensor], actions_mean: Dict[str, mindspore.Tensor] = None, agent_ids: mindspore.Tensor = None, agent_key: str = None, rnn_hidden: Dict[str, List[mindspore.Tensor]] | None = None)[source]¶

Get critic values via critic networks.

Parameters:

observation (Dict[str, Tensor]) – The input observations for the policies.
actions_mean (Dict[str, Tensor]) – The mean actions of each agent’s neighbors.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states of critic representation.

Returns:

The new RNN hidden states of critic representation. values (dict): The evaluated critic values.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

property parameters_model¶

class xuance.mindspore.policies.categorical_marl.TarMAC_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: IC3Net_Policy

forward(observation: Dict[str, mindspore.Tensor], agent_ids: mindspore.Tensor | None = None, avail_actions: Dict[str, mindspore.Tensor] = None, agent_key: str = None, rnn_hidden: Dict[str, List[mindspore.Tensor]] | None = None, alive_ally: dict | None = None)[source]¶

coordination_graph¶

class xuance.mindspore.policies.coordination_graph.Coordination_Graph(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

set_coordination_graph()[source]¶

class xuance.mindspore.policies.coordination_graph.DCG_payoff(*args: Any, **kwargs: Any)[source]¶

Bases: DCG_utility

construct(hidden_states_n, edges_from=None, edges_to=None)[source]¶

class xuance.mindspore.policies.coordination_graph.DCG_utility(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

construct(hidden_states_n)[source]¶

core¶

class xuance.mindspore.policies.core.ActorNet(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

The actor network for deterministic policy, which outputs activated continuous actions directly.

Parameters:

state_dim (int) – The input state dimension.
action_dim (int) – The dimension of continuous action space.
hidden_sizes (Sequence[int]) – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.

construct(x: mindspore.Tensor, avail_actions: mindspore.Tensor | None = None)[source]¶: Returns the output of the actor. :param x: The input tensor. :type x: Tensor :param avail_actions: The actions mask values when use actions mask, default is None. :type avail_actions: Optional[Tensor]

class xuance.mindspore.policies.core.BasicQhead(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

A base class to build Q network and calculate the Q values.

Parameters:

state_dim (int) – The input state dimension.
n_actions (int) – The number of discrete actions.
hidden_sizes – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.

construct(x: mindspore.Tensor)[source]¶: Returns the output of the Q network. :param x: The input tensor. :type x: Tensor

class xuance.mindspore.policies.core.BasicRecurrent(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

Build recurrent neural network to calculate Q values.

construct(x: mindspore.Tensor, h: mindspore.Tensor, c: mindspore.Tensor = None)[source]¶: Returns the rnn hidden and Q-values via RNN networks.

class xuance.mindspore.policies.core.C51Qhead(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

A base class to build Q network and calculate the distributional Q values.

Parameters:

state_dim (int) – The input state dimension.
n_actions (int) – The number of discrete actions.
atom_num (int) – The number of atoms.
hidden_sizes – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.

construct(x: mindspore.Tensor)[source]¶

Returns the discrete action distributions. :param x: The input tensor. :type x: Tensor

Returns:: The probability distribution of the discrete actions.
Return type:: dist_probs

class xuance.mindspore.policies.core.CategoricalActorNet(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

The actor network for categorical policy, which outputs a distribution over all discrete actions.

Parameters:

state_dim (int) – The input state dimension.
action_dim (int) – The dimension of continuous action space.
hidden_sizes (Sequence[int]) – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.

construct(x: mindspore.Tensor, avail_actions: mindspore.Tensor | None = None)[source]¶

Returns the stochastic distribution over all discrete actions. :param x: The input tensor. :type x: Tensor :param avail_actions: The actions mask values when use actions mask, default is None. :type avail_actions: Optional[Tensor]

Returns:: CategoricalDistribution(action_dim), a distribution over all discrete actions.
Return type:: self.dist

distribution(logits: mindspore.Tensor)[source]¶

class xuance.mindspore.policies.core.CategoricalActorNet_SAC(*args: Any, **kwargs: Any)[source]¶

Bases: CategoricalActorNet

The actor network for categorical policy in SAC-DIS, which outputs a distribution over all discrete actions.

Parameters:

state_dim (int) – The input state dimension.
action_dim (int) – The dimension of continuous action space.
hidden_sizes (Sequence[int]) – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.

construct(x: mindspore.Tensor, avail_actions: mindspore.Tensor | None = None)[source]¶

Returns:: CategoricalDistribution(action_dim), a distribution over all discrete actions.
Return type:: self.dist

distribution(logits: mindspore.Tensor)[source]¶

class xuance.mindspore.policies.core.CriticNet(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

The critic network that outputs the evaluated values for states (State-Value) or state-action pairs (Q-value).

Parameters:

input_dim (int) – The input dimension (dim_state or dim_state + dim_action).
hidden_sizes (Sequence[int]) – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.

construct(x: mindspore.Tensor)[source]¶: Returns the output of the Q network. :param x: The input tensor. :type x: Tensor

class xuance.mindspore.policies.core.DuelQhead(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

A base class to build Q network and calculate the dueling Q values.

Parameters:

state_dim (int) – The input state dimension.
n_actions (int) – The number of discrete actions.
hidden_sizes – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.

construct(x: mindspore.Tensor)[source]¶

Returns the dueling Q-values. :param x: The input tensor. :type x: Tensor

Returns:: The dueling Q-values.
Return type:: q

class xuance.mindspore.policies.core.GaussianActorNet(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

The actor network for Gaussian policy, which outputs a distribution over the continuous action space.

Parameters:

state_dim (int) – The input state dimension.
action_dim (int) – The dimension of continuous action space.
hidden_sizes (Sequence[int]) – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.

construct(x: mindspore.Tensor)[source]¶

Returns the stochastic distribution over the continuous action space. :param x: The input tensor. :type x: Tensor

Returns:: A distribution over the continuous action space.
Return type:: self.dist

distribution(mu: mindspore.Tensor, std: mindspore.Tensor)[source]¶

class xuance.mindspore.policies.core.GaussianActorNet_SAC(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

The actor network for Gaussian policy in SAC, which outputs a distribution over the continuous action space.

Parameters:

state_dim (int) – The input state dimension.
action_dim (int) – The dimension of continuous action space.
hidden_sizes (Sequence[int]) – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.

construct(x: mindspore.Tensor)[source]¶

Returns the stochastic distribution over the continuous action space. :param x: The input tensor. :type x: Tensor

Returns:: A distribution over the continuous action space.
Return type:: self.dist

distribution(mu: mindspore.Tensor, std: mindspore.Tensor)[source]¶

class xuance.mindspore.policies.core.QMIX_FF_mixer(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

The feedforward mixer without the constraints of monotonicity.

construct(values_n, states)[source]¶

Returns the feedforward total Q-values.

Parameters:

values_n – The individual Q-values.
states – The global states.

class xuance.mindspore.policies.core.QMIX_mixer(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

The QMIX mixer. (Monotonicity)

Parameters:

dim_state (int) – The dimension of global state.
dim_hidden (int) – The size of rach hidden layer.
dim_hypernet_hidden (int) – The size of rach hidden layer for hyper network.
n_agents (int) – The number of agents.

construct(values_n, states)[source]¶

Returns the total Q-values for multi-agent team.

Parameters:

values_n – The individual values for agents in team.
states – The global states.

Returns:

The total Q-values for the multi-agent team.

Return type:

q_tot

class xuance.mindspore.policies.core.QRDQNhead(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

A base class to build Q networks for QRDQN policy.

Parameters:

state_dim (int) – The input state dimension.
n_actions (int) – The number of discrete actions.
atom_num (int) – The number of atoms.
hidden_sizes – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.

construct(x: mindspore.Tensor)[source]¶

Returns the quantiles of the distribution. :param x: The input tensor. :type x: Tensor

Returns:: The quantiles of the action distribution.
Return type:: quantiles

class xuance.mindspore.policies.core.QTRAN_alt(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

The basic QTRAN module.

Parameters:

dim_state (int) – The dimension of the global state.
action_space (Dict[str, Discrete]) – The action space for all agents.
dim_hidden (int) – The dimension of the hidden layers.
n_agents (int) – The number of agents.
dim_utility_hidden (int) – The dimension of the utility hidden states.
use_parameter_sharing (bool) – Whether to use parameters sharing trick.

construct(states: mindspore.Tensor, hidden_state_inputs: mindspore.Tensor, actions_onehot: mindspore.Tensor)[source]¶

Calculating the joint Q and V values.

Parameters:

states (Tensor) – The global states.
hidden_state_inputs (Tensor) – The joint hidden states inputs for QTRAN network.
actions_onehot (Tensor) – The joint onehot actions for QTRAN network.

Returns:

The evaluated joint Q values. v_jt (Tensor): The evaluated joint V values.

Return type:

q_jt (Tensor)

class xuance.mindspore.policies.core.QTRAN_base(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

The basic QTRAN module.

Parameters:

dim_state (int) – The dimension of the global state.
action_space (Dict[str, Discrete]) – The action space for all agents.
dim_hidden (int) – The dimension of the hidden layers.
n_agents (int) – The number of agents.
dim_utility_hidden (int) – The dimension of the utility hidden states.
use_parameter_sharing (bool) – Whether to use parameters sharing trick.

construct(states: mindspore.Tensor, hidden_state_inputs: mindspore.Tensor, actions_onehot: mindspore.Tensor)[source]¶

Calculating the joint Q and V values.

Parameters:

states (Tensor) – The global states.
hidden_state_inputs (Tensor) – The joint hidden states inputs for QTRAN network.
actions_onehot (Tensor) – The joint onehot actions for QTRAN network.

Returns:

The evaluated joint Q values. v_jt (Tensor): The evaluated joint V values.

Return type:

q_jt (Tensor)

class xuance.mindspore.policies.core.VDN_mixer(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

The value decomposition networks mixer. (Additivity)

construct(values_n, states=None)[source]¶

deterministic¶

class xuance.mindspore.policies.deterministic.BasicQnetwork(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

The base class to implement DQN based policy

Parameters:

action_space (Discrete) – The action space, which type is gym.spaces.Discrete.
representation (Module) – The representation module.
hidden_size (Sequence[int]) – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

construct(observation: mindspore.Tensor)[source]¶

Returns the output of the representation, greedy actions, and the evaluated Q-values.

Parameters:: observation – The original observation input.
Returns:: The hidden state output by the representation. argmax_action: The greedy actions. evalQ: The evaluated Q-values.
Return type:: outputs

copy_target()[source]¶

target(observation: mindspore.Tensor)[source]¶

Returns the output of the representation, greedy actions, and the evaluated Q-values via target networks.

Parameters:: observation – The original observation input.
Returns:: The hidden state output by the representation. argmax_action: The greedy actions from target networks. targetQ: The evaluated Q-values output by target Q-network.
Return type:: outputs_target

trainable_params(recurse=True)[source]¶

class xuance.mindspore.policies.deterministic.C51Qnetwork(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

The policy for C51 distributional deep Q-networks.

Parameters:

action_space (Discrete) – The action space, which type is gym.spaces.Discrete.
atom_num (int) – The number of atoms.
v_min (float) – The lower bound of value distribution.
v_max (float) – The upper bound of value distribution.
representation (ModuleType) – The representation module.
hidden_size (Sequence[int]) – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

construct(observation: numpy.ndarray | dict)[source]¶

Returns the output of the representation, greedy actions, and the evaluated Z-values.

Parameters:: observation – The original observation input.
Returns:: The hidden state output by the representation. argmax_action: The greedy actions. eval_Z: The evaluated Z-values.
Return type:: outputs

copy_target()[source]¶

target(observation: numpy.ndarray | dict)[source]¶

Returns the output of the representation, greedy actions, and the evaluated Z-values via target networks.

Parameters:: observation – The original observation input.
Returns:: The hidden state output by the representation. argmax_action: The greedy actions from target networks. target_Z: The evaluated Z-values output by target Z-network.
Return type:: outputs_target

class xuance.mindspore.policies.deterministic.DDPGPolicy(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

The policy of deep deterministic policy gradient.

Parameters:

action_space (Space) – The action space.
representation (Module) – The representation module.
actor_hidden_size (Sequence[int]) – List of hidden units for actor network.
critic_hidden_size (Sequence[int]) – List of hidden units for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.

Qaction(observation: numpy.ndarray | dict, action: mindspore.Tensor)[source]¶: Returns the evaluated Q-values of state-action pairs.

Qpolicy(observation: numpy.ndarray | dict)[source]¶: Returns the evaluated Q-values by calculating actions via actor networks.

Qtarget(observation: numpy.ndarray | dict)[source]¶: Returns the evaluated Q-values via target networks.

construct(observation: mindspore.Tensor)[source]¶

Returns the output of the actor representations, and the actions.

Parameters:: observation – The original observation input.
Returns:: The output of the actor representations. act: The actions calculated by the actor.
Return type:: outputs

soft_update(tau=0.005)[source]¶

class xuance.mindspore.policies.deterministic.DRQNPolicy(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

construct(observation: mindspore.Tensor, *rnn_hidden: mindspore.Tensor)[source]¶

copy_target()[source]¶

init_hidden(batch)[source]¶

init_hidden_item(rnn_hidden, i)[source]¶

target(observation: numpy.ndarray | dict, *rnn_hidden: mindspore.Tensor)[source]¶

class xuance.mindspore.policies.deterministic.DuelQnetwork(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

The policy for deep dueling Q-networks.

Parameters:

action_space (Space) – The action space, which type is gym.spaces.Discrete.
representation (ModuleType) – The representation module.
hidden_size (Sequence[int]) – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

construct(observation: mindspore.Tensor)[source]¶

copy_target()[source]¶

target(observation: mindspore.Tensor)[source]¶

trainable_params(recurse=True)[source]¶

class xuance.mindspore.policies.deterministic.MPDQNPolicy(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

Atarget(state)[source]¶

Qeval(state, action, input_q)[source]¶

Qpolicy(state, input_q)[source]¶

Qtarget(state, action)[source]¶

con_action(state)[source]¶

construct()[source]¶

soft_update(tau=0.005)[source]¶

class xuance.mindspore.policies.deterministic.NoisyQnetwork(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

The policy for noisy deep Q-networks.

Parameters:

action_space (Discrete) – The action space, which type is gym.spaces.Discrete.
representation (Module) – The representation module.
hidden_size – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

construct(observation: mindspore.Tensor)[source]¶

Returns the output of the representation, greedy actions, and the evaluated Q-values.

Parameters:: observation – The original observation input.
Returns:: The hidden state output by the representation. argmax_action: The greedy actions. evalQ: The evaluated Q-values.
Return type:: outputs

copy_target()[source]¶

noisy_parameters(is_target=False)[source]¶

target(observation: mindspore.Tensor)[source]¶

Returns the output of the representation, greedy actions, and the evaluated Q-values via target networks.

Parameters:: observation – The original observation input.
Returns:: The hidden state output by the representation. argmax_action: The greedy actions from target networks. targetQ: The evaluated Q-values output by target Q-network.
Return type:: outputs_target

trainable_params(recurse=True)[source]¶

update_noise(noisy_bound: float = 0.0)[source]¶: Updates the noises for network parameters.

class xuance.mindspore.policies.deterministic.PDQNPolicy(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

Atarget(state)[source]¶

Qeval(state, action)[source]¶

Qpolicy(state)[source]¶

Qtarget(state, action)[source]¶

con_action(state)[source]¶

construct()[source]¶

soft_update(tau=0.005)[source]¶

class xuance.mindspore.policies.deterministic.QRDQN_Network(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

The policy for quantile regression deep Q-networks.

Parameters:

action_space (Discrete) – The action space, which type is gym.spaces.Discrete.
quantile_num (int) – The number of quantiles.
representation (ModuleType) – The representation module.
hidden_size (Sequence[int]) – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

construct(observation: mindspore.Tensor)[source]¶

Returns the output of the representation, greedy actions, and the evaluated Z-values.

Parameters:: observation – The original observation input.
Returns:: The hidden state output by the representation. argmax_action: The greedy actions. eval_Z: The evaluated Z-values.
Return type:: outputs

copy_target()[source]¶

target(observation: mindspore.Tensor)[source]¶

Returns the output of the representation, greedy actions, and the evaluated Z-values via target networks.

Parameters:: observation – The original observation input.
Returns:: The hidden state output by the representation. argmax_action: The greedy actions from target networks. target_Z: The evaluated Z-values output by target Z-network.
Return type:: outputs_target

trainable_params(recurse=True)[source]¶

class xuance.mindspore.policies.deterministic.SPDQNPolicy(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

Atarget(state)[source]¶

Qeval(state, action, input_q)[source]¶

Qpolicy(state, input_q)[source]¶

Qtarget(state, action)[source]¶

con_action(state)[source]¶

construct()[source]¶

soft_update(tau=0.005)[source]¶

class xuance.mindspore.policies.deterministic.TD3Policy(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

The policy of twin delayed deep deterministic policy gradient.

Parameters:

action_space (Space) – The action space.
representation (Module) – The representation module.
actor_hidden_size (Sequence[int]) – List of hidden units for actor network.
critic_hidden_size (Sequence[int]) – List of hidden units for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.

Qaction(observation: mindspore.Tensor | dict, action: mindspore.Tensor)[source]¶: Returns the evaluated Q-values of state-action pairs.

Qpolicy(observation: mindspore.Tensor | dict)[source]¶: Returns the evaluated Q-values by calculating actions via actor networks.

Qtarget(observation: mindspore.Tensor | dict)[source]¶: Returns the evaluated Q-values via target networks.

construct(observation: mindspore.Tensor | dict)[source]¶

Returns the output of the actor representations, and the actions.

Parameters:: observation – The original observation input.
Returns:: The output of the actor representations. act: The actions calculated by the actor.
Return type:: outputs

soft_update(tau=0.005)[source]¶

deterministic_marl¶

class xuance.mindspore.policies.deterministic_marl.BasicQnetwork(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

The base class to implement DQN based policy

Parameters:

action_space (Optional[Dict[str, Discrete]]) – The action space, which type is gym.spaces.Discrete.
n_agents (int) – The number of agents.
representation (ModuleDict) – A dict of the representation module for all agents.
hidden_size (Sequence[int]) – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters’ initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – Other arguments.

Qtarget(observation: Dict[str, mindspore.Tensor], agent_ids: Dict[str, mindspore.Tensor], agent_key: str = None, rnn_hidden: Dict[str, List[mindspore.Tensor]] | None = None)[source]¶

Returns the Q^target of next observations and actions pairs.

Parameters:

observation (Dict[Tensor]) – The observations.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. q_target: The evaluations of Q^target.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

construct(observation: Dict[str, mindspore.Tensor], agent_ids: mindspore.Tensor = None, avail_actions: Dict[str, mindspore.Tensor] = None, agent_key: str = None, rnn_hidden: Dict[str, List[mindspore.Tensor]] | None = None, **kwargs)[source]¶

Returns actions of the policy.

Parameters:

observation (Dict[str, Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. argmax_action (Dict[str, Tensor]): The actions output by the policies. evalQ (Dict[str, Tensor])： The evaluations of observation-action pairs.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

copy_target()[source]¶

property parameters_model¶

class xuance.mindspore.policies.deterministic_marl.DCG_policy(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

construct(observation: mindspore.Tensor, agent_ids: mindspore.Tensor, *rnn_hidden: mindspore.Tensor, avail_actions=None)[source]¶

copy_target()[source]¶

class xuance.mindspore.policies.deterministic_marl.Independent_DDPG_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

Atarget(next_observation: Dict[str, mindspore.Tensor], agent_ids: mindspore.Tensor = None, agent_key: str = None, rnn_hidden: Dict[str, List[mindspore.Tensor]] | None = None)[source]¶

Returns the next actions by target policies.

Parameters:

next_observation (Dict[Tensor]) – The observations of next step.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. next_actions (Dict[Tensor]): The next actions.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

Qpolicy(observation: Dict[str, mindspore.Tensor], actions: Dict[str, mindspore.Tensor], agent_ids: mindspore.Tensor = None, agent_key: str = None, rnn_hidden: Dict[str, List[mindspore.Tensor]] | None = None)[source]¶

Returns Q^policy of current observations and actions pairs.

Parameters:

observation (Dict[Tensor]) – The observations.
actions (Dict[Tensor]) – The actions.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. q_eval: The evaluations of Q^policy.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

Qtarget(next_observation: Dict[str, mindspore.Tensor], next_actions: Dict[str, mindspore.Tensor], agent_ids: mindspore.Tensor = None, agent_key: str = None, rnn_hidden: Dict[str, List[mindspore.Tensor]] | None = None)[source]¶

Returns the Q^target of next observations and actions pairs.

Parameters:

next_observation (Dict[Tensor]) – The observations of next step.
next_actions (Dict[Tensor]) – The actions of next step.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. q_target: The evaluations of Q^target.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

construct(observation: Dict[str, mindspore.Tensor], agent_ids: mindspore.Tensor = None, agent_key: str = None, rnn_hidden: Dict[str, List[mindspore.Tensor]] | None = None)[source]¶

Returns actions of the policy.

Parameters:

observation (Dict[Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. actions (Dict[Tensor]): The actions output by the policies.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

property parameters_actor¶

property parameters_critic¶

soft_update(tau=0.005)[source]¶

class xuance.mindspore.policies.deterministic_marl.MADDPG_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: Independent_DDPG_Policy

Qpolicy(joint_observation: mindspore.Tensor, joint_actions: mindspore.Tensor, agent_ids: mindspore.Tensor = None, agent_key: str = None, rnn_hidden: Dict[str, List[mindspore.Tensor]] | None = None)[source]¶

Returns Q^policy of current observations and actions pairs.

Parameters:

joint_observation (Tensor) – The joint observations of the team.
joint_actions (Tensor) – The joint actions of the team.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. q_eval: The evaluations of Q^policy.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

Qtarget(joint_observation: mindspore.Tensor, joint_actions: mindspore.Tensor, agent_ids: mindspore.Tensor = None, agent_key: str = None, rnn_hidden: Dict[str, List[mindspore.Tensor]] | None = None)[source]¶

Returns the Q^target of next observations and actions pairs.

Parameters:

joint_observation (Tensor) – The joint observations of the team.
joint_actions (Tensor) – The joint actions of the team.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. q_target: The evaluations of Q^target.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

class xuance.mindspore.policies.deterministic_marl.MATD3_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: MADDPG_Policy, Cell

Returns Q^policy of current observations and actions pairs.

Parameters:

joint_observation (Tensor) – The joint observations of the team.
joint_actions (Tensor) – The joint actions of the team.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The evaluations of Q^policy calculated by critic A. q_eval_B (Dict[Tensor]): The evaluations of Q^policy calculated by critic B. q_eval (Dict[Tensor]): The evaluations of Q^policy averaged by critic A and Critic B.

Return type:

q_eval_A (Dict[Tensor])

Returns the Q^target of next observations and actions pairs.

Parameters:

joint_observation (Tensor) – The joint observations of the team.
joint_actions (Tensor) – The joint actions of the team.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The evaluations of Q^target.

Return type:

q_target (Dict[Tensor])

property parameters_critic¶

soft_update(tau=0.005)[source]¶

class xuance.mindspore.policies.deterministic_marl.MFQnetwork(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

The base class to implement Mean Field Reinforcement Learning - MFQ.

Parameters:

action_space (Optional[Dict[str, Discrete]]) – The action space, which type is gym.spaces.Discrete.
n_agents (int) – The number of agents.
representation (ModuleDict) – A dict of the representation module for all agents.
hidden_size (Sequence[int]) – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters’ initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – Other arguments.

Qtarget(observation: Dict[str, mindspore.Tensor], actions_mean: Dict[str, mindspore.Tensor], agent_ids: Dict[str, mindspore.Tensor], agent_key: str = None, rnn_hidden: Dict[str, List[mindspore.Tensor]] | None = None)[source]¶

Returns the Q^target of next observations and actions pairs.

Parameters:

observation (Dict[Tensor]) – The observations.
actions_mean (Dict[str, Tensor]) – The mean of each agent’s neighbors.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. q_target: The evaluations of Q^target.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

construct(observation: Dict[str, mindspore.Tensor], agent_ids: mindspore.Tensor = None, actions_mean: Dict[str, mindspore.Tensor] = None, avail_actions: Dict[str, mindspore.Tensor] = None, agent_key: str = None, rnn_hidden: Dict[str, List[mindspore.Tensor]] | None = None)[source]¶

Returns actions of the policy.

Parameters:

observation (Dict[Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
actions_mean (Dict[str, Tensor]) – The mean actions of each agent’s neighbors.
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. argmax_action (Dict[str, Tensor]): The actions output by the policies. evalQ (Dict[str, Tensor])： The evaluations of observation-action pairs.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

copy_target()[source]¶

get_boltzmann_policy(q)[source]¶

get_mean_actions(actions: Dict[str, mindspore.Tensor], agent_mask_tensor: mindspore.Tensor, batch_size: int)[source]¶

property parameters_model¶

class xuance.mindspore.policies.deterministic_marl.MixingQnetwork(*args: Any, **kwargs: Any)[source]¶

Bases: BasicQnetwork

Q_tot(individual_values: Dict[str, mindspore.Tensor], states: mindspore.Tensor | None = None)[source]¶

Returns the total Q values.

Parameters:

individual_values (Dict[str, Tensor]) – The individual Q values of all agents.
states (Optional[Tensor]) – The global states if necessary, default is None.

Returns:

The evaluated total Q values for the multi-agent team.

Return type:

evalQ_tot (Tensor)

Qtarget_tot(individual_values: Dict[str, mindspore.Tensor], states: mindspore.Tensor | None = None)[source]¶

Returns the total Q values with target networks.

Parameters:

individual_values (Dict[str, Tensor]) – The individual Q values of all agents.
states (Optional[Tensor]) – The global states if necessary, default is None. (Shape: batch * dim_state)

Returns:

The evaluated total Q values calculated by target networks.

Return type:

q_target_tot (Tensor)

copy_target()[source]¶

trainable_params(recurse=True)[source]¶

class xuance.mindspore.policies.deterministic_marl.Qtran_MixingQnetwork(*args: Any, **kwargs: Any)[source]¶

Bases: BasicQnetwork

The base class to implement weighted value-decomposition based policy.

Parameters:

action_space (Optional[Dict[str, Discrete]]) – The action space, which type is gym.spaces.Discrete.
n_agents (int) – The number of agents.
representation (ModuleDict) – A dict of the representation module for all agents.
mixer (Module) – The mixer module that mix together the individual values to the total value.
qtran_mixer (Module) – The feedforward mixer module that mix together the individual values to the total value.
hidden_size (Sequence[int]) – List of hidden units for fully connect layers.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – Other arguments.

Q_tot(individual_values: Dict[str, mindspore.Tensor], states: mindspore.Tensor | None = None)[source]¶

Returns the total Q values.

Parameters:

individual_values (Dict[str, Tensor]) – The individual Q values of all agents.
states (Optional[Tensor]) – The global states if necessary, default is None.

Returns:

The evaluated total Q values for the multi-agent team.

Return type:

evalQ_tot (Tensor)

Q_tran(states: mindspore.Tensor, hidden_states: Dict[str, mindspore.Tensor], actions: Dict[str, mindspore.Tensor], agent_mask: Dict[str, mindspore.Tensor] = None, avail_actions: Dict[str, mindspore.Tensor] = None)[source]¶

Returns the total Q values.

Parameters:

states (Tensor) – The global states.
hidden_states (Dict[str, Tensor]) – The hidden states.
actions (Dict[str, Tensor]) – The executed actions.
agent_mask (Dict[str, Tensor]) – Agent mask values, default is None.
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.

Returns:

The evaluated joint Q values. v_jt (Tensor): The evaluated joint V values.

Return type:

q_jt (Tensor)

Q_tran_target(states: mindspore.Tensor, hidden_states: Dict[str, mindspore.Tensor], actions: Dict[str, mindspore.Tensor], agent_mask: Dict[str, mindspore.Tensor] = None, avail_actions: Dict[str, mindspore.Tensor] = None)[source]¶

Returns the total Q values.

Parameters:

states (Tensor) – The global states.
hidden_states (Dict[str, Tensor]) – The hidden states.
actions (Dict[str, Tensor]) – The executed actions.
agent_mask (Dict[str, Tensor]) – Agent mask values, default is None.
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.

Returns:

The evaluated joint Q values. v_jt (Tensor): The evaluated joint V values.

Return type:

q_jt (Tensor)

Qtarget(observation: Dict[str, mindspore.Tensor], agent_ids: Dict[str, mindspore.Tensor], agent_key: str = None, rnn_hidden: Dict[str, List[mindspore.Tensor]] | None = None)[source]¶

Returns the Q^target of next observations and actions pairs.

Parameters:

observation (Dict[Tensor]) – The observations.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. rep_hidden_state (Dict[str, Tensor]): The hidden states. q_target: The evaluations of Q^target.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

Returns actions of the policy.

Parameters:

observation (Dict[Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. rep_hidden_state (Dict[str, Tensor]): The hidden states. argmax_action (Dict[str, Tensor]): The actions output by the policies. evalQ (Dict[str, Tensor])： The evaluations of observation-action pairs.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

copy_target()[source]¶

property parameters_model¶

class xuance.mindspore.policies.deterministic_marl.Weighted_MixingQnetwork(*args: Any, **kwargs: Any)[source]¶

Bases: MixingQnetwork

copy_target()[source]¶

q_centralized(observation: Dict[str, mindspore.Tensor], agent_ids: Dict[str, mindspore.Tensor], agent_key: str = None, rnn_hidden: Dict[str, List[mindspore.Tensor]] | None = None)[source]¶

Returns the centralised Q value.

Parameters:

observation (Dict[Tensor]) – The observations.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. evalQ_cent (Tensor): The evaluated centralised Q values.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

q_feedforward(individual_values: Dict[str, mindspore.Tensor], states: mindspore.Tensor | None = None)[source]¶

Returns the total Q values with feedforward mixer networks.

Parameters:

individual_values (Dict[str, Tensor]) – The individual Q values of all agents.
states (Optional[Tensor]) – The global states if necessary, default is None.

Returns:

The evaluated total Q values for the multi-agent team.

Return type:

evalQ_tot (Tensor)

target_q_centralized(observation: Dict[str, mindspore.Tensor], agent_ids: Dict[str, mindspore.Tensor], agent_key: str = None, rnn_hidden: Dict[str, List[mindspore.Tensor]] | None = None)[source]¶

Returns the centralised Q value with target networks.

Parameters:

observation (Dict[Tensor]) – The observations.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. q_target_cent (Tensor): The evaluated centralised Q values with target networks.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

target_q_feedforward(individual_values: Dict[str, mindspore.Tensor], states: mindspore.Tensor | None = None)[source]¶

Returns the total Q values with target feedforward mixer networks.

Parameters:

individual_values (Dict[str, Tensor]) – The individual Q values of all agents.
states (Optional[Tensor]) – The global states if necessary, default is None.

Returns:

The evaluated total Q values for the multi-agent team.

Return type:

q_target_tot (Tensor)

trainable_params(recurse=True)[source]¶

dreamer¶

class xuance.mindspore.policies.dreamer.DreamerV2Policy(*args: Any, **kwargs: Any)[source]¶

Bases: Module

actor_critic_forward(posteriors: torch.Tensor, recurrent_states: torch.Tensor, terms: torch.Tensor) → Dict[str, List[Any]][source]¶

static compute_lambda_values(rewards: torch.Tensor, values: torch.Tensor, continues: torch.Tensor, bootstrap: torch.Tensor = None, horizon: int = 15, lmbda: float = 0.95) → torch.Tensor[source]¶

hard_update()[source]¶

model_forward(obs: torch.Tensor, acts: torch.Tensor, is_first: torch.Tensor) → Tuple[torch.distributions.Independent, torch.distributions.Independent, torch.distributions.Independent, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor][source]¶

class xuance.mindspore.policies.dreamer.DreamerV3Policy(*args: Any, **kwargs: Any)[source]¶

Bases: Module

actor_critic_forward(posteriors: torch.Tensor, recurrent_states: torch.Tensor, terms: torch.Tensor) → Dict[str, List[Any]][source]¶

model_forward(obs: torch.Tensor, acts: torch.Tensor, is_first: torch.Tensor) → Tuple[SymLogDistribution, TwoHotEncodingDistribution, torch.distributions.Independent, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor][source]¶

soft_update(tau=0.02)[source]¶

gaussian¶

class xuance.mindspore.policies.gaussian.ActorCriticPolicy(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

Actor-Critic for stochastic policy with Gaussian distributions. (Continuous action space)

Parameters:

action_space (Space) – The continuous action space.
representation (Module) – The representation module.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

construct(observation: mindspore.Tensor)[source]¶

Returns the hidden states, action distribution, and values.

Parameters:: observation – The original observation of agent.
Returns:: The outputs of representation. a_mean: The mean variable of the gaussian distribution. a_std: The standard deviation of the gaussian distribution. value: The state values output by critic.
Return type:: outputs

class xuance.mindspore.policies.gaussian.ActorPolicy(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

Actor for stochastic policy with Gaussian distributions. (Continuous action space)

Parameters:

action_space (Space) – The continuous action space.
representation (Module) – The representation module.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

construct(observation: mindspore.Tensor | dict)[source]¶

Returns the hidden states, action distribution.

Parameters:: observation – The original observation of agent.
Returns:: The outputs of representation. a_mean: The mean variable of the gaussian distribution. a_std: The standard deviation of the gaussian distribution.
Return type:: outputs

class xuance.mindspore.policies.gaussian.PPGActorCritic(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

Actor-Critic for PPG with Gaussian distributions. (Continuous action space)

Parameters:

action_space (Space) – The continuous action space.
representation (Module) – The representation module.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

construct(observation: mindspore.Tensor | dict)[source]¶

Returns the actors representation output, action distribution, values, and auxiliary values.

Parameters:: observation – The original observation of agent.
Returns:: The outputs of actor representation. a_mean: The mean variable of the gaussian distribution. a_std: The standard deviation of the gaussian distribution. value: The state values output by critic. aux_value: The auxiliary values output by aux_critic.
Return type:: policy_outputs

class xuance.mindspore.policies.gaussian.SACPolicy(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

Actor-Critic for SAC with Gaussian distributions. (Continuous action space)

Parameters:

action_space (Space) – The continuous action space.
representation (Module) – The representation module.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.

Qaction(observation: mindspore.Tensor | dict, action: mindspore.Tensor)[source]¶

Returns the evaluated Q-values for current observation-action pairs.

Parameters:

observation – The original observation.
action – The selected actions.

Returns:

The Q-value calculated by the first critic network. q_2: The Q-value calculated by the other critic network.

Return type:

q_1

Qpolicy(observation: mindspore.Tensor | dict)[source]¶

Feedforward and calculate the log of action probabilities, and Q-values.

Parameters:: observation – The original observation of an agent.
Returns:: The log of action probabilities. q_1: The Q-value calculated by the first critic network. q_2: The Q-value calculated by the other critic network.
Return type:: log_action_prob

Qtarget(observation: mindspore.Tensor | dict)[source]¶

Calculate the log of action probabilities and Q-values with target networks.

Parameters:: observation – The original observation of an agent.
Returns:: The log of action probabilities. target_q: The minimum of Q-values calculated by the target critic networks.
Return type:: log_action_prob

construct(observation: mindspore.Tensor | dict)[source]¶

Returns the output of actor representation and samples of actions.

Parameters:: observation – The original observation of an agent.
Returns:: The outputs of the actor representation. act_sample: The sampled actions from the distribution output by the actor.
Return type:: outputs

soft_update(tau=0.005)[source]¶

gaussian_marl¶

class xuance.mindspore.policies.gaussian_marl.Basic_ISAC_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

Basic_ISAC_Policy: The basic policy for independent soft actor-critic.

Parameters:

action_space (Optional[Dict[str, Box]]) – The continuous action space.
n_agents (int) – The number of agents.
actor_representation (ModuleDict) – A dict of representation modules for each agent’s actor.
critic_representation (ModuleDict) – A dict of representation modules for each agent’s critic.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – Other arguments.

Qpolicy(observation: Dict[str, mindspore.Tensor], actions: Dict[str, mindspore.Tensor], agent_ids: mindspore.Tensor = None, agent_key: str = None, rnn_hidden_critic_1: Dict[str, List[mindspore.Tensor]] | None = None, rnn_hidden_critic_2: Dict[str, List[mindspore.Tensor]] | None = None)[source]¶

Returns Q^policy of current observations and actions pairs.

Parameters:

observation (Dict[Tensor]) – The observations.
actions (Dict[Tensor]) – The actions.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden_critic_1 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_1 representation.
rnn_hidden_critic_2 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_2 representation.

Returns:

The updated rnn states for critic_1_representation. rnn_hidden_critic_new_2: The updated rnn states for critic_2_representation. q_1: The evaluation of Q values with critic 1. q_2: The evaluation of Q values with critic 2.

Return type:

rnn_hidden_critic_new_1

Qtarget(next_observation: Dict[str, mindspore.Tensor], next_actions: Dict[str, mindspore.Tensor], agent_ids: mindspore.Tensor = None, agent_key: str = None, rnn_hidden_critic_1: Dict[str, List[mindspore.Tensor]] | None = None, rnn_hidden_critic_2: Dict[str, List[mindspore.Tensor]] | None = None)[source]¶

Returns the Q^target of next observations and actions pairs.

Parameters:

next_observation (Dict[Tensor]) – The observations of next step.
next_actions (Dict[Tensor]) – The actions of next step.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden_critic_1 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_1 representation.
rnn_hidden_critic_2 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_2 representation.

Returns:

The updated rnn states for critic_1_representation. rnn_hidden_critic_new_2: The updated rnn states for critic_2_representation. q_target: The evaluations of Q^target.

Return type:

rnn_hidden_critic_new_1

construct(observation: Dict[str, mindspore.Tensor], agent_ids: mindspore.Tensor = None, avail_actions: Dict[str, mindspore.Tensor] = None, agent_key: str = None, rnn_hidden: Dict[str, List[mindspore.Tensor]] | None = None)[source]¶

Returns actions of the policy.

Parameters:

observation (Dict[Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The hidden variables of the RNN.

Returns:

The new hidden variables of the RNN. actions (Dict[Tensor]): The actions output by the policies.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

property parameters_actor¶

property parameters_critic¶

soft_update(tau=0.005)[source]¶

class xuance.mindspore.policies.gaussian_marl.MAAC_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: Cell

MAAC_Policy: Multi-Agent Actor-Critic Policy with Gaussian distributions.

Parameters:

action_space (Box) – The continuous action space.
n_agents (int) – The number of agents.
representation_actor (ModuleDict) – A dict of representation modules for each agent’s actor.
representation_critic (ModuleDict) – A dict of representation modules for each agent’s critic.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – Other arguments.

Returns actions of the policy.

Parameters:

observation (Dict[str, Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
avail_actions (Dict[str, Tensor]) – Actions mask values, default is None.
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states of actor representation.

Returns:

The new RNN hidden states of actor representation. pi_dists (dict): The stochastic policy distributions.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

get_values(observation: Dict[str, mindspore.Tensor], agent_ids: mindspore.Tensor = None, agent_key: str = None, rnn_hidden: Dict[str, List[mindspore.Tensor]] | None = None)[source]¶

Get critic values via critic networks.

Parameters:

observation (Dict[str, Tensor]) – The input observations for the policies.
agent_ids (Tensor) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states of critic representation.

Returns:

The new RNN hidden states of critic representation. values (dict): The evaluated critic values.

Return type:

rnn_hidden_new (Optional[Dict[str, List[Tensor]]])

property parameters_model¶

value_tot(values_n: mindspore.Tensor, global_state=None)[source]¶

class xuance.mindspore.policies.gaussian_marl.MASAC_Policy(*args: Any, **kwargs: Any)[source]¶

Bases: Basic_ISAC_Policy

Basic_ISAC_Policy: The basic policy for independent soft actor-critic.

Parameters:

action_space (Box) – The continuous action space.
n_agents (int) – The number of agents.
actor_representation (ModuleDict) – A dict of representation modules for each agent’s actor.
critic_representation (ModuleDict) – A dict of representation modules for each agent’s critic.
actor_hidden_size (Sequence[int]) – A list of hidden layer sizes for actor network.
critic_hidden_size (Sequence[int]) – A list of hidden layer sizes for critic network.
normalize (Optional[ModuleType]) – The layer normalization over a minibatch of inputs.
initialize (Optional[Callable[..., Tensor]]) – The parameters initializer.
activation (Optional[ModuleType]) – The activation function for each layer.
activation_action (Optional[ModuleType]) – The activation of final layer to bound the actions.
use_distributed_training (bool) – Whether to use multi-GPU for distributed training.
**kwargs – Other arguments.

Qpolicy(joint_observation: mindspore.Tensor | None = None, joint_actions: mindspore.Tensor | None = None, agent_ids: mindspore.Tensor = None, agent_key: str = None, rnn_hidden_critic_1: Dict[str, List[mindspore.Tensor]] | None = None, rnn_hidden_critic_2: Dict[str, List[mindspore.Tensor]] | None = None)[source]¶

Returns Q^policy of current observations and actions pairs.

Parameters:

joint_observation (Optional[Tensor]) – The joint observations of the team.
joint_actions (Optional[Tensor]) – The joint actions of the team.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden_critic_1 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_1 representation.
rnn_hidden_critic_2 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_2 representation.

Returns:

The updated rnn states for critic_1_representation. rnn_hidden_critic_new_2: The updated rnn states for critic_2_representation. q_1: The evaluations of Q^policy with critic 1. q_2: The evaluations of Q^policy with critic 2.

Return type:

rnn_hidden_critic_new_1

Qtarget(joint_observation: mindspore.Tensor | None = None, joint_actions: mindspore.Tensor | None = None, agent_ids: mindspore.Tensor = None, agent_key: str = None, rnn_hidden_critic_1: Dict[str, List[mindspore.Tensor]] | None = None, rnn_hidden_critic_2: Dict[str, List[mindspore.Tensor]] | None = None)[source]¶

Returns the Q^target of next observations and actions pairs.

Parameters:

joint_observation (Optional[Tensor]) – The joint observations of the team.
joint_actions (Optional[Tensor]) – The joint actions of the team.
agent_ids (Dict[Tensor]) – The agents’ ids (for parameter sharing).
agent_key (str) – Calculate actions for specified agent.
rnn_hidden_critic_1 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_1 representation.
rnn_hidden_critic_2 (Optional[Dict[str, List[Tensor]]]) – The RNN hidden states for critic_2 representation.

Returns:

The updated rnn states for critic_1_representation. rnn_hidden_critic_new_2: The updated rnn states for critic_2_representation. q_target: The evaluations of Q^target.

Return type:

rnn_hidden_critic_new_1