multi_agent_rl

coma_learner

COMA: Counterfactual Multi-Agent Policy Gradients Paper link: https://ojs.aaai.org/index.php/AAAI/article/view/11794 Implementation: MindSpore

class xuance.mindspore.learners.multi_agent_rl.coma_learner.COMA_Learner(config: Namespace, model_keys: List[str], agent_keys: List[str], policy: mindspore.nn.Cell, callback)

Bases: LearnerMAS

actor_forward_fn(bs, obs, actions, values_pred_dict, agent_mask, avail_actions, IDs, epsilon)

build_optimizer()

build_training_data(sample: dict | None, use_parameter_sharing: bool | None = False, use_actions_mask: bool | None = False, use_global_state: bool | None = False)

Prepare the training data.

Parameters:

• sample (dict) – The raw sampled data.

• use_parameter_sharing (bool) – Whether to use parameter sharing for individual agent models.

• use_actions_mask (bool) – Whether to use actions mask for unavailable actions.

• use_global_state (bool) – Whether to use global state.

Returns:

The formatted sampled data.

Return type:

sample_Tensor (dict)

critic_forward_fn(bs, batch_size, obs, state, actions, agent_mask, returns, IDs)

update(sample, epsilon=0.0)

commnet_learner

class xuance.mindspore.learners.multi_agent_rl.commnet_learner.CommNet_Learner(config: Namespace, model_keys: List[str], agent_keys: List[str], policy: torch.nn.Module, callback)

Bases: IPPO_Learner

build_optimizer()

update_rnn(sample)

dcg_learner

DCG: Deep coordination graphs Paper link: http://proceedings.mlr.press/v119/boehmer20a/boehmer20a.pdf Implementation: MindSpore

class xuance.mindspore.learners.multi_agent_rl.dcg_learner.DCG_Learner(config: Namespace, model_keys: List[str], agent_keys: List[str], policy: mindspore.nn.Cell, callback)

Bases: LearnerMAS

class PolicyNetWithLossCell(*args: Any, **kwargs: Any)

Bases: Cell

construct(s, o, a, label, *rnn_hidden)

act(hidden_states, avail_actions=None)

get_graph_values(hidden_states, use_target_net=False)

get_hidden_states(obs_n, *rnn_hidden, use_target_net=False)

q_dcg(hidden_states, actions, states=None, use_target_net=False)

update(sample)

iac_learner

Independent Advantage Actor Critic (IAC) Paper link: https://ojs.aaai.org/index.php/AAAI/article/view/11794 Implementation: Pytorch

class xuance.mindspore.learners.multi_agent_rl.iac_learner.IAC_Learner(config: Namespace, model_keys: List[str], agent_keys: List[str], policy: mindspore.nn.Cell, callback)

Bases: LearnerMAS

build_optimizer()

build_training_data(sample: dict | None, use_parameter_sharing: bool | None = False, use_actions_mask: bool | None = False, use_global_state: bool | None = False)

Prepare the training data.

Parameters:

• sample (dict) – The raw sampled data.

• use_parameter_sharing (bool) – Whether to use parameter sharing for individual agent models.

• use_actions_mask (bool) – Whether to use actions mask for unavailable actions.

• use_global_state (bool) – Whether to use global state.

Returns:

The formatted sampled data.

Return type:

sample_Tensor (dict)

forward_fn(*args)

update(sample)

ic3net_learner

class xuance.mindspore.learners.multi_agent_rl.ic3net_learner.IC3Net_Learner(config: Namespace, model_keys: List[str], agent_keys: List[str], policy: torch.nn.Module, callback)

Bases: CommNet_Learner

build_training_data(sample: dict | None, use_parameter_sharing: bool | None = False, use_actions_mask: bool | None = False, use_global_state: bool | None = False)

Prepare the training data.

Parameters:

• sample (dict) – The raw sampled data.

• use_parameter_sharing (bool) – Whether to use parameter sharing for individual agent models.

• use_actions_mask (bool) – Whether to use actions mask for unavailable actions.

• use_global_state (bool) – Whether to use global state.

Returns:

The formatted sampled data.

Return type:

sample_Tensor (dict)

update_rnn(sample)

iddpg_learner

Independent Deep Deterministic Policy Gradient (IDDPG) Implementation: MindSpore

class xuance.mindspore.learners.multi_agent_rl.iddpg_learner.IDDPG_Learner(config: Namespace, model_keys: List[str], agent_keys: List[str], policy: mindspore.nn.Cell, callback)

Bases: LearnerMAS

forward_fn_actor(obs, ids, mask_values, agent_key)

forward_fn_critic(obs, actions, ids, mask_values, q_target, agent_key)

update(sample)

ippo_learner

Multi-Agent Proximal Policy Optimization (MAPPO) Paper link: https://arxiv.org/pdf/2103.01955.pdf Implementation: MindSpore

class xuance.mindspore.learners.multi_agent_rl.ippo_learner.IPPO_Learner(config: Namespace, model_keys: List[str], agent_keys: List[str], policy: mindspore.nn.Cell, callback)

Bases: IAC_Learner

forward_fn(*args)

update(sample)

iql_learner

Independent Q-learning (IQL) Implementation: MindSpore

class xuance.mindspore.learners.multi_agent_rl.iql_learner.IQL_Learner(config: Namespace, model_keys: List[str], agent_keys: List[str], policy: mindspore.nn.Cell, callback)

Bases: LearnerMAS

forward_fn(obs, actions, agt_mask, avail_actions, ids, q_target, agent_key)

update(sample)

isac_learner

Independent Soft Actor-critic (ISAC) Implementation: MindSpore

class xuance.mindspore.learners.multi_agent_rl.isac_learner.ISAC_Learner(config: Namespace, model_keys: List[str], agent_keys: List[str], policy: mindspore.nn.Cell, callback)

Bases: LearnerMAS

forward_fn_actor(obs, ids, mask_values, agent_key)

forward_fn_alpha(log_pi_eval_i, key)

forward_fn_critic(obs, actions, ids, mask_values, backup, agent_key)

update(sample)

isacdis_learner

Independent Soft Actor-critic (ISAC) with discrete action spaces. Implementation: Pytorch

members:

undoc-members:

show-inheritance:

maddpg_learner

Multi-Agent Deep Deterministic Policy Gradient Paper link: https://proceedings.neurips.cc/paper/2017/file/68a9750337a418a86fe06c1991a1d64c-Paper.pdf Implementation: MindSpore Trick: Parameter sharing for all agents, with agents’ one-hot IDs as actor-critic’s inputs.

class xuance.mindspore.learners.multi_agent_rl.maddpg_learner.MADDPG_Learner(config: Namespace, model_keys: List[str], agent_keys: List[str], policy: mindspore.nn.Cell, callback)

Bases: LearnerMAS

forward_fn_actor(batch_size, obs, obs_joint, actions, ids, mask_values, agent_key)

forward_fn_critic(obs_joint, actions_joint, ids, mask_values, q_target, agent_key)

update(sample)

mappo_learner

Multi-Agent Proximal Policy Optimization (MAPPO) Paper link: https://arxiv.org/pdf/2103.01955.pdf Implementation: MindSpore

class xuance.mindspore.learners.multi_agent_rl.mappo_learner.MAPPO_Learner(config: Namespace, model_keys: List[str], agent_keys: List[str], policy: mindspore.nn.Cell, callback)

Bases: IPPO_Learner

forward_fn(*args)

update(sample)

masac_learner

Multi-agent Soft Actor-critic (MASAC) Implementation: MindSpore

class xuance.mindspore.learners.multi_agent_rl.masac_learner.MASAC_Learner(config: Namespace, model_keys: List[str], agent_keys: List[str], policy: mindspore.nn.Cell, callback)

Bases: LearnerMAS

forward_fn_actor(batch_size, obs, obs_joint, ids, mask_values, agent_key)

forward_fn_alpha(log_pi_eval_i, key)

forward_fn_critic(obs_joint, actions_joint, ids, mask_values, backup, agent_key)

update(sample)

masacdis_learner

Multi-agent Soft Actor-critic (MASAC) with discrete action spaces. Implementation: Pytorch

class xuance.mindspore.learners.multi_agent_rl.masacdis_learner.MASACDIS_Learner(config: Namespace, model_keys: List[str], agent_keys: List[str], policy: torch.nn.Module, callback)

Bases: ISAC_Learner

update(sample)

update_rnn(sample)

matd3_learner

Multi-Agent TD3

class xuance.mindspore.learners.multi_agent_rl.matd3_learner.MATD3_Learner(config: Namespace, model_keys: List[str], agent_keys: List[str], policy: mindspore.nn.Cell, callback)

Bases: LearnerMAS

forward_fn_actor(batch_size, obs, obs_joint, actions, ids, mask_values, agent_key)

forward_fn_critic(obs_joint, actions_joint, ids, mask_values, q_target, agent_key)

update(sample)

mfac_learner

MFAC: Mean Field Actor-Critic Paper link: http://proceedings.mlr.press/v80/yang18d/yang18d.pdf Implementation: Pytorch

class xuance.mindspore.learners.multi_agent_rl.mfac_learner.MFAC_Learner(config: Namespace, model_keys: List[str], agent_keys: List[str], policy: mindspore.nn.Cell, callback)

Bases: IPPO_Learner

build_actions_mean_input(sample: dict | None, use_parameter_sharing: bool | None = False)

forward_fn(*args)

update(sample)

mfq_learner

MFQ: Mean Field Q-Learning Paper link: http://proceedings.mlr.press/v80/yang18d/yang18d.pdf Implementation: MindSpore

class xuance.mindspore.learners.multi_agent_rl.mfq_learner.MFQ_Learner(config: Namespace, model_keys: List[str], agent_keys: List[str], policy: mindspore.nn.Cell, callback)

Bases: LearnerMAS

build_actions_mean_input(sample: dict | None, use_parameter_sharing: bool | None = False)

forward_fn(obs, actions, act_mean, agt_mask, avail_actions, ids, q_target, agent_key)

update(sample)

qmix_learner

Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning Paper link: http://proceedings.mlr.press/v80/rashid18a/rashid18a.pdf Implementation: MindSpore

class xuance.mindspore.learners.multi_agent_rl.qmix_learner.QMIX_Learner(config: Namespace, model_keys: List[str], agent_keys: List[str], policy: mindspore.nn.Cell, callback)

Bases: LearnerMAS

forward_fn(state, obs, actions, agt_mask, avail_actions, ids, q_tot_target)

update(sample)

qtran_learner

QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning Paper link: http://proceedings.mlr.press/v97/son19a/son19a.pdf Implementation: MindSpore

class xuance.mindspore.learners.multi_agent_rl.qtran_learner.QTRAN_Learner(config: Namespace, model_keys: List[str], agent_keys: List[str], policy: mindspore.nn.Cell, callback)

Bases: LearnerMAS

forward_fn(bs, batch_size, state, obs, actions, state_next, obs_next, agent_mask, avail_actions, avail_actions_next, rewards_tot, terminals_tot, IDs)

update(sample)

tarmac_learner

class xuance.mindspore.learners.multi_agent_rl.tarmac_learner.TarMAC_Learner(config: Namespace, model_keys: List[str], agent_keys: List[str], policy: torch.nn.Module, callback)

Bases: IC3Net_Learner

vdac_learner

Value Decomposition Actor-Critic (VDAC) Paper link: https://ojs.aaai.org/index.php/AAAI/article/view/17353 Implementation: MindSpore

class xuance.mindspore.learners.multi_agent_rl.vdac_learner.VDAC_Learner(config: Namespace, model_keys: List[str], agent_keys: List[str], policy: mindspore.nn.Cell, callback)

Bases: IAC_Learner

forward_fn(*args)

update(sample)

vdn_learner

Value Decomposition Networks (VDN) Paper link: https://arxiv.org/pdf/1706.05296.pdf Implementation: MindSpore

class xuance.mindspore.learners.multi_agent_rl.vdn_learner.VDN_Learner(config: Namespace, model_keys: List[str], agent_keys: List[str], policy: mindspore.nn.Cell, callback)

Bases: LearnerMAS

forward_fn(obs, actions, agt_mask, avail_actions, ids, q_tot_target)

update(sample)

wqmix_learner

Weighted QMIX Paper link: https://proceedings.neurips.cc/paper/2020/file/73a427badebe0e32caa2e1fc7530b7f3-Paper.pdf Implementation: MindSpore

class xuance.mindspore.learners.multi_agent_rl.wqmix_learner.WQMIX_Learner(config: Namespace, model_keys: List[str], agent_keys: List[str], policy: mindspore.nn.Cell, callback)

Bases: LearnerMAS

forward_fn(state, obs, actions, agt_mask, avail_actions, ids, target_value)

update(sample)