policy_gradient

a2c_learner

Advantage Actor-Critic (A2C) Implementation: TensorFlow2

class xuance.tensorflow.learners.policy_gradient.a2c_learner.A2C_Learner(config: Namespace, policy: tensorflow.keras.Model, callback)

Bases: Learner

forward_fn(obs_batch, act_batch, ret_batch, adv_batch)

learn(*inputs)

update(**samples)

ddpg_learner

Deep Deterministic Policy Gradient (DDPG) Paper link: https://arxiv.org/pdf/1509.02971.pdf Implementation: TensorFlow2

class xuance.tensorflow.learners.policy_gradient.ddpg_learner.DDPG_Learner(config: Namespace, policy: tensorflow.keras.Model, callback)

Bases: Learner

actor_forward_fn(obs_batch)

critic_forward_fn(obs_batch, act_batch, next_batch, rew_batch, ter_batch)

learn_actor(*inputs)

learn_critic(*inputs)

update(**samples)

mpdqn_learner

Multi-pass parameterised deep Q network (MP-DQN) Paper link: https://arxiv.org/pdf/1905.04388.pdf Implementation: TensorFlow2

class xuance.tensorflow.learners.policy_gradient.mpdqn_learner.MPDQN_Learner(policy: tensorflow.keras.Model, optimizers: Sequence[tensorflow.keras.optimizers.Optimizer], device: str = 'cpu:0', model_dir: str = './', gamma: float = 0.99, tau: float = 0.01)

Bases: Learner

update(obs_batch, act_batch, rew_batch, next_batch, terminal_batch)

npg_learner

class xuance.tensorflow.learners.policy_gradient.npg_learner.NPG_Learner(config: Namespace, policy: torch.nn.Module, callback)

Bases: Learner

compute_fisher_information(params, obs, act)

update(**samples)

pdqn_learner

Parameterised deep Q network (P-DQN) Paper link: https://arxiv.org/pdf/1810.06394.pdf Implementation: TensorFlow2

class xuance.tensorflow.learners.policy_gradient.pdqn_learner.PDQN_Learner(policy: tensorflow.keras.Model, optimizers: Sequence[tensorflow.keras.optimizers.Optimizer], device: str = 'cpu:0', model_dir: str = './', gamma: float = 0.99, tau: float = 0.01)

Bases: Learner

update(obs_batch, act_batch, rew_batch, next_batch, terminal_batch)

pg_learner

Policy Gradient (PG) Paper link: https://proceedings.neurips.cc/paper/2001/file/4b86abe48d358ecf194c56c69108433e-Paper.pdf Implementation: TensorFlow2

class xuance.tensorflow.learners.policy_gradient.pg_learner.PG_Learner(config: Namespace, policy: tensorflow.keras.Model, callback)

Bases: Learner

forward_fn(obs_batch, act_batch, ret_batch)

learn(*inputs)

update(**samples)

ppg_learner

Phasic Policy Gradient (PPG) Paper link: http://proceedings.mlr.press/v139/cobbe21a/cobbe21a.pdf Implementation: TensorFlow2

class xuance.tensorflow.learners.policy_gradient.ppg_learner.PPG_Learner(config: Namespace, policy: tensorflow.keras.Model, callback)

Bases: Learner

auxiliary_forward_fn(*args)

critic_forward_fn(obs_batch, ret_batch)

learn_auxiliary(*inputs)

learn_critic(*inputs)

learn_policy(*inputs)

policy_forward_fn(obs_batch, act_batch, adv_batch, old_log_prob_batch)

update()

update_auxiliary(**samples)

update_critic(**samples)

update_policy(**samples)

ppo_learner

Proximal Policy Optimization (PPO) with clip trick Paper link: https://arxiv.org/pdf/1707.06347.pdf Implementation: TensorFlow2

class xuance.tensorflow.learners.policy_gradient.ppo_learner.PPO_Learner(config: Namespace, policy: tensorflow.keras.Model, callback)

Bases: Learner

estimate_total_iterations()

Estimated total number of training iterations

forward_fn(obs_batch, act_batch, ret_batch, adv_batch, old_logp)

learn(*inputs)

update(**samples)

ppokl_learner

Proximal Policy Optimization with KL divergence (PPO-KL) Paper link: https://arxiv.org/pdf/1707.06347.pdf Implementation: TensorFlow2

class xuance.tensorflow.learners.policy_gradient.ppokl_learner.PPOKL_Learner(config: Namespace, policy: tensorflow.keras.Model, callback)

Bases: Learner

forward_fn(*args)

learn(*inputs)

update(**samples)

sac_learner

Soft Actor-Critic with continuous action spaces (SAC) Paper link: http://proceedings.mlr.press/v80/haarnoja18b/haarnoja18b.pdf Implementation: TensorFlow2

class xuance.tensorflow.learners.policy_gradient.sac_learner.AlphaLayer(*args: Any, **kwargs: Any)

Bases: Model

class xuance.tensorflow.learners.policy_gradient.sac_learner.SAC_Learner(config: Namespace, policy: tensorflow.keras.Model, callback)

Bases: Learner

actor_forward_fn(obs_batch)

alpha_forward_fn(log_pi)

critic_forward_fn(obs_batch, act_batch, rew_batch, next_batch, ter_batch)

learn_actor(*inputs)

learn_alpha(*inputs)

learn_critic(*inputs)

update(**samples)

sacdis_learner

Soft Actor-Critic with discrete action spaces (SAC-Discrete) Paper link: https://arxiv.org/pdf/1910.07207.pdf Implementation: TensorFlow2

class xuance.tensorflow.learners.policy_gradient.sacdis_learner.AlphaLayer(*args: Any, **kwargs: Any)

Bases: Model

class xuance.tensorflow.learners.policy_gradient.sacdis_learner.SACDIS_Learner(config: Namespace, policy: tensorflow.keras.Model, callback)

Bases: Learner

actor_forward_fn(obs_batch)

alpha_forward_fn(log_pi)

critic_forward_fn(obs_batch, act_batch, rew_batch, next_batch, ter_batch)

learn_actor(*inputs)

learn_alpha(*inputs)

learn_critic(*inputs)

update(**samples)

spdqn_learner

Split parameterised deep Q network (SP-DQN) Paper link: https://arxiv.org/pdf/1810.06394.pdf Implementation: TensorFlow2

class xuance.tensorflow.learners.policy_gradient.spdqn_learner.SPDQN_Learner(policy: tensorflow.keras.Model, optimizers: Sequence[tensorflow.keras.optimizers.Optimizer], device: str = 'cpu:0', model_dir: str = './', gamma: float = 0.99, tau: float = 0.01)

Bases: Learner

update(obs_batch, act_batch, rew_batch, next_batch, terminal_batch)

td3_learner

Twin Delayed Deep Deterministic Policy Gradient (TD3) Paper link: http://proceedings.mlr.press/v80/fujimoto18a/fujimoto18a.pdf Implementation: TensorFlow2

class xuance.tensorflow.learners.policy_gradient.td3_learner.TD3_Learner(config: Namespace, policy: tensorflow.keras.Model, callback)

Bases: Learner

actor_forward_fn(obs_batch)

critic_forward_fn(obs_batch, act_batch, rew_batch, next_batch, ter_batch)

learn_actor(*inputs)

learn_critic(*inputs)

update(**samples)