policy_gradient

a2c_agent

class xuance.tensorflow.agents.policy_gradient.a2c_agent.A2C_Agent(config: Namespace, envs: DummyVecEnv | SubprocVecEnv | None = None, observation_space: gymnasium.spaces.Space | None = None, action_space: gymnasium.spaces.Space | None = None, callback: BaseCallback | None = None)[source]

Bases: OnPolicyAgent

The implementation of A2C agent.

Parameters:

  • config – the Namespace variable that provides hyperparameters and other settings.

  • envs – the vectorized environments.

  • callback – A user-defined callback function object to inject custom logic during training.

ddpg_agent

class xuance.tensorflow.agents.policy_gradient.ddpg_agent.DDPG_Agent(config: Namespace, envs: DummyVecEnv | SubprocVecEnv | None = None, observation_space: gymnasium.spaces.Space | None = None, action_space: gymnasium.spaces.Space | None = None, callback: BaseCallback | None = None)[source]

Bases: OffPolicyAgent

The implementation of DDPG agent.

Parameters:

  • config – the Namespace variable that provides hyperparameters and other settings.

  • envs – the vectorized environments.

  • callback – A user-defined callback function object to inject custom logic during training.

get_actions(observations: numpy.ndarray, test_mode: bool | None = False)[source]

Returns actions and values.

Parameters:

  • observations (np.ndarray) – The observation.

  • test_mode (Optional[bool]) – True for testing without noises.

Returns:

The actions to be executed.

Return type:

actions

mpdqn_agent

class xuance.tensorflow.agents.policy_gradient.mpdqn_agent.MPDQN_Agent(config: Namespace, envs: Gym_Env, callback: BaseCallback | None = None)[source]

Bases: PDQN_Agent

The implementation of MPDQN agent.

Parameters:

  • config – the Namespace variable that provides hyperparameters and other settings.

  • envs – the vectorized environments.

  • callback – A user-defined callback function object to inject custom logic during training.

npg_agent

class xuance.tensorflow.agents.policy_gradient.npg_agent.NPG_Agent(config: Namespace, envs: DummyVecEnv | SubprocVecEnv | None = None, observation_space: gymnasium.spaces.Space | None = None, action_space: gymnasium.spaces.Space | None = None, callback: BaseCallback | None = None)[source]

Bases: OnPolicyAgent

The implementation of NPG agent.

Parameters:

  • config – the Namespace variable that provides hyperparameters and other settings.

  • envs – the vectorized environments.

  • callback – A user-defined callback function object to inject custom logic during training.

pdqn_agent

class xuance.tensorflow.agents.policy_gradient.pdqn_agent.PDQN_Agent(config: Namespace, envs: Gym_Env, callback: BaseCallback | None = None)[source]

Bases: Agent

The implementation of PDQN agent.

Parameters:

  • config – the Namespace variable that provides hyperparameters and other settings.

  • envs – the environments.

  • callback – A user-defined callback function object to inject custom logic during training.

end_episode(episode)[source]
get_actions(obs)[source]
pad_action(disaction, conaction)[source]
test(env_fn, test_episodes)[source]
train(train_steps=10000)[source]
train_epochs(n_epochs=1)[source]

pg_agent

class xuance.tensorflow.agents.policy_gradient.pg_agent.PG_Agent(config: Namespace, envs: DummyVecEnv | SubprocVecEnv | None = None, observation_space: gymnasium.spaces.Space | None = None, action_space: gymnasium.spaces.Space | None = None, callback: BaseCallback | None = None)[source]

Bases: OnPolicyAgent

The implementation of PG agent.

Parameters:

  • config – the Namespace variable that provides hyperparameters and other settings.

  • envs – the vectorized environments.

  • callback – A user-defined callback function object to inject custom logic during training.

get_terminated_values(observations_next: numpy.ndarray, rewards: numpy.ndarray = None)[source]

Returns values for terminated states.

Parameters:

  • observations_next (np.ndarray) – The terminal observations.

  • rewards (np.ndarray) – The rewards for terminated states.

Returns:

The values for terminal states.

Return type:

values_next

ppg_agent

class xuance.tensorflow.agents.policy_gradient.ppg_agent.PPG_Agent(config: Namespace, envs: DummyVecEnv | SubprocVecEnv | None = None, observation_space: gymnasium.spaces.Space | None = None, action_space: gymnasium.spaces.Space | None = None, callback: BaseCallback | None = None)[source]

Bases: OnPolicyAgent

The implementation of PPG agent.

Parameters:

  • config – the Namespace variable that provides hyperparameters and other settings.

  • envs – the vectorized environments.

  • callback – A user-defined callback function object to inject custom logic during training.

batch_actions(observations: numpy.ndarray)[source]

Return batch of actions with shape = [num_envs, batch_size, dim_obs]

get_actions(observations: numpy.ndarray, return_dists: bool = False, return_logpi: bool = False)[source]

Returns actions and values.

Parameters:

  • observations (np.ndarray) – The observation.

  • return_dists (bool) – Whether to return dists.

  • return_logpi (bool) – Whether to return log_pi.

Returns:

The actions to be executed. values: The evaluated values. dists: The policy distributions. log_pi: Log of stochastic actions.

Return type:

actions

get_aux_info(policy_output: dict = None)[source]

Returns auxiliary information.

Parameters:

policy_output (dict) – The output information of the policy.

Returns:

The auxiliary information.

Return type:

aux_info (dict)

train(train_steps)[source]

Run the main on-policy training loop.

This method interacts with the training environments to collect rollouts from the current policy, stores transitions in the on-policy trajectory buffer, and triggers policy/value updates when the buffer is full. The loop advances in vectorized steps (one iteration corresponds to self.n_envs environment steps).

Parameters:

train_steps (int) – Number of rollout collection iterations to run. Each iteration steps all vectorized environments once, so the total number of environment steps is approximately train_steps * self.n_envs.

Returns:

A dictionary containing aggregated training information and logged metrics collected during training.

Return type:

dict

Notes

  • This method assumes that training environments (self.train_envs)

    and the trajectory buffer (self.memory) have already been initialized.

  • After collecting horizon_size steps per environment, the buffer becomes full and the agent computes

    bootstrapped terminal values, finalizes trajectory segments via finish_path, and performs n_epochs optimization passes over mini-batches using train_epochs.

  • Episode termination and reset logic are handled per environment,

    and episode-level statistics are reported via callbacks.

ppo_agent

class xuance.tensorflow.agents.policy_gradient.ppo_agent.PPO_Agent(config: Namespace, envs: DummyVecEnv | SubprocVecEnv | None = None, observation_space: gymnasium.spaces.Space | None = None, action_space: gymnasium.spaces.Space | None = None, callback: BaseCallback | None = None)[source]

Bases: OnPolicyAgent

The implementation of PPO agent.

Parameters:

  • config – the Namespace variable that provides hyperparameters and other settings.

  • envs – the vectorized environments.

  • callback – A user-defined callback function object to inject custom logic during training.

get_aux_info(policy_output: dict = None)[source]

Returns auxiliary information.

Parameters:

policy_output (dict) – The output information of the policy.

Returns:

The auxiliary information.

Return type:

aux_info (dict)

train(train_steps)[source]

Run the main on-policy training loop.

This method interacts with the training environments to collect rollouts from the current policy, stores transitions in the on-policy trajectory buffer, and triggers policy/value updates when the buffer is full. The loop advances in vectorized steps (one iteration corresponds to self.n_envs environment steps).

Parameters:

train_steps (int) – Number of rollout collection iterations to run. Each iteration steps all vectorized environments once, so the total number of environment steps is approximately train_steps * self.n_envs.

Returns:

A dictionary containing aggregated training information and logged metrics collected during training.

Return type:

dict

Notes

  • This method assumes that training environments (self.train_envs)

    and the trajectory buffer (self.memory) have already been initialized.

  • After collecting horizon_size steps per environment, the buffer becomes full and the agent computes

    bootstrapped terminal values, finalizes trajectory segments via finish_path, and performs n_epochs optimization passes over mini-batches using train_epochs.

  • Episode termination and reset logic are handled per environment,

    and episode-level statistics are reported via callbacks.

ppokl_agent

class xuance.tensorflow.agents.policy_gradient.ppokl_agent.PPOKL_Agent(config: Namespace, envs: DummyVecEnv | SubprocVecEnv | None = None, observation_space: gymnasium.spaces.Space | None = None, action_space: gymnasium.spaces.Space | None = None, callback: BaseCallback | None = None)[source]

Bases: OnPolicyAgent

The implementation of PPO agent with KL divergence.

Parameters:

  • config – the Namespace variable that provides hyperparameters and other settings.

  • envs – the vectorized environments.

  • callback – A user-defined callback function object to inject custom logic during training.

get_aux_info(policy_output: dict = None)[source]

Returns auxiliary information.

Parameters:

policy_output (dict) – The output information of the policy.

Returns:

The auxiliary information.

Return type:

aux_info (dict)

train(train_steps)[source]

Run the main on-policy training loop.

This method interacts with the training environments to collect rollouts from the current policy, stores transitions in the on-policy trajectory buffer, and triggers policy/value updates when the buffer is full. The loop advances in vectorized steps (one iteration corresponds to self.n_envs environment steps).

Parameters:

train_steps (int) – Number of rollout collection iterations to run. Each iteration steps all vectorized environments once, so the total number of environment steps is approximately train_steps * self.n_envs.

Returns:

A dictionary containing aggregated training information and logged metrics collected during training.

Return type:

dict

Notes

  • This method assumes that training environments (self.train_envs)

    and the trajectory buffer (self.memory) have already been initialized.

  • After collecting horizon_size steps per environment, the buffer becomes full and the agent computes

    bootstrapped terminal values, finalizes trajectory segments via finish_path, and performs n_epochs optimization passes over mini-batches using train_epochs.

  • Episode termination and reset logic are handled per environment,

    and episode-level statistics are reported via callbacks.

sac_agent

class xuance.tensorflow.agents.policy_gradient.sac_agent.SAC_Agent(config: Namespace, envs: DummyVecEnv | SubprocVecEnv | None = None, observation_space: gymnasium.spaces.Space | None = None, action_space: gymnasium.spaces.Space | None = None, callback: BaseCallback | None = None)[source]

Bases: OffPolicyAgent

The implementation of SAC agent.

Parameters:

  • config – the Namespace variable that provides hyperparameters and other settings.

  • envs – the vectorized environments.

  • callback – A user-defined callback function object to inject custom logic during training.

get_actions(observations: numpy.ndarray, test_mode: bool | None = False)[source]

Returns actions and values.

Parameters:

  • observations (np.ndarray) – The observation.

  • test_mode (Optional[bool]) – True for testing without noises.

Returns:

The actions to be executed. values: The evaluated values. dists: The policy distributions. log_pi: Log of stochastic actions.

Return type:

actions

spdqn_agent

class xuance.tensorflow.agents.policy_gradient.spdqn_agent.SPDQN_Agent(config: Namespace, envs: Gym_Env, callback: BaseCallback | None = None)[source]

Bases: PDQN_Agent, Agent

The implementation of SPDQN agent.

Parameters:

  • config – the Namespace variable that provides hyperparameters and other settings.

  • envs – the vectorized environments.

  • callback – A user-defined callback function object to inject custom logic during training.

train(train_steps=10000)[source]

td3_agent

class xuance.tensorflow.agents.policy_gradient.td3_agent.TD3_Agent(config: Namespace, envs: DummyVecEnv | SubprocVecEnv | None = None, observation_space: gymnasium.spaces.Space | None = None, action_space: gymnasium.spaces.Space | None = None, callback: BaseCallback | None = None)[source]

Bases: DDPG_Agent

The implementation of TD3 agent.

Parameters:

  • config – the Namespace variable that provides hyperparameters and other settings.

  • envs – the vectorized environments.

  • callback – A user-defined callback function object to inject custom logic during training.