qlearning_family

c51_agent

class xuance.torch.agents.qlearning_family.c51_agent.C51_Agent(config: Namespace, envs: DummyVecEnv | SubprocVecEnv | None = None, observation_space: gymnasium.spaces.Space | None = None, action_space: gymnasium.spaces.Space | None = None, callback: BaseCallback | None = None)

Bases: DQN_Agent

The implementation of C51DQN agent.

Parameters:

• config – the Namespace variable that provides hyperparameters and other settings.

• envs – the vectorized environments.

• callback – A user-defined callback function object to inject custom logic during training.

ddqn_agent

class xuance.torch.agents.qlearning_family.ddqn_agent.DDQN_Agent(config: Namespace, envs: DummyVecEnv | SubprocVecEnv | None = None, observation_space: gymnasium.spaces.Space | None = None, action_space: gymnasium.spaces.Space | None = None, callback: BaseCallback | None = None)

Bases: DQN_Agent

The implementation of Double DQN agent.

Parameters:

• config – the Namespace variable that provides hyperparameters and other settings.

• envs – the vectorized environments.

• callback – A user-defined callback function object to inject custom logic during training.

dqn_agent

class xuance.torch.agents.qlearning_family.dqn_agent.DQN_Agent(config: Namespace, envs: DummyVecEnv | SubprocVecEnv | None = None, observation_space: gymnasium.spaces.Space | None = None, action_space: gymnasium.spaces.Space | None = None, callback: BaseCallback | None = None)

Bases: OffPolicyAgent

The implementation of Deep Q-Networks (DQN) agent.

Parameters:

• config – the Namespace variable that provides hyperparameters and other settings.

• envs – the vectorized environments.

• callback – A user-defined callback function object to inject custom logic during training.

drqn_agent

class xuance.torch.agents.qlearning_family.drqn_agent.DRQN_Agent(config: Namespace, envs: DummyVecEnv | SubprocVecEnv | None = None, observation_space: gymnasium.spaces.Space | None = None, action_space: gymnasium.spaces.Space | None = None, callback: BaseCallback | None = None)

Bases: OffPolicyAgent

get_actions(obs, egreedy=0.0, rnn_hidden=None)

Returns actions and values.

Parameters:

• observations (np.ndarray) – The observation.

• test_mode (Optional[bool]) – True for testing without noises.

Returns:

The actions to be executed. values: The evaluated values. dists: The policy distributions. log_pi: Log of stochastic actions.

Return type:

actions

test(test_episodes: int, test_envs: DummyVecEnv | SubprocVecEnv | None = None, close_envs: bool = True) → list

Evaluate the current policy in a vectorized environment.

This method runs evaluation episodes using test_envs and returns the per-episode scores. During evaluation, actions are selected in deterministic (test) mode and optional RGB-array frames can be recorded for video logging when rendering is enabled.

Parameters:

• test_episodes (int) – Total number of evaluation episodes to run across all vectorized environments.

• test_envs (Optional[DummyVecEnv | SubprocVecEnv]) – Vectorized environments used for evaluation. Must not be None.

• close_envs (bool) – Whether to close test_envs before returning. Set this to False if test_envs is managed externally and will be reused after evaluation.

Returns:

A list of episode scores collected during evaluation.

Return type:

list

Notes

• This method resets the evaluation environments at the beginning of testing and steps them

  until test_episodes episodes are completed.

• When render_mode == “rgb_array” and self.render is True, the method records frames and logs the

  best-scoring episode as a video.

• By default, this implementation updates obs_rms during testing. If you want to avoid contaminating

  training statistics, consider guarding this update with a dedicated flag (e.g., update_rms=False).

train(train_steps: int) → dict

Run the main off-policy training loop.

This method interacts with the training environments for a fixed number of environment steps, collects transitions, stores them in the replay buffer, and periodically updates the policy using sampled mini-batches.

Training proceeds in a step-based manner (not episode-based): at each iteration, the agent selects actions, steps the environments, logs intermediate information via callbacks, and performs policy updates when the configured conditions are met.

Parameters:

train_steps (int) – Total number of environment steps to execute for training. The actual loop advances in increments of self.n_envs steps per iteration due to vectorized environments.

Returns:

A dictionary containing aggregated training information and

logged metrics collected during the training process.

Return type:

dict

Notes

• This method assumes that training environments (self.train_envs) and the replay buffer (self.memory) have already been initialized.

• Exploration behavior (e.g., epsilon-greedy or action noise) is applied during training and updated

  dynamically based on the current training step.

• Policy updates are triggered periodically according to

  self.training_frequency after self.start_training steps.

• Episode termination and reset logic are handled per environment

  instance, and episode-level statistics are logged via callbacks.

dueldqn_agent

class xuance.torch.agents.qlearning_family.dueldqn_agent.DuelDQN_Agent(config: Namespace, envs: DummyVecEnv | SubprocVecEnv | None = None, observation_space: gymnasium.spaces.Space | None = None, action_space: gymnasium.spaces.Space | None = None, callback: BaseCallback | None = None)

Bases: DQN_Agent

The implementation of DuelDQN agent.

Parameters:

• config – the Namespace variable that provides hyperparameters and other settings.

• envs – the vectorized environments.

• callback – A user-defined callback function object to inject custom logic during training.

noisydqn_agent

class xuance.torch.agents.qlearning_family.noisydqn_agent.NoisyDQN_Agent(config: Namespace, envs: DummyVecEnv | SubprocVecEnv | None = None, observation_space: gymnasium.spaces.Space | None = None, action_space: gymnasium.spaces.Space | None = None, callback: BaseCallback | None = None)

Bases: Agent

get_actions(obs)

test(test_episodes: int, test_envs: DummyVecEnv | SubprocVecEnv | None = None, close_envs: bool = True) → list

train(train_steps: int) → dict

train_epochs(n_epochs=1)

perdqn_agent

class xuance.torch.agents.qlearning_family.perdqn_agent.PerDQN_Agent(config: Namespace, envs: DummyVecEnv | SubprocVecEnv | None = None, observation_space: gymnasium.spaces.Space | None = None, action_space: gymnasium.spaces.Space | None = None, callback: BaseCallback | None = None)

Bases: DQN_Agent

The implementation of Per-DQN agent.

Parameters:

• config – the Namespace variable that provides hyperparameters and other settings.

• envs – the vectorized environments.

• callback – A user-defined callback function object to inject custom logic during training.

train(train_steps)

Run the main off-policy training loop.

This method interacts with the training environments for a fixed number of environment steps, collects transitions, stores them in the replay buffer, and periodically updates the policy using sampled mini-batches.

Training proceeds in a step-based manner (not episode-based): at each iteration, the agent selects actions, steps the environments, logs intermediate information via callbacks, and performs policy updates when the configured conditions are met.

Parameters:

train_steps (int) – Total number of environment steps to execute for training. The actual loop advances in increments of self.n_envs steps per iteration due to vectorized environments.

Returns:

A dictionary containing aggregated training information and

logged metrics collected during the training process.

Return type:

dict

Notes

• This method assumes that training environments (self.train_envs) and the replay buffer (self.memory) have already been initialized.

• Exploration behavior (e.g., epsilon-greedy or action noise) is applied during training and updated

  dynamically based on the current training step.

• Policy updates are triggered periodically according to

  self.training_frequency after self.start_training steps.

• Episode termination and reset logic are handled per environment

  instance, and episode-level statistics are logged via callbacks.

train_epochs(n_epochs=1)

qrdqn_agent

class xuance.torch.agents.qlearning_family.qrdqn_agent.QRDQN_Agent(config: Namespace, envs: DummyVecEnv | SubprocVecEnv | None = None, observation_space: gymnasium.spaces.Space | None = None, action_space: gymnasium.spaces.Space | None = None, callback: BaseCallback | None = None)

Bases: DQN_Agent

The implementation of QRDQN agent.

Parameters:

• config – the Namespace variable that provides hyperparameters and other settings.

• envs – the vectorized environments.

• callback – A user-defined callback function object to inject custom logic during training.