Split Parameterized Deep Q-Network (SP-DQN)¶

Split Parameterized Deep Q-Network (SP-DQN) is an extension of the traditional Deep Q-Network (DQN) designed to improve the efficiency and scalability of the Q-learning algorithm in large-scale problems. SP-DQN splits the Q-network into multiple parameterized parts, each corresponding to a different action space. By decoupling these parts, SP-DQN can effectively reduce the training complexity and memory requirements while maintaining high performance.

This table lists some general features about SP-DQN algorithm:

Features of SP-DQN	Values	Description
On-policy	❌	The evaluate policy is the same as the target policy.
Off-policy	✅	The evaluate policy is different from the target policy.
Model-free	✅	No need to prepare an environment dynamics model.
Model-based	❌	Need an environment model to train the policy.
Discrete Action	✅	Deal with discrete action space.
Continuous Action	❌	Deal with continuous action space.

Q-Learning Recap¶

Q-Learning is a model-free RL algorithm where the agent learns a Q-value function \(Q(s, a)\), which estimates the expected cumulative reward of taking action \(a\) in state \(s\) and following the optimal policy thereafter. The Bellman equation for Q-learning is given by:

\[ Q(s, a) \leftarrow Q(s, a) + \alpha [ r + \gamma \max_{a'}Q(s', a') - Q(s, a) ]. \]

Where \(\alpha\) is the learning rate, \(r\) is the reward, \(\gamma\) is the discount factor, and \(s'\) is the next state.

Split Parameterized Q-Learning¶

The key idea behind SP-DQN is to split the Q-network into multiple parameterized parts. Each part is responsible for estimating the Q-values corresponding to a specific subset of actions. This structure reduces the computational complexity of training, as the different parts of the network can be trained independently or in parallel.

SP-DQN applies parameter splitting to the Q-function, allowing for more efficient representation and computation in environments with large action spaces.

Deep Q-Network with Split Parameters¶

SP-DQN improves on DQN by using a split network architecture. The neural network is divided into multiple parts, each corresponding to a specific action category. The general steps involved are as follows:

Action Space Split: Split the action space into disjoint subsets, and assign each subset to a different parameterized network.
Independent Network Training: Train each parameterized network independently, which reduces training complexity and memory usage.
Consolidated Q-Function: The Q-function is computed by combining the outputs of each independent network, producing the final Q-value for each action.

The loss function for training SP-DQN is similar to DQN:

\[ L = \mathbb{E}_{(s, a, s', r) \sim \mathcal{D}}[(y - Q(s, a; \theta))^2], \]

where \(y = r + \gamma \max_{a'}{Q(s', a'; \theta^{-})}\), and \(\theta^{-}\) represents the parameters of the target network.

\(\epsilon\)-Greedy Exploration¶

SP-DQN uses the same \(\epsilon\)-greedy exploration policy as DQN:

\[\begin{split} \pi(s) = \begin{cases} \arg\max_{a}Q(s, a) & \text{with probability } 1-\epsilon, \\ \text{a random action} & \text{with probability } \epsilon. \end{cases} \end{split}\]

This policy ensures that the agent explores the environment randomly with probability \(\epsilon\) and exploits the learned policy otherwise.

Algorithm¶

The full algorithm for training SP-DQN is presented in Algorithm 1:

Framework¶

The overall agent-environment interaction of SP-DQN, as implemented in XuanCe, is illustrated in the figure below:

Run SP-DQN in XuanCe¶

Before running SP-DQN in XuanCe, you need to prepare a conda environment and install xuance following the installation steps.

Run Build-in Demos¶

After completing the installation, you can open a Python console and run SP-DQN directly using the following commands:

import xuance
runner = xuance.get_runner(algo='spdqn',
                           env='classic_control',  # Choices: classic_control, box2d, atari.
                           env_id='CartPole-v1',  # Choices: CartPole-v1, LunarLander-v3, ALE/Breakout-v5, etc.
                           )
runner.run()  # Or runner.benchmark()

Run With Self-defined Configs¶

If you want to run SP-DQN with different configurations, you can build a new .yaml file, e.g., my_config.yaml. Then, run SP-DQN by the following code block:

import xuance as xp
runner = xp.get_runner(algo='spdqn',
                       env='classic_control',  # Choices: classic_control, box2d, atari.
                       env_id='CartPole-v1',  # Choices: CartPole-v1, LunarLander-v3, ALE/Breakout-v5, etc.
                       config_path="my_config.yaml",  # The path of my_config.yaml file should be correct.
                       )
runner.run()  # Or runner.benchmark()

Run With Custom Environment¶

If you would like to run XuanCe’s SP-DQN in your own environment that was not included in XuanCe, you need to define the new environment following the steps in New Environment Tutorial. Then, prepare the configuration file spdqn_myenv.yaml.

After that, you can run SP-DQN in your own environment with the following code:

import argparse
from xuance.common import load_yaml
from xuance.environment import REGISTRY_ENV
from xuance.environment import make_envs
from xuance.torch.agents import SP_DQN_Agent

configs_dict = load_yaml(file_dir="spdqn_myenv.yaml")
configs = argparse.Namespace(**configs_dict)
REGISTRY_ENV[configs.env_name] = MyNewEnv

envs = make_envs(configs)  # Make parallel environments.
Agent = SP_DQN_Agent(config=configs, envs=envs)  # Create a SP-DQN agent from XuanCe.
Agent.train(configs.running_steps // configs.parallels)  # Train the model for numerous steps.
Agent.save_model("final_train_model.pth")  # Save the model to model_dir.
Agent.finish()  # Finish the training.

Citation¶

@article{he2017split,
  title={Split Parameterized Deep Q-Network},
  author={He, Xun and Zhang, Xiang and Liu, Jian and Lu, Yun},
  journal={arXiv preprint arXiv:1707.02785},
  year={2017}
}