Distributed Deep Reinforcement Learning with
Wideband Sensing for Dynamic Spectrum Access
Umuralp Kaytaz∗ , Seyhan Ucar‡ , Baris Akgun† and Sinem Coleri∗
Department of Electrical and Electronics Engineering, Koc University Istanbul Turkey∗
Department of Computer Engineering, Koc University Istanbul Turkey†
InfoTech Labs, Toyota Motor North America R&D, Mountain View, CA ‡
{ukaytaz@ku.edu.tr, seyhan.ucar@toyota.com, baakgun@ku.edu.tr, scoleri@ku.edu.tr}
Abstract—Dynamic Spectrum Access (DSA) improves spectrum utilization by allowing secondary users (SUs) to opportunistically access temporary idle periods in the primary
user (PU) channels. Previous studies on utility maximizing
spectrum access strategies mostly require complete network
state information, therefore, may not be practical. Model-free
reinforcement learning (RL) based methods, such as Q-learning,
on the other hand, are promising adaptive solutions that do not
require complete network information. In this paper, we tackle
this research dilemma and propose deep Q-learning originated
spectrum access (DQLS) based decentralized and centralized
channel selection methods for network utility maximization,
namely DEcentralized Spectrum Allocation (DESA) and Centralized Spectrum Allocation (CSA), respectively. Actions that
are generated through centralized deep Q-network (DQN) are
utilized in CSA whereas the DESA adopts a non-cooperative
approach in spectrum decisions. We use extensive simulations
to investigate spectrum utilization of our proposed methods for
varying primary and secondary network sizes. Our findings
demonstrate that proposed methods significantly outperform
traditional methods, including slotted-Aloha and random assignment, while %88 of optimal channel access is achieved.
Index Terms—Cognitive radio, dynamic spectrum access,
deep reinforcement learning, medium access control (MAC).
I. I NTRODUCTION
Cognitive Radio (CR) is a promising technology for
emerging wireless communication systems due to its efficient
utilization of the frequency bands. CR enhances temporal and
spatial efficiency by exploiting temporary idle periods, a.k.a
spectrum holes in the frequency usage. Dynamic Spectrum
Access (DSA) plays a central role in CR networks by
allowing secondary users (SUs) to opportunistically access
spectrum holes in the PU channels. Network utility maximization during DSA requires efficient frequency selection
and minimal interference. Previous studies on DSA related
channel selection strategies mostly require complete network
state information, so, may not be practical [1]–[3]. Modelfree reinforcement learning (RL) based methods, such as Qlearning, on the other hand, are promising solutions that do
not require complete network information [4].
Deep Reinforcement Learning (DRL) adopts deep neural
network (DNN) architectures for approximating the objective
values during the learning [5]. High-dimensional spectrum
assignment problem, which includes large number of actions
and states space, can be solved via DRL without any prior
knowledge about the network state and/or environment.
Up to now, several works with RL and/or deep RL (DRL)
have been proposed with the aim of interference avoidance
and/or utility maximization in CR networks. DRL based
frequency allocation has been investigated for maximizing
the number of successful transmissions in [6]. A similar
approach has been proposed in [7] to maximize the network
utility during spectrum access via Deep Q-network (DQN).
Deep Q-learning (DQL) and narrowband sensing policy
has been examined for heterogeneous networks that operate
different MAC protocols [8]. However, narrowband sensing
adopted in these studies limits the usage of multiple spectral
opportunities. [9] examines multi-agent RL with wideband
sensing for learning sensing policies in a distributed manner.
Different from the proposed architecture, we focus on spectrum assignment and primary system frequency allocations.
Wideband sensing with DQN based access policy has been
investigated for utility maximization of a single SU [10].
However, existence of multiple cognitive agents in different
primary network settings and effect of centralized policy
learning is omitted in the presented work.
In this paper, we propose DQL and wideband sensing
based multiple SU utility maximization for the first time
in the literature. The original contribution of this study
is threefold. First, we develop a Markov Decision Process
(MDP) formulation for utility maximization during secondary
system decision making based on wideband sensing. Second,
we propose novel decentralized and centralized spectrum selection methods based on wideband sensing, namely DEcentralized Spectrum Allocation (DESA) and Centralized Spectrum Allocation (CSA), respectively. Third, we investigate
the spectrum utilization of the proposed spectrum allocation
methods for varying primary and secondary network sizes.
The remainder of the paper is organized as follows. Section II describes the system model. Section III presents the
formulation and RL based learning methods. DQN architecture and DQL-based spectrum allocation algorithms are
provided in Section IV. Section V details simulation setup
and analyzes experimental outcomes. Finally, we present our
concluding remarks in Section VI.
II. S YSTEM M ODEL
Fig. 1 represents IEEE 802.22 standard and Wireless Regional Area Network (WRAN) adopted network architecture.
PUs and SUs are assumed to be WRAN Base Station (BS)
IV. A PPROACH
q π (s, a) = Eπ
X
∞
γ k rt+k+1 st = s, at = a
k=0
(2)
There exists an optimal deterministic stationary policy for
each MDP model [11]. This policy maximizes the expected
reward returned from a state in the MDP model with unknown transition probabilities. Optimal state-value function
v∗ and optimal action-value function q∗ for all s ∈ S and
a ∈ A under policy π can be denoted as follows;
v∗ (s) = max v π (s)
(3)
q∗ (s, a) = max q π (s, a)
(4)
π
π
C. Q-Learning
Q-learning is a widely-used control algorithm for policy
independent approximation of optimal action-value function
(q∗) based on Bellman optimality equations. In the absence
of state transition probabilities P, optimal policy calculation
requires approximation of optimal value functions, q∗ and v∗,
for utility maximization. Bellman optimality equations allow
policy independent representations and recursive calculation
for optimal value functions [4]. These equations are;
Next, we describe our DQN architecture and algorithms for
DQL based spectrum allocation. First, we explain our DQN
architecture detailing hyperparameters selected for the learning procedure. Later on, we present wideband sensing based
DQL algorithm for single SU frequency selections. Finally,
we elaborate on our proposed centralized and decentralized
DRL architectures for multi-user utility maximization during
secondary system spectrum decisions.
A. Deep Q-Network
Q-learning calculation requires memorization and representation of Q-values during the expected reward calculation.
DQN is a deep reinforcement learning method that uses deep
neural network architecture for value storage and expected
utility approximation during Q-learning procedure. Using
DQN allows complex mapping of high-dimensional statespace representations to action-values [6], [7]. Our DQN
training procedure uses experience replay technique. With
this approach, we ensure convergent and stable training of
DQN model by sampling past experience at each batch and
training on decorrelated examples [12].
Table II: Hyperparameters for DQN
v∗ (s) = max E[ rt+1 + γv∗ (st+1 ) | st = s, at = a ]
a
(5)
Hyperparameter
Number of episodes
Number of time slots
Number of layers
Batch size B
Memory size
Learning rate α
Discount factor γ
Exploration rate ǫ
Epsilon decay
Loss function
Optimizer
q∗ (s, a) = E[ rt+1 + γ max q∗ (st+1 , at+1 ) | st = s, at = a ]
at+1
(6)
Learning algorithm uses multiple experiences and one-step
look-ahead approach for recursively approximating q∗ values,
in which, experience represents a sample taken from the MDP
model. An Experience at time slot t can be denoted using
et = hst , at , rt , st+1 i. Q-learning uses the following update
rule for solving the MDP formulation and approximating q∗
values;
q(s, a) ←
− q(s, a) + ∆q(s, a)
(7)
using learning rate α, ∆q(s, a) is defined as;
h
i
∆q(s, a) = α rt + γ max q(st+1 , at ) − q(st , at )
at
Table I: Reinforcement Learning Notation
Notation
γ
π
rt
st
at
v π (s)
q π (s, a)
v∗ (s)
q∗ (s, a)
et = hst , at , rt , st+1 i
Description
Discount factor
Policy
Reward observed at time t
State at time t
Action taken at time t
State-value function under policy π
Action-value function under policy π
Optimal state-value function
Optimal action-value function
Experience at time t
(8)
Value
200
10000
5
10
2000
0.001
0.95
1.0 −
→ 0.01
0.997
Huber
Adam
Feed-forward network consists of 5 fully connected layers,
where input is the state si . Input layer is of size 1 x N and
output layer is a vector of action set A. Hidden layers are
of size 30, 20, 10 and use ReLU activation function that
computes f (x) = max(x, 0). Last layer of the
P network
uses softmax activation function σ(xj ) = exj / i exi for
predicting Q-value of each action. We chose Adam gradient
descent optimization algorithm for updating weights during
DQN training [13]. Huber loss has been chosen as the loss
function during back propagation due to its robustness against
outliers [14]. At each iteration i, Adam algorithm is used to
update weights θi of the DQN for minimizing loss function
Li (θi ) =
(
1
2 (yi
2
− Qi )
c|yi − Qi | − 21 c2
f or |yi − Qi | ≤ c
otherwise
(9)
where yi = E[ ri+1 + γ maxa′ q∗ (si+1 , a′ )] is calculated
with the DQN using weights θi−1 from previous iteration,
Qi = q(si , a|θi ) and c = 1.0 as presented in Table II.
Algorithm 1: DQL originated Spectrum Access (DQLS)
Input: S ←
− primary system channel occupation
Output: DQN target
Data: M em : Memory, b : minibatch
train
1 Initialize DQN
, DQN target ←
− DQN (S, A)
2 for each episode Et do
3
for each state st ∈ Et do
4
Compute at = DQN train .get action(st )
5
at ←
− exploration with prob. ǫ
6
Compute rt = DQN train .get reward(st , at )
7
Get next state st+1
8
Store M em.store(et = hst , at , rt , st+1 i)
9
Update st ←
− st+1
10
if |M em| > B then
11
Compute b = M em.sample(B)
12
for each ei in b do
13
Compute
t = E[ ri+1 + γ maxai+1 q∗ (si+1 , ai+1 )]
14
Compute DQN train .train(si , t)
15
weights
Update DQN target ←−−−−− DQN train
B. DQL originated Spectrum Access Algorithm (DQLS)
Algorithm 1 is executed for learning spectrum decision
policies based on detected PU channel allocations. Learning is triggered upon receiving primary system channel
information from the environment using wideband sensing.
First, two DQN implementations (DQN train , DQN target )
are initialized for the experience replay procedure using
state space S and action space A (Line 1). Training lasts
for a predetermined amount of episodes (Lines 2-16). At
each episode randomly generated PU spectrum decisions are
determined using wideband sensing (Lines 3-15). Action at
corresponding to the highest expected utility is returned by
the DQN train given the current PU channel occupations
st (Line 4). Training procedure uses ǫ-greedy policy for
choosing random actions over actions that return highest
expected utility with probability ǫ (Line 5). Exploration
is necessary in order to find better action selections that
result in higher reward signals. We linearly decrease the
0.997
exploration rate ǫ (1.0 −−−→ 0.01) by epsilon decay for a balanced exploration-exploitation trade-off. Immediate reward
obtained by the selected action is recorded for the rest of the
training procedure (Line 6).
After getting the channel occupation in the subsequent time
slot current state is updated and an experience et is stored
in the memory M em using action at , states st , st+1 and
immediate reward rt (Lines 7-9). As the experience count
surpasses the predetermined batch size B, experiences are
randomly sampled from the memory to form a minibatch
and train on decorrelated experiences (Lines 10-11). Based
on the information contained in each experience, Q-value
Q(s, a) of that state-action pair is calculated and DQN model
DQN train is trained (Lines 12-14). After each episode
learned weights by the training model DQN train is used
for updating the target model DQN target (Line 15).
Algorithm 2: Centralized Spectrum Allocation (CSA)
Input: K ←
−number of SUs, N ←
− number of PUs
Output: Setaction : Secondary system actions
Data: si ←
− state during i-th slot
1 Compute |A| ←
− (N + 1)K
action
2 Initialize Set
= {}, assignedch = 0
target
3 Train DQN
, DQN train ←
− DQLS
target
4 Compute a = DQN
.get action(si )
ch
5 while assigned
6= K do
ch
6
Update |A| ←
− (N + 1)K−assigned −1
7
Compute P U index = ⌊(a/|A|)⌋
8
Update Setaction .append(P U index)
9
Update a = a mod |A|
10
Update assignedch += 1
C. Centralized Spectrum Allocation Algorithm (CSA)
Centralized DQL architecture uses Centralized Spectrum
Allocation Algorithm (CSA) for learning spectrum decision
policies. After obtaining primary system channel occupation
patterns using distributed sensing [9], Algorithm 2 is run for
determining secondary system spectrum decisions. Initially
the number of possible actions, action space size, is calculated considering each of K SUs has (N+1) actions for
N detected PU channels (Line 1). An empty set of actions
Setaction is created with a variable assignedch for counting
the number of assigned SU actions (Line 2). Centralized
DQN is trained and an action number for encoding all
individual actions of K SUs is computed using single DQN
based DQLS algorithm (Lines 3-4). Action code a, generated
by the central DQN for a given state si , is decoded into
separate action numbers until all SUs have an assigned action
(Lines 5-10). For each individual SU action this procedure
keeps the action of remaining SUs as a subset and divides
the action code to possible number of remaining actions
ch
(N + 1)K−assigned −1 (Lines 6-7). During the rest of the
procedure action code representing the cumulative actions of
remaining SUs and number assigned SU actions are updated
(Lines 9-10). After the calculation of the current channel
assigned to the SU, immediate reward signal is obtained from
the environment and total utility is updated (Lines 12-13).
D. Decentralized Spectrum Allocation Algorithm (DESA)
Decentralized policy learning architecture consists of noncooperative secondary BSs that perform independent wideband sensing and policy learning during DSA. Initially, each
secondary BS SU performs wideband sensing individually
and determines primary system channel allocations. Based
on spectrum occupancy information, SUs are trained using
the proposed DQLS algorithm (Lines 1-2). Upon completion
of the training procedure, secondary BSs choose the S
action and primary channel index, which provide the highest
expected utility given state sj at the current time slot j (Lines
3-4).
500
3
4
for each SU ∈ secondary system do
target
Compute aj = DQNSU
.get action(sj )
V. S IMULATIONS
A. Simulation Setup
Simulations of CSA and DESA algorithms have been
implemented for centralized and decentralized spectrum decision scenarios using open-source neural-network library
Keras [15] and numerical computation library TensorFlow
[16]. CSA uses a single DQN for generating secondary
system actions during DSA. Central DQN in the CSA is
aware of the primary system channels, which are detected by
the SUs using wideband sensing approach. During the DESA,
each SU has been modeled as a learning cognitive agent
that is capable of performing sensing and policy learning
operations independently. We have modeled PU channels as
independent 2-state Markov chains that can either be in 1
(occupied) or 0 (vacant) state. Similar to previous work in
[10], namely DQN-based access policy (DQNP), we have
implemented DSA for N = 20 channels each assigned to a
different PU with randomly generated transition probability
matrices {Pi }20
i=1 .
Network utility performance under CSA and DESA have
been compared with classical slotted-Aloha protocol and
optimal policy performance. Expected channel throughput
under slotted-Aloha protocol at a given time-slot i is given
by ni pi = (1 − pi )ni −1 , where pi represents transmission
probability and ni is number of SUs in the CR environment [7]. For the optimal policy performance comparison
we have implemented fixed-pattern channel switching based
optimal policy (OP) algorithm proposed in [6]. Each of the
simulations were carried out for T = 10, 000 time slots and
E = 200 episodes. We have used average SU utility per
episode as the performance evaluation metric for representing
average cumulative reward gained by SUs at each episode.
B. Simulation Results
We first present simulation results for spectrum decisions
of a single SU coexisting with multiple PUs in the CR
environment. Fig. 2 shows average SU utility per episode
obtained by a single SU spectrum decisions under DQLS.
Note that network scenario with 1 SU and 20 PUs corresponds to the architecture proposed for DQNP. We have
two observations from Fig. 2. First, average SU throughput
400
350
300
250
200
DQLS 1 SU 15 PU
DQLS 1 SU 20 PU (DQNP)
DQLS 1 SU 25 PU
OP 1 SU 25 PU
OP 1 SU 20 PU
OP 1SU 15 PU
150
100
50
0
0
50
100
150
200
Episodes
Figure 2: DQLS versus optimal channel access policy
increases as the number of detected PU channels decreases.
Low cardinality of the action space enables SU to learn a
better channel selection policy. Therefore, a higher value of
cumulative utility is obtained at the end of each episode as
the number of PU per SU decreases. Second, convergence
to DQN-based access policy takes longer with increasing
number of detected PU channels. Increasing complexity in
the CR network results in higher dimensionality of action
and state space representations, hence policy learning from
channel occupancy observations slows down.
Fig 3 shows average spectrum selection performance of
SUs under CSA. During simulations, the number of PU
channels is set to 20 while the number of base stations in
the secondary system vary as 2, 3 and 4. Similar to the
results presented in Fig 2, SUs perform better with DQN
based channel selection strategy as the number of available
PU channels per SU decreases. Different from slotted-Aloha
based medium access, average utility under CSA increases
with increasing number of SU. Furthermore, CSA-based
spectrum policy performance surpasses random access and
slotted-Aloha based spectrum decisions as the number of SU
increases to 4. On the other hand, 52% of optimal policy
performance is achieved.
Fig 4 depicts the performance of non-cooperative SUs
under DESA. Similar to the results depicted in Fig 2 and
Average SU Utility per Episode
Algorithm 3: Decentralized Spectrum Allocation
(DESA)
Input: S ←
− primary system channel occupation
Output: aj : PU channel at j-th slot
Data: sj ←
− state at j-th slot
1 for each SU ∈ secondary system do
target
train
, DQNSU
←
− DQLS
2
Train DQNSU
Average SU Utility per Episode
450
550
500
450
400
350
300
250
200
150
100
50
0
−50
−100
OP 20 PU 4 SU
OP 20 PU 2 SU
CSA 20 PU 4 SU
CSA 20 PU 2 SU
Slotted Aloha 20 PU 4 SU
Slotted Aloha 20 PU 2 SU
Random Access Policy
0
50
100
150
200
Episodes
Figure 3: Spectrum policy performance under CSA
Average SU Utility per Episode
R EFERENCES
550
500
450
400
350
300
250
200
150
100
50
0
−50
−100
OP 20 PU 4 SU
OP 20 PU 2 SU
DESA 20 PU 4 SU
DESA 20 PU 2 SU
Slotted Aloha 20 PU 4 SU
Slotted Aloha 20 PU 2 SU
Random Access Policy
0
50
100
150
200
Episodes
Figure 4: Spectrum policy performance under DESA
Fig 3, DESA-based channel access results show that increasing number of SUs increases the average performance
of DQN-based spectrum policy. It’s evident that DESAbased spectrum assignment outperforms other approaches.
Furthermore, this performance gap drastically increases as
the number of SU reaches to 4. Compared to results obtained
by centralized spectrum assignment under CSA, we observe
that independent policy learning in the decentralized scenario
results in higher average network utility values. Furthermore,
88% of optimal policy performance is achieved as 20 PUs
and 4 SUs are available in the CR environment.
VI. C ONCLUSION AND F UTURE W ORK
In this paper, we present multi-agent deep reinforcement
learning based spectrum selection with wideband sensing
capability for multi-user utility maximization during DSA.
Initially, DQN-based spectrum decisions of a single SU coexisting with multiple PUs have been derived. Later on, we propose novel algorithms for DQL and wideband sensing based
centralized and decentralized spectrum selection, namely
DESA and CSA. Through simulations, we demonstrate that
the performance of DQN-based frequency selection increases
as the number of available PU channels per SU decreases.
Moreover, we observe that independent policy learning in
non-cooperative manner under DESA results in more effective spectrum decisions than centralized spectrum assignment
under CSA. Overall, our proposed methods improve the spectrum utilization compared to traditional spectrum assignment
methods in which 88% and 52% of optimal channel access
are achieved by DESA and CSA, respectively.
Going forward, our future work will focus on extending
both the system model and proposed algorithms. For the
system model, we will concentrate on efficient utilization of
OFDMA resource blocks and analyze aggregated interference
at primary receivers during secondary system transmissions.
Considering the great potential of DRL-based technologies
for DSA, we also plan to work on creating an open source
dataset and implementing other DRL approaches such as
policy gradient methods.
[1] K. Wang and L. Chen, “On optimality of myopic policy for restless
multi-armed bandit problem: An axiomatic approach,” IEEE Transactions on Signal Processing, vol. 60, no. 1, pp. 300–309, Jan 2012.
[2] S. H. A. Ahmad, M. Liu et al., “Optimality of myopic sensing in
multichannel opportunistic access,” IEEE Transactions on Information
Theory, vol. 55, no. 9, pp. 4040–4050, Sep. 2009.
[3] Q. Zhao, L. Tong et al., “Decentralized cognitive mac for opportunistic
spectrum access in ad hoc networks: A pomdp framework,” Selected
Areas in Communications, IEEE Journal on, vol. 25, pp. 589 – 600,
05 2007.
[4] R. S. Sutton and A. G. Barto, Reinforcement Learning: An
Introduction, 2nd ed. The MIT Press, 2018. [Online]. Available:
http://incompleteideas.net/book/the-book-2nd.html
[5] V. Mnih, K. Kavukcuoglu et al., “Human-level control through deep
reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533,
Feb. 2015. [Online]. Available: http://dx.doi.org/10.1038/nature14236
[6] S. Wang, H. Liu et al., “Deep reinforcement learning for dynamic
multichannel access in wireless networks,” IEEE Transactions on
Cognitive Communications and Networking, vol. 4, no. 2, pp. 257–
265, June 2018.
[7] O. Naparstek and K. Cohen, “Deep multi-user reinforcement learning for distributed dynamic spectrum access,” IEEE Transactions on
Wireless Communications, vol. 18, no. 1, pp. 310–323, Jan 2019.
[8] Y. Yu, T. Wang, and S. C. Liew, “Deep-reinforcement learning multiple
access for heterogeneous wireless networks,” in IEEE International
Conference on Communications (ICC), May 2018, pp. 1–7.
[9] J. Lunden, S. R. Kulkarni et al., “Multiagent reinforcement learning
based spectrum sensing policies for cognitive radio networks,” IEEE
Journal of Selected Topics in Signal Processing, vol. 7, no. 5, pp.
858–868, Oct 2013.
[10] H. Q. Nguyen, B. T. Nguyen et al., “Deep q-learning with multiband
sensing for dynamic spectrum access,” in 2018 IEEE International
Symposium on Dynamic Spectrum Access Networks (DySPAN), Oct
2018, pp. 1–5.
[11] R. Bellman, “The theory of dynamic programming,” Bull. Amer.
Math. Soc., vol. 60, no. 6, pp. 503–515, 11 1954. [Online]. Available:
https://projecteuclid.org:443/euclid.bams/1183519147
[12] M. Andrychowicz, F. Wolski et al., “Hindsight experience
replay,” in Advances in Neural Information Processing Systems
30, I. Guyon, U. V. Luxburg et al., Eds.
Curran
Associates, Inc., 2017, pp. 5048–5058. [Online]. Available:
http://papers.nips.cc/paper/7090-hindsight-experience-replay.pdf
[13] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”
CoRR, vol. abs/1412.6980, 2015.
[14] P. J. Huber, “Robust estimation of a location parameter,” Ann. Math.
Statist., vol. 35, no. 1, pp. 73–101, 03 1964. [Online]. Available:
https://doi.org/10.1214/aoms/1177703732
[15] F. Chollet et al., “Keras,” https://github.com/fchollet/keras, 2015.
[16] M. Abadi, P. Barham et al., “Tensorflow: A system for large-scale
machine learning,” in 12th USENIX Symposium on Operating
Systems Design and Implementation (OSDI 16), 2016, pp. 265–283.
[Online]. Available: https://www.usenix.org/system/files/conference/
osdi16/osdi16-abadi.pdf