Int. j. inf. tecnol.
https://doi.org/10.1007/s41870-018-0130-3
ORIGINAL RESEARCH
Spectrum management in cognitive radio ad-hoc network using
Q-learning
Shiraz Khurana1 • Shuchita Upadhayaya1
Received: 25 November 2017 / Accepted: 28 March 2018
 Bharati Vidyapeeth’s Institute of Computer Applications and Management 2018
Abstract Cognitive radio ad-hoc networks (CRAHN’s) is
the new buzzword wherever spectrum congestion is a
matter of discussion. It was proposed to handle spectrum
insufficiency, by shifting the traffic to those portions of
spectrum which are not in use at that time, without
affecting the transmissions of legitimate users. In this paper
firstly we have discussed about CRAHN’s, secondly we
have discussed about reinforcement learning and then
challenges which are to be handled to implement rein-
forcement learning in CRAHN’s. Reinforcement learning
can be implemented in many areas in cognitive radio ad
hoc networks, like spectrum sensing, spectrum manage-
ment, spectrum mobility, and spectrum sharing. Primarily
in this paper our focus is on spectrum management. We
propose a reinforcement learning technique to handle
spectrum management. Our approach uses Q learning
algorithm (implemented in python) to prove that if rein-
forcement learning is used in this area, there will be drastic
improvement in performance of CRAHN’s and it enables
efficient assignment of spectrum by maximizing long-term
reward. We then execute our algorithm against various
network scenarios. In the end we have concluded our
experiment’s result, which shows that Q learning model
takes some episodes to learn, after that it gives us highly
optimized result, which saves secondary users to interrupt
primary users with an extremely good accuracy.
& Shuchita Upadhayaya
shuchita_bhasin@yahoo.com
Shiraz Khurana
shiraz.khurana@gmail.com
1
Department of Computer Science & Applications,
Kurukshetra University, Kurukshetra, Haryana, India
Keywords CRAHN  Reinforcement learning  Machine
learning  Mobile agent
1 Introduction
Cognitive radio ad hoc networks was emerged to solve
problems associated with wireless networking due to lim-
ited available spectrum. The problem of spectrum inade-
quacy can be removed by exploiting the usage of existing
wireless spectrum intelligently. Earlier the spectrum was
allocated using static spectrum allocation policies which
partitioned the radio spectrum into two parts: licensed and
unlicensed bands. Licensed bands are given to licensed
users which don’t transmit at all the times results into
spectral resource waste, examples of licensed bands are
mobile carriers, television broadcasting etc., Unlicensed
users operate in unlicensed bands which support variety of
applications [e.g., sensor networks, personal area networks
(PAN), mesh networks, wireless local area networks
(WLANs) etc.], increased number of users has caused
congestion in this category of band [1].
The concept of dynamic spectrum access (DSA) was
proposed to improve these spectrum insufficiency issues.
The software defined radio (SDR) allows the development
of such devices which can be adjusted to function in a wide
spectrum band, such devices are called cognitive radio
(CR) devices [2]. A cognitive radio (CR) is able to alter its
transmitting parameters like (operating spectrum, modula-
tion, transmission power, and communication technology)
by interacting with its operating environment. Thus CR
technology can be utilized to sort the issue of uneven
spectrum usage and inefficiency by identifying and trans-
mitting through the seldom used portions of the spectrum.
A network which is constituted with such kind of radio
123

devices is known as cognitive radio network (CRN). CRNs
without any fixed infrastructure support are called as dis-
tributed network, in which nodes communicate with each
other in an ad-hoc manner [3], and this is called cognitive
radio ad hoc networks (CRAHNs) as shown in Fig. 1.
CRAHNs aims at utilizing scarcely used spectrum bands to
increase overall spectrum efficiency.
In licensed band 1: PU are not using the spectrum so it’s
being used by cognitive user, in licensed band 2: the
spectrum (channel) is being used by primary users, so it is
not free. So every time before using the spectrum CU sense
the spectrum whether it is being used by PU or not.
Reinforcement learning (RL) is a type of machine
learning in which we allow machine to learn and change its
behaviour based on previous input/output and feedback
received from the environment. This behaviour can be
learnt once and for all, or keep on adapting as time goes.
The history of (RL) [4] is inspired by two different aspects:
one is psychology of animal learning and the other is
problem of optimal control and its solution using value
function and dynamic programming. The Q learning
algorithm that we are going to use in this paper is based on
reinforcement learning. The rest of the paper is divided into
following sections: in Sect. 1.1 we have discussed about an
agent’s role in reinforcement learning. In Sect. 1.2 we
review the literature and discuss the challenges in imple-
mentation of reinforcement learning in CRAHN’s. In
Sect. 2 we have defined the whole workflow of imple-
menting Q learning algorithm in CRAHN’s environment.
Section 3 represents the experimental results and we con-
clude our paper in Sect. 4.
1.1 Role of agent in reinforcement learning
An agent is the one that can view or perceive its environ-
ment through its sensor (i.e., collecting percept) and can act
upon that environment through its actuators. In case of
CRAHNs the performance, environment, sensors and
Fig. 1 Cognitive radio ad hoc network
123
Int. j. inf. tecnol.
actuators (PESA) for the agent is given below in Table 1.
This is just an abstracted view, there can be other param-
eter as well.
The next section suggests the challenges which are to be
handled while implementing reinforcement learning in
CRAHN’s.
1.2 Reinforcement leaning with CRAHN
(challenges)
CRAHN’s support two kinds of users: cognitive user (CU)
and primary user (PU). PU are users which have a legiti-
mate access to the spectrum and CU is also known as
secondary user (SU) can opportunistically access the
spectrum when it not being used by the PU [5]. But it also
makes sure that the licensed PUs immediately get the
spectrum for transmissions whenever it needs. So SU
should always look for vacant spectrum slot which is also
known as spectrum holes. So now when a primary user in
not using its allocated spectrum, the spectrum can be uti-
lized by secondary user. In case sudden appearance of
primary user, there can be two possible choices for sec-
ondary user to continue the operation. (1) The secondary
user jumps to another frequency of spectrum holes. (2) It
can operate in the same spectrum but it has to reduce its
transmission power so that there will be no disturbance to
primary user. CRAHNs are multi-hop wireless networks, in
which reinforcement learning techniques can be introduced
to boost the overall performance of such networks. There
are numerous challenges which should be kept in mind
before implementing any of the reinforcement learning
techniques in CRAHNs.
1.2.1 Challenges [6]
• Multifunctional: An RL agent can be used to observe
many different functionalities related to transmission
e.g., availability of radio resources, channel hetero-
geneity, cost of each transmission, power requirement,
delay (back off and switching) and mobility etc., The
main thing to focus here is that all these parameters are
to be stored in a state, receive an aggregate feedback
and optimize a general goal as a whole, e.g. throughput.
• Ambidextrous: RL techniques not only exploit the
fittest solution but also explore the solution space of the
behaviour to the dynamic spectrum. Since due to many
factors the path may get change, so exploration is also
to be done.
• Multi-agent: RL algorithm is not intended just to
implement a single agent functionality on a single node,
but it can also work in a distributed environment so that
total workload is reduced.
Int. j. inf. tecnol.
Table 1 PESA description for agent
Performance
Latency, throughput, response time, jitter, error rate, network utilization, arrival
rate, bandwidth, packet loss, reliability, security and power consumption
• Instability: In RL techniques after observing current
percept from the environment the cognitive radio (CR)
node need to perform action after learning from the
environment, so that feedbacks about the total cost of
each state-action pair are collected. Sometimes it may
choose the suboptimal solution to explore the state
space which is not exploited yet. It can be considered as
not getting stuck in local maxima. This activity may
lead to short-term performance degradation. So this can
make the network as instable.
• Convergence rate: In RL technique chance of getting
better solution highly depends on the number of percept
feed into the training data set. More is number of
training data sets more are the chances to get optimal
solution. The amount of training data might reach to
infinite as well. However, in wireless networks this is
not realistic. So the convergence rate is slow for
CRAHNs protocols. However, this is not a fundamental
property for CRAHN due to the non-stationary charac-
teristics of the environment.
• Impact of learning cost on power: The increased cost of
the learning process may impact the power constrained
devices. It also has more memory requirements.
2 Workflow for reinforcement learning
In RL technique an agent interacts with environment at
discrete point of time. At each step, RL agent observes the
environment and selects action from the set of possible
action and receives a reward or punishment from the
environment for that specific action taken. The agent
receives feedback about the appropriateness of its response
[7].
Fig. 2 Agent life cycle
Environment Sensor Actuators
Mobile nodes working in Switches, Switches,
cooperative manner servers, servers,
routers routers
Figure 2 shows agent’s interaction with the environ-
ment, if any agent takes some action, then the environment
changes its state from St to St?1. In the same way a reward
is given by environment. Figure 3 shows the relationship
between the agent’s action, state and reward. For instance,
agent is at state St and takes action At got a reward Rt?1
and moves to state St?1.
RL agent’s life is a Markov decision process. So let’s
have some assumption related to our agent. An agent
interacts with its environment in discrete time steps. The
objective of it is to collect maximum possible reward [8].
• S is the set of distinct states agent can perceive from
environment.
• A is the set of distinct actions which the agent can
perform.
• At each discrete time step t, the agent senses the state s,
chooses an action a, and executes it.
• The environment responds with a reward or punishment
r (s, a) and a new state s ? 1 = T (s, a). We denote
reward as a number, which will have greater value for
favourable, less for unfavourable. R: S 9 A ? R.
• The reinforcement and next-state functions are not
known to the agent. It depends only on the current state
and action.
• Reward specify what agent need to achieve, not how to
achieve it.
2.1 Q learning [9]
It is a technique of reinforcement learning. Q-learning is an
intelligent algorithm which can handle problems with
stochastic transitions and rewards. It works by making a
policy by which action-state pair will return optimal
results. A policy (p) is a rule that the agent follows in
selecting actions, given the state it is in. When the corre-
sponding action value function is identified, the optimal
policy can be constructed by simply selecting the action
Fig. 3 Relationship between agent state & environment
123

with the highest value in each state. It has been shown that
[4] for any finite MDP, Q-learning finally finds an optimal
policy, meaning that the expected value of total reward
returns in all subsequent steps from the current state is the
maximum achievable. It can be applied in any of following
fields in CRAHN: spectrum sensing, spectrum decision,
spectrum mobility, and spectrum sharing. Our prime focus
in this paper is on spectrum management. Experiment
results show that Q learning is able to sustain in a high
network load as compared to traditional techniques. During
the initial period of time when the Q learning agent is
learning from the example, it might show some strain in
performance but that is almost negligible. But when the
load is high it actually outperforms the traditional one. Q
learning not only manages into higher loads but also when
there is occurrence of primary user.
2.2 Q-learning algorithm [10, 11]
Step 1 set the value of discount factor ‘c’
Step 2 create reward matrix R
Step 3 initialize Q matrix by zero
Step 4 for each episode repeat the following steps:
Step 5 select a random start state, from that start state
select a random valid action state
Step 6 compute Q (state, action) = R (state, action) ? c
* Max [Q (next state, all actions)]
Step 7 set the next state as the current state
X
1
p¼ cRat ðSt ; Stþ1 Þ
t¼0
The algorithm consists of various terms which are defined
as follows:
Discount factor (‘c’): the value of discount factor (‘c’)
determines how much importance you want to give to
future rewards. Its value is always in between 0 and 1. Say,
you go to a state which is further away from the goal state,
but from that state the chances of encountering a state with
primary user is less, so, here the future reward is more,
even though the instant reward is less. A value close to zero
will not allow the agent to explore those choices in which
current reward is less but long term reward is more, while a
factor approaching to one will make it strive for a long-
term high reward.
Reward matrix (R): This matrix contains the reward
which agent will receive after moving to all possible set of
states. In reward matrix row and column denotes states and
action respectively. The way network is available we enter
the value in reward matrix.
Initial condition (Q matrix): it is an iterative algorithm;
it implicitly assumes an initial condition before the first
123
Int. j. inf. tecnol.
update. First time an action is taken the values from reward
matrix is used to set the value of Q matrix. This will make
an immediate learning in case of fixed deterministic
rewards. This process can be repeated many times based on
the number of episodes the user wants to proceed. In this
also the matrix row denotes states and column denotes
action.
So initially we start with Q matrix with all entries as 0,
in the very first step, all possible action will have equal
weightage, now agent can choose any action randomly.
After taking that action environment will return some
numerical reward, which is stored for future actions. So, it
means initially when Q matrix is empty or almost empty
then agent will do more exploration and once entries are
filled and learning is complete, it tries to exploit the
existing solution by selecting the maximum reward avail-
able. The ultimate goal of an agent is to maximize this
score. For each state we have to find an action that gen-
erates optimal values. The long term reward for each state
will be considered as optimal action.
3 Experiment
In our example we have taken a 20 node network which
can be represented using a 20 * 5 matrix, we can consider
these 20 nodes as states (like agent will be in which state at
a particular time) and 5 operations: up, down, left, right and
no change (in this case the agent will remain at the same
state). To simulate the whole experiment, we have used
Python language and numpy library. Initially we have
taken the Q matrix as all zero values. We have generated
the position where the primary user is appearing, where the
goal is and which state is the start state (all these positions
are generated randomly). For our experiment state space
Fig. 4 given below.
We have created a reward matrix in which: ? 1 value is
assigned for all movements (from one node to another with
an intention to find Goal) where neither there is a primary
Fig. 4 State space diagram
Int. j. inf. tecnol.
user nor goal state. - 10 value is assigned for all those
moves which results into states which have primary user,
? 10 value is assigned for all those moves which result
into goal state. - 1 value is assigned when an agent stays
on the same state instead of approaching towards the goal
and - 10 in case when the state contains primary user. The
objective of an agent is to maximize its score. We have
initialized the Q matrix by all 0’s initially.
Then we have created a transition matrix in which we
have recorded all set of actions both valid and invalid, e.g.,
we cannot move left or up from state 1 (our start state)
since this is an invalid move, in the same way right and
down are valid moves. In this fashion we store all actions
for each state in a matrix (invalid move are stored by - 1)
and valid moves are stored by resulting state number like,
if we are at state 1 and we perform a down action, we reach
are state 5 and so on for every state. Now it’s time to make
our agent to learn from the environment and then test it,
whether it reaches goal on time or not. We generate 5
random states, 3 states as primary user and 1 state as start
state and 1 state as goal state. For goal state we can use 2
different approaches:
1. Generate goal state randomly for each iteration.
2. Make goal state as fix.
We can use both approaches. In the first approach it will
take extra time as compared to the second one to learn, and
the result will be totally random in nature that is for every
set of iteration it will take different episode to learn and
different results. The second approach we fix the goal state,
now we repeat the experiment for 8 episodes (this number
can be chosen based upon when you see that there is not
much change in the value of Q matrix).
Every time we start the experiment, we can choose a
random start state to begin. Now for the training phase we
randomly generate the action at each state. Now for each
state we repeat the experiment several times. Based on the
reward received we assign a weight to it. For instance, if
you are at state 16 the action ‘down’ has more weight as
compare to ‘left’ or ‘up’, similarly in state 20 we have
more weight for ‘no action’ as compare to all other. This
way we can find the best possible channel selection from
start to goal node. This way agent will learn and remember
the moves which give a good reward for every state.
In case there is sudden appearance of primary user in
any state the reward of that state will start to vanish (again
agent will back to learning phase), and we can choose the
best possible next option. After executing for a specified
number of iteration, now our agent has learnt and as an
intermediate output we have got the Q matrix with weight
values for each state-action pair.
We have written code in python based on Markov
decision process model. Q learning algorithm calculates
PU hit Chart
100%
Success Percentage
50%
0%
NONONONO
NO NO yes
yes yes
1 2 3 4
5 6 7
8 >=9
Number of Episode & Network Trained (Yes or No)
PU hit Result
Fig. 5 Results
number of times we hit a state in which to primary user is
active. As a result, we get a Q matrix which has assigned
certain weight to each state action pair, now based on these
values we can predict future path from starting state to the
goal state. The maximum value for a specific action
according to state is preferably chosen, in case there is tie
between various values of q matrix corresponding to a
specific state then we can choose any of this action
randomly.
Now we can stop learning when we see our Q matrix is
getting stabled or not much change is there, we start at a
random start state, and our objective is to reach goal state
without hitting any primary user. We have repeated this
experiment 1000 times to calculate the performance of our
agent and validity of our Q-matrix. This is done after
learning for 7 episodes. The accuracy and graphs are shown
below in Fig. 5. The results are the following: our network
takes 7 episodes to get trained, before that we encountered
PU in the path to reach the goal infinite times, as our matrix
is not stable, it may have stuck in infinite loop as well. In
7th training episode on an average there will be 10 PU hits
(in 1000) iteration with accuracy of 99%. If we train our
network for 8 or more than 8 times, it never hits on the cell
which contains PU (for a 20 node network). Next time
whenever PU appears on another cell, the same process is
repeated that is training and execution.
4 Conclusion and future scope
In this paper firstly we have introduced the challenges
which we need to encounter before implementing rein-
forcement learning into cognitive radio ad hoc networks
(CRAHNs). Then we have suggested that using Q learning
approach, we can get optimized result. We have written
one program in python based on Q learning algorithm to
calculate number of hits to primary user. As a result, we get
a Q matrix which has assigned certain weight to each state-
123

action pair. Based on these values we can predict future
path from starting state to the goal state. The experiment
results show that after few episodes of learning the agent
never go to state which contains, primary user. The future
work will be focused on implementing similar kind of
learning in construction of routing table for CRAHN’s.
References
1. Akyildiz IF, Lee WY, Chowdhury KR (2009) CRAHNs: cogni-
tive radio ad hoc networks. Ad Hoc Netw 7(5):810–836. https://
doi.org/10.1016/j.adhoc.2009.01.001
2. Akyildiz IF, Lee WY, Vuran MC, Mohanty S (2006) NeXt
generation/dynamic spectrum access/cognitive radio wireless
networks: a survey. Comput Netw 50(13):2127–2159. https://doi.
org/10.1016/j.comnet.2006.05.001
3. Cesana M, Cuomo F, Ekici E (2011) Routing in cognitive radio
networks: challenges and solutions. Ad Hoc Netw 9(3):228–248.
https://doi.org/10.1016/j.adhoc.2010.06.009
123
Int. j. inf. tecnol.
4. Al-Rawi HAA, Yau KLA (2013) Routing in distributed cognitive
radio networks: a survey. Wireless Pers Commun
69(4):1983–2020. https://doi.org/10.1007/s11277-012-0674-7
5. Sutton RS, Barto AG (1998) Reinforcement learning: an intro-
duction. Bradford Books, Cambridge
6. Russell SJ et al (2010) Artificial intelligence: a modern approach.
Prentice-Hall Inc, Upper Saddle River, NJ, USA. ISBN 0-13-
103805-2
7. Akyildiz IF, Lee WY, Chowdhury KR (2009) Spectrum man-
agement in cognitive radio ad hoc networks. IEEE Netw
23(4):6–12. https://doi.org/10.1109/MNET.2009.5191140
8. Wu C, Chowdhury K, Di Felice M, Pourpeighambar B, Dehghan
M, Sabaei M (2010) Spectrum management of cognitive radio
using multi-agent reinforcement learning categories and subject
descriptors. Auton Agents Multiagent Syst 106:1705–1712.
https://doi.org/10.1016/j.comcom.2017.02.013
9. Felice MD, Chowdhury K, Wu C, Bononi L (2010) Learning-
based spectrum selection in cognitive radio ad hoc networks.
Auton Agents Multiagent Syst. https://doi.org/10.1007/978-3-
642-13315-2_11
10. Aneek Das.https://medium.com/towards-data-science/introduc
tion-to-q-learning-88d1c4f2b49c. Accessed 04 Oct 2017
11. http://mnemstudio.org/path-finding-q-learning.htm. Accessed 04
Oct 2017