Application of Reinforcement Learning in Cognitive Radio Networks: Models and Algorithms

Algorithm 3

RL algorithm for joint dynamic channel selection and channel sensing [34].

Repeat

(a) Choose action 𝑎^𝑖_𝑡∈𝐴

        𝑎^𝑖_𝑡=argmax_𝑎𝑄^𝑖_𝑏,𝑡𝐬_𝐭,𝑎=argmax_𝑎∑_𝐬∈𝐒𝐛(𝐬)𝑄^𝑖_𝑡(𝐬,𝑎)

(b) Receive delayed reward 𝑟^𝑖_𝑡+1𝐬^𝐢_𝐭+𝟏,𝑎^𝑖_𝑡

(c) Update belief 𝐛(𝐬^𝐢_𝐭)

(d) Update 𝑄-value:

   𝑄^𝑖_𝑡+1𝐬^𝐢_𝐭,𝑎^𝑖_𝑡=𝑄^𝑖_𝑡𝐬^𝐢_𝐭,𝑎^𝑖_𝑡+𝛼⋅𝐛(𝐬^𝐢_𝐭)⋅𝑟^𝑖_𝑡+1𝐬^𝐢_𝐭+𝟏,𝑎^𝑖_𝑡+𝛾max_𝑎∈𝐴𝑄^𝑖_𝑏,𝑡+1𝐬^𝐢_𝐭+𝟏,𝑎−𝑄^𝑖_𝑡𝐬^𝐢_𝐭,𝑎^𝑖_𝑡