This paper presents a Q-learning approach to state-based planning of behaviour-based walking robots. The learning process consists of a teaching stage and an autonomous learning stage. During the teaching stage, the robot is instructed to operate in some interesting areas of the solution space to accumulate some prior knowledge. Then, the learning is switched to the autonomous learning stage to let the robot explore the solution space based on its prior knowledge. Experiments are conducted in the RoboCup domain and results show a good performance of the proposed method.