Models for TAMP problems are complex and challenging to develop. The high-dimensional sensory-motor space and the required integration of metric and symbolic state variables augment the challenges. Machine learning addresses these challenges at both the acting level and the planning level. But ML in robotics faces specific problems: lack of massive data; experiments needed for RL are scarce, very expensive, and difficult to reproduce; realistic sensory-motor simulators remain computationally costly; and expert human input for RL, e.g., for specifying or shaping reward functions or giving advices, is scarce and costly. The functions learned tend to be narrow: transfer of learned behaviors and models across environments and tasks is challenging. This chapter presents approaches for learning reactive sensory-motor skills using deep RL algorithms and methods for learning heuristics to guide a TAMP planner avoiding computation on unlikely feasible movements.

Acting with robots and sensory-motor devices demands the combined capabilities of reasoning both on abstract actions and on concrete motion and manipulation steps. In the robotics literature, this is referred to as "task-aware planning," i.e., planning beyond motion and manipulation. In the AI literature, it is referred to as "combined task and motion planning" (TAMP). This class of TAMP problems, which includes task, motion, and manipulation planning, is the topic of this part. The challenge in TAMP is the integration of symbolic models for task planning with metric models for motion and manipulation. This part introduces the representations and techniques for achieving and controlling motion, navigation, and manipulation actions in robotics. It discusses motion and manipulation planning algorithms, and their integration with task planning in TAMP problems. It covers learning for the combined task and motion-manipulation problems.

Acting, planning and learning are critical cognitive functions for an autonomous actor. Other functions, such as perceiving, monitoring, and goal reasoning, are also needed and can be essential in many applications. This chapter briefly surveys a few such functions and their links to acting, planning, and learning. Section 24.1 discusses perceiving and information gathering: how to model and control perception actions in order to recognize the state of the world and detect objects, events, and activities relevant to the actor while performing its tasks. It discusses semantic mapping and anchoring sensor data to symbols. Section 24.2 is about monitoring, that is, detecting and interpreting discrepancies between predictions and observations, anticipating what needs be monitored, and controlling monitoring actions. Goal reasoning in Section 24.3 is about assessing the relevance of current goals, from observed evolutions, failures, and opportunities to achieve a higher-level assigned mission.

AI acting systems, or actors – which may be embodied in physical devices such as robots or in abstract procedures such as web-based service agents – require several cognitive functions, three of which are acting, planning, and learning. Acting is more than just the sensory-motor execution of low-level commands: there is a need to decide how to perform a task, given the context and changes in the environment. Planning involves choosing and organizing actions that can achieve a goal and is done using abstract actions that the agent will need to decide how to perform. Learning is important for acquiring knowledge about expected effects, which actions to perform and how to perform them, and how to plan; and acting and planning can be used to aid learning. This chapter introduces the scientific and technical challenges of developing these three cognitive functions and the ethical challenge of doing such development responsibly.

In this chapter, we propose different approaches to planning with nondeterministic models. We describe three techniques for planning with nondeterministic state transition systems: And/Or graph search (Section 12.1), planning based on determinization techniques (Section 12.2), and planning via symbolic model checking (Section 12.3). We then present techniques for planning by synthesis of input/output automata (Section 12.4). We finally briefly discuss techniques for behavior tree generation (Section 12.5).

This chapter is about planning with hierarchical refinement methods. A plan guides the acting engine RAE with informed choices about the best methods for the task and context at hand. We consider an optimizing planner to find methods maximizing a utility function. In principle, the planner may rely on an exact dynamic programming optimization procedure. An approximation approach is more adapted to the online guidance of an actor. We describe a Monte Carlo tree search planner, called UPOM, parameterized for rollout depth and number of rollouts. It relies on a heuristic function for estimating the remainder of a rollout when the depth is bounded. UPOM is an anytime planner used in a receding horizon manner. This chapter relies on chapters 8, 9, and 14. It presents refinement planning domains and outlines the approach. Section 15.2 proposes utility functions and an optimization procedure. The planner is developed in Section 15.3.

This chapter is about representing state-transition systems and using them in acting. The first section gives formal definitions of state-transition systems and planning problems, and a simple acting algorithm. The second section describes state-variable representations of state-transition systems, and the third section describes several acting procedures that use this representation. The fourth section describes classical representation, an alternative to state-variable representation that is often used in the AI planning literature.

The chapters in Part I are about acting, planning, and learning using deterministic state-transition (or "classical planning") models. The relative ease of constructing and using such models can make them desirable even though most real-world environments do not satisfy all of their underlying assumptions. The chapters in this part also introduce several concepts that will be used throughout the book, such as state-variable representation.

This part of the book is about planning, acting, and learning approaches in which time is explicit. It describes several algorithms and methods for handling durative and concurrent activities with respect to a predicted dynamics. Acting with temporal models raises dispatching and temporal controllability issues that rely heavily on planning concepts.

Nondeterministic models, like probabilistic models (see Part III), drop the assumption that an action applied in a state leads to only one state. The main difference with probabilistic models is that nondeterministic models do not have information about the probability distribution of transitions. In spite of this, the main motivation for acting, planning, and learning using nondeterministic models is the same as that of probabilistic approaches, namely, the need to model uncertainty: most often, the future is never entirely predictable without uncertainty. Nondeterministic models might be thought to be a special case of probabilistic models with a uniform probability distribution. This is not the case. In nondeterministic models we do not know that the probability distribution is uniform; we simply do not have any information about the distribution.

HTN planning algorithms require a set of HTN methods that provide knowledge about potential problem-solving strategies. Typically these methods are written by a domain expert, but this chapter is about some ways to learn HTN methods from examples. It describes how to learn HTN methods in learning-by-demonstration situations in which a learner is given examples of plans for various tasks, and also in situations where the learner is given only the plans and must infer what tasks the plans accomplish. The chapter also speculates briefly about prospects for a “planning-to-learn” approach in which a learner generates its own examples using a classical planner.

Learning for nondeterministic models can take advantage of most of the techniques developed for probabilistic models (Chapter 10). Indeed, note that in reinforcement learning (RL), probabilities of action transitions are not needed, so RL techniques can be applied to nondeterministic models too. For instance, we can use the algorithms for Q-learning, parametric Q-learning, and deep Q-learning. However, these algorithms do not give explicit description models of actions. In this chapter, we therefore discuss some intuitions and also some challenges of how the techniques for learning deterministic action specifications could be extended to deal with nondeterministic models. Note, however, that learning lifted action schemas in nondeterministic models is still an open problem.

Temporal models are quite rich, allowing concurrency and temporal constraints to be handled. But the development of the temporal models is a bottleneck, to be eased with machine learning techniques. In this chapter, we first briefly address the problem of learning heuristics for temporal planning (Section 19.1). We then consider the issue of learning durative action schema and temporal methods (Section 19.2). The chapter outlines the proposed approaches, based on techniques seen earlier in the book, without getting into detailed descriptions of the corresponding procedures.

This chapter addresses the issues of acting with temporal models . It presents methods for handling dynamic controllability (Section 18.1), dispatching (Section 18.2), and execution and refinement of a temporal plan (Section 18.3). It proposes methods for acting with a reactive temporal refinement engine (Section 18.4), planning with Monte Carlo rollouts (Section 18.5), and integrating planning and acting (Section 18.6).