Humanoid Motion Learning Framework
This framework uses Adversarial Motion Priors to encourage the policy to imitate reference motions while simultaneously learning task-specific behaviors.
It provides flexible control over policy observations through conditioning on commands, motion triggers, and motion phase, and over credit assignment and the learning proccess through motion-specific reward functions, termination criteria, and configurable AMP-style rewards.
Point-Goal Locomotion + Standing
Overview
A motion manifest defines which motion clips are available to be sampled from and their respective command observations, triggers, rewards, and termination criteria.
motions:
walk_to_goal:
file: data/walk.npz
sample_prob: 1.0
amp: true
commands:
goal_x: [-5.0, 5.0]
goal_y: [-5.0, 5.0]
zero_command_prob: 0.25
rewards:
progress:
weight: 1.0
scale: 30.0
target_hit:
weight: 0.5
scale: 0.5
termination: unhealthy
The environment manages everything and exposes the necessary information so the agent can apply the AMP reward when appropriate.
Experiments
Triggered Dance Skills with Phase Conditioning
A single policy learns multiple dance behaviors from reference motions. Discrete behavior triggers select the desired skill, while motion phase provides progress through its execution.
Compositional Skill Learning with Goal-Conditioned Locomotion
A single policy learns multiple dance behaviors from reference motions. Discrete behavior triggers select the desired skill, while motion phase provides progress through its execution.
Compositional Skill Learning with Goal-Conditioned Locomotion
A single policy learns multiple dance behaviors from reference motions. Discrete behavior triggers select the desired skill, while motion phase provides progress through its execution.
Discussion and limitations
Adversarial Motion Priors encourages a policy to produce motions that resemble a reference distribution. A discriminator is trained beside the policy to supply a smooth reward signal for how well motions fall on the manifold defined by the reference clips. As such, the method struggles to extrapolate between different reference motion clips as transitions often do not fall on this manifold and there is an ambigious signal. Additionally, the method struggles learning some motions due to Mode collapse. 1. I experienced difficulty composing multiple motions in a single policy. 2. Tracking reward produced
Notes
My background notes on the foundations of reinforcement learning and the ideas behind this framework.