← Anthony Roumi

Humanoid Motion Learning Framework

Reinforcement Learning · Imitation Learning

Code / Notes


This framework uses Adversarial Motion Priors to encourage the policy to imitate reference motions while simultaneously learning task-specific behaviors.

It provides flexible control over policy observations through conditioning on commands, motion triggers, and motion phase, and over credit assignment and the learning proccess through motion-specific reward functions, termination criteria, and configurable AMP-style rewards.

Point-Goal Locomotion + Standing

Learned Policy
Reference Motion

Overview

A motion manifest defines which motion clips are available to be sampled from and their respective command observations, triggers, rewards, and termination criteria.

motions:
  walk_to_goal:
    file: data/walk.npz
    sample_prob: 1.0
    amp: true

    commands:
      goal_x: [-5.0, 5.0]
      goal_y: [-5.0, 5.0]

    zero_command_prob: 0.25

    rewards:
      progress:
        weight: 1.0
        scale: 30.0
      target_hit:
        weight: 0.5
        scale: 0.5

    termination: unhealthy
    

The environment manages everything and exposes the necessary information so the agent can apply the AMP reward when appropriate.

Experiments

Triggered Dance Skills with Phase Conditioning

A single policy learns multiple dance behaviors from reference motions. Discrete behavior triggers select the desired skill, while motion phase provides progress through its execution.

Placeholder for commanded versus measured velocity or goal-point tracking

Compositional Skill Learning with Goal-Conditioned Locomotion

A single policy learns multiple dance behaviors from reference motions. Discrete behavior triggers select the desired skill, while motion phase provides progress through its execution.


Compositional Skill Learning with Goal-Conditioned Locomotion

A single policy learns multiple dance behaviors from reference motions. Discrete behavior triggers select the desired skill, while motion phase provides progress through its execution.

Discussion and limitations

Adversarial Motion Priors encourages a policy to produce motions that resemble a reference distribution. A discriminator is trained beside the policy to supply a smooth reward signal for how well motions fall on the manifold defined by the reference clips. As such, the method struggles to extrapolate between different reference motion clips as transitions often do not fall on this manifold and there is an ambigious signal. Additionally, the method struggles learning some motions due to Mode collapse. 1. I experienced difficulty composing multiple motions in a single policy. 2. Tracking reward produced

Notes

My background notes on the foundations of reinforcement learning and the ideas behind this framework.