A Systematic Ablation Study of SR2L and Domain
Randomization for Quadruped Locomotion
Anand Patel
School of Computer Science and Applied Mathematics
University of the Witwatersrand, Johannesburg
Email: anand.patel@students.wits.ac.za
Abstract—Quadruped robots require robustness to sensor noise
and actuator failures, yet combining training methods remains
unexplored. We systematically compare Smooth Regularized RL
(SR2L), Domain Randomization (DR), and their combination
across 38,000 episodes on RealAnt.
Key findings: (1) Negative synergy—combining SR2L+DR
underperforms DR alone by 67.8% under maximum stress
(p < 0.001, d = 2.68). Training dynamics reveal catastrophic
collapse at 14-20M steps due to gradient conflict. (2) Implicit
robustness—VecNormalize provides 146% improvement (p <
0.001), explaining universal noise tolerance across all models.
(3) Cross-distribution generalization—SR2L achieves 106-114%
retention on Poisson and salt-and-pepper noise despite Gaussianonly training.
Our findings challenge assumptions of additive robustness
benefits. For robustness: specialize training to dominant failure
modes rather than multi-objective approaches. Comprehensive
methodology includes checkpoint-based convergence analysis,
VecNormalize ablation, and factorial joint×noise testing across
9 systematic experiments.
Index Terms—Quadruped Locomotion, Robustness, Domain
Randomization, Smooth Regularization, Reinforcement Learning, PPO, Ablation Study, Negative Synergy
I. R ELATED W ORK
A. Robustness Methods: SR2L and Domain Randomization
Smooth Regularized Reinforcement Learning (SR2L) [1]
introduces smoothness constraints through a regularization
term LSR2L = E[∥πθ (s) − πθ (s + δ)∥2 ] that penalizes policy
sensitivity to observation perturbations, creating robustness to
sensor noise. Related approaches enforce Lipschitz continuity [2] or adversarial perturbations [3]. Domain randomization
(DR) [4], [5] trains on randomized environment parameters;
for locomotion, this includes joint failure randomization with
curriculum learning [6]. While both methods address robustness individually, no prior work systematically evaluates their
interaction when combined.
B. Multi-Objective Learning and Negative Transfer
Multi-objective RL aims to optimize competing objectives
simultaneously [7]. Convention suggests additive benefits from
combining robustness methods. However, recent work on
negative transfer [8] shows that some objective combinations
interfere during training. Our work demonstrates that SR2L
smoothness penalties and DR adaptation requirements create
conflicting gradients.
C. Quadruped Locomotion and Implicit Robustness
Deep RL has achieved impressive quadruped locomotion [9]–[11], typically using DR as an implicit robustness
mechanism. VecNormalize [12], a standard observation normalization technique, is typically viewed as a training stabilizer. Our controlled ablation reveals VecNormalize provides substantial implicit robustness (146% improvement,
p < 0.001), suggesting observation normalization deserves
recognition as a robustness mechanism with implications for
fair method comparison.
II. I NTRODUCTION
A. Motivation and Problem Statement
Quadruped robots operating in unstructured environments
face two fundamental failure modes: sensor degradation (noisy
joint encoders from vibration, electromagnetic interference,
or wear) and actuator failures (joint malfunctions, reduced
torque, or complete mechanical breakdown). While reinforcement learning (RL) has achieved impressive locomotion capabilities in simulation [9], [10], policies trained under ideal
conditions exhibit catastrophic performance degradation when
deployed on physical hardware experiencing these real-world
perturbations.
Two distinct robustness paradigms have emerged to address
this issue:
Smooth Regularized RL (SR2L) [1] injects zero-mean
Gaussian noise into observations during training while penalizing policy sensitivity to these perturbations. The regularization
term Lsmooth = Es∼D [∥πθ (s) − πθ (s + δ)∥2 ] encourages local
Lipschitz continuity, creating policies robust to continuous
sensor noise. SR2L is designed specifically for observation
noise robustness.
Domain Randomization (DR) [4], [5] randomly varies
environment parameters during training, exposing the agent to
diverse conditions. For actuator failures, DR randomly disables
joints (setting torque to zero) to force the policy to learn
compensatory behaviors. DR targets structural and mechanical
failure modes.
Despite extensive work on these individual methods, a
critical question remains unanswered: do SR2L and DR provide complementary or redundant robustness? The natural
hypothesis is that these methods should be complementary:
SR2L targets continuous sensor noise through smoothness
regularization, while DR targets discrete structural failures
through randomized exposure. Different failure modes affecting different system components suggest additive robustness
benefits when combined. Multi-randomization approaches in
prior DR work [13] support this intuition, showing that combining visual and dynamics randomization improves sim-toreal transfer. However, these methods might instead interfere,
with competing training objectives degrading overall performance—a possibility unexplored in prior literature.
B. Research Questions and Approach
We conduct a systematic 4-way ablation study across approximately 38,000 evaluation episodes, comparing:
• M1 (Baseline): PPO without robustness training
• M2 (SR2L): PPO with smooth regularization (λ = 0.001,
σ = 0.01)
• M3 (DR): PPO with 3-phase curriculum domain randomization (50-60% episodes with 1-2 joint failures during
training)
• M4 (Combo): PPO with both SR2L and curriculum DR
Our evaluation protocol spans 8 experiments:
1) Baseline performance (ideal conditions)
2) Sensor noise robustness (12 Gaussian noise levels, σ ∈
[0, 3.0])
3) Extended noise types (Gaussian, Poisson, salt-andpepper with SNR matching)
4) Joint failure robustness (8 individual joints with 2second delayed locking)
5) Combined stress (simultaneous noise and joint failures)
6) Per-joint deep dive (velocity profiling and anatomical
patterns)
7) Validation suite (controlled ablation of VecNormalize,
stochastic resonance testing)
8) Full factorial ablation (8 joints × 4 noise conditions with
recovery time tracking)
Contributions. We make four key contributions:
C1: Quantified interference in multi-method training.
We provide quantitative evidence that combining SR2L and
DR produces interference rather than synergy. Under the
highest stress condition (dual joint failure + high noise), M4
(combo) underperforms M3 (DR alone) by 67.8% (0.72m vs
2.22m, p < 0.001, Cohen’s d = +2.68), demonstrating that
multi-objective robustness training creates conflicting objectives that severely degrade performance.
C2: Discovery of implicit robustness mechanism.
Through controlled ablation, we identify VecNormalize as an
unrecognized robustness mechanism. Models with VecNormalize achieve 146% better baseline performance (11.2m vs
4.5m) and maintain 96% retention at σ = 0.1 despite no
robustness training. This running statistics normalization acts
as an implicit low-pass filter, explaining unexpected noise
tolerance in baseline models.
C3: Discovery of SR2L noise tolerance and generalization. Through systematic multi-distribution noise testing
(Experiment 2B), we demonstrate that SR2L-trained policies exhibit noise-induced regularization, achieving up to
114% performance at moderate noise levels (σ = 0.050.10). Remarkably, despite training on Gaussian noise only,
SR2L generalizes to novel distributions: 106.6% average retention on Poisson noise and 112.8% on salt-and-pepper noise.
This cross-distribution transfer demonstrates that smoothness
regularization creates general noise robustness rather than
distribution-specific adaptation.
C4: Comprehensive robustness methodology. We introduce temporal recovery metrics (time to regain 50% baseline
velocity post-failure), multi-distribution noise testing with
SNR equivalence, and full factorial joint×noise interaction
analysis. This methodology enables mechanistic understanding
beyond static retention percentages.
Deployment recommendation. Based on approximately
38,000 episodes, we recommend M3 (DR alone) for real-world
deployment: 3.73m average absolute distance under joint failures (superior to M4’s 3.38m), 104.7% noise retention, acceptable 29% baseline sacrifice, and simpler training than multimethod approaches. The combined method (M4) provides no
benefit over M3 while requiring increased training complexity
and sacrificing an additional 23% baseline performance.
III. M ETHODOLOGY
A. Experimental Platform and Environment
All experiments use the RealAnt-v0 quadruped robot simulated in MuJoCo physics engine [14]. The RealAnt has
8 actuated degrees of freedom (2 joints per leg: hip and
ankle), organized as shown in Figure 1. The 29-dimensional
observation space consists of: (1) torso linear velocities in x, y,
z (3D), (2) torso z-position (1D), (3) sin and cos of torso Euler
angles for roll, pitch, yaw (6D), (4) torso angular velocities
(3D), (5) joint angular positions (8D), and (6) joint angular
velocities (8D). The action space consists of 8 continuous
angular position set-points for the joints.
Joint numbering follows the convention: Joint 1
(hip 1/ankle 1) = front-left leg, Joint 2 (hip 2/ankle 2)
= front-right leg (camera-facing), Joint 3 (hip 3/ankle 3) =
rear-left leg, Joint 4 (hip 4/ankle 4) = rear-right leg (camerafacing). Hip joints are proximal (closer to torso), while ankle
joints are distal (end of leg). This anatomical layout is critical
for interpreting joint failure results in Section 4.3, particularly
the worst-case performance of camera-facing ankle joints.
The environment uses a forward locomotion reward structure (SuccessRewardWrapper) that provides reward proportional to forward velocity: rt = 100 · vx2 , where vx is the
robot’s velocity in the forward direction. This custom wrapper
implements the velocity-based reward shaping described by
Sutton & Barto [15], using exponential scaling (v 2 ) rather than
linear velocity to strongly incentivize faster gaits. Episodes
terminate after 1000 timesteps or if the robot falls (torso height
< 0.2m). This reward structure encourages fast, stable forward
locomotion without requiring explicit goal positions.
B. Model Training Configurations
We train four distinct models using Proximal Policy Optimization (PPO) [16] to enable systematic ablation analysis:
Phase 2 (10-20M steps): 50% episodes with exactly 1
random joint locked
• Phase 3 (20-32M steps): 60% episodes with 1-2 random
joints locked
Joint locking implemented by setting affected joint torques to
zero. Total training: 32M timesteps.
M4 (Combined): PPO with both SR2L smoothness regularization AND curriculum domain randomization. Identical
curriculum schedule to M3, but with SR2L loss (λ = 0.001,
σ = 0.01) applied throughout all phases. Total training: 30M
timesteps.
All models use identical policy architecture: 2-layer MLP
with [128, 128] hidden units and tanh activation. PPO hyperparameters: learning rate 3×10−4 , batch size 1536, 8 epochs per
update, clip range 0.15, GAE λ = 0.97, discount γ = 0.995.
•
RealAnt Joint Numbering Convention
Top-Down View (8 Actuated Joints: 4 Hips +
4 Ankles)
Front Left
Front Right
2
1
Ankle 2
Ankle1
Hip 2
Hip 1
Torso
Hip 3
Ankle 3
C. Observation Normalization
Hip 4
All models use VecNormalize [12] to maintain running
statistics (mean and standard deviation) of observations and
rewards. This normalization is applied during both training and
evaluation, with statistics frozen during evaluation rollouts.
This is critical infrastructure that provides implicit robustness
benefits across all models.
Ankle 4
3
4
Rear Left
Rear Right
D. Evaluation Protocol
Fig. 1. RealAnt joint numbering convention. Top-down view showing the
8 actuated joints (4 hips + 4 ankles) and their anatomical positions. Each
leg has two joints: a proximal hip joint (larger circles) and a distal ankle
joint (smaller circles). Joints are numbered 1-4 corresponding to front-left,
front-right, rear-left, and rear-right legs respectively. Note that joints 2 and 4
are camera-facing (positioned on the right side when robot walks left→right),
which affects failure observability and recovery strategies. This diagram serves
as reference for interpreting per-joint robustness results throughout the paper.
M1 (Baseline): Standard PPO trained in clean conditions
with no robustness augmentation. Uses SuccessRewardWrapper only. Trained for 10M timesteps on RealAnt-v0. This
establishes the performance ceiling under ideal conditions.
M2 (SR2L): PPO with Smooth Regularized Reinforcement
Learning [1]. Augments the standard PPO loss with a smoothness regularization term:
Ltotal = LP P O + λ · Es∼D [∥πθ (s) − πθ (s + δ)∥2 ]
(1)
where δ ∼ N (0, σ 2 I) with σ = 0.01 applied only to joint
observation dimensions (13-28), and λ = 0.001. Perturbations
applied during training but not to the actual environment.
Trained for 20M timesteps. While training uses Gaussian
noise only, we evaluate generalization to Poisson and salt-andpepper noise distributions (Experiment 2B) to assess whether
smoothness regularization transfers across noise types.
M3 (Domain Randomization): PPO with 3-phase curriculum domain randomization targeting joint failures. Training
schedule:
•
Phase 1 (0-10M steps): Clean training, no failures
We conduct 9 systematic experiments totaling approximately 38,000 evaluation episodes:
Experiment 1: Baseline Performance (400 episodes) Establish baseline locomotion capabilities without any perturbations. Each model evaluated for 100 episodes in nominal
RealAnt-v0 environment with no sensor noise or joint failures.
Experiment 2: Sensor Noise Robustness (4,800 episodes)
- Test robustness to sensor observation noise at escalating
severity levels. Gaussian noise (µ = 0, σ) added to joint
position/velocity observations. Tested 12 noise levels: σ ∈
{0.00, 0.01, 0.02, 0.03, 0.04, 0.05, 0.07, 0.10, 0.15, 0.20, 0.25, 0.30}.
Range spans 0× to 30× M2’s training noise level (σ = 0.01).
100 episodes per model per noise level.
Experiment 3: Joint Failure Robustness (3,200 episodes)
- Evaluate adaptation to individual joint actuator failures. Each
of 8 joints (4 hips, 4 ankles) locked at neutral position for
entire episode. Tests catastrophic single-joint failure scenarios.
100 episodes per model per joint.
Experiment 4: Combined Stress Testing (2,400 episodes)
- Assess performance under simultaneous sensor and actuator
failures. 6 combined scenarios crossing 3 joint failure rates
{10%, 30%, 50%} with 2 noise levels {σ = 0.00, σ = 0.05}.
Episode-level random joint selection with specified probability.
100 episodes per model per scenario.
Experiment 5: Per-Joint Deep Dive (4,800 episodes) High-resolution analysis of individual joint vulnerabilities with
velocity profiling. Each of 8 joints tested individually with
locked actuator for entire episode. 150 episodes per joint for
statistical power in velocity variance analysis. Total: 4 models
× 8 joints × 150 episodes.
E. Performance Metrics
For each evaluation episode, we record:
• Distance traveled: Total forward distance over 1000
timesteps
• Average velocity: Mean forward velocity v̄x
• Success rate: Percentage of episodes achieving > 2m
distance
• Cumulative reward: Total reward accumulated
• Fall rate: Percentage of episodes ending in termination
• Retention percentage: (Faulty performance / Baseline
performance) × 100%
• Recovery time: Time to regain 50% baseline velocity
after fault injection
Statistical significance assessed using paired t-tests with
Bonferroni correction for multiple comparisons (α = 0.05).
IV. R ESULTS
We present comprehensive evaluation results across approximately 38,000 episodes, organized chronologically by
stress condition. All statistical tests use paired t-tests with
Bonferroni correction (α = 0.05/n) where n is the number
of comparisons. Retention percentage is defined as (faulty
performance / baseline performance) × 100%.
A. Baseline Performance
Figure 1 shows performance under ideal conditions (no
sensor noise, no joint failures). M1 achieved the highest
baseline performance (11.20 ± 0.16m, 95% CI: [11.17, 11.23],
n = 100), significantly outperforming all robustness-trained
models (all p < 0.001). M2 (SR2L) achieved 8.75 ± 2.35m
(22% sacrifice), M3 (DR) achieved 7.96 ± 2.33m (29%
sacrifice), and M4 (Combined) achieved 5.34 ± 3.75m (52%
sacrifice).
The 52% baseline sacrifice by M4 is substantially larger
than M3’s 29% sacrifice, despite both models undergoing
domain randomization training. This suggests training interference rather than complementary benefits. The pairwise
comparison between M3 and M4 did not achieve statistical
significance after Bonferroni correction (p = 0.024 > αadj =
Baseline Performance Comparison
(Clean Environment, No Failures)
M3 Reference (7.96m)
16
Distance Traveled (m)
Experiment 6: Delayed Joint Locking (3,200 episodes)
- Test adaptation to mid-episode actuator failures. 2-second
delay before locking joints (simulating sudden failure). Tested
across all 8 individual joints. 100 episodes per model per joint.
Experiment 7: Joint-Noise Interaction Analysis (12,800
episodes) - Systematic ablation of joint failure × sensor noise
interactions. Full factorial design crossing 8 joints with 4
noise levels {σ = 0.00, σ = 0.05, σ = 0.10, σ = 1.0}.
Tests whether failures combine additively or synergistically.
100 episodes per model per combination (32 conditions).
Experiment 8: Detailed Combined Stress (6,400
episodes) - Extended combined stress testing with additional noise levels. 16 scenarios crossing 4 joint failure rates
{0%, 10%, 30%, 50%} with 4 noise levels {σ = 0.00, σ =
0.02, σ = 0.05, σ = 0.10}. 100 episodes per model per
scenario.
14
12
11.20m
29% sacrifice
for robustness
8.75m
10
7.96m
52% sacrifice
(interference)
5.34m
8
6
4
2
0
100%
93%
92%
64%
M1
(Baseline)
M2
(SR2L)
M3
(DR)
M4
(Combined)
Model Configuration
Fig. 2. Baseline performance comparison across four training methods.
Left: Average distance traveled in ideal conditions (no noise, no failures).
Center: Success rates (episodes >5m). Right: Failure rates (falls/immobility).
M1 achieves highest performance (11.20m) but lacks robustness training. M3
(DR) and M4 (Combined) show substantial baseline sacrifices (29% and 52%
respectively), with M4’s combined training producing the largest performance
cost. Error bars: 95% confidence intervals. All pairwise differences p < 0.001
(Bonferroni corrected, n = 100 episodes per model).
0.0083), though the large effect size (Cohen’s d = +0.84)
indicates a practically meaningful difference. This exemplifies
the distinction between statistical and practical significance:
with n = 100 episodes, we have sufficient power to detect
that a moderate-to-large effect (d=0.84) exists, even when it
falls short of our conservative significance threshold.
All robustness training methods incurred baseline performance costs, with combined training (M4) showing the largest
sacrifice. M4’s high standard deviation (3.75m) indicates inconsistent performance, with some episodes achieving reasonable distances while others exhibited near-zero locomotion.
B. Training Dynamics and Convergence
Figure 3 presents training dynamics reconstructed from
checkpoint evaluation at regular intervals throughout training.
This checkpoint-based approach provides more rigorous convergence analysis than W&B training logs, as it uses the exact
evaluation protocol applied to final models.
Convergence patterns. M1 (Baseline) achieved rapid convergence within 2M steps, reaching a stable plateau at 11m distance with minimal variance (CV: 2.1%). M2 (SR2L) showed
smooth, monotonic improvement over 20M steps, stabilizing
at 9m with very low variance (CV: 3.8%)—consistent with
SR2L’s smoothness objective. M3 (DR) exhibited high training
variance throughout all 32M steps, with frequent oscillations
between 3m and 9m reflecting continuous adaptation to randomized joint failures. Critically, M3 did not achieve a stable
plateau—the 15.7% coefficient of variation in the final 10M
steps indicates ongoing adaptation rather than convergence.
M4’s catastrophic collapse. M4 (Combined) experienced
catastrophic forgetting at 14-20M steps, with episode rewards
collapsing from 8m to near-zero (robot unable to walk).
This 6M-step failure period preceded partial recovery to 78m by 30M steps, though final performance remained below
Training Dynamics: Convergence Analysis from Checkpoint Evaluation
Sensor Noise Robustness: VecNormalize is the Hidden Hero
(All Models Benefit, SR2L Redundant)
(a) Reward Progression: Convergence via Checkpoint Evaluation
12
M1 (Baseline)
M2 (SR2L)
M3 (DR)
M4 (Combined)
12
8
4
2
0
M1 (Baseline)
M2 (SR2L)
M3 (DR)
M4 (Combined)
2
4
0
5
10
15
20
Training Steps (millions)
25
30
(b) Locomotion Distance: Performance Over Training
8
6
4
2
10
0.00
Mean Distance (m)
M1 WITHOUT VecNormalize
VecNormalize Benefit
SR2L Training Noise
10× Training Noise
10
6
Distance Traveled (m)
Mean Episode Reward
10
8
6
4
M1 (Baseline)
M2 (SR2L)
M3 (DR)
M4 (Combined)
2
0
0
5
10
15
20
Training Steps (millions)
25
30
Fig. 3. Training dynamics reconstructed from checkpoint evaluation.
(a) Episode reward progression showing convergence patterns: M1 rapid
convergence (2M steps), M2 smooth monotonic improvement (20M steps),
M3 high variance reflecting DR adaptation (32M steps). Most critically,
M4 exhibits catastrophic collapse at 14-20M steps (reward → 0) with
incomplete recovery, providing direct evidence of gradient conflict between
SR2L smoothness and DR adaptation objectives. (b) Locomotion distance over
training confirming reward patterns correlate with actual forward movement.
Checkpoints evaluated every 500K-1M steps using identical protocol as final
evaluation (n = 10 episodes per checkpoint). This collapse is not visible in
sparse W&B training logs (4-6% sampling), highlighting the importance of
checkpoint-based convergence analysis.
M3. The collapse occurred during the training phase when
both SR2L smoothness penalties and DR joint failures were
simultaneously active. Neither M2 (SR2L alone) nor M3 (DR
alone) exhibited such instability.
Training variance. The final 10M steps of training reveal
distinct stability profiles: M1 shows coefficient of variation
(CV) of 2.1%, M2 shows 3.8%, M3 shows 15.7%, and M4
shows 12.4%. M3’s high variance reflects oscillations between
3m and 9m throughout training, while M4 shows moderate
variance but lower absolute performance, never recovering to
pre-collapse levels.
C. Sensor Noise Robustness
Figure 2 presents results across 12 Gaussian noise levels
ranging from σ = 0.00 to σ = 0.30. At σ = 0.10 (10×
M2’s training noise level), models achieved: M1: 10.95 ±
0.16m (97.8% retention), M2: 8.38 ± 2.60m (95.8% retention), M3: 8.33 ± 1.66m (104.7% retention), and M4:
7.91 ± 1.40m (148.1% retention). M4’s anomalous 148%
retention reflects baseline inconsistency (5.34m baseline vs
M3’s 7.96m)—absolute performance remains lower than M3.
Statistical tests comparing performance at σ = 0.10 against
baseline (Bonferroni α = 0.05/4 = 0.0125) revealed: M1
showed significant degradation (p < 0.001, d = +1.58), while
M2 (p = 0.318, d = +0.15) and M3 (p = 0.213, d = −0.18)
showed no significant changes.
0.05
0.10
0.15
0.20
Observation Noise Level ( )
0.25
0.30
Fig. 4. Sensor noise robustness across 12 noise levels. Top left: Average
distance vs noise standard deviation (σ). Top right: Retention percentage
heatmap showing all models maintain >95% at σ = 0.10 (10× M2 training
level). Bottom left: VecNormalize ablation showing 146% improvement
(WITH: 11.20m, WITHOUT: 4.55m, p < 0.001). Bottom right: M2 (SR2L)
multi-distribution generalization achieving 101-114% retention on Gaussian,
Poisson, and salt-and-pepper noise distributions. The unexpected universal
robustness traces to VecNormalize’s implicit low-pass filtering, while SR2L
exhibits noise-induced regularization with cross-distribution transfer. n = 100
episodes per condition.
VecNormalize ablation. Controlled ablation testing (Experiment 6) compared M1 with versus without VecNormalize. M1
with VecNormalize: 11.20m baseline, 10.75m at σ = 0.10. M1
without VecNormalize (trained from scratch, identical 10M
step training, same PPO hyperparameters): 4.55m baseline,
4.35m at σ = 0.10. This represents 146% improvement at
baseline (t = 88.82, p < 0.001) and 147% improvement at
σ = 0.10.
D. Joint Failure Robustness
Figure 4 analyzes performance when individual joints were
locked throughout episodes, simulating catastrophic actuator
failures. M3 achieved the highest average absolute distance
under joint failures: 3.73m across 8 joints, outperforming M1
(3.57m), M4 (3.38m), and M2 (2.24m). M3’s 10.5% advantage
over M4 (3.73m vs 3.38m) demonstrates superior real-world
robustness.
When expressed as retention percentage (secondary metric),
results appear contradictory: M4 shows 63.2% ± 34.8% average retention, higher than M3’s 46.9% ± 24.0%. However,
M4’s substantially lower baseline (5.34m vs M3’s 7.96m)
inflates this metric—a small absolute distance appears as high
retention when divided by a small baseline. Statistical comparison of retention percentages found no significant difference
between M3 and M4 (p = 0.086, n.s.), Cohen’s d = −0.55.
Anatomical Patterns. Hip joints consistently outperformed
ankle joints across all models: M1 (hips: 48.6%, ankles:
15.2%), M2 (hips: 35.5%, ankles: 15.7%), M3 (hips: 52.2%,
ankles: 41.6%), M4 (hips: 84.9%, ankles: 41.5%). However,
none of these differences achieved statistical significance after
Bonferroni correction (all p > 0.0125). Front legs showed
non-significant trends toward better retention than rear legs
(all p > 0.05). Camera position effects on ankle joints were
not statistically significant (all p > 0.05).
Joint Failure Analysis: Anatomical Patterns and Model Performance
(Which joints are hardest to compensate for?)
40
20
0
Hip
80
60
40
0
Ankle
(c) M3 (DR) Per-Joint Performance
(Ranked Best to Worst)
100
Camera-Away
33.5%
35.1%
39.4%
Hip 2
Hip 4
50.3%
Ankle 3
52.8%
67.4%
Ankle 2
Good (>50%)
OK (30-50%)
84.3%
Poor (<30%)
Hip 1
0
20
40
60
Retention (%)
80
100
Average Retention (%)
Hip 3
Ankle 1
80
60
5.34m
4
2
0
110
M2 (SR2L)
100% (No degradation)
104.7%
105
100
95
95.8%
90
M2 (SR2L)
M3 (DR)
(b) Sensor Noise ( =0.10)
10× Training Noise
L)
M2 (SR2
M3 (DR)
69%
46.9%
50
40
30
25.6%
12%
29%
33%
4.12m
4
3
2
1
L)
M2 (SR2
12%
M3 (DR)
12%
9%
92%
130%
49%
80%
73%
65%
25%
64%
19%
Hip_1
Ankle_1
Hip_2
Ankle_2
Hip_3
Ankle_3
Hip_4
Ankle_4
30
Retention (%)
51%
L+DR)
M4 (SR2
M3 (DR)
20
44%
3.42m
2.12m
M4 (SR2L+DR)
10
65%
5
0
L)
R)
(e) Per-Joint
L+DR)
M3 (DRetention
2 (SR2 Failure
R2Heatmap
M(Green
= Good,
RedM=4 (SPoor)
28%
(d) Combined Stress
Noise + Joint Failures
6
100% (No degradation)
20
L+DR)
37%
M4 (SR2L+DR)
(c) Joint Failures 63.2%
Average Across 8 Joints
60
M4 (SR2
29%
0
40
7.96m
6
85
M1 (Baseline)
M2 (SR2L)
M3 (DR)
M4 (Combined)
14.2%
Ankle 4
Camera-Facing
(d) Front vs Rear Legs
(Locomotion Asymmetry)
(a) Baseline Performance (No Failures)
Tradeoff: Robustness Training Sacrifices Speed
148.1%
8
115
20
8.75m
10
Distance (m)
60
M1 (Baseline)
M2 (SR2L)
M3 (DR)
M4 (Combined)
12
Avg Retention (%)
80
100
Distance Traveled (m)
M1 (Baseline)
M2 (SR2L)
M3 (DR)
M4 (Combined)
Average Retention (%)
Average Retention (%)
100
Robustness Training Methods: SR2L vs DR vs Combined
(Which training approach handles different failure modes best?)
(b) Camera Position Effect
(Does visibility matter?)
Retention (%)
(a) Joint Type Comparison
Hips vs Ankles
40
44%
50
13%
60
KEY FINDINGS: M2 (SR2L) excels at sensor noise but weak on joint failures | M3 (DR) dominates joint failures | M4 (Combined) shows NO SYNERGY - worse than M3 alone
20
0
Front Legs
Rear Legs
KEY FINDINGS: Ankles harder than hips | Rear legs harder than front | Camera position has minimal effect | M3 (DR) consistently best
Fig. 5. Individual joint failure robustness analysis. Top: Per-model
retention heatmap across 8 joints (4 hips, 4 ankles). M3 (DR) shows most
consistent robustness with 47% average retention. M4 (Combined) shows
misleading 63% retention due to low baseline—absolute distance reveals
M3 superior (3.73m vs 3.38m). Bottom left: Absolute distance comparison
showing M3’s practical advantage. Bottom right: Anatomical patterns showing
hip superiority trends (not statistically significant after Bonferroni correction).
Ankle 4 (rear-right, camera-facing) emerges as universal worst-case joint
across all models. n = 100 episodes per joint per model.
These results indicate that while anatomical patterns exist, individual joint variation and model-specific adaptation
strategies dominate robustness outcomes. M3 showed the
most consistent adaptation across joint types, with moderate
variance (std: 24.0%) compared to M4’s high variance (std:
34.8%).
E. Robustness Method Comparison
Figure 3 presents comprehensive comparison of M2 (SR2L),
M3 (DR), and M4 (Combined) across all evaluation dimensions.
Combined stress performance. Under ”Ultimate Challenge” conditions (hip 4 + ankle 3 failure + σ = 0.10
noise), M3 achieved 2.22 ± 0.70m (27.9% retention) while
M4 achieved 0.72 ± 0.44m (13.4% retention). This 67.8%
performance difference was highly significant (t = 18.45,
p < 0.001, Cohen’s d = +2.68, n = 100 paired episodes).
Additional combined stress scenarios: ”Extreme Dual Failure” (hip 1 + ankle 2 + σ = 0.05): M3 achieved 4.50 ±
0.67m (56.6% retention) versus M4’s 3.49 ± 1.30m (65.4%
retention). ”Mild Combined” (ankle 2 + σ = 0.05): M3
achieved 4.67 ± 1.71m (58.7% retention) versus M4’s 3.39
± 1.53m (63.6% retention). M3’s absolute distance exceeded
M4 in all scenarios.
M2 specialization pattern. M2 (SR2L) maintained 95.8%
retention under noise alone (σ = 0.10) but averaged only
2.24m (25.6% retention) under joint failures—the lowest of
all models. M4 sacrificed 52% baseline performance versus
M3’s 29%, without compensating robustness advantage under
combined stress.
Fig. 6. Comprehensive robustness method comparison addressing RQ1
and RQ2. Top: Performance across combined stress scenarios showing M3
(DR alone) consistently outperforming M4 (Combined SR2L+DR). The ”Ultimate Challenge” (hip 4 + ankle 3 + σ = 0.10) reveals 67.8% performance
difference (M3: 2.22m, M4: 0.72m, p < 0.001, d = +2.68). Middle: Specialization patterns showing M2 excels at noise (>100% retention) but fails
at joint failures (25.6% retention). Bottom: Trade-off analysis demonstrating
M4’s 52% baseline sacrifice produces no robustness advantage over M3’s 29%
sacrifice, providing quantitative evidence of training interference rather than
synergy. Error bars: standard deviation. n = 100 episodes per condition.
F. Joint-Noise Interaction Effects
Figure 5 presents full factorial analysis crossing 8 individual joint failures with 4 sensor noise levels (σ ∈
{0.00, 0.05, 0.10, 1.0}), totaling 32 conditions per model (n =
100 episodes per condition, 3,200 total episodes).
For M3 (DR), retention percentages ranged from 91%
(hip 1, σ = 0.00) to 13.7% (ankle 4, σ = 1.0). Ankle 4
(rear-right, camera-facing) showed lowest retention across all
models under highest noise: M1: 7.7%, M2: 6.5%, M3: 13.7%,
M4: 7.8%.
M3’s worst-case retention (13.7%) exceeded M2 (6.5%)
and approached M1 (7.7%). M4 achieved 7.8% worst-case
retention. Ankle joints showed greater noise sensitivity than
hip joints across all models. Performance degradation from
combined stressors approximated the product of individual
retention rates rather than exhibiting compounding effects.
V. D ISCUSSION
A. Mechanisms of Training Interference
Our results demonstrate that combining SR2L and DR
produces interference rather than synergy (67.8% underperformance under highest stress, p < 0.001). We propose that
this stems from fundamental incompatibility between their
training objectives, creating conflicting gradient signals during
optimization.
Mathematical Formulation. M4’s total loss function combines three components:
Ltotal = LP P O +λSR2L ·E[∥πθ (s)−πθ (s+δ)∥2 ]+LDR (2)
where LP P O maximizes locomotion rewards, LSR2L penalizes policy changes under observation perturbations, and LDR
represents rewards collected under randomized joint failures.
91%
89%
89%
90%
Ankle 1
30%
32%
27%
36%
Hip 2
53%
46%
47%
48%
Ankle 2
55%
55%
58%
38%
Hip 3
36%
41%
42%
23%
Ankle 3
54%
45%
55%
56%
Hip 4
42%
49%
51%
45%
Ankle 4
17%
17%
14%
14%
=0.00
=0.00
=0.10
=1.00
Sensor Noise Level
Interaction Type (Hip_1 + =0.10)
488%
400
300
319%
309%
204%
200
100
0
M1 (B
aseline)
M2
(SR2L)
M
3 (DR)
M4 (Com
bined)
80
60
40
20
0
Worst-Case Scenario Performance
60
Perfect Additivity
Worst-Case Retention (%)
Additivity Score (%)
500
100
Retention (%)
Failed Joint
M3 (DR) Retention: Joint Failure × Sensor Noise
Hip 1
50
40
30
20
10
13.7%
7.7%
7.7%
aseline)
)
2 (SR2L
0
M1 (B
M
7.8%
M
3 (DR)
bined)
M4 (Com
Fig. 7. Full factorial joint-noise interaction analysis. Heatmaps showing retention percentage across 8 joint failures × 4 noise levels (σ ∈
{0.00, 0.05, 0.10, 1.0}) for all four models. Progressive degradation visible as
both stressors intensify. Ankle 4 emerges as universal worst-case (M1: 7.7%,
M2: 6.5%, M3: 13.7%, M4: 7.8% at highest noise), with M3 maintaining best
worst-case robustness. Ankle joints show greater noise sensitivity than hips.
Performance degradation approximates product of individual retention rates
(additivity, not synergy), supporting independent failure mode hypothesis.
This demonstrates sensor noise and joint failures stress different locomotion
subsystems. n = 100 episodes per condition (3,200 total episodes for this
analysis).
for failed joints while SR2L penalizes those same policy
changes.
After partial recovery, M4 achieves 5.34m baseline (33%
worse than M3) with 3.38m under failures (10.5% worse than
M3), sacrificing baseline performance without compensating
robustness benefit. The large effect size (Cohen’s d = +2.68
for Ultimate Challenge) and consistency across multiple scenarios confirm this is not a training artifact but fundamental
incompatibility between the two objectives.
Implications. This finding contradicts the common assumption that combining robustness methods yields additive
benefits. Multi-objective training can produce negative transfer
when objectives conflict at the gradient level [8]. Individuals
should analyze whether robustness methods target independent
failure modes (potentially compatible) or require opposing policy characteristics (incompatible). For quadruped locomotion,
discrete structural adaptation (DR) fundamentally opposes
continuous smoothness (SR2L).
B. Model Behavior and Specialization
M3’s high training variance (15.7% CV) reflects continuous
adaptation to diverse joint failure conditions rather than instability, enabling superior robustness (3.73m average) compared
to M4’s incomplete recovery (3.38m). M2 demonstrates clear
specialization: excellent noise robustness (95.8% retention at
During gradient descent, these components generate:
10× training noise) but poor joint failure adaptation (25.6%
∇θ LP P O : Maximize forward locomotion
(3) retention). The smoothness constraint prevents reactive, noise∇θ LSR2L = 2E[(πθ (s) − πθ (s + δ)) · (∇πθ (s) − ∇πθ (s + δ))] sensitive behaviors required for failure compensation—the
(4) interference mechanism observed in M4. Joint×noise multiplicative degradation indicates sensor noise and joint failures
∇θ LDR : Maximize reward under failures (requires adaptation)
stress different locomotion subsystems, yet combining their
(5)
training methods produces gradient conflict rather than modThe Conflict. SR2L penalizes policy sensitivity to obser- ular robustness.
vation perturbations—not all policy changes, but specifically
those triggered by noise. When a joint fails during DR C. Implicit and Explicit Robustness Mechanisms
training, the observation vector changes drastically (zero veVecNormalize provides 146% baseline improvement (p <
locity/position for failed joint). Effective failure compensation 0.001), acting as adaptive low-pass filtering that explains M1’s
requires noise-sensitive behaviors: precise weight distribution 97.8% noise retention at 10× training levels despite no explicit
adjustments based on exact remaining joint states. These training [12]. This has methodological implications: robustness
compensatory policies must react strongly to small observation comparisons must control for VecNormalize effects. SR2L
differences to maintain balance with asymmetric actuation. achieves noise-induced regularization (up to 114% perforHowever, SR2L’s gradient penalizes exactly these sensitive, mance at moderate noise) with cross-distribution generalizareactive behaviors, as they increase ∥πθ (s) − πθ (s + δ)∥2 tion (106.6% on Poisson, 112.8% on salt-and-pepper despite
under observation perturbations. The result: gradients point in Gaussian-only training). While reminiscent of beneficial noise
opposing directions, creating a compromise policy that neither effects [17], [18], our phenomenon is more accurately charadapts strongly (as DR requires) nor maintains smoothness (as acterized as noise-induced regularization—smoothness creates
SR2L requires).
policies that tolerate and sometimes benefit from observation
Empirical Evidence. The training dynamics (Figure 3) perturbations.
directly support this mechanism. M4’s catastrophic collapse at
14-20M steps—where the policy temporarily loses locomotion D. Deployment Recommendations
Based on 38,000 evaluation episodes across 9 systematic
capability entirely—occurs precisely when both SR2L penalties and DR joint failures are simultaneously active. Neither experiments, we recommend:
M2 (SR2L alone) nor M3 (DR alone) exhibit such instability,
For sensor-noise-dominant environments: Deploy M2
confirming the collapse stems from their interaction. The (SR2L). Achieves 95.8%-114% retention across noise types
collapse represents a period where opposing gradients prevent with only 22% baseline sacrifice. Generalizes across noise
stable learning: DR pushes toward aggressive compensation distributions without retraining.
For actuator-failure-dominant environments: Deploy M3
(DR). Achieves 3.73m absolute distance under failures (best of
all models) with acceptable 29% baseline sacrifice. Consistent
performance across joint types.
For mixed environments: Deploy M3 (DR), not M4 (combined). M3 maintains 104.7% noise retention while providing
superior joint failure robustness. M4 offers no advantage
(10.5% worse under failures, similar noise performance) while
sacrificing additional 23% baseline performance and requiring
more complex training.
General principle: Specialize training to expected dominant failure mode rather than attempting general robustness
through multi-method approaches. Our results demonstrate
that multi-objective robustness training can produce interference when methods have conflicting gradient signals.
E. Limitations and Future Work
Our study has several limitations. First, simulation-only
evaluation: we used RealAnt simulation rather than physical
hardware; sim-to-real transfer may reveal additional challenges
including actuator dynamics, sensing delays, and physical
contact modeling that could affect robustness comparisons.
Second, training duration confound: models trained for
different durations (M1: 10M, M2: 20M, M3: 32M, M4:
30M steps), potentially confounding method effectiveness with
training time. While M1 and M2 achieved clear convergence
plateaus (CV: 2.1% and 3.8% respectively), M3 exhibited
15.7% coefficient of variation in final 10M steps, indicating
ongoing adaptation rather than true convergence. This suggests
M3’s superior robustness stems from both the DR method and
extended training. A fair comparison would require training all
models to convergence or comparing at matched checkpoints
(e.g., all at 10M steps). Third, single SR2L configuration:
we tested one hyperparameter setting (λ = 0.001, σ = 0.01);
a sweep over and might reveal parameter regimes where
interference reduces or synergy emerges. Fourth, our DR
curriculum was hand-designed; learned curricula might better
balance objectives. Finally, we focused on PPO; other algorithms (SAC, TD3) might exhibit different interaction patterns.
Future work should explore: (1) alternative combination
strategies that avoid gradient conflict (e.g., sequential training,
importance-weighted objectives), (2) theoretical analysis of
when robustness methods are compatible vs. conflicting, (3)
extension to other locomotion platforms and failure modes, (4)
physical robot validation of our deployment recommendations.
VI. C ONCLUSION
This work provides the first systematic quantitative comparison of robustness methods for quadruped locomotion,
revealing both unexpected mechanisms and important negative
results. Through approximately 38,000 evaluation episodes
across four training methods, we establish clear evidence that
(1) combining SR2L and DR produces interference rather
than synergy (67.8% performance degradation under highest
stress, p < 0.001), (2) VecNormalize provides previously
unrecognized implicit robustness (146% improvement, p <
0.001), and (3) SR2L exhibits noise-induced regularization
with remarkable cross-distribution generalization (up to 114%
performance at moderate noise levels across Gaussian, Poisson, and salt-and-pepper distributions).
Our negative result—that multi-objective robustness training
underperforms specialized approaches—contradicts the natural
hypothesis of additive benefits. Through training dynamics
analysis (Figure 3), we demonstrate that this stems from fundamental incompatibility: SR2L’s smoothness penalty opposes
DR’s adaptation requirement when joint failures induce large
observation changes, leading to catastrophic collapse during
training. This finding has immediate practical implications:
practitioners should train specialized models for specific failure modes rather than attempting to handle all robustness
dimensions simultaneously.
The discovery that VecNormalize provides substantial robustness without explicit training suggests that running statistics normalization acts as implicit low-pass filtering, attenuating high-frequency sensor noise. This mechanism, operating
independently of the learning algorithm, explains why all
models maintained >95% performance at 10× training noise
levels. Future quadruped RL systems should leverage this
effect deliberately rather than treating normalization as a mere
training stabilizer.
Deployment Recommendations. For real-world quadruped
systems, we recommend: (1) M3-style domain randomization
for joint/actuator failure robustness (47% retention vs. 32%
baseline), (2) VecNormalize for implicit sensor noise filtering
(no training cost), (3) specialized models per failure mode
rather than combined training, (4) absolute distance metrics
over retention percentages when baselines differ substantially.
This work contributes quantitative evidence, methodological frameworks (comprehensive ablation methodology with
38,000 episodes, multi-distribution generalization testing, controlled VecNormalize ablation), and theoretical insights (mathematical formulation of gradient conflict, discovery of VecNormalize’s implicit robustness mechanism, characterization
of noise-induced regularization) that advance understanding
of robustness in learned locomotion policies. The finding that
specialized training outperforms multi-objective approaches
challenges conventional wisdom and provides actionable guidance for deploying robust quadruped systems.
R EFERENCES
[1] Q. Shen, Y. Li, H. Jiang, Z. Wang, and T. Zhao, “Deep reinforcement
learning with robust and smooth policy,” in International Conference on
Machine Learning (ICML). PMLR, 2020, pp. 8707–8718.
[2] T. Yu, S. Sugano, J. Nakanishi, and J. Morimoto, “L2c2: Locally
lipschitz continuous constraint towards stable and smooth reinforcement
learning,” in International Conference on Machine Learning (ICML).
PMLR, 2022, pp. 25 688–25 707.
[3] L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta, “Robust adversarial reinforcement learning,” in International Conference on Machine
Learning (ICML). PMLR, 2017, pp. 2817–2826.
[4] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel,
“Domain randomization for transferring deep neural networks from
simulation to the real world,” in IEEE/RSJ International Conference
on Intelligent Robots and Systems (IROS). IEEE, 2017, pp. 23–30.
[5] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Sim-toreal transfer of robotic control with dynamics randomization,” in IEEE
International Conference on Robotics and Automation (ICRA). IEEE,
2018, pp. 3803–3810.
[6] F. Muratore, F. Ramos, G. Turk, W. Yu, M. Gienger, and J. Peters,
“Robot learning from randomized simulations: A review,” Frontiers
in Robotics and AI, vol. 9, 2022, comprehensive review including
curriculum DR.
[7] C. Liu, X. Xu, and D. Hu, “Multiobjective reinforcement learning: A
comprehensive overview,” IEEE Transactions on Systems, Man, and
Cybernetics: Systems, vol. 45, no. 3, pp. 385–398, 2015.
[8] T. Standley, A. Zamir, D. Chen, L. Guibas, J. Malik, and S. Savarese,
“Which tasks should be learned together in multi-task learning?” in
International Conference on Machine Learning (ICML). PMLR, 2020,
pp. 9120–9132, negative transfer in multi-task learning.
[9] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun,
and M. Hutter, “Learning agile and dynamic motor skills for legged
robots,” Science Robotics, vol. 4, no. 26, p. eaau5872, 2019.
[10] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning
quadruped locomotion over challenging terrain,” in Science Robotics,
vol. 5, no. 47, 2020.
[11] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, M. Khadiv, and M. Hutter,
“Learning robust perceptive locomotion for quadrupedal robots in the
wild,” Science Robotics, vol. 7, no. 62, 2022.
[12] P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford,
J. Schulman, S. Sidor, Y. Wu, and P. Zhokhov, “Openai baselines,” https:
//github.com/openai/baselines, 2017, vecNormalize implementation.
[13] B. Mehta, M. Diaz, F. Golemo, C. J. Pal, and L. Paull, “Active domain
randomization,” in Conference on Robot Learning (CoRL). PMLR,
2020, pp. 1162–1176, multiple randomization strategies.
[14] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for modelbased control,” IEEE/RSJ International Conference on Intelligent Robots
and Systems (IROS), pp. 5026–5033, 2012.
[15] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction,
2nd ed. MIT Press, 2018.
[16] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347,
2017.
[17] L. Gammaitoni, P. Hänggi, P. Jung, and F. Marchesoni, “Stochastic
resonance,” Reviews of Modern Physics, vol. 70, no. 1, p. 223, 1998.
[18] M. D. McDonnell and D. Abbott, “What is stochastic resonance?
definitions, misconceptions, debates, and its relevance to biology,” PLoS
Computational Biology, vol. 5, no. 5, p. e1000348, 2009.
0
You can add this document to your study collection(s)
Sign in Available only to authorized usersYou can add this document to your saved list
Sign in Available only to authorized users(For complaints, use another form )