Auditing Informal Scenario-Based Fragility Analyses

Audit informal scenario-based fragility analyses by checking coverage, joint stress, worst-case identification, and criterion alignment. Distinguish one-at-a-time (OAT) estimates from true robust worst-case losses, identify worst vertices for linear-in-uncertainty objectives over box uncertainty sets, and flag stochastic-vs-robust criterion mismatches.

Step 1 of 157%

Tutorial

Introduction

A scenario-based fragility analysis evaluates a decision $x$ by computing the outcome $f(x, u)$ for each $u$ in a finite list of scenarios $\{u^{(1)}, \dots, u^{(K)}\} \subseteq \mathcal{U}$ , where $\mathcal{U}$ is the set of plausible parameter values. The analysis is informal when the scenarios are chosen by intuition or convention, rather than by solving

\max_{u \in \mathcal{U}} \operatorname{loss}(x, u).

To audit an informal analysis, check four dimensions:

Coverage. Do the tested $u^{(i)}$ reach the extremes of $\mathcal{U}$ ?
Joint stress. Are parameters varied jointly, not one at a time?
Worst-case identification. Is the true worst point $u^\star = \arg\max_{u \in \mathcal{U}} \operatorname{loss}(x, u)$ approximated by some tested $u^{(i)}$ ?
Binding constraints. Are the scenarios that make a constraint tight actually included?

A scenario list can pass every individual test and still leave the decision fragile, simply because the worst point in $\mathcal{U}$ was never tried.

For a small illustration, take $x = 10$ with loss $L(u) = u \cdot x$ and $u \in [-3, 5]$ . An analyst who tests $u = 0, 2, 4$ reports a maximum loss of $40$ . The audit flags coverage: the right edge $u = 5$ gives $L = 50$ , which exceeds the reported maximum.