Validation & Testing

VegasAfterglow includes a comprehensive validation framework to ensure numerical accuracy and physical correctness. The validation suite is located in the validation/ directory.

Running Validation

Prerequisite: Install with validation support, which includes per-stage CPU profiling and PDF report dependencies:

pip install -e ".[test]" --config-settings=cmake.define.AFTERGLOW_PROFILE=ON

The unified validation runner orchestrates all tests and generates reports:

# Run full validation suite (benchmark + regression + PDF report)
python validation/run_validation.py --all

# Run only benchmark tests
python validation/run_validation.py --benchmark

# Run only regression tests
python validation/run_validation.py --regression

# Check existing results without re-running tests
python validation/run_validation.py --check-only

# Fast mode: skip resolution convergence scans (timing + regression only)
python validation/run_validation.py --all --fast

# Control parallelism (default: all CPU cores)
python validation/run_validation.py --all -j 4

--fast is useful for a quick sanity check — it runs performance timing and regression tests but skips the resolution convergence scans, so it completes significantly faster. The PDF report generated in fast mode omits the convergence detail pages.

The validation runner returns exit code 0 on success and 1 on failure, making it suitable for CI/CD pipelines.

Benchmark Tests

Benchmark tests measure computational performance and verify numerical convergence. Located in validation/benchmark/.

What is tested:

  • Performance Timing: Execution time for various jet/medium/radiation configurations

  • Resolution Convergence: Convergence analysis in three dimensions:

    • Phi (azimuthal): Tests values from 0.15 to 0.3 radians

    • Theta (polar): Tests values from 0.5 to 1.25 radians

    • Time: Tests values from 10 to 25 points per decade

  • Configuration Matrix: All combinations of jet types (tophat, gaussian, powerlaw), media (ISM, wind), and viewing angles

Pass Criteria:

Status

Mean Error

Max Error

PASS

< 5%

< 15%

ACCEPTABLE

< 5%

>= 15%

FAIL

>= 5%

any

Regression Tests

Regression tests verify that simulation outputs match theoretical predictions from GRB afterglow theory. Located in validation/regression/.

Shock Dynamics Tests:

Validates power-law scaling relations Q ~ t^alpha for:

  • Lorentz factor (u)

  • Radius (r)

  • Magnetic field (B)

  • Particle number (N_p)

Across three evolutionary phases:

  • Coasting: Free expansion (Gamma ~ const)

  • Blandford-McKee: Self-similar relativistic deceleration

  • Sedov-Taylor: Non-relativistic regime

Characteristic Frequency Tests:

Verifies time evolution of:

  • Injection frequency (nu_m)

  • Cooling frequency (nu_c)

  • Self-absorption frequency (nu_a)

  • Maximum synchrotron frequency (nu_M)

Spectral Shape Tests:

Checks power-law spectral indices beta = d(log F)/d(log nu) across five regimes:

  • Regime I: nu_a < nu_m < nu_c (slow cooling)

  • Regime II: nu_m < nu_a < nu_c

  • Regime III: nu_a < nu_c < nu_m (fast cooling)

  • Regime IV: nu_c < nu_a < nu_m

  • Regime V: nu_c < nu_m < nu_a

Tolerances:

  • Shock dynamics: 0.1

  • Characteristic frequencies: 0.1

  • Spectral shapes: 0.15

Validation Reports

The validation runner generates a comprehensive PDF report (validation/comprehensive_report.pdf) containing:

  1. Title Page: Version info, commit hash, platform details

  2. Table of Contents: Clickable navigation with page numbers

  3. Benchmark Section:

    • Guide pages explaining how to interpret results

    • Overview plots (on-axis and off-axis configurations)

    • Convergence summary grid with color-coded pass/fail status

    • Detailed convergence plots for each configuration

  4. Regression Section:

    • Guide pages with theoretical background

    • Summary grid comparing measured vs expected power-law exponents

    • Detailed diagnostic plots for shock dynamics and frequencies

Directory Structure

validation/
├── run_validation.py          # Unified CLI runner
├── comprehensive_report.pdf   # Generated PDF report
├── benchmark/
│   ├── benchmark_suite.py     # Benchmark test implementation
│   ├── configs.py             # Test configurations
│   └── results/               # JSON output files
├── regression/
│   ├── run_regression.py      # Regression test runner
│   └── results/               # JSON output files
└── visualization/
    ├── dashboard.py           # PDF report generator
    ├── common.py              # Shared utilities
    ├── benchmark_plots.py     # Benchmark visualizations
    ├── regression_plots.py    # Regression visualizations
    └── guides/                # Markdown guide documents

Profiling Per-Stage CPU Cost

When installed with validation support (see above), per-stage CPU profiling is automatically enabled. The benchmark overview page will show a stacked bar chart breaking down CPU time by internal C++ computation stage:

Profiling data is also accessible directly from Python when built with profiling:

import VegasAfterglow as ag

model = ag.Model(jet=jet, medium=medium, observer=obs, fwd_rad=rad)

# Reset, run, and retrieve stage timing
ag.Model.profile_reset()
result = model.flux_density(t, nu)
print(ag.Model.profile_data())
# {'total': 6.4, 'shock_dynamics': 5.1, 'syn_photons': 0.4, ...}

To return to the normal (zero-overhead) build:

pip install -e .

CI/CD Integration

The validation framework is designed for CI/CD pipelines:

# Example GitHub Actions workflow
- name: Run validation
  run: python validation/run_validation.py --all

- name: Upload report
  uses: actions/upload-artifact@v3
  with:
    name: validation-report
    path: validation/comprehensive_report.pdf

The validation runner:

  • Returns exit code 0 only if all tests pass

  • Generates reports even when tests fail (for debugging)

  • Supports parallel execution for faster CI runs

  • Produces machine-readable JSON results