Interactive Methods Workflow

Digital Twin + Reinforcement Learning
Step-by-Step Pipeline

An interactive walkthrough of the six-stage methodology for personalised adaptive chemotherapy dosing — from digital twin construction to safety-validated RL dose optimisation.

① Digital Twin
② PBPK Model
③ RL Formulation
④ SAC Algorithm
⑤ Safety Architecture
⑥ Virtual Patients
1
2
3
4
5
6
Step 1 of 6 — Digital Twin Architecture
🧬
Step 1
Digital Twin Architecture
Three-subsystem patient-specific computational model
▼

The Digital Twin (DT) is a patient-specific computational model comprising three tightly coupled subsystems that mirror real-time patient physiology and tumour dynamics.

PBPK Gompertz ODE Haematological Model Patient-Specific
I

Multi-Compartment PBPK Engine

Governs drug distribution and elimination across plasma, tumour, liver, kidney, and peripheral tissue compartments.

II

Gompertz Tumour Growth ODE

Represents tumour growth dynamics modulated by drug effect — capturing both intrinsic growth and pharmacodynamic kill.

III

Haematological Toxicity Model

Tracks absolute neutrophil count (ANC) as a proxy for bone marrow suppression — the primary safety constraint.

💡
Key insight: The three subsystems are tightly coupled — drug concentration from PBPK feeds into both the tumour ODE (pharmacodynamic kill) and the ANC model (toxicity), enabling realistic patient simulation before any real-world administration.
💊
Step 2
Physiologically-Based Pharmacokinetic (PBPK) Model
Five-compartment drug disposition with Gompertz pharmacodynamics
▼

Drug disposition is described by a five-compartment PBPK model. Mass-balance ODEs govern drug transport between compartments according to blood flow rates, tissue volumes, and partition coefficients calibrated to literature-reported values.

5 Compartments Mass-Balance ODEs Emax PD Model Calibrated Parameters
dV/dt = r·V·ln(K/V) − E(Ctumour)·V
E(C) = Emax·C / (EC50 + C)
Gompertz growth ODE with Emax pharmacodynamic kill term. V = tumour volume (cm³); r = intrinsic growth rate; K = carrying capacity; C = drug concentration.
3.0 L
Plasma Volume (Vp)
0.18 h⁻¹
Elimination Rate (kel)
0.012 day⁻¹
Gompertz Growth (r)
1.2 µg/mL
EC50
100 cm³
Carrying Capacity (K)
0.045–0.050
Emax Range (day⁻¹)
⚗️
Calibration: All PBPK parameters were calibrated against literature-reported values for representative antineoplastic agents, with patient-specific Emax values fitted to individual tumour response profiles.
🎯
Step 3
Reinforcement Learning Formulation
Constrained Markov Decision Process (CMDP) design
▼

The dosing optimisation problem is formulated as a Constrained Markov Decision Process (CMDP). The state space, action space, reward function, and safety constraint are carefully designed to balance tumour kill efficacy with patient safety.

CMDP 6D State Space Continuous Action Composite Reward ANC Constraint
State Vector — 6 Dimensions
V(t)
Tumour Volume
[0, 100] cm³
Cp(t)
Plasma Drug Conc.
[0, 50] µg/mL
ANC(t)
Absolute Neutrophil Count
[0, 8000] cells/µL
d(t)
Day of Cycle
[1, 21] days
Dcum
Cumulative Dose
[0, ∞) mg
ΔV/Δt
Tumour Rate of Change
cm³/day
Composite Reward Function
R(s,a) = w₁·ΔV_kill − w₂·max(0, ANC_thresh − ANC) − w₃·a/D_max − w₄·𝕀[ANC < ANC_thresh]

+ Tumour Kill

Positive reward for tumour volume reduction per step.

w₁ = 1.0

− ANC Penalty

Negative reward when ANC drops below safety threshold.

w₂ = 0.5

− Dose Efficiency

Proportional penalty for excessive dosing beyond therapeutic need.

w₃ = 0.1

− Hard Constraint

Large penalty for any ANC safety constraint violation.

w₄ = 100
⚠️
Safety constraint: ANC ≥ 1,500 cells/µL must be maintained at all times. Violations trigger a −100 reward penalty and automatic dose suspension — regardless of RL agent output.
🤖
Step 4
Soft Actor-Critic (SAC) Algorithm
Maximum-entropy RL for robust continuous-action dosing
▼

SAC was selected as the primary RL algorithm based on its maximum-entropy objective, which promotes robust exploration, and its demonstrated superiority in continuous-action domains — critical for precise dose recommendation.

Max-Entropy RL Continuous Action Space Actor-Critic Gaussian Policy Auto-Temperature α
J(π) = Σt E(s,a)~π [ R(st, at) + α·H(π(·|st)) ]
SAC maximum-entropy objective. α = temperature parameter (auto-tuned) controlling exploration-exploitation balance; H = policy entropy term encouraging diverse action selection.
Closed-Loop Dosing Cycle
📡
Stage 1
State Observation
→
🧠
Stage 2
Policy Evaluation
→
🛡️
Stage 3
Safety Validation
→
💉
Stage 4
Dose Administration
→
🔄
Stage 5
Policy Update
[0, 150]
Action Space (mg)
Gaussian
Policy Distribution
Auto
Temperature α Tuning
Dual Q
Critic Architecture
🔬
Why SAC? Compared to DDPG, TD3, and PPO, SAC's entropy maximisation prevents premature policy collapse in the sparse-reward medical dosing domain, enabling the agent to discover diverse safe dosing strategies across heterogeneous patient profiles.
🛡️
Step 5
Three-Tier Safety Architecture
Hierarchical fail-safe system preventing unsafe dose administration
▼

A hierarchical safety system operates at three levels to prevent unsafe dose recommendations from reaching the virtual patient. Each tier provides an independent layer of protection, operating sequentially before any dose is administered.

Pharmacological Haematological Protocol-Level Fail-Safe
1

Pharmacological Layer

Hard dose bounds enforced by action clipping to [0, 150 mg]. No RL output can exceed this absolute pharmacological limit regardless of policy output.

2

Haematological Layer

ANC-based dose hold — if predicted ANC falls below 1,500 cells/µL, dose is automatically set to zero regardless of RL output. Protects against neutropenia.

3

Protocol Layer

Minimum inter-dose interval (48 h) and maximum cycle dose cap enforced by rule-based override. Mimics real-world clinical treatment protocols.

🛡️
Clinical relevance: The three-tier architecture mirrors real-world oncology protocols — pharmacological limits from drug labelling, haematological monitoring from clinical guidelines, and cycle scheduling from standard-of-care protocols.
👥
Step 6
Virtual Patient Cohort
Three heterogeneous patient profiles for simulation validation
▼

Three virtual patient profiles were defined to represent the spectrum of clinical phenotypes encountered in practice — from high-burden average responders to low-burden favourable responders with varying pharmacokinetic clearance profiles.

3 Patients Heterogeneous PK Variable Sensitivity 180-Day Simulation
Patient Initial Tumour Vol. Emax Clearance Clinical Phenotype
P1 20 cm³ 0.045 day⁻¹ Standard High tumour burden, average responder
P2 12 cm³ 0.042 day⁻¹ High Low burden, favourable responder
P3 15 cm³ 0.044 day⁻¹ Slow Intermediate burden, partial responder
Simulation Results — AUC Tumour Burden Reduction
43.0%
P1 AUC Reduction
(1607.8 → 916.1)
42.2%
P2 AUC Reduction
(976.6 → 564.7)
43.8%
P3 AUC Reduction
(1230.8 → 692.3)
📊
Generalisation: The consistent ~43% AUC reduction across all three heterogeneous patient profiles demonstrates the robustness of the RL-DT framework — the SAC agent successfully adapts its dosing strategy to each patient's unique pharmacokinetic and tumour response profile.