Automating ETL Pipeline Recovery with RL Agents

System Architecture and Logic

The system automates ETL failure remediation by separating concerns into three distinct layers: deterministic anomaly detection, a Q-learning policy for action selection, and an external safety override. When an AWS Glue job fails, an Amazon EventBridge trigger initiates a Lambda function that gathers logs from CloudWatch and schema metadata from the AWS Glue Data Catalog. The system constructs a state representation based on failure category, risk level, and data quality metrics, which the Q-learning policy uses to select one of six actions: retry, schema coercion, rollback, quarantine, escalate, or log.

Safety and Validation

The safety layer operates outside the learned policy to enforce operational constraints. If the policy proposes a passive action for a critical anomaly, the safety layer overrides it to trigger an escalation. The system treats escalation as a first-class outcome rather than a failure, recognizing the boundary of its own authority. Every proposal, override, and execution result is logged to an audit trail to ensure the system remains interpretable for human engineers. The implementation uses Q-learning specifically because the state and action spaces are small, allowing for direct inspection of Q-tables to verify why the agent chose a specific response.

Performance and Evaluation

In synthetic benchmarks across 30 controlled runs, the system achieved a mean resolution time of 5.24 minutes, compared to a manual baseline of 2.5 working days. This represents a 99.85% reduction in mean time to recovery (MTTR). The rule-based anomaly detector achieved a precision of 1.0, a recall of 0.8, and an F1 score of 0.889. The RL policy matched the performance of a hand-defined deterministic policy, demonstrating that the value of the agent lies in its ability to learn action preferences from outcomes rather than inherent superiority over simple rules.