Agent-Based Hyperparameter Optimization for Gradient Boosted Trees

Why This Problem Matters in Practice

Hyperparameter tuning consumes a disproportionate amount of experimentation time, yet most tuning failures stem from recurring structural issues — not random chance. Experienced practitioners can spot these patterns, but automated optimizers only see scalar objective values.

What Is New or Different

This work reframes hyperparameter optimization as an iterative reasoning process rather than a pure search problem. Intermediate diagnostic artifacts (parameter importance, generalization gaps, plateau signals) become first-class inputs that guide subsequent decisions. Encoding this reasoning via agents enables systematic reuse of expert heuristics that are otherwise applied informally.

Scope and Limitations

The case study uses LightGBM as the sample demo, but the architecture is generic and can be applied to any ML model. The talk explicitly discusses scenarios where agent-based optimization adds limited value or introduces unnecessary complexity.

Audience Takeaways

Attendees will gain: - A blueprint for putting an LLM in any decision loop with guardrails - If you do ML: a new way to think about HPO - If you don't: a reusable pattern for agent-driven automation