Probing triple Higgs production via $4τ2b$ decay channel at a 100 TeV hadron collider

Abstract: A comprehensive study of triple Higgs boson production in the $4\tau 2b$ decay final state is performed for the first time at a future 100 TeV hadron collider. The analysis incorporates modified Higgs self-couplings via trilinear Higgs self-coupling $c_3$ and quartic Higgs self-coupling $d_4$, enabling for a model-independent investigation of potential new physics effects. Higgs bosons are reconstructed using both resolved and boosted techniques. To optimize sensitivity across different kinematic regions, we introduce a novel event categorization strategy based on the triple Higgs invariant mass spectrum and the multiplicity of boosted Higgs bosons. In addition to a traditional cut-based analysis, a Boosted Decision Tree (BDT) approach is employed to exploit multivariate correlations among kinematic observables, leading to a significant improvement in sensitivity. Our result demonstrates that the $4\tau 2b$ channel provides a viable pathway for probing the Higgs quartic coupling, complementing the existing multi-Higgs production studies, and could reach 5 $\sigma$ in significance for $c_3 \lesssim -1$ and $d_4 \gtrsim 10$ in the scanned range.