Histone deacetylase 8 (HDAC8), a pivotal epigenetic regulator orchestrating chromatin remodeling and transcriptional control, has emerged as a compelling anticancer target owing to its frequent dysregulation in malignancies. The development of isoform-selective inhibitors, however, confronts substantial challenges because of the pronounced structural conservation across HDAC isoforms. This study developed a high-efficiency drug screening pipeline primarily leveraging artificial learning (AI) models and molecular simulation techniques, and further evaluated the biological activity of computationally prioritized compounds through enzymatic inhibition assays and cellular experiments. The structurally novel compound Cmpd.16 was successfully identified, exhibiting promising inhibitory activity against HDAC8 (IC50 = 0.16 ± 0.01 μM) and moderately inhibited HCT116 cell proliferation mainly through the induction of apoptosis and G2/M phase cell cycle arrest. Subsequently, molecular dynamics simulations were employed to investigate the binding conformation of Cmpd.16 within the active pocket, and the key residues identified through free energy decomposition further elucidated the drug-target binding mode. Collectively, Cmpd.16, featuring 1H-pyrazolo[3,4-b]pyridine scaffold, was discovered via an efficient screening workflow, offering valuable insights to guide the rational design of HDAC8-targeted agents.