NEK2 is a conserved mitotic regulator critical for cell cycle progression. Aberrant expression of NEK2 has been found in a variety of human cancers, making it an attractive molecular target for the design of novel anticancer therapeutics. In the present study, we have identified a novel compound MBM-5, which was found to bind to NEK2 with high affinity by docking simulations study. MBM-5 potently inhibited NEK2 kinase activity in vitro in a concentration-dependent manner. MBM-5 also suppressed cellular NEK2 kinase activity, as evidenced by the decreased phosphorylation of its substrate Hec1 on S165 in a concentration- and time-dependent manner. This inhibition impeded mitotic progression by inducing chromosome segregation defects and cytokinesis failure; therefore leading to accumulation of cells with ≥4N DNA content, which finally underwent apoptosis. More importantly, MBM-5 treatment effectively suppressed the tumor growth of human gastric and colorectal cancer cells xenografts. Taken together, we demonstrated that MBM-5 effectively inhibited the kinase activity of NEK2 and showed a potential application in anti-cancer treatment regimens.