DNA methyltransferases (DNMTs) are important epigenetic regulatory enzymes involved in gene expression corresponding to many diseases including cancer. As one of the major enzymatically active mammalian DNMTs, DNMT3A has been regarded as an attractive target for the treatment of cancer particularly in hematological malignancy. Discovery of promising inhibitors toward this target with low toxicity, adequate activity and target selectivity is therefore pivotal in the development of novel cancer therapy and the inhibitory mechanism investigation. In this study, a multistep structure-based virtual screening and in vitro bioassays were conducted to search for potent novel DNMT3A inhibitors. Compound DY-46 was then identified as a promising new scaffold candidate (IC50 = 1.3 ± 0.22 μM) that can occupy both the SAM-cofactor pocket and the cytosine pocket of DNMT3A. Further similarity searching led to the discovery of compound DY-46-2 with IC50 of 0.39 ± 0.23 μM, which showed excellent selectivity against DNMT1 (33.3-fold), DNMT3B (269-fold) and G9a (over 1000-fold). These potent compounds significantly inhibited cancer cell proliferation and showed low cytotoxicity in peripheral blood mononuclear cells. This study provides a promising scaffold for the further development of DNMT3A inhibitors, and the possibility to design proper analogs with broad or specific selectivity.