The soil microbial diversity in the gangue accumulation area is severely stressed by a variety of heavy metals, while the influence of long-term recovery of herbaceous plants on the ecological structure of gangue-contaminated soil is to be explored. Therefore, we analysed the differences in physicochemical properties, elemental changes, microbial community structure, metabolites and expression of related pathways in soils in the 10- and 20-year herbaceous remediation areas of coal gangue. Our results showed that phosphatase, soil urease, and sucrase activities of gangue soils significantly increased in the shallow layer after herbaceous remediation. However, in zone T1 (10-year remediation zone), the contents of harmful elements, such as Thorium (Th; 1.08-fold), Arsenic (As; 0.78-fold), lead (Pb; 0.99-fold), and uranium (U; 0.77-fold), increased significantly, whereas the soil microbial abundance and diversity also showed a significant decreasing trend. Conversely, in zone T2 (20-year restoration zone), the soil pH significantly increased by 1.03- to 1.06-fold and soil acidity significantly improved. Moreover, the abundance and diversity of soil microorganisms increased significantly, the expression of carbohydrates in soil was significantly downregulated, and sucrose content was significantly negatively correlated with the abundance of microorganisms, such as Streptomyces. A significant decrease in heavy metals was observed in the soil, such as U (1.01- to 1.09-fold) and Pb (1.13- to 1.25-fold). Additionally, the thiamin synthesis pathway was inhibited in the soil of the T1 zone; the expression level of sulfur (S)-containing histidine derivatives (Ergothioneine) was significantly up-regulated by 0.56-fold in the shallow soil of the T2 zone; and the S content in the soil significantly reduced. Aromatic compounds were significantly up-regulated in the soil after 20 years of herbaceous plant remediation in coal gangue soil, and microorganisms (Sphingomonas) with significant positive correlations with benzene ring-containing metabolites, such as Sulfaphenazole, were identified.