CDK8/19 kinases, which mediate transcriptional reprogramming, have become an active target for cancer drug discovery. Several small-molecule CDK8/19 inhibitors showed in vivo efficacy and two have entered clinical trials, with no significant toxicities reported. However, Clarke et al. (eLife 2016; 5; e20722) found severe systemic toxicity associated with two potent CDK8/19 inhibitors, Cmpd3 (CCT251921) and Cmpd4 (MSC2530818), and suggested that their toxicity was due to on-target effects. Here, we compared five CDK8/19 inhibitors: Cmpd3, Cmpd4, Senexin B, 16-didehydro-cortistatin A (dCA) and 15w, in different assays. Only Cmpd4 showed striking toxicity in developing zebrafish. In cell-based assays for CDK8 and CDK19 inhibition, Cmpd3, Cmpd4, dCA and 15w showed similar low-nanomolar potency and efficacy against CDK8 and CDK19, while Senexin B was less potent. Only dCA produced sustained inhibition of CDK8/19-dependent gene expression. While toxicity of different compounds did not correlate with their effects on CDK8 and CDK19, kinome profiling identified several off-target kinases for both Cmpd3 and Cmpd4, which could be responsible for their toxicity. Off-target activities could have been achieved in the study of Clarke et al. due to high in vivo doses of Cmpd3 and Cmpd4, chosen for the ability to inhibit STAT1 S727 phosphorylation in tumor xenografts. We show here that STAT1 S727 phosphorylation is induced by various cytokines and stress stimuli in CDK8/19-independent manner, indicating that it is not a reliable pharmacodynamic marker of CDK8/19 activity. These results illustrate the need for careful off-target analysis and dose selection in the development of CDK8/19 inhibitors.