He was was referring to their abrupt and unaccounted for appearance:
The phrase “Darwin’s abominable mystery” is frequently used with reference to a range of outstanding questions about the evolution of the plant group today known as the angiosperms. Here, I seek to more fully understand what prompted Darwin to coin the phrase in 1879, and the meaning he attached to it, by surveying the systematics, paleobotanical records, and phylogenetic hypotheses of his time. In the light of this historical research, I argue that Darwin was referring to the origin only of a subset of what are today called angiosperms: a (now obsolete) group equivalent to the “dicotyledons” of the Hooker and Bentham system. To Darwin and his contemporaries, the dicotyledons’ fossil record began abruptly and with great diversity in the Cretaceous, whereas the gymnosperms and monocotyledons were thought to have fossil records dating back to the Carboniferous or beyond. Based on their morphology, the dicotyledons were widely seen by botanists in Darwin’s time (unlike today) as more similar to the gymnosperms than to the monocotyledons. Thus, morphology seemed to point to gymnosperm progenitors of dicotyledons, but this hypothesis made the monocotyledons, given their (at the time) apparently longer fossil record, difficult to place. Darwin had friendly disagreements about the mystery of the dicotyledons’ abrupt appearance in the fossil record with others who thought that their evolution must have been more rapid than his own gradualism would allow. But the mystery may have been made “abominable” to him because it was seen by some contemporary paleobotanists, most notably William Carruthers, the Keeper of Botany at the British Museum, as evidence for divine intervention in the history of life. Subsequent developments in plant systematics and paleobotany after 1879 meant that Darwin’s letter was widely understood to be referring to the abrupt appearance of all angiosperms when it was published in 1903, a meaning that has been attached to it ever since.
Richard J. A. Buggs, “The origin of Darwin’s “abominable mystery”” at [publication]
Unfortunately, the article is paywalled.