As a member of the subfamily Caprinae of the family Bovidae, the muskox is more closely related to sheep and goats than to oxen. It has been placed in its own genus, Ovibos — Latin for sheep-ox. It is one of the two largest extant members of Caprinae, along with the similarly sized takin.
While the takin and muskox were once considered possibly closely related, the takin lacks common ovibovine features, such as the muskox's specialized horn morphology, and genetic analysis shows that their lineages actually separated early in caprine evolution.
Instead, the muskox's closest living relatives appear to be the gorals of the genus Naemorhedus, nowadays common in many countries of central and east Asia. The vague similarity between takin and muskox must therefore be considered an example of convergent evolution.
The modern muskox is the last member of a line of ovibovines that first evolved in temperate regions of Asia and adapted to a cold tundra environment late in its evolutionary history. They lived alongside our lovely Mammoths and would have competed for the same plant resources as those much larger beasts.
Muskox ancestors with sheep-like high-positioned horns — horn cores being mostly over the plane of the frontal bones, rather than below them as in modern muskoxen — first left the temperate forests for the developing grasslands of Central Asia during the Pliocene, expanding into Siberia and the rest of northern Eurasia.
Later migration waves of Asian ungulates, including the high-horned muskox, reached Europe and North America during the first half of the Pleistocene. The first well-known muskox, the "shrub-ox" Euceratherium, crossed to North America over an early version of the Bering Land Bridge two million years ago and prospered in the American southwest and Mexico. Euceratherium was larger yet more lightly built than modern muskoxen, resembling a giant sheep with massive horns, and preferred hilly grasslands.
A genus with intermediate horns, Soergelia, inhabited Eurasia in the early Pleistocene, from Spain to Siberia, and crossed to North America during the Irvingtonian (1.8 million years to 240,000 years ago), soon after Euceratherium. Unlike Euceratherium, which survived in America until the Pleistocene-Holocene extinction event, Soergelia was a lowland dweller that disappeared fairly early, displaced by more advanced ungulates, such as the "giant muskox" Praeovibos (literally "before Ovibos").
The low-horned Praeovibos was present in Europe and the Mediterranean 1.5 million years ago, colonized Alaska and the Yukon one million years ago and disappeared half a million years ago. Praeovibos was a highly adaptable animal that appears associated with cold tundra (reindeer) and temperate woodland (red deer) faunas alike.
During the Mindel glaciation 500,000 years ago, Praeovibos was present in the Kolyma river area in eastern Siberia in association with many Ice Age megafauna that would later coexist with Ovibos, in the Kolyma itself and elsewhere, including wild horses, reindeer, woolly mammoth and stag-moose.
It is debated, however, if Praeovibos was directly ancestral to Ovibos, or both genera descended from a common ancestor since the two occurred together during the middle Pleistocene. Defenders of ancestry from Praeovibos have proposed that Praeovibos evolved into Ovibos in one region during a period of isolation and expanded later, replacing the remaining populations of Praeovibos.
Two more Praeovibos-like genera were named in America in the 19th century, Bootherium and Symbos, which are now identified as the male and female forms of a single, sexually dimorphic species, the "woodland muskox", Bootherium bombifrons. Bootherium inhabited open woodland areas of North America during the Late Pleistocene, from Alaska to Texas and maybe even Mexico, but was most common in the Southern United States, while Ovibos replaced it in the tundra-steppe to the north, immediately south of the Laurentian ice sheet.
Modern Ovibos appeared in Germany almost one million years ago and were common in the region through the Pleistocene. Muskoxen had also reached the British Isles. Both Germany and Britain were just south of the Scandinavian ice sheet and covered in the tundra during cold periods, but Pleistocene muskoxen are also rarely recorded in more benign and wooded areas to the south like France and Green Spain, where they coexisted with temperate ungulates like red deer and aurochs. Likewise, the muskox is known to have survived in Britain during warm interglacial periods.
Today's muskoxen are descended from others believed to have migrated from Siberia to North America between 200,000 and 90,000 years ago, having previously occupied Alaska (at the time united to Siberia and isolated periodically from the rest of North America by the union of the Laurentide and Cordilleran Ice Sheets during colder periods) between 250,000 and 150,000 years ago.
After migrating south during one of the warmer periods of the Illinoian glaciation, non-Alaskan American muskoxen would be isolated from the rest in the colder periods. The muskox was already present in its current stronghold of Banks Island 34,000 years ago, but the existence of other ice-free areas in the Canadian Arctic Archipelago at the time is disputed.
Along with the bison and the pronghorn, the muskox was one of a few species of Pleistocene megafauna in North America to survive the Pleistocene/Holocene extinction event and live to the present day. The muskox is thought to have been able to survive the last glacial period by finding ice-free areas (refugia) away from prehistoric peoples.
Fossil DNA evidence suggests that muskoxen were not only more geographically widespread during the Pleistocene, but also more genetically diverse. During that time, other populations of muskoxen lived across the Arctic, from the Ural Mountains to Greenland. By contrast, the current genetic makeup of the species is more homogenous. Climate fluctuation may have affected this shift in genetic diversity: research indicates colder periods in Earth's history are correlated with more diversity and warmer periods with more homogeneity.