The Drúedain were considered Edain by the Elves, and especially seen as a part of the Halethrim. At the beginning of the Second Age, some of these Drúedain even accompanied the remnants of the Edain to the island of Númenor and had become Númenóreans, even if they largely stayed a tribe on their own. With the rise of the Sea-Kings and later Sauron, however, most of these Drûghic Númenóreans left Númenor, joining colonists and Elendili on the western coasts of Middle-Earth, and later retreating to the forests and mountains, eventually rejoining tribes of their old lost kinsmen. By the time of the Downfall, none remained in Númenor.
The Drughu comprised the original inhabitants of southern Gondor. The Drughu originally migrated westwards from the lands south of Mordor at the beginning of the First Age, and their large clans occupied the lowlands on both sides of the White Mountains. With the arrival of the Daen peoples, who in their own tongue referred to the Drughu as "Puchael" (Dn. "Demons"), the Drúedain found themselves subsequently driven out of the lowlands and forced into the mountain wilderness. Daen hunters and warriors often killed the Drughu out of fear — or for sport. The Dúnedain knew of this strange people only through a branch of their kindred that appeared in Beleriand as allies of the Elves and Edain. The Adan ancestors of the Dúnedain called these strange, diminutive folk the "Drúedain." While exceptionally distinct from their Second-born brethren, though, they were ultimately still Men—Hildor—just like the blessed Edain, and many perished in the struggle against Morgoth.
A remnant of these people survived the War of Wrath, and some accompanied the Edain to Númenor. As the Shadow fell upon Númenor, though, the Drughu gradually migrated back to Middle-earth with the Faithful, establishing themselves in a forest of northern Gondor which the Dúnedain named "Tawar-in-Druedain."
The greater part of the Drughu, however, had never departed their ancient homelands, though by the time of Gondor's founding (S.A. 3320) their numbers had dwindled, and they had withdrawn into the high fastness of the Ered Nimrais or the houseless wilderness of Andrast. Much later in the Third Age, when the Rohirrim settled the lands north of the White Mountains, they spoke much of the Drughu of the forest, whom they called "Woses" (Ro. "Wild Men"). By that time no other Drû clans were known to exist in the White Mountains, and so the Rohirrim never perceived that the Woses of Tawar-in-Drúedain were connected with the "Pukel-men" of Dunlending legend. Most Men in southern Gondor had also forgotten the origins of their Drúadan neighbors, and joined the Rohirrim in denying the humanity of the "Wild Men."