Families with multiple male children with intellectual disability (ID) are usually suspected of having disease due to a X-linked mode of inheritance and genetic studies focus on analysis of segregating variants in X-linked genes. However, the genetic cause of ID remains elusive in approximately 50% of affected individuals. Here, we report the analysis of next-generation sequencing data in 274 affected individuals from 135 families with a family history suggestive of X-linked ID. Genetic diagnoses were obtained for 19% (25/135) of the families, and 24% (33/135) had a variant of uncertain significance. In 12% of cases (16/135), the variants were not shared within the family, suggesting genetic heterogeneity and phenocopies are frequent. Of all the families with reportable variants (43%, 58/135), we observed that 55% (32/58) were in X-linked genes, but 38% (22/58) were in autosomal genes, while the remaining 7% (4/58) had multiple variants in genes with different modes on inheritance. This study highlights that in families with multiple affected males, X linkage should not be assumed, and both individuals should be considered, as different genetic etiologies are common in apparent familial cases.