Phox homology (PX) domains are membrane interacting domains that bind to phosphatidylinositol phospholipids or phosphoinositides, markers of organelle identity in the endocytic system. Although many PX domains many bind the canonical endosome-enriched lipid PtdIns3P, others interact with alternative phosphoinositides, and a precise understanding of how these specificities arise has remained elusive. We have systematically screened all human PX domains for their phospholipid preferences using liposome binding assays, biolayer interferometry and isothermal titration calorimetry. These analyses define four distinct classes of human PX domains that either bind specifically to PtdIns3P, non-specifically to various di- and tri-phosphorylated phosphoinositides, bind both PtdIns3Pand other phosphoinositides, or associate with none of the lipids tested. A comprehensive evaluation of PX domain structures reveals two distinct binding sites that explain these specificities, providing a basis for defining and predicting the functional membrane interactions of the entire PX domain protein family.