Prostate cancer is characterized by a high degree of heterogeneity, which poses a major challenge to precision therapy and drug development. In this review, we discuss how nongenetic factors contribute to heterogeneity of prostate cancer. We also discuss tumor heterogeneity and phenotypic switching related to anticancer therapies. Lastly, we summarize the challenges targeting the tumor environments, and emphasize that continued exploration of tumor heterogeneity is needed in order to offer a personalized therapy for advanced prostate cancer patients.