This phase II trial tests how well enfortumab vedotin works for treating patients with squamous cell carcinoma of the penis that has spread to other places in the body (metastatic) or cannot be removed by surgery (unresectable). Enfortumab vedotin is a monoclonal antibody, enfortumab, linked to an anticancer drug called vedotin. It works by helping the immune system to slow or stop the growth of tumor cells. Enfortumab attaches to a protein called nectin-4 on tumor cells in a targeted way and delivers vedotin to kill them.