This phase I/II trial studies the side effects and best dose of avelumab with M6620 in treating patients with deoxyribonucleic acid (DNA) damage repair (DDR) deficient solid tumors that have spread to other places in the body (metastatic) or cannot be removed by surgery (unresectable). DDR deficiency refers to a decrease in the ability of cells to respond to damaged DNA and to repair the damage, which can be caused by genetic mutations. Immunotherapy with monoclonal antibodies, such as avelumab, may help the body's immune system attack the cancer, and may interfere with the ability of tumor cells to grow and spread. M6620 may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. Giving avelumab together with M6620 may help to control DDR deficient metastatic or unresectable solid tumors.