This phase I trial tests the safety, side effects and best dose of epcoritamab alone and epcoritamab with polatuzumab vedotin, rituximab, cyclophosphamide, doxorubicin, and prednisone (pola-R- mini-CHP) for the treatment of diffuse large B cell lymphoma (DLBCL) in elderly or unfit patients. Epcoritamab is a monoclonal antibody that may interfere with the ability of cancer cells to grow and spread. A monoclonal antibody is a type of protein that can bind to certain targets in the body, such as molecules that cause the body to make an immune response (antigens). Polatuzumab vedotin is a monoclonal antibody, called polatuzumab, linked to a chemotherapy drug, called monomethyl auristatin E. Polatuzumab is a form of targeted therapy because it attaches to specific molecules (receptors) on the surface of cancer cells, known as B-cell receptors, and delivers monomethyl auristatin E to kill them. Rituximab is a monoclonal antibody. It binds to a protein called cluster of differentiation antigen 20 (CD20), which is found on B cells (a type of white blood cell) and some types of cancer cells. This may help the immune system kill cancer cells. Cyclophosphamide is in a class of medications called alkylating agents. It works by damaging the cell's DNA and may kill cancer cells. It may also lower the body's immune response. Doxorubicin is in a class of medications called anthracyclines. Doxorubicin damages the cell's DNA and may kill cancer cells. It also blocks a certain enzyme needed for cell division and deoxyribonucleic acid (DNA) repair. Prednisone is in a class of medications called corticosteroids. It is used to reduce inflammation and lower the body's immune response to help lessen the side effects of chemotherapy drugs. Epcoritamab alone and epcoritamab with pola-R-mini-CHP may be safe, tolerable and/or effective in treating DLBCL in elderly or unfit patients.