This phase II trial tests the effect of autologous tumor infiltrating lymphocytes (TILs) in combination with interleukin-2 (aldesleukin) in treating patients with gastrointestinal stromal tumors (GIST) that has spread to nearby tissue or lymph nodes (locally advanced), that has come back after a period of improvement (recurrent) or that has spread from where it first started (primary site) to other places in the body (metastatic). Autologous TILs are made using the patient's own tumor cells collected from a previous surgery. Lymphocytes (a type of white blood cell) are a part of the immune system that helps the body fight infections. Lymphocytes are found in tumor tissue cells because they are working to attack the tumor. The cells from the tumor are grown in a lab to create more immune cells (lymphocytes). This may help the immune system find and destroy any remaining tumor cells. Aldesleukin is a form of interleukin-2, a cytokine made by leukocytes, that is made in the laboratory. Aldesleukin may help white blood cells and T cells regulate the immune response. Chemotherapy, such as cyclophosphamide and fludarabine, are given before receiving TIL to help kill tumor cells in the body and helps make room for the treatment. Colony-stimulating factors, such as filgrastim, may increase the production of blood cells and may help the immune system recover from the side effects of chemotherapy. Giving autologous TILs in combination with aldesleukin may be safe, tolerable, and/or effective in treating patients with locally advanced, recurrent or metastatic GIST.