This study explored the effect of ubiquitin-specific protease 4 (USP4) on mitochondrial dynamics in esophageal squamous cell carcinoma (ESCC). USP4 and phosphoglycerate mutase 5 (PGAM5) expression in ESCC tissues was measured. ESCC cells were subjected to gain- and loss-of-function experiments, followed by examinations of proliferation, invasion, migration, apoptosis, light chain 3 (LC3), P62, Bcl2-associated X (Bax), B-cell lymphoma 2 (Bcl2), Cytochrome c (CytC), caspase3, mitochondrial membrane potential (MMP), mitochondrial reactive oxygen species (mtROS), and ROS. An in vivo tumor formation model of ESCC was established for in vivo verification. The relationship between USP4 and PGAM5 was analyzed. USP4 and PGAM5 expression was high in ESCC tissues. Mechanistically, USP4 eliminated K48-linked ubiquitin chains to maintain PGAM5 stability. PGAM5 knockdown impaired malignant behaviors of ESCC cells, reduced LC-3I-to-LC-3II conversion, increased mtROS, ROS, and P62 levels, activated the caspase-dependent mitochondrial pathway, and decreased MMP. Overexpressing USP4 increased MMP and Bcl2 expression while lowering Bax, CytoC, cleaved caspase3, and mtROS levels in ESCC cells, which was nullified by PGAM5 knockdown. Ectopic USP4 accelerated tumor growth in mice, which was reversed by PGAM5 knockdown. Collectively, USP4 promotes cell growth and excessive mitophagy in ESCC by stabilizing PGAM5.