He hailed from Donghai 東海 (modern Tengxian 滕縣, Shandong) and studied the Gongyang Commentary 公羊傳 to the Confucian Classic Chunqiu 春秋 "Spring and Autumn Annals". With this educational background he was made court gentleman (lang 郎), then aide (changshi 長史) of the Prince of Chu 楚.
After a short period of leave he became rectifier (sizhi 司直) of the Counsellor-in-chief (chengxiang 丞相). Recommended by Shi Dan 師丹 he was made Chamberlain for Law Enforcement (tingwei 廷尉), then regional inspector (cishi 刺史) of Qingzhou 青州, and governor (taishou 太守) of the commandery of Runan 汝南 and then Jiujiang 九江.
In 2 BCE he entered the central government as Chamberlain for attendants (guangluxun 光祿勳) and General to the right (you jiangjun 右將軍). A year later he succeeded Kong Guang 孔光 in the office of Grand Minister of Education (dasitu 大司徒). In 1 CE Ma Gong was given the title of Marquis of Fude 扶德侯, in 5 CE he was made Minister of War (dasima 大司馬).
He requested to retire when he was involved in a dispute between the powerful minister Wang Mang 王莽 and Emperor Ping 漢平帝 (r. 1 BCE-5 CE) about the posthumous title of Empress Dowager Fu 傅太后. When Wang Mang usurped the throne of the Han dynasty, Ma Gong acceded to take over the post of Grand Mentor (taifu). He died in office.