Fahim, a leader of the Tajik ethnic minority, was senior vice president under President Hamid Karzai, who will step down at elections next month as NATO combat forces pull out of Afghanistan after 13 years of fighting the Taliban.
Aged 56, Fahim was accused of being a ruthless strongman who maintained his own militia forces, but he also received American support as Afghanistan struggled for stability following the fall of the Taliban regime in 2001.
"With extreme sorrow, Marshal Mohammad Qasim Fahim, the first vice president of Afghanistan, passed away due to an illness," the palace said in a statement.
It said that President Karzai expressed his deep condolences over the "uncompensable loss" of "a patriot and a great mujahid (holy warrior)".
Three days of national mourning will begin on Monday, with the national flag flown at half-mast across the country.
Fahim served as a influential but brutal intelligence officer under iconic rebel commander Ahmad Shah Massoud, fighting against the Soviet occupation in 1980s and against the Taliban's 1996-2001 regime.
Massoud was killed by an al-Qaida bomb two days before the 9/11 attacks in the United States in 2001.
The Taliban were then ousted in a US-led operation for sheltering al-Qaida — and Fahim emerged as a leading figure in Afghan politics during the international military and civilian intervention that is now winding now.
With the strong backing from Washington, Fahim was appointed to the key position of defence minister in 2002, where he was criticised for appointing mainly Tajik senior officers and failing to build a representative national army.
He also served as a vice president from 2002-2004, but was removed by Karzai from both jobs in 2004 in one of the many power struggles that have undermined efforts to develop the war-torn and impoverished country.
Fahim returned as senior vice-president in the 2009 election as Karzai sought to broaden his support base and build alliances.
At the time, the Human Watch Rights group said it was "appalled" to see the former warlord selected by Karzai's government.
"Fahim's critics dismissed him as a semi-literate, self-appointed field marshal, and one of the principal obstacles to Afghan unity because of his alleged ruthless threats, beatings and general thuggery," according to the Essential Field Guide to Afghanistan released this year.
Fahim was famous for his love of "buzkashi", the violent and popular Afghan sport in which horseriders wrestle over a dead calf and try to drop the animal into a chalk circle.
He maintained a large stable of buzkashi horses, and retained loyal support in the Panjshir valley where he was raised and where the sport is closely followed.
He also owned extensive property in Kabul, which he was often accused of obtaining through extortion or corruption.
Fahim died from complications caused by diabetes and heart problems, according to officials who declined to be named.
His death is likely to have limited impact on the April 5 election to choose a successor to Karzai, who is constitutionally barred from standing for a third term in office.