Besides the body style, there are about 50 chapters of improvements between the 2003 and 2004 model years in the 2004 New Car Features manual.
In addition, there were only about 53,000 2001-2003 Prius imported into North America compared to over a million 2004-2009 Prius so it's easier to find parts and mechanics for the 2004-2009. (Not that I've had any problems with my 2001, but I like my 2004 better and would recommend buying one of them before getting a 2001-2003 Prius).
My 2001 gets about 48 mpg while the 2004 has a lifetime average of 57 mpg over 115,000 miles (mostly my wife drives the 2001 and doesn't keep a logbook).
2004 Prius MPG from the logbook. (Complete years only):
2003-2004 -- 50.8 mpg 17,628 miles
2005 -- 52.6 mpg 14,688 miles
2006 -- 56.3 mpg 16174 miles
2007 -- 57.3 mpg 18384 miles
2008 -- 59.9 mpg 21755 miles
2009 -- 61.4 mpg 16177 miles
2010 -- 65.2 mpg 12134 miles
www.ctvbc.ctv.ca