Extremely Slow Terminal 2003 Server

I have a 2003 Terminal server with dual quad 2.33GHz proc and 32gig of memory
windows SP2. I have 20-30 users connecting to this machine daily they are
complaining that is takes for every just to log in most of the time they get
profiles can't be loaded. I have applied every fix MS suggests regarding this
issue still no luck. They all say the performance is very poor most of the
time it takes 10 min.. just for office to load. I look at the performance
monitor when they are complaining and none of the proc's are working and only
about a 1-3 gig of memory is being used? Any help would be greatly
appreciated.

Thanks,