The main problem would be the drag. This won't hurt the Prius any but it will reduce mpg. Some people put the kayaks in the Prius (depending upon the kind of kayak and how much stuff is inside already). Other people tow a small trailer (under 500 lbs.) occasionally.
A good idea is to get a Scangauge (
http://www.scangauge.com/ ) so that you can watch the engine and inverter temperatures. Note that hitches for the Prius are for rack mounting so they don't have safety chain eyelets or lighting hook-ups. You'll have to get these fabricated.
So far no one has reported any damage (short or long term) and people have been towing lightweight trailers since the Prius first appeared in the US in 2000.
IMHO, the reason that Toyota doesn't set a towing limit is:
1. The suspension and brakes aren't designed for towing.
2. There is no frame reinforcement.
3. (and perhaps most importantly). If they set a 500 lbs. limit, people would tow 5,000 lbs. So by setting a 0 lbs. limit people will only tow 250 lbs. to 500 lbs and will be cautious about doing it.