I had this problem,   and D-link had me RMA the router,   so i sent it back,  got the new one,  upgraded to the newest firmware,. and blammo,  exact same problem,  
i turned off wirless, and its been running fine for hours,  after rebooting every 5-10 minutes.
Unfortunately this is not really a solution, as i bought this unit specifically for the wireless N.
From reading on broadbandreports forums,  this is happening to many people,  so its obviously buggy firmware.