Use this time to make sure that your Blunderbuss is loaded. There are a few points when you can just catch up with him and kill him.

College of Winterhold - Revealing the Unseen. This will let you slip in through the break in the fence that you used to escape the grounds last time. All the noisy blasts scares de Fayet and he goes off running towards the docks.

Keep doing these thing until De Fayet reaches his destination, be ready for an all-out brawl! He will pull out his sword.