I have just noticed that YouTube have added a few small features to make everything a little bit better.

YouTube progress barHovering over a place in the progress bar will show a balloon with the time on it, which can help with finding previous times easier.