Up the Line by Robert Silverberg
Being a Time Courier was one of the best jobs Judson Daniel Elliott III ever had. It was tricky, though, taking group after group of tourists back to the same historic event without meeting yoruself coming or going. Trickier still was avoiding the temptation to become intimately involved with the past and interfere with events to come. The deterrents for any such actions were frighteningly effective. So Judson Daniel Elliott played by the book. Then he met a lusty Greek in Byzantium who showed him how rules were made to be broken...and set him on a family-history-go-round that would change his past and his future forever!