Taylor Swift and his ex-Boyfriend Conor Kennedy enjoyed romantic weekends