Peter can't help but think that Alec's just telling him these things to make him feel better about ruining his goal.

"You don't have to do this anymore, so why do you do it?"

Even Peter's not sure why he asks it. Maybe it's because he thinks that there's got to be some sort of goal -- some sort of objective -- that Alec's not admitting to, even if it's not the whole four hour thing like Peter thinks it is.