The verse states, ‘In all your ways you should know him’, meaning that all a person’s actions, even tending his physical needs, should be done for the sake of Hashem to enable him to learn Torah and do mitzvas. When he does so, even his mundane actions are considered a mitzva since he is doing them in order to serve Hashem. This is true only if whenever he can he utilizes his time to learn Torah and do mitzvas, but if he squanders his free time in idle activities, he demonstrates that his mundane activities were not aimed at allowing him to learn, because when he had time he did not utilize it. According to this, not only did he lose the time he didn’t learn, but also the time he was involved in taking care of his bodily needs that could have been counted as service of Hashem, and by wasting the time he had left he showed that this time was also not used in Hashem’s service.
