You (and Mister) could tell by the arrogant look on their faces that they were implying a "...so keep out, stupid old geezer!" at the end of that line.
Also, if you think that there actually needs to be some sort of reason to justify the beating, they pretty much deserved the punishment in this case. Arrogant brats.
Well they still are Children of Yukkuri then. But they kinda deserved it, they are not cute and nice as the koremis, who are thanking nicely the Mister!