Not a French native, but I thought T'en fais pas is "Don't Worry"? Since it's similar in meaning to "Ne t'inquiète pas". Would "Vas-y mollo" work better in this case? (lit. "Go there gently") Unless, of course, you were intent on the Yukkuri saying that to the context of "fuggedaboutit!"