He says why he agrees to leave. Karen approached him alone so he could fight her off, Senjougahara came with help, he can't fight this off. If he didn't agree, i would be more scared of what she would do than what Araragi would do, even considering what happened to Karen. I should probably pray that Araragi and Senjougahara would never get into a serious argument in their love affair, that would likely end unwell for our vampire-ish boy lol.