Should I give him space and time after this?

Sounds like you are in one of those confusing/ complicated friendship where there are more feelings involved that are being repressed. I say give yourself time from each other... as in more than two days. I'm talking a few WEEKS. This usually helps to gather your feelings and see if you can get past the person or if you truly do need them in your life.