M4A1-S: Easily my favorite between the two. It does have a smaller clip, but I tend to play a stealthy role, which means I am quiet and precise. I have good aim so I am not bothered between the 10 bullet difference. The M4A1-s has 30 less bullets over-all. That kind of hurts it for me. I have to remember that I can't spam through smoke or walls, at the risk of having no ammo later.
M4A4: I am not bothered by the increased recoil. I practice controlling my spray, so it really isn't an issue. The issue is that it is so loud and clunky for my play style. Recently I've been using just the M4A4 so I can be more of a rounded player, but if there was an M4A4 and an M4A1-S side by side, I would pick up the M4A1-S.
It really is preference (as you can see from the other replies).
