Well, in terms of losing your userbase one after another, once they figure out that your site allows people to spread their malware, of course.
As long as you don't explicitly execute uploaded files on you server, I don't think so. Haven't seen any code that does it automatically. But then again: Never say never. There's always a chance that someone figures a way to do it.
Apart from that always present risk, I'd give it a "No, it can not."
Furthermore, this depends on which OS you are running your site on. Linux systems are somewhat safe due to the simple fact, that there just is little malware for that OS. More importantly is it a matter of proper configuration. In server-land (Linux/Windows alike, though Linux again clearly has the advantage) you should always have your deamons (services) run with their own user. Giving them only the priviliges (access to files, network interfaces, what not...) they really need to do their thing. This way, worst case, your site itself would be harmed, but without having your whole machine compromised, so replaying a backup would solve the problem rather easily.