The question is what we want: do we want videos that appeal to everyone, or videos that appeal to a large number of people? If people downvote consistently, then videos which tend to inspire either love or hate will not make the cut, since the 'hates' will effectivly cancel out the 'loves'. This also means that niche videos which appeal to people with a certain hobby but are boring to everyone else won't tend to make it to the front either.

I would prefer no downvoting myself, just because I would prefer to find one really interesting video (that may not be to everyone's taste) than to see the same ten videos that are circulating around everywhere else.