There are projects (http://git-annex.branchable.com/ and https://github.com/schacon/git-media) to try and get around this issue but then you are not really using git to manage the files directly anymore.