Yes that's how git works. However joey has made git annex, that uses git for versioning, but doesn't store all binary data in git, hence avoiding the exact problem you point out.