For it to be open source it must "made available with a license in which the copyright holder provides the rights to study, change, and distribute the software to anyone and for any purpose."
If you publish a github repo with no LICENSE file that specifies an actual open source license it's all rights reserved by default.
It seems rms would agree with my definition - at least partly. From one of his GNU essays[1]:
Nearly all open source software is free software, but
there are exceptions. First, some open source licenses
are too restrictive, so they do not qualify as free
licenses. For example, “Open Watcom” is nonfree because
its license does not allow making a modified version
and using it privately.
He considers Open Watcom to be open source but nonfree, when it has a license that would not qualify as open source by your definition. I think you're using the OSI definition? I tend to think of open source software more like rms does in his essay. If I can see and read the code, the source is open for me to peruse, then I consider it open source. Might be a total legal clusterf*ck to work with, but still, I have access to the source. I don't think the issue is as clear cut as no license - all rights reserved[2].
I'm interested in what would be the name for software released on a GitHub public repo, but with all rights reserved?
RMS is focused on the four freedoms. Open Source licenses which don't provide those freedoms he considers to be nonfree.
In the extreme case, you can read the source, but all rights are reserved, as you mentioned. Java is the highest profile project that existed this way until it was finally open-sourced. You could read the source, but were not allowed to really do anything with it. As such, it was not only non-free, but also not open source.
