Nobody deems any amount of radiation to be a safe dose. Instead, they work under a linear no-threshold model where 50 millirems each might cause 10 cancers in N million people exposed, while 200 millirems each will cause 40 cancers in the same population - absorbed dose is supposed to be directly proportional to harm in large groups of people.
There are suspicions that this is maybe too conservative, and that we have repair mechanisms for small quantities of radiation that don't exist for larger quantities, but this directly contradicts the establishment line on the matter, and would be impossible to ethically test.
Placing limits on workplace or general exposure would not represent "safety", but merely a threshold at which regulatory mechanisms kick in. Per http://www.nrc.gov/images/about-nrc/radiation/factoid2-lrg.g... , background dose aboveground is about 310mrem/yr, and we average about twice that when taking into account human activities (medical imaging and radon mostly, I would expect).
There are populated places on earth with high, naturally occurring background radiation, one such place is in Iran, what have been the long term health effects on people in places like that? I imagine studies have been done.
Have a read up on radiation hormesis. It looks like small doses might be beneficial. The linear no threshold model is all fine and good at high doses, but low down the data is poor. http://www.ncbi.nlm.nih.gov/pubmed/15673519
As effect size goes down, statistically significant data gets harder and harder to collect and control effectively. We have tiny amounts of research funding, and even if we took in seven billion people as a dataset, there is some non-negligible dose below which this ceases to be statistically significant.
"a worker at Los Primos is exposed to about 300 millirem per year (100 per year is deemed the highest "safe" dose)"
should be "5000-10000 per year is deemed the highest safe dose".