I've been doing something similar for the last month or so but I decided that Project Gutenberg data was fine for me since what I am producing (haiku poetry) is slightly more formal than how real people talk.
I also ran up against the problems of storing huge data structures in memory - my list of n-grams had something like 57 million entries, which I was able to manipulate into a dictionary with 9 million keys.
Unfortunately this was still too large so I turned to the Redis key-value store. I'd been somewhat dubious about the benefit of these so-called "NoSQL" databases but I have to say that Redis was perfect for storing these huge data structures and the Python libraries made the whole thing really easy to work with. I recommend you give it a try if memory issues are limiting what you can do.
I also ran up against the problems of storing huge data structures in memory - my list of n-grams had something like 57 million entries, which I was able to manipulate into a dictionary with 9 million keys.
Unfortunately this was still too large so I turned to the Redis key-value store. I'd been somewhat dubious about the benefit of these so-called "NoSQL" databases but I have to say that Redis was perfect for storing these huge data structures and the Python libraries made the whole thing really easy to work with. I recommend you give it a try if memory issues are limiting what you can do.