Pre-Training Large Memory Language Models with Internal and External Knowledge