In Ch. 14 the Giver explains that Jonas and The Giver must hold all the memories because:
"It gives us wisdom. Without wisdom I could not fulfill my function of advising the Committee of Elders when they call upon me" (Ch. 14).
Even the Elders who make all of the decisions for the community do not have these communal memories. They depend on the Giver to advise them based on his memories of what has happened before.
Ultimately, the Giver explains, "everyone would be burdened and pained" if the community held onto the memories and "they don't want that." So, one unlucky person, the Receiver of Memory, gets to be the memory bank for the entire community.
Explanation:
