Answer:
The author begins the poem and ends it that way to emphasize the past that will not return.
Explanation:
In this poem the author emphasizes his memories that are part of the past and of his childhood.
He constantly repeats "I remember" and names various things he remembers from when he was little.
The poem ends with the phrase "but that was yesterday." and it makes us understand that those memories are already part of a past that will never come back.
That is why he chooses these words both to begin and to end his poem.
