One way of thinking about it is to consider our current economy - both activity and infrastructure - as a baseline, then think about growth in terms of how long it would take for everything to double. Keep in mind how many hundreds or thousands of years it took for things to be built-up and paved over and populated and systematized as they are.
At 1% growth (compounding), everything doubles in 72 years. Imagine a child born into a world fully utilizing its resources, but with a similar-sized spare planet handy to fuel further growth. If growth is at 1%, during that child's natural lifetime the spare planet will be full and fully utilized; "the wall" will be reached in one lifetime.
If growth is 3%, the spare planet will be fully utilized in 24 years or so. Then, to keep things going at "historic averages", 2 more planets of similar size would be needed. Assuming there are two more planets handy, both of these will be filled up and done in about 24 more years. At which point the individual in the example will be 48, and would be likely to say "we still need 3% growth to deal with growth in the workforce - find more planets!". Assuming four more planets are conveniently at hand, then happily growth continues for another 24 years, when all 8 planets are full and more planets are needed again. And so on...
Needless to say, we don't have any other planets, and growth has put us up against the limits of many resources currently. Even if one doesn't consider the effects of our energy habits on the atmosphere and climate, and the large-scale extinctions of the other species we share the planet with, as a species our poverty may be growing faster than our wealth.
This is an interesting article on exponential growth - up to how long current rates of growth would take to require the energy of the entire galaxy to sustain (its not all that long!):
http://www.theoildrum.com/node/8155