Re-opening an old thread, I know, but an interesting tidbit I just picked up seems to belong here and I will throw in a couple of other thoughts.
Until a large apartment complex was built in NYC in the 1880s, the largest structure ever built in what is now the US was in Chaco Canyon, NM. It was a 5 story (in places) free standing "fort/castle" built of sandstone around 1000 AD and abandoned before 1200 AD. There are several different theories as to why it was abandoned, but I think the weather theory makes the most sense. It had a couple of centuries of unusually wet weather and what was before and is again desert could be farmed. From about 1175-1200, there was a drought.
I think there were a couple of things that limited the size of their structures. One was their history. Many tribes were nomadic, following herds or searching for more abundant forage. The other was lack of need. Why do you make large buildings? If you don't have much room to expand. If you want fortifications. They had plenty of room and without canons, siege engines or cavalry the justification for the effort involved in walling a fortress is not that strong.