Pottery fragments from prehistoric settlements have provided evidence that people used these as sieves for making cheese, which was one of the early forms of dairy that people could digest.  Most people were lactose intolerant at the time:

During the most recent ice age, milk was essentially a toxin to adults because — unlike children — they could not produce the lactase enzyme required to break down lactose, the main sugar in milk. But as farming started to replace hunting and gathering in the Middle East around 11,000 years ago, cattle herders learned how to reduce lactose in dairy products to tolerable levels by fermenting milk to make cheese or yogurt.

The article goes on to say that European man was later able to drink milk into adulthood through a “genetic mutation” which happened thousands of years later.  Also,  something of which I was unaware:

Young children almost universally produce lactase and can digest the lactose in their mother's milk. But as they mature, most switch off the lactase gene. Only 35% of the human population can digest lactose beyond the age of about seven or eight

The map on the site shows a higher percentage of people from northern Europe as having the ability to drink milk into adulthood through the ability to produce lactase.  I guess that since my ancestry originated from that area, I also have that ability.  I didn't realize so many people could not drink milk.