Field stones, vertical wooden boards, horizontal wooden boards, wattle and daub or plaster with nice thick wood beams, granite or limestone blocks for the exterior walls. Various tiles for the roof.
How about just for S&Gs you throw in a crane's nest for the roof?
The way you combine a stone lower level with a plaster upper level in some of the choices for textures is good . Field stones and granite or limestone blocks would look good below the plaster. A whole building made of granite or limestone blocks would be impressive, especially for a library.