Recent advancements in deep learning have revolutionized the reconstruction of spatially-varying surface reflectance of real-world objects. Many existing methods have successfully recovered high-quality reflectance maps using a remarkably limited number of images captured by a lightweight handheld camera and a flash-like light source. As the samples become sparse, the choice of the sampling set has a significant impact on the results. To determine the best sampling set for each material while ensuring minimal capture costs, we introduce an appearance-aware adaptive sampling method in this paper. We model the sampling process as a sequential decision-making problem, and employ a deep reinforcement learning (DRL) framework to solve it. At each step, an agent (NBVL Planner), after trained on a specially designed dataset, plans the next best view-lighting (NBVL) pair based on the appearance of the material recognized so far. Once stopped, the sequence of the NBVLs constitutes the best sampling set for the material. We show, through extensive experiments on both synthetic materials and real-world cases, that the best sampling set extracted by our method outperforms other sampling sets, especially for challenging materials featuring globally-varying specular reflectance.