Modeling 3D objects with realistic surface appearance is a challenging and time-demanding task. While a rough approximation of the overall object shape can be quickly modeled by an experienced modeler, or retrieved from a shape repository, endowing a 3D shape with realistic surface appearance, consisting of spatially variant fine scale geometric detail and reflectance, can be extremely time consuming. In this paper, a method that allows modelers to quickly extract a non-parametric appearance model from a single photograph and to easily apply it on various 3D shapes was presented. The extraction was assisted by a user-provided proxy, whose geometry roughly approximates that of the object in the image. A novel technique was used to align and deform the proxy shape so as to match the reference object, thereby enabling accurate joint recovery of geometric detail and reflectance. The correlations between the recovered geometry at various scales and the spatially varying appearance constitute a non-parametric model. The extracted appearance model can then be easily applied to the proxy by our normal transfer algorithm, greatly assisting in modeling detailed and realistic 3D models.