Abstract: Automatical grouping algorithm on large-scale image set ,which is an important part of the scene reconstruction system, can help users organize the image set contents quickly. A hierarchical image grouping algorithm based on bag-of-words(BOW) was proposed. Firstly, each image is projected to a superhigh dimensional visual word vector by a multiple paths quantization (MPQ) method, and this step is so-called coarse grouping. Then, feature matching is carried out in every divided group and an affine invariant constraint is proposed to get rid of the incorrect matching features. This step is so-called refined grouping which can improve image grouping accuracy. The precision-recall curves show that the refined grouping can obviously improve the accucy of coase grouping ,and better grouping accucy can be achieved when using constraints.