Joint sparse representation has been widely used for hyperspectral image classification in recent years, however, the equal weight assigned to each neighbouring pixel is less realistic, especially for the edge areas, and one fixed scale is not appropriate for the entire image extent. To overcome these problems, we propose an adaptive local neighbour selection strategy suitable for hyperspectral image classification. We also introduce a multi-level joint sparse model based on the proposed adaptive local neighbour selection strategy. This method can generate multiple joint sparse matrices on different levels based on the selected parameters, and the multi-level joint sparse optimization can be performed efficiently by a simultaneous orthogonal matching pursuit algorithm. Tests on three benchmark datasets show that the proposed method is superior to the conventional sparsity representation methods and the popular support vector machines.