Graph Neural Network (GNN) introduces deep neural networks into graph structure data. It has achieved advanced performance in many fields such as traffic prediction, recommendation systems, and computer vision, which has received extensive attention from the academic community. Most of the existing research on graph neural networks focuses on graph convolution, while graph pooling is usually ignored. Although there are also some graph pooling methods, most of the current pooling methods are based on top-k node selection. In the top-kbased pooling method, unselected nodes will be directly discarded, which will cause the loss of feature information during the pooling process. In this paper, we propose a novel graph pooling operator, called Hierarchical Graph Pooling with Self-Adaptive Cluster Aggregation (HGP-SACA), which uses a sparse and differentiable method to capture the graph structure. Before using top-k for cluster selection, the unselected clusters and the selected clusters in the neighbor perform an n-hop feature information aggregation. The merged clusters which contain neighborhood clusters are used for top-k selection , which can enhance the function of the unselected clusters. Through extensive theoretical analysis and experimental verification on multiple datasets, our experimental results show that combining the existing GNN architecture with HGP-SACA can achieve state-of-the-art results on multiple graph classification benchmarks, which proves the effectiveness of our proposed model.