The invention relates to a spectrum pool system optimization method based on an optimal wavelet filter, specifically comprising the steps: step 1, setting an initial value; step 2, creating a waveletfilter; step 3, comparing symbol rates of the randomly generated wavelet packet tree structure and Mallat structure , wherein the symbol rates of the two structures are equal, and returning to the step 2 if not equal; step 4, coding; step 5, using the initial filter and the objective function value thereof as the current solution, and putting the solution into a taboo table; step 6, checking the taboo table, and regenerating a new filter in the neighborhood; step 7, comparing the symbol rates of the newly generated wavelet filter structure and the Mallat structure, updating the taboo table ifthe symbol rates of the two structures are equal, and returning to the step 6 if not equal; and step 8, comparing the objective function value of the new filter with the objective function value in the current solution to generate an optimal wavelet filter, and modulating the cognitive user signal by using the optimal wavelet filter to produce a smaller ICI. The invention is capable of reducing the interference affecting authorized users and optimizing the performance of the spectrum pool system.