Good books to choose good problemss
There are many good math book originally published in Russia.
Sergey Dorichenko: A Moscow Math Circle: Week-by-Week Problem Sets (MSRI Mathematical Circles Library)
Boris A. Kordemsky: The Moscow Puzzles: 359 Mathematical Recreations (Dover Recreational Math)
Perhaps the best topic of high school math research is combinatorial game theory. It seems that our group is the first to do research on combinatorial game as a group of high school students, but there are going to appear some high school group of students in the U.S. are beginning to do research on combinatorial game under the guidance of some mathemticians.
L R Haff and W J Garner:An Introduction to Combinatorial Game TheoryMay 2, 2018
Richard Nowakowski and David Wolfe:Lessons in Play: An Introduction to Combinatorial Game TheoryJul 2, 2007
If you really want to discover new things, it is better not to choose very famous problmes such as Rieman conjecture, 3m+1 problem, etc. It is also not good to study very popolar theme such as Fibonacci sequences. These famous problems and these popular themes have been studied by many mathematicians, and it is very difficult to discover new things.