The title “prince” is not some random noble title.
A prince is a son of a king, and a king is a ruler.
In the new movie, we learned that part of the curse actually affected the villagers as well, as their memories were wiped of any memory of the prince or his castle.
Which, among other things, left poor Mr. Potts at loose ends, apparently for years.