Perhaps Toby doesn’t want to see “her” because he knows he’d have to bring Kate, which could cause some drama between her and whoever is there.
Before we get too wrapped up in This Is Us Toby and Kate divorce theories, it should be acknowledged that this could be a red herring.
Keep in mind this is several years into the future, so there’s really no telling what family feuds and drama have unfolded over time.
Toby might be one of those guys that takes off his wedding ring when he goes to sleep, and Kate might be in the next room oblivious to the phone call taking place.