What’s clear is the song’s certainly not about the very young kids Blake Lively and Ryan Reynolds have (their oldest was born in 2014).
Thus, the James in question in the song is not specifically Ryan Reynolds or Blake Lively’s kid, and may or may not be a boy.
This also seemingly indicates it’s probable Taylor Swift just thought of James and Inez for the song because of the connection to her friends and not as an Easter egg to incorporate “Betty” and out the third daughter’s name.