The world-building of Outlander has enhanced the series from the beginning; unfortunately, by cramming the plot of Drums of Autumn into only 13 episodes of Season 4, some of the character journeys were lost in the world-building.
The season(s) based on Drums of Autumn would need to be fairly heavy on exposition for future seasons to have a solid base.
Unlike Claire and Jamie’s trip to the West Indies, their time in America isn’t going to last for half a book or so before they move on.