I don't have a ton of mods and I know it is a mod issue as if I take the mod off, I can go right into the castle no issue. I was able to go in and out of DC to talk to The Ron.
For now, I'm going to continue to work through the Sim Settlements questline. At the beginning of the game, it was Abernathy Farm. It is definitely a cell issue, but the cell seems to move around. I've ran into the same issue with using either Vortex to load everything up or going through Fallout 4 mod's engine. Sure, so it is the following (and loaded into this order):