I always inferred that Fredo lived on the compound
I think more than an inference can be drawn. When Deanna is shrieking after the attempted hit she says something about them bieng "right outside MY window."
Maybe they drove up to the place because they went to a casino after the communion service so Deanna could have a couple of cocktails and Fredo a couple of cocktail waitreses.