You pose an interesting question, dt. But I don't see a conflict in his behavior, or even a sharp transition:
For dramatic effect, Frankie was a bit broadly portrayed at the party. I think the buffoonish behavior was FFC's way of contrasting Michael's "swell" and "legitimate" pose at Anthony's party with a stark reminder of where his roots were. It was, in effect, another example of Michael's struggle to bridge two worlds--the estates of Nevada and the streets of New York. Frankie was also there to emphasize another Michael transition--trusting Roth rather than "your own kind."
Frankie was acutely aware of his own limitations--"I, I don't have your head for the big deals..." That's probably one of the reasons Michael chose him to succeed Clemenza. But it didn't mean he was dumb. I was pleased, but not surprised, that he knew the jig was up as soon as he saw his brother sitting next to Michael. His history reading wasn't a surprise either--he had a lot of time on his hands on that Air Force base. And, reading history told him what he needed to do to make sure his family was taken care of. It's among the many reasons why Frankie is my favorite character in the Trilogy.