Swanson tried to resist the European excursion, but it was pointless. “When his mind was made up,” she later wrote, “there was not a big enough lever in the world to move him. I might argue all day, but I knew he would only out-argue me.” So she agreed to “throw a shawl over my scarlet letter and have tea with his wife and my husband and the vicar, doubtless, not to mention the press.”73

Rose Kennedy would always deny that her husband had anything other than a professional relationship with Swanson. According to Eunice Kennedy Shriver, her mother didn’t even hear rumors about the affair until the 1960s, and then waved them off. The claim of ignorance seems hard to believe. More likely, Rose knew about the romance while it was going on but chose to suppress it, to pretend it wasn’t there. “Mrs. Kennedy had this amazing knack for shutting out anything she did not want to know or face or deal with, and conversely of actually believing what she wanted to believe,” an employee in the household recalled. Perhaps some small part of Rose even condoned the relationship, or at least understood it, in view of what appears to have been her highly circumscribed view of what constituted proper sexual behavior for a devout Catholic—in essence, outside of procreation, intercourse should be sharply limited, in both frequency and duration—and her knowledge that it conflicted with her husband’s sex drive.74

For his part, Joe Kennedy knew that his philandering was wrong, that adultery was contrary to God’s word. But he also believed in confession and the forgiveness of sins. So he went on straying from the marriage bed. Obsessively focused on winning, on conquest, he always wanted more, more, more—in all areas of life. A journalist who knew him speculated that for Kennedy, a mistress was “another thing that a rich man had—like caviar. It wasn’t sex, it was part of the image…his idea of manliness.”75

Never, it seems, did Kennedy seriously consider leaving Rose for another woman, not even for Gloria Swanson. The actress hints in her memoirs that he contemplated forsaking Rose for her, but the notion is far-fetched.76 She was a huge catch, one of the most alluring women in the entire world, and dating her gave Kennedy prestige in Hollywood and—even more so—among his astonished Harvard friends. But he spent less time with her than he might have if the affair were as important to him as Swanson wanted it to be. For Kennedy, family was ultimately sacrosanct, even if he often had an odd and callous way of showing it. His preferred arrangement, common enough among men of his station, was having a wife at home and girlfriends away from home. By the fall of 1929 the affair had run its course, and Kennedy was readying to leave Hollywood and the film business behind—$5 million richer, thirty pounds lighter, and fighting an ulcer.77

He needed to rein himself in, he understood, needed to return to the more buttoned-up Northeast, to his wife and kids. Earlier in 1929, P. J. Kennedy had become deathly ill, at age seventy-one. (Mary Augusta had died in 1923.) Joe had spent several days by his father’s bedside in Boston in the final days; when P. J. seemed to rally, the son took it as a sign he could return to California, only for death to come on May 18. Nor did he travel back from Hollywood for the funeral (he sent Rose and Joe Junior in his place), a decision he regretted instantly. Never the introspective sort, Kennedy nonetheless understood that, having turned forty and with his father now gone, he needed to get his priorities in order, lest he lose his family. He had to be home, and home was not the wild west of glitzy young Southern California.78

And besides, even with his gargantuan work ethic, Kennedy found it hard to give sufficient attention from Los Angeles to the chattering stock ticker in New York. Now, as he gradually transitioned back to the East, he could afford to do that. He sensed big things were in the offing on Wall Street and made a major decision, one that went against the crowd and the consensus of expert opinion: he got out. Guy Currier, a well-connected and flamboyant lawyer to whom Kennedy often turned for advice, had warned him that the stock market seemed inflated, edgy, precarious, the danger signs flashing all around. Kennedy, predisposed toward pessimism, agreed, and he systematically went about liquidating much of his vast portfolio, even as the bankers and industrialists and traders around him stayed bullish. When prices began falling in September 1929, he stood at a safe distance. He remained there when the bottom fell out on October 29, Black Tuesday.79

One of Kennedy’s Harvard classmates, historian Frederick Lewis Allen, captured the moment in his classic work Only Yesterday:

The big gong had hardly sounded in the great hall of the Exchange at ten o’clock Tuesday morning before the storm broke out in full force. Huge blocks of stock were thrown upon the market for what they would bring. Five thousand shares, ten thousand shares appeared at a time on the laboring ticker at fearful recessions in price. Not only were innumerable small traders being sold out, but big ones, too, protagonists of the new economic era who a few weeks before had counted themselves millionaires. Again and again the specialist in a stock would find himself surrounded by brokers fighting to sell—and nobody at all even thinking of buying.80