“If you have to ask…”
The anecdote is legendary: a financier asks J.P. Morgan what it costs to maintain his yacht. Morgan, with a flick of his cigar, famously replies, “If you have to ask, you can’t afford it.” For over a century, this phrase has been canonized as the ultimate status symbol of the ultra-wealthy. It suggests that once one reaches a certain level of fiscal arrival, the pedestrian mechanics of accounting, budgeting, and performance tracking become beneath them.
However, in the modern era, this cultural disdain for granular financial detail has mutated from a badge of honor into a silent, systematic destroyer of dynastic wealth. After spending 27 years founding and leading TIGER 21—a premier global network for the world’s most successful entrepreneurs—and subsequently managing my own family office, I have observed a sobering truth: the arrogance of “not asking” has migrated from the maintenance of yachts to the management of investment portfolios.
The Disconnect: Building Wealth vs. Managing Wealth
To understand why so many family offices underperform, one must first distinguish between the two distinct disciplines of wealth creation and wealth management.
Successful entrepreneurs are, by definition, masters of concentrated conviction. They identify a market inefficiency, marshal extraordinary resources, and accept asymmetric risks to build an enterprise. They possess an intimate, granular understanding of every lever within their business. They know their margins, their churn rates, and their competitive moats.
Yet, once a liquidity event occurs—the sale of that business—these same individuals often enter the global capital markets with a surprisingly passive approach. They transition from being the "architect" of their own fortune to being a "client" of the global financial apparatus. They hand their capital over to a constellation of consultants, private equity managers, and investment committees, often losing the very rigor that made them wealthy in the first place.
The result is a phenomenon I call the "Performance Gap." While these entrepreneurs might have achieved double-digit returns through concentrated business ownership, the average family office portfolio, after accounting for fees, complexity, and diversification, often compounds at a mere 6% to 7% annually.
The Chronology of Compounding Erosion
To appreciate the gravity of this performance gap, one must look at the unforgiving mathematics of compounding over a single generation—roughly 30 years.
Consider a baseline of $1 million. If that capital is invested in the broad U.S. equity market, which has historically returned approximately 10% annually, that $1 million grows to roughly $17.5 million over three decades. Conversely, that same $1 million, compounded at the typical family office rate of 6.5%, grows to only $5.74 million.
The cost of this "management" is not merely the fee paid to the consultant; it is the $11.76 million in forfeited growth.
The Evolution of the Family Office
- The Liquidity Event: The entrepreneur exits their business. Their wealth is at its peak of potential, fueled by the 20%+ returns associated with active business building.
- The Institutionalization Phase: The family creates a formal structure, hiring external advisors. The focus shifts from "creation" to "preservation."
- The Diversification Trap: Seeking to avoid the "risk" of concentration, the family office adopts a broad asset allocation strategy, often ignoring the reality that excessive diversification is simply a hedge against ignorance.
- The Stagnation Plateau: After several cycles, the portfolio settles into a 6–7% growth pattern, significantly lagging the broader indices and failing to account for the actual risks taken.
Supporting Data: Why "The Scoreboard" is Often Wrong
In the world of professional sports, Billy Beane’s "Moneyball" revolution succeeded because it identified that the traditional metrics—batting average, RBIs, and home runs—were often poor predictors of winning. Instead, they focused on on-base percentage and efficiency.
Wealth management is in desperate need of its own Moneyball moment. Currently, most family offices measure success by the wrong variables: asset allocation, manager reputation, or access to exclusive "club" deals. These are, at best, vanity metrics.
Surveys across the industry consistently show that family office performance fluctuates wildly, tethered to the whims of the market. In a bull market, they report 15% returns and feel brilliant; in a correction, they report a fraction of a percent or a loss and blame "macro conditions." This volatility obscures the reality: they are rarely beating the market over a full cycle, yet they are paying significantly higher fees to attempt to do so.
Official Perspectives on Governance
The consensus among institutional wealth advisors and family office consultants is that the failure is rarely due to the selection of a specific stock or fund, but rather a failure of governance.
According to guidelines from the Securities and Exchange Commission (SEC) and the Financial Industry Regulatory Authority (FINRA), transparency in fee structures and investment mandates is the legal standard. However, legal compliance is not the same as strategic excellence. Many families, in an effort to maintain privacy or control, create opaque reporting structures that hide the true "cost" of their management.
True governance requires asking the uncomfortable questions:
- What is the objective? Is it absolute growth, inflation protection, or multi-generational preservation?
- What is the cost of complexity? Every additional layer of management adds a fee, increases administrative drag, and dilutes the potential for alpha.
- Who is accountable? If the portfolio underperforms the S&P 500 for five consecutive years, is there a mechanism to pivot?
Implications: A New Discipline for the Ultra-Wealthy
The "Morgan Mentality"—the idea that one shouldn’t bother with the numbers—is a relic that no longer serves the modern family office. To survive and thrive in a world of high-velocity capital, wealthy families must adopt a new, rigorous discipline.
1. Multi-Horizon Performance Tracking
Stop measuring your portfolio solely by annual calendar returns. Implement a dashboard that tracks performance across three horizons: short-term liquidity needs, medium-term growth objectives, and long-term legacy goals. Furthermore, categorize assets not just by class (e.g., "stocks vs. bonds"), but by "liquidity and risk tiers." This reveals whether you are being compensated for the illiquidity premium you are accepting.
2. Acknowledge the Skills Gap
The skills required to manage a fortune are fundamentally different from those required to create one. The entrepreneur is an operator; the family office must be an allocator. If you are not an expert allocator, you must find one who is—not merely a salesman who can sell you a "private" opportunity.
3. Define the Purpose
Before the next quarterly investment committee meeting, the family must answer the "Why." If the family office is designed for capital preservation, then the 6% return is acceptable. But if the objective is wealth growth, then the current performance is an indictment of the strategy. You cannot optimize for two conflicting goals simultaneously.
4. Measure What Matters (The "Moneyball" Mandate)
If you do not measure your performance against a realistic benchmark, you are flying blind. Businesses understand that "what gets measured gets managed." By consistently tracking your returns against the broad index, you will quickly see if your expensive consultants and complex strategies are actually adding value or merely eroding your capital through fee leakage.
Conclusion: The Cost of Silence
The tragedy of the modern family office is not that it fails, but that it fails while believing it is succeeding. By hiding behind the myth of the "successful investor" who doesn’t need to count pennies, many families are allowing their legacy to evaporate.
Measuring performance is not an act of greed; it is an act of stewardship. The choice is simple: you can either maintain the facade of the untouchable financier who never asks about the cost, or you can audit your portfolio, embrace the data, and protect the wealth you worked a lifetime to create.
The next time your family office reports your returns, do not just look at the bottom line. Ask about the costs, the risks, and the alternatives. Because if you don’t ask, you are almost certainly paying for it—more than you can afford.
