Venture capital funds are reaping the rewards of a booming market as returns continue to improve. But the dizzying climb in startup valuations has some worried that the market euphoria is turning irrational.

“It’s going too well. I’m not actually that smart. It can’t continue like this,” said Matt Harris, a partner at Bain Capital Ventures. Over the past 12 months, five of his portfolio companies either went public—including Flywire Corp. and AvidXchange Holdings Inc.—or have been preparing to list. Others raised private capital...

Venture capital funds are reaping the rewards of a booming market as returns continue to improve. But the dizzying climb in startup valuations has some worried that the market euphoria is turning irrational.

“It’s going too well. I’m not actually that smart. It can’t continue like this,” said Matt Harris, a partner at Bain Capital Ventures. Over the past 12 months, five of his portfolio companies either went public—including Flywire Corp. and AvidXchange Holdings Inc.—or have been preparing to list. Others raised private capital at higher valuations. Mr. Harris said he has been surprised both by the traction companies are getting and how well they are being received in public and private markets.

Liquidity and valuations jumped sharply this year for venture-backed companies after picking up pace in the second half of 2020. The value of U.S.-based venture-backed companies that went public or were acquired this year through the third quarter totaled $582.5 billion, up from $289 billion in 2020 and surpassing half a trillion dollars for the first time ever, according to the latest PitchBook-NVCA Venture Monitor report. Numerous startups once backed by venture capital have been trading at market capitalizations of double-digit billions of dollars on public markets, including trading app Robinhood Markets Inc. and software-development startup GitLab Inc., which debuted last week.

That liquidity, combined with rising valuations of still-private companies, is generating very strong returns for venture funds. The top quartile of global venture funds raised in 2018 had a net investment rate of return of 42% since inception, as of the third quarter of this year, beating out private equity and stock market benchmarks, according to Preqin Ltd. Even the lower-quartile funds are returning more than 10%, per the data.

The rise in valuations is driven by the general influx of capital in the venture market, from both traditional firms that have raised larger funds, and hedge funds and other nontraditional venture investors.

Some of the growth in valuations is surprising, especially when the valuation of a startup multiplies within months, said Kirsten Morin, co-head of global venture capital at investment firm abrdn PLC, where she invests in venture funds.

“There are such smart people making these investment decisions, who am I to question the total addressable market that they see on the horizon in the areas they are pursuing,” Ms. Morin said. She added that many in the industry have been raising caution about high valuations for years now, yet “they keep climbing.”

The unrealized portion of global venture-capital portfolios skyrocketed to $1.33 trillion by March 2021, up from $803 billion in December 2019, according to Preqin Ltd.

The increase in startup values is happening amid a backdrop of macroeconomic trends that continue to encourage risk-on investment strategies such as venture capital, said Cameron Joyce, vice president of research insights at Preqin. In particular, 10-year bond yields continue to stay low for now, which helps justify further investment in the riskier venture-capital assets, he said.

At the same time, technology companies are doing well, overall, growing revenue and attracting more customers, with many tech trends such as digital payments and e-commerce only accelerating through the pandemic, Mr. Joyce said.

Short-term increases in valuations, however, aren’t that meaningful in venture capital, which requires long periods of holding illiquid assets, said Mike Larsen, managing director at investment advisory firm Cambridge Associates.

“Are there great companies executing well and scaling rapidly? Yes. But there is also a lot of noise, and noise seldom corresponds with long-term performance. The spike in IRRs [investment rates of return] will dissipate and in a few years we’ll have a clearer picture,” Mr. Larsen said.

Scott Sandell, managing general partner at venture firm New Enterprise Associates, said he noticed that more board meetings for startups have been a breeze because of strong performance.

Earlier this year, he had a board call scheduled for an hour that ended in 26 minutes for an early-stage productivity software startup called Merge API Inc. The same company recently sent board members an update. “The update was comprehensive, transparent, and full of fantastic progress. When I finished reading it, I thought to myself that we don’t even need to have a board meeting,” Mr. Sandell said. Instead of convening a meeting, the board members got a to-do list, he said.

Mr. Harris of Bain Capital Ventures echoed the sentiment in terms of traction in his own portfolio.

Joelle Kayden, founder and managing partner of Accolade Partners.

Photo: Accolade Partners

“I don’t think everything is working. But more things are working than I’ve experienced in my career. And what ‘working’ looks like—how high is up—is unprecedented,” he added.

The very success of venture capital, however, is already leading to problems.

Investing in new deals is getting harder for venture investors. Competition and valuations rise in tandem, diminishing the amount of equity each investor can grab. U.S. startups raised $82.8 billion in the third quarter, a 73% increase over the year-ago investment amount and exceeding many annual totals of prior years, according to the PitchBook-NVCA report.

Overcapitalized startups, meanwhile, are burning through their cash by competing with each other for talent and customers, said Joelle Kayden, founder and managing partner of Accolade Partners, a fund of funds that invests in venture funds, including Accel and Andreessen Horowitz.

At the same time, the stock market has turned jittery in recent weeks. The total value of venture-backed initial public offerings in the most recent quarter declined from the prior quarter, the PitchBook-NVCA data show, though was still well above all other quarterly totals in the past decade.

For now, Ms. Kayden said, she can’t complain. Accolade, founded in 2000, has had the best 18 months in its history in terms of its own fundraising and the performance of the funds in its portfolio. “It feels a little surreal,” she added.

Write to Yuliya Chernova at yuliya.chernova@wsj.com