Ryan Seddon makes clear the potential performance problem with cross-page View Transitions:
… on a slow or spotty network, the transition may appear as if the screen is freezing, as the browser waits for the page to load before it can transition smoothly between the two screens—this is not ideal.
But also that our new friend the Speculation Rules API is a potential remedy:
Combining these two helps mitigate the original tradeoff of the “pause” between navigations while the browser loads the next document. With speculative prerender, it can render the page before the user clicks, making the transition near-instant.
Both these APIs are Chrome’n’friends only, so I guess it’s a you-break-it you-fix-it deal. They are also both progressive enhancements so no grave harm in using them now, unless you consider potentially unused pre-renders too wasteful.