Maybe I’m mistaken and I admittedly didn’t watch the whole video, but I’m not sure this would be possible with their technique would it? From what I can tell there’s no rigging or traditional animation, is there? At least not from the AI generated visuals.
1 Like
Great analysis from IGN here
I’m trying to avoid spoilers, but is this preview from a newer build or the same reveal trailers from earlier this year?
Na… Just the analysis of the E3 video
Nothing new there
2 Likes
Is it like the Gears 5 video by them? Because I remember that was surprisingly good…for a IGN video even more so, lol.
But I can imagine that they talk about the very inconsistent framerate here.
Don’t think so. It runs real time as of a year ago, and algorithms only get better over time. So a decade from now we should be way past that from a compute pov.
1 Like
It can do the same thing from any angle that there is footage for. The challenge would be getting those angles, but it’s certainly doable (it’s done for photogrammetry en masse already and it’s done in much more complex/cumbersome conditions for mocap).
I think inside of a decade there will be games that bypass rasterization in favor of frame-by-frame image generation using ML like in the GTA example video.
There are still fundamental issues by using sprites in a game. The rotations will be far from seamless and all of those frame data will need to be stored within memory. I also don’t think any of us can call mocap cumbersome considering it’s been an industry standard for decades now.
This is a very nice experiment with ML but I’d be surprised if any sports studio would find this ideal or the way of the future.
Says who? It selects the sprite frame by frame. This isn’t a static sprite planted in the world that rotates to face the camera. It’s changing the sprite based on what is happening in the scene every single frame.
Mocap is super cumbersome. It requires tons of very specific cameras (unlike this setup would), a whole compute heavy rig to help stitch things together, lots of manual labor work by animators, and the actors have to wear awkward setups/cameras. Plus, it takes a ton of work to then add in things like clothing and other decor to the characters since those can’t be captured in mocap.
Just bc it’s the standard doesn’t make it not cumbersome. Especially compared to just having multiple cameras and letting the actors do their thing and letting the AI do the hard/tedious work.
I’m saying none of us have enough experience to judge how cumbersome it is or isn’t.
And I still think you’re overlooking the limitations of a sprite based system. Just because they are generated using ML, that doesn’t mean they could avoid the traps of using sprites, could they? Or are they generated on the fly so the frames of animation don’t have to be stored in memory? I just find it interesting that the video focuses mostly on player behavior than player fidelity.
The issue with sprites traditionally is that they are often static billboards. If you are constantly updating the sprite based on the frame, you won’t ever see those seams as the player. Just as you don’t see them in the tennis clip. Note that even there the players are fully animated, fully spinning/rotating and turning to the camera, moving closer/further from the net/camera and even can jump vertically. So provided the sprites are available for the AI to select, I don’t see any issue with it.
Note the GTA video too…there it’s generating all the assets it sees in the scene, frame by frame, and drawing those in each frame the player sees. It’s also a fairly old video but it’s a rly neat ML use case. They can make it run much faster too by replacing the rasterization the game code does as is (they left it in for their comparisons/experiments). They can vastly simplify what the AI is fed and avoid running the rendering from the game code in parallel with the ML work.
My point wrt mocap is that it is a bunch of cameras + tons of other stuff. That has to be less cumbersome than just having the cameras and not all the other stuff. Major studios already are using ML for animation (Ubisoft’s teams use it as does some of EA’s, for instance).
I dunno what ya mean about the player fidelity comment…? It looks totally photoreal to me. And it’s the actual player’s movements. What do ya mean?
It’s easy for them to have fluid animations with sprites when the camera is static. The real challenge comes when you have a dynamic camera. In fact outside of the players and the ball, everything in the scenes are static. So I’m not going to make assumptions on the fluidity and flexibility of the sprites until I see the application used in a scenario better comparable to today’s games.
And by fidelity, I mean void of the obvious artifacts and visual errors we see in the video. You can clearly see the players skate around at times and what I’d probably best describe as masking artifacts where parts of the edges of the sprites fade or fizzle in and out. This looks like a high end late 90s PC tennis game when looking at it blindly. This is why I hope they take this ML behavioral system and apply it to the fidelity and animations seen in today’s sports games.
Yeah that was why I was qualifying everything wrt assuming they had the imagery from the various angles. Any artifacts you have to go in search of will be easily sorted within a decade from better algorithms + vastly more compute.
Saying it looks like a 90’s level videogame is comical. You don’t think that, lol. Looking at it casually looks exactly like a live action actual game being played to my eye. I’ve shown it to multiple ppl who thought that was what it was. It looks a LOT more photoreal than any existing tennis game I’ve ever seen.
I don’t mean a 90s video game in a literal sense. Obviously we didn’t have games with digitized sprites with those animations, systems didn’t have the memory to store that many frames of animation, but everything else outside of the animations does remind me of it. I’m sure plenty of people would mistaken it for a live action tennis match at a glance. Doesn’t take long to see it’s not though. The artifacts and errors stick out too much for me.
It’ll be interesting to see where this tech goes but I’m more interested in the fidelity of rasterization improving than us moving to digitized ML sprites.
1 Like
This should be good for things like NBA 2k, cameras are everywhere in basketball arenas. And mocap never looks just right in their games, like if every action is recorded without the other players on the field.
3 Likes
You are right, their technique is completely different and is based on the idea of AI analyzing video, but carries out the animations through 2D sprites instead of creating 3D animations.
2 Likes
4 Likes
Here’s the written article, which I skimmed. Seems like:
- Series X is mostly stable and sticks to target resolution in all modes
- PS5 has more trouble with frame rate and resolution (needs system level options to enable 120Hz)
- Series S tries for 1440p60 and is stable but sticks between 80-90%
- Switch tries 1080/720p for docked/handheld, but also sticks to about 80%. Seems like pretty big sacrifices to run at all, and it has frame-pacing issues with 30fps
I’m pretty poorly versed in these things, so apologies if I misinterpreted anything. As with everything, check the source yourself and read carefully to avoid miscommunications!
6 Likes
You’re pretty spot on in your summary. I’d add that the Series S version does not offer 120fps or ray tracing modes.
Really good results on all platforms really outside of the PS5 120fps mode, that mode needs some additional work.
Surprised there’s a Switch version and it turned out as well as it did.
Really need to get this when I get the money. Was always interested in this since the first Microsoft showcase.
1 Like
it’s about 90 minutes long so hold out for a good sale, 
1 Like