Microsoft is releasing Direct ML apis for the general public

This is really neat!

1 Like

That’s what i was wondering. The frame has to be completed first, only then it can be used to upscale correctly. I don’t think super sampling will be running in parallel.

Although, i am not sure of using the shaders first for rendering the frame and then using it for super sampling within the given frame time is possible or not. If this can work, then no worries of GPU utilisation or partition.

Is the XBX gpu capable of ai upscaling without the upscaling process itself taking to much gpu time?

This is the real question


  • Render in 4k = x time

  • Render in 640x480 AND upscale = y AND z time

We must assume that y time is way smaller than x time

The question is y AND z combined are also smaller than x time?

For DLSS on Nvidia gpus the answer is a very big yes for all games that support it thus far.

For consoles, I would expect similar results, if a method is too costly for them devs will just switch to something else.

1 Like

Nvidia’s approach lets them start rasterizing the next frame and more evenly spreads the ML workload out over the frame. Not sure exactly how that works though.

We don’t need to speculate on if it is possible using shaders a la how it will work on XSX…it’s already being done exactly for this kinda thing at XGS. MS didn’t go out of their way to put INT operation capabilities into the shader core so they can do precisely this exact SS without testing and researching and knowing 100% for certain it worked, lol. The question now is how well SS works, how it works with VRS, if the texture upres approach is more efficient, etc.

1 Like

Just to be clear: We know for certain that it will yield notable improvements to IQ and framerates. How big is anyone’s guess right now, but the speculation around if it is plausible is misguided.

AMD’s offering to MS initially did not include INT8/INT4 capabilities. MS specifically asked those be included ON THE BASIS of already having researched using those capabilities for ML (they even called out SS specifically). They didn’t just guess that it could maybe be plausible, they don’t make decisions about silicone changes with guesswork. They saw how much research was steering towards using INT operations in SS and other ML tasks (that shift started among researchers in 2017/2018). MS is well aware of that and asked AMD to tweak the shader cores after doing their own R&D with it.

They have directly explained this in their DF interview as well as in their marketing for XSX’s ML capabilities. The question is how well will it work and how it combines with other innovations in the platform.

1 Like

Thanks @TavishHill for the links.

I saw Digital Foundry as the source for INT8, found nothing else. I read a lot of stuff from NVidia about thier solution and they themselfes only mention FP16 performane and metrics, but mostly older material. Could be they improved their solution.

And I thought about precision alot yesterday and you can probably get away with lower stuff (even without quantization of the network weights), because in TAA/DLSS cases the network has only to determine which samples of current or previous frames to reject or accept for a given pixel location (i.e. the trained rectification heuristics)

Regarding the parallel execution of SuperResolution and normale GPU work, this is rather difficult, because SuperResolution uses almost all compute power in the GPU when it runs. So you can interleave it asynchronously with other work, when they don’t use the GPU or only parts of it. This is certainly possible, because normale rendering work doesn’t utilize GPU 100% all the time and some stuff doesn’t use compute a lot, like shadow mapping. But this makes SuperResolution not faster.

Exciting times. Will be very interesting how Microsoft adapts this for Xbox.

The licence in the sample is MIT/Apache, I could not find a mention of NVidia. I thought this very surprising. So this isn’t NVidias model they first demonstrated with DirectML at GDC. Its the one from models/vision/super_resolution/sub_pixel_cnn_2016 at master · onnx/models · GitHub. This isn’t really useful for games, because it only upscales single images without additional temporal information from previous frames and depth and motion buffers.

1 Like

My guess is they won’t be using it spread out much across the frame time. Nvidia’s earlier work didn’t spread it out either. Since there can be 75% fewer pixels to shade, there should be plenty of time left over in a frame to dedicate to ML work. There’s lots of other ML use cases that can make sense earlier in the frame time too, but those are probably not latency sensitive stuff.

MS has said ML will be used for super sampling, resolution scaling of textures, npc behavior, animation improvements, etc:

“One of the studios inside Microsoft has been experimenting with using ML models for asset generation. It’s working scarily well. To the point where we’re looking at shipping really low-res textures and having ML models uprez the textures in real-time. You can’t tell the difference between the hand-authored high-res texture and the machine-scaled-up low-res texture, to the point that you may as well ship the low-res texture and let the machine do it” -James Gwertzman

“…making NPCs much smarter, providing vastly more lifelike animation, and greatly improving visual quality.” -XSX Tech Glossary article

Also, this is from Hot Chips presentation:

So I think we know what they plan to use it for, just a matter of how well it works in practice and (possibly more challenging) how game designers can make use of it especially for npc behavior.

1 Like

Wish this was the Backwards Compatibility team, lol. I guess this approach would reduce storage and RAM utilization.

1 Like

And I/O throughput needs, DL size and SSD footprint. :slight_smile:

I see no reason it couldn’t work on BC games too. Something very similar has been done on PC after all.


For me its when we can expect to see it. I think it is a while away, if it is used at all. Has the smell of PS4 Pro RPM to me.

More fun use cases for ML in gaming (not at run time afaik):

Also, another example EA noted was that Mass Effect’s Remastered trilogy got texture boosts using ML super sampling on those older textures. They upres’d many thousands of textures that way. :slight_smile:

1 Like

So will Xbox Series X and S get this and will it be a game changer?

1 Like

Well MS had AMD add additional features to their SOC for this purpose, so hopefully it pans out


Yes. Literally changing the games. This is still a wide open field of research. It is unlike most other tech advancements in that it’s not just hw features built for specific problem cases. Our imagination collectively is a limiting factor atm for ML. The obvious uses are potentially huge game changers right outta the gate though (image upscaling for textures/frames). Some other stuff will need to be carefully explored by game designers though, like using ML for npc AI or systems.

1 Like

Cool cannot wait than hope it will make hitting 60fps 4k or maybe 120fps 4k easier.

The approach from Nvidia was to train the network specifically for a game first and then deploy the DLSS network AI to upscale that specific game.

Direct ML approach is rather different.

  1. Super resolution is just a part/option in Direct ML.

  2. So far, the above example ( video resolution upscale from 540p to 1080p ) is the only one which comes close to be implemented for games upscaling

  3. So far it seems to be a general model, applicable for any 540p video.

We may see it first to be implemented in a game from Microsoft studios. Project Mara i believe.

Microsoft studios may use a specifically developed network for project Mara.

Xbox may develope a generic solution as a part of GDK which can be deployed for any game. It could simply be a better replacment for anti aliasing techniques and may offer 10-15% of net performance improvements. Don’t expect it to do wonders at this stage.

To gain performance increments like Nvidia’s DLSS, specifically trained networks are required. I believe, only Xbox game studios and studios which may have partnership with Xbox will go this route.

At the least, we will have a good anti aliasing solution for the all games. And the best thing be that games will hit the target more and more consistently. We may also have games finally hitting 4k & 1440p targets. Games which are already running at 4k & 1440p may have PC high/ultra graphics. We may achieve 120fps without compromising output resolution.

Note: I only called it a ‘good anti aliasing’ solution because a generic network may not be able to compete with DLSS by a long shot. In that case, it might not be worthy enough to call it a resolution upscaler.

What will happen if input a 720p or 30 fps video instead. Will it simply crash or reject the input?


Have you tried it? Thanks!

If DirectML provides better results than the regular temporal upsampling then it’s already a win. I would even add it as a OS level feature if it works well.

They should use it to SS the dashboard to 4k so @Shpeshal_Nick will shut up about it. :slight_smile: