Give Me 4K or Nothing

While I'm not a fan of Nvidia's consumer practices, I can't argue that the software they have is fantastic.

I’ve enjoyed VSR quite a bit in the past week or so. Of course there are some videos that don’t work well with it… but I agree with Linus that a good source being upscaled tends to look better than PLclip’s native 4K. Nowwwww a garbage source being upscaled isn’t going to fundamentally change it. But I’m sure it’ll improve as time goes on :)

In my opinion, the future of video codec lies in AI upscaling combined with extra data. You would encode your stream at 1080p, and then the AI would handle 90% of the upscaling, with the remaining 10% being perfected by providing hints to the AI. It's worth noting that the size of these hints could vary depending on the AI's performance. Looking even further ahead, I believe we may see an encoder AI and decoder AI that work so well together that they can generate just enough bytes for optimal video reproduction.

I find VSR works great for animated content like Cartoons and Anime.
Cleans up aliasing on the line art from lower resolution uploads and sharpens up the text for Subtitles in Anime nicely.

Me with my 2080TI after hearing 40 and 30 series GPU's: "What am I chopped liver?"

*OK FOR THOSE WHO WONDERING HOW TO MAKE THIS WORK:* u only need to download the nvidia's current drivers (make sure is games ready GDR, it wont work with studio's) go into nvidia's control panel go all the way down to "Adjust Video Image settings" and at the right side checkmark the RTX video enhancement set quality to 4 if u want 4k. You will also need latest chrom/edge to use this it wont work with firefox atm.
Enjoy!

The ideas about encoding repeating patterns and moving objects are pretty similar to how compression already works. But I'm sure AI will make it a lot more efficient.

I've moved my plex watching from the Plex Windows desktop app to the web app to take advantage of the NVidia 4k upscaling. It works SOOO well for TV shows that were only aired in 1080p!

I still don't understand how you guys do a whole 2+ hour stream and still do everything else. I respect the work ethic

The optimization topic reminds me a lot of SVG. No image format scales as well as the format that is literally just the math to redraw the image at any resolution. Similarly, I could see a new video format that is designed to draw the video frame by frame rather than a fixed stream of pre-rendered images

Linus' idea sounds like replay-files in games. they only save the information about what happened and the video is generated in-engine where you can move the camera around freely so no graphical information is saved in the replay at all. for podcasts that are always in front of the same background that would work, the viewer just has to download the information about the set and the people once and can use it every episode. wouldn't work for any other kind of video though and it would make it harder when, for example in wan-show, someone shares their screen with the stream

I'm playing around with it right now. It looks surpassingly good for upscaling lower res video, but once you go past the 1080 resolution, you can see a hint of the similar DLSS smudginess depending on what's on screen.

This compression thing is a really cool idea for some theatre kid programming student to do as some thesis for a research project. I really want to see this go somewhere in the next 5 years.

300 watts of power demand, when all your doing is watching a youtube video, is honestly hilarious lmao it very well describes the current state of PC hardware

What's actually going to happen is that new video codecs are going to be created (experimental codecs are already being worked on) that have AI reconstruction built in on the codec level, and the trained neural net is part of the spec - that way everyone has the same decoder and gets the same experience. But the bitrate requirements are an order of magnitude lower.
No need to add metadata to modify an H.264 video, when it's far more efficient to _replace_ H.264 with something completely designed for AI from the ground up.

Nice! Ive had this idea for a while when it comes to scifi holograms / VR Presence --- send a super low quality transmission over and let the other end artificially upscale it and fill in missing details. In most use cases, you don't need the actual details, but the absence of details can be jarring

Reminds me of the days of midi sound cards where music would sound different depending on what card you had

I use it to upscale shows, don't really care about it for PLclip videos as it's not as important to me. But it's pretty good. I watch a lot of 1080p content on a 1440p screen and that causes some MINOR fuzziness but VSR clears it up pretty well. Only thing is, at quality 4... damn. GPU uses like 310 watts and 30-40% usage. Which is why I actually believe Nvidia when they say the 20 series cards couldn't keep up and need additional development time

Honestly the jump from 1080p to 1440p alone is such a big difference. I am really happy with 2k, anything beyond i don't notice unless the display gets bigger than 32 inch and more

I would love we talk more about the good old MPV player. It's basically the same as VSR, people create algorithm, sometime neural nets, and play/upscale/correct videos with it. It is life changing.
Problems are you cannot open DRM content (twitch and youtube, live or not, work though) and there is no good integration in the browser.
Better is, you can choose you algorithm depending on you hardware and what you are watching.

The current high tech encodings for video are almost "mocap" already (they move sections of the video as groups of pixels). But yeah, an AI could take that idea even further.
For example the "fuzzy snow" example is good. It'd need the encoder to know how to remove the "snow/confetti" and then tell the AI "Just draw snow/confetti back in please". :P

Watching this with 1080p to 4k upscaling with RTX super resolution. It looks so amazing. Only issues is the laptop 3080 I have is utilizing at 30% gpu usage. Makes the laptop noisy.

Video patterns being upscaled differently like you can play a midi file with different sound fonts is a hilarious idea.

I really like that question that someone asked what is the bitrate of the floatplane and Luke just made his way out of the answer. Do someone know what is the bitrate of the floatplane?

You would just need to do some of the processing server side statically. Then send that preprocessed data to the user who would do some more processing (or none, but turn that data into an upscaled format). Maybe they've tried it and the bandwidth use is too high sending pre-processed data? Still pretty incredible.,

4k tv have gotten pretty cheap over the past few years but 4k monitors priced refuse to budge.

I just want Nvidia to release a new shield with updated 4K upscaling

I use it, works on everything from anime to live-streams of video games, I wish there was a quick way to toggle it though btcause when the gpu becomes busy with a game or otherwise it bogs down

I wonder if Nvidia can use frame generation to turn 30 to 60 fps video or higher?

Works pretty well on older 360p/480p VHS type video that's been uploaded to YT, not perfect but still does a pretty decent job at increasing the quality and removing artifacts.

I absolutely love my 1440P w HDR. It's going to be a while before I have a real need to upgrade to 4k

maybe in the future, the video description will contain an URL for AI-upscaling to retrieve the dataset to upscale the video in the best way, a few high res layers for the AI to work in-between or something. and if big-tech has their way, there might even be an add layer to dump adds into parts of the video image.

They are explaining how video compression already works. They already encode to move objects and patterns etc.

When doing content creation (not on this channel so don't look for it I don't advertise) 4k recording would be way too expensive as the file sizes are massive and I don't like to delete my footage. This is good news.

Imagine if servers could send neural network fragments to users that are fine tuned for their particular content. Or imagine if there was some kind of hashing so they could borrow fragments you've already downloaded. I could imagine the server and client being like "Draw a giraffe here" - 'What's a giraffe?' - "It's like a horse but longer" - 'okay done'

Nvidia Canvas is pretty cool. I'm writing a fanfic and described an environmental scene. Three moons of different colours, two rising while the third descended over a mountain range. One of my reviewers asked for an image of it. Which obviously didn't exist.
Jumped onto Canvas, painted a sky and a mountain range in a few seconds, cut them apart in a photoshop program and added in the moons. Took me an hour to take something that was an image in my mind and make a version of it other people could see.
It obviously still has serious limitations. I could draw the entire scene via AI, just the 'backdrop' and then had to make and layer the moons manually. But as these tools get stronger, I could probably 'draw' an entire scene like that in Canvas if they add more element options to paint with. The trick then would be finding ways to retain consistency with previously generated designs so that each time you want to make a new one, it doesn't start all over and you get a totally different look in every image.

This and missing HDR support will probably drive me from firefox to chrome *grumpy*

Soon we’ll be able to turn any film into any kind of quality footage we want. Like ok, let’s take the series Friends and tell it we want it to look like it was filmed on expensive Hollywood Cameras for movies like the ones Christopher Nolan uses. Later, we’ll be able to have it take an episode and tell it to turn it into a dark film directed by Christopher Nolan. Literally all the pieces are being developed right now, and I thought music was going to be the hard part but Googles recent diffusion text2music paper showed it will be able to do it a hellva lot better and we’re a lot closer than I expected. The only thing that seems to be the thing it would need to get right is extreme subtlety in multiple dimensions, writing acting cinematography sound music etc etc. That’s going to be a huge hurdle and until a few months ago I’d have said we were way far away from that. But now i know we’re way closer. Of course we’ll start simply, but the question will be how much better, and how fast will the the improvements will be with the release of each new model.
There was a gap of like 5-6 months with Midjourney ver 4 to 5, and they said they expect their next version to be released within 2 months! The “AI can’t do hands” meme mockery will be something that we will barely remember.

I am SO glad Linus mentioned the power draw of RTX VSR. Gamers Nexus did an excellent video but for some reason they literally did not mention power draw whatsoever. I don’t care much about power draw, but I don’t really want to suck back 330W on my 3080 just to watch PLclip videos when CPU bound games won’t even use that much lmao

I love this feature so far. Wake up one day to an extra thing on my gpu. Not bad

For those who aren't aware RTX VSR is currently built to run with the 30 series' sparse matrix improvements, this is why it requires work it'll run on 20 series.

What your talking about with AI compression could be done with an Autoencoder network. The encode side is a large network that condenses into fewer vectors while the decoding side takes a few vectors and expands into the original size, the AI is then trained for encode/decode accuracy and the compression results can be absolutely wild. Toss an AI upscaling network like Nvidia's over top of the output and you could probably compress videos to a tenth of their original size without losing viewing quality. Of course, doing all of this on the fly would require insane computing power.

Can anyone tell me how (the 4k upscaler) it would affect say VCR rips to digital?? Or even a VCR stream to screen output??
Linus, might be a good video opp

8:37 yeah that's the main issue with streaming video games and the MAIN reason to turn all the way down or off if u can the particles effects in games as no matter if u have a 4090 it will still tank your streaming image quality cos of that compression with bitrate

7:00 What you're talking about is called an auto-encoder. A lossy machine learning compression algorithm that's basically a big neural network with a bottle neck in the middle. You split the model at the bottle neck - the first half becomes the encoder, and the second half is the decoder.
It has an advantage over regular compression because it's able to leverage the predictability of the content it's compressing.

VSR works "fine" on lower power devices (if you ignore the glaring bugs that make it nearly impossible to use) but if we are talking about power consumption, I think nvidia just doesnt know how to manage higher power GPUS and just tries to throw the entire GPU at it instead of what is needed.

Compression is the biggest killer of quality. 1080p offline often looks better than 4k online

Imagine how this could affect game streaming. As little computation locally as possible regarding what happens in the game, but then heavy computation regarding rendering the streamed image with supporting meta data to be interpreted by the target device...

AMD could easily make a VFSR version of FSR, and I think Intel could make a XeVSS version of their XeSS, for similar results, but ideally also available for rendering videos. And the advantage of FSR is that it can work on all GPUs, including GPUs integrated in CPUs (aka. APUs).

"Floatplane would be DRM protected too... well, don't worry about that"
put a pin in that Lafr

god that looks good, video is smoother even on native youtube resolution

my eyes when luke's hand thru mic's hand stand after he lift it up above to be seen become 3d something out of the adobe cut without final filter is already something and I'm testing this new super resolution

I wish I could get float plan but I can never get my validation email!!!

They dropped upscaling for ages. It used to be something with that nvidia dvd/video player back in the day.

There is already a video chat codec that transmits a still image and motion data of the other person and reconstructs video at the destination.

I hope they expand functionality to firefox browser as well

We switched back to the Apple TV for AirPlay and I forgot how much i relied on the nvidia shield AI upscaling!

Too bad even a 3060 can't handle the lowest Up Scale setting without dropping the video's framerate to like 40fps at 95+% usage. I wish they added an option to upscale to 1080p and just remove artifacts instead of pushing for 4k

I tried it and I thought it looked horrible, especially at level 4. That typical AI enhanced water colour effect was all over it, especially on close ups of faces. Weirdly I didn't have that problem with the Shield's upscaling.

So the NFL confetti cannon now shoots out square cards with 'confetti goes here' QR codes... Cool!

Hmm, I haven't been able to see a difference between VSR on or off. How do you know its working?

This... Might make me buy a Nvidia GPU again.

I just tried this watching this video, and it didn't look great to me? No matter what setting I had it set to it made linus' beard and neck look extremely AI generated by smoothing it out like crazy

Me enjoying upscaled 4K videos.
My gpu: 🔥

PLclip 4K can have banding depending on how you shoot it and the bit rate you use. There are youtubers who do extremely high quality 4k Videos with no banding. This will just allow more low quality videos.

What will REALLY bake your noodle is how we can produce a 360p (etc) resolution video designed to make the AI upscaling work better, by having the video compressor (essentially) use the same AI in reverse. (By “we” I mean “they” as in video tech wonks. I’m just an investor.)

Did he say 300w to watch a damn upscaled video? My jaw fell open and there’s no way in hell I’m enabling that.

I wonder if Kodi can upscale our libraries of videos ? 😊

Opera Gx have been doing this for a while now. its nice.

So does it work on Netflix? If so how will a vendor like Netflix be able to charge for hi-res streams if people can just upscale the low-res streams?

Considering who's behind it, I could see this being used in cloud gaming to improve responsiveness. Basically ship video signal at lower quality to reduce the amount of data sent, then enhance on client machine.

Okay, when do we get it on actual browsers? (firefox)

I have been following this and now it is here I see no difference. But, High bitrate 1080P from PLclip IS impressive. Nvidia thing seems like bs.

Too bad I can't use that in Firefox on Linux

Haven't been impressed by 4k yet, just looks like sharper 1080p with 4x the performance requirements

This feature has been available on Nvidia SHIELD TV devices since November 2019 IIRC. The Shield TV uses an underpowered Tegra X1+ ARM SoC (essentially a Nintendo Switch running at stock clock speed), can anyone explain why the F**K it took Nvidia 3 years to bring the feature to their discrete PC GPUs that are an order of magnitude more powerful than the Tegra X1??? Honestly, I'm baffled they didn't debut this feature on PC and then optimize it for ARM. The Tegra X1+ doesn't use more than 20 watts while gaming so the AI upscaling can't use 300 watts for sure, so why is it so unpolished on PC and why does it seem like the requirements to make it work are a lot more complicated? The Tegra X1 is on Maxwell GPU architecture, and Nvidia can't even promise this will work on Turing?!? It doesn't make any sense!! lol

we free users are locked at 1080p in xxi century. after years of 4k now we miss that feature

If .. the power consumption is this high, the first thing has to be to drastically lower the power usage.
If that can be achieved, streaming will be heavily reduced bandwidth, which overall would be good for everyone. The downside is, nobody will see the real source anymore, so a loss of data and reality would come along with that. Which in itself offers room for bad actors as well.

finally linus duke nukem in 4k!!!!

I think at some point we will get a netflix type subscription where you give a input from 1 line to multiple pages where you describe the type of content you want to watch and its story and you get a generated video.

what about 4k to 4k or 1080 to 1080 to deal with the insane over compression they do on YT?

Can someone test this out with TheSlomoguys glitter video where they broke PLclip video compression?

Wait, you can upscale Netflix?
Isn't Netflix 4k a higher price tier?
I wonder how long until upscaling Netflix gets disabled

4:57 a quick 700mv undervolt would fix that :P

imagine how much bitrate it would save if theu stopped waving their arms as theu speak

I hope this moves to Firefox, too. I am seriously considering moving AWAY from Chrome b/c of the way too dominant position over the web the way Internet Explorer had- and we NEED an alternative to grow and force the world to NOT comply ONLY to Chrome...

They were basically just describing Adobe Flash

A motte is something that others can't do (CUDA). Everyone can do upscaling. Entertainment systems have been doing upscaling for a decade. The AI upscaling was announced within a week for both Intel and AMD. Personally, I'm sick of upscaling. I want pixel perfect 4k because hitting the limitations of technology is how you get better tech. Downgrading your processes to enable more with less is how you get taken as a consumer. Projectors rendering 3-4 1080p frames for 4k. Games with no optimizations relying on AI Super Sampling to make it playable. Coax hitting 1Gb instead of Fiber to the home so you still have shit upload.

I feel like im taking crazy pills. I have a 40 series card, updated every possible bit of software on my pc and I dont notice a single difference on this video at any resolution with vsr on, which im watching on a 4k tv.

There's barely any difference when you turn it on. So it'll just end up forgotten in a matter of weeks. Which is a shame. But at least better technology already exists. MadVR/NGU upscaling still gives better sharpness and it's been around for many years. I really wanted a successor to it but this ain't it.

I don't need upscaling, I just need youtube to stop changing my quaility settings from 4k down to garbage. Linus says almost no one uses 4k on youtube, well maybe it's because youtube keeps turning it off

I tried it while watching this video and in my own opinion it looks bad. It just makes everything look like a moving painting. As someone stated in another comment, it might work better for animated videos so i'm gonna have to try that out. But for now I doubt i'll be using it

Netflix: quality above 480p requires a more premium membership for $6 more a month.
Me: lol jokes on you, I'll just buy a 4090 and save myself the cash over a 40 year period.

Ok guys, time to start uploading wan clips in 4k. Pretty please.

VSR unfortunately will NOT fix your favorite anime

Linus the future you are envisioning is here.
Edit:
It's called "Event Based MetaVision" from France if I recall correctly.

Didn't PLclip want to remove 4K from free users? Only for premium? hah! Nvidia Upscale here it comes!

surely this should be avaliable for older GPUs as well, ai upscaling cant be that demanding for high end 10 series gpus, i really have no idea though, but id be happy for my 10 series gpu to be at 90% usage if im watching youtube, i would love to get rid of youtubes crap compression

If a tegra X1 can do it why can't 9, 10, or 20 series capable?

As the owner of a 6700xt, I was initially very jealous of nvidia users when I heard about this feature. But if it doesn't work on Firefox, it's a non-starter for me - I have no desire to switch to Chrome or Edge.