AV1@Scale: Film Grain Synthesis, The Awakening

Regarding aesthetics, I don't think AV1 synthesized grain takes into account the size of the grains in the source video, so chunky grain from an old film source, with its big silver halide crystals, will appear as fine grain in the synthesis, which looks wrong (this might be mitigated by a good film denoiser). It also doesn't model film's separate color components properly, but supposedly that doesn't matter because Netflix's video sources are often chroma subsampled to begin with: https://norkin.org/pdf/DCC_2018_AV1_film_grain.pdf

Disclaimer: I just read about this stuff casually so I could be wrong.

If you have a few static frames and average them, you improve SNR by retaining the unchanged signal and having the purely random noise cancel itself out. Retaining noise itself is not useful.

I suspect the effect you might be seeing is either just an aesthetic preference for the original grain behavior, or that you are comparing low bandwidth content with heavy compression artifacts like smoothing/low pass filtering (not storing fine detail saves significant bandwidth) to high bandwidth versions that maintain full detail, entirely unrelated to the grain overlaid on top.

That's true, but at a given bitrate (until you get to very high bitrates), the compressed original will usually look worse and less sharp because so many bits are spent trying to encode the original grain. As a result, that original grain tends to get "smeared" over larger areas, making it look muddy. You lose sharpness in areas of the actual scene because it's trying (and often failing) to encode sharp grains.

Film Grain Synthesis makes sense for streaming where bandwidth is limited, but I'll agree that in the examples, the synthesized grain doesn't look very grain-like. And, depending on the amount and method of denoising, it can definitely blur details from the scene.

I would love for them to provide an option to view it with film simulation vs without.

One of my favorite movies of all time, The Holdovers, did film simulation extremely well. It's set in the '70s so it attempts to look like a movie of that era.

It looked great to me, but if you're an actual film nerd you're going to notice a lot of things aren't exactly accurate.

Maybe in the near future we'll see Netflix being able to process some post effects on the client. So if you're color blind, you get a mode for that. If you don't want fake grain you can turn it off.

A few things to note:

- still-frames are also a mediocre way to evaluate video quality.

- a theoretically perfect[1] noise-removal filter will always look less detailed than the original source, since your brain/eye system will invent more detail for a noisy image than for a blurry image.

1: By which I mean a filter that preserves 100% of the non-grain detail present, not one that magically recovers detail lost due to noise.

that's an understatement. it just looks like RGB noise effect was added. film grain does not look like RGB noise. to me, film grain is only one part of what gave film the film look. the way the highlights bloom rather than clip. it also was more natural/organic/some descriptive other than the ultrasharp of modern digital acquisition. using some SoftFX or Black Mist type filters help, but it's just not the same as it is a digital vs analog type of acquisition. all of these attempts at making something look like it's not just keep falling down in the same ways. but hey, there's a cool tech blog about it this time. film grain filters have been around for a long time, yet people just don't care for them. even in Blu-ray time frame, there was attempts at removing the grain in the encode and applying it in playback. Netflix isn't coming up with anything new, and apparently nothing exciting either based on the results.

Look around you: nearly all surfaces have some kind of fine texture and are not visually uniform. When this is recorded as video, the fine texture is diminished due to things like camera optics, limited resolution, and compression smoothing. Film grain supplies some of the high frequency visual stimulus that was lost.

Our eyes and brains like that high frequency stimulation and aren't choosy about whether the exact noise pattern from the original scene is reproduced. That's why the x265 video encoder (which doesn't have grain synthesis since it produces H.265 video) has a psy-rd parameter that basically says, "try to keep the compressed video as 'energetic' as the original, even if the energy isn't in the exact same spot", and even a psy-rdoq parameter that says, "prefer higher 'energy' in general". These parameters can be adjusted to make a compressed video look better without needing to store more data.

A child watching a Buster Keaton skit and gasping and giggling and enjoying it is going to have a different subjective aesthetic experience of the media than a film critic who knows exactly what type of film and camera were used, and what the meaning of all the different abstractions imply about the scene, and the fabric of Keaton's costume, and so on, and so forth.

Subjective aesthetic preferences are in the realm of cognition - we need a formal theory of intelligence mapped to the human brain, and all of these subjective phenomena collapse into individualized data processing and initial conditions.

There's something about film grain contrasted against clean cel animation which might make it easier for people to suspend disbelief. They are conditioned to think that absence of grain is associated with unreal animation, particular types of media, and CGI. Home video and news and so forth had grain and low quality, so grain gets correlated with "real". In my view, there's nothing deeper than that - we're the product of our times. In 40 years, media will have changed, and it may be that film grain is associated with surrealism, or edited out completely, as it's fundamentally noise.

I don't know the psychovisuals behind that. Maybe it adds some high frequencies that compression often washes out, or maybe acts like some kind of dithering.

As for your eyes, I am pretty sure that they have grain, that's how quantum physics work, you just don't perceive it because your brain filters it out. But again, I don't know how it interacts with film grain.

They definitely do at night when it's dark out. There's a kind of "sparkling" or "static" that comes in faint light.

Fortunately, our eyes have way better sensitivity than cameras. But the "realism" just comes from how it was captured using the technology of the day. It's no different from phonograph hiss or the way a CRT signal blurs. The idea is to be "real" to the technology that the filmmaker used, and the way they knew their movie would be seen.

It's the same way Van Gogh's brush strokes were real to his paintings. You wouldn't want his oil paintings sanded down to become flat. It's the reality of the original medium. And so even when we have a digital print of the film, we want to retain as much of the reality of the original as we can.

It might be that there is a large part of the population that still has that association.

Cinephiles are also more likely to watch older (i.e. with grain) movies that ARE well shot and beautiful (which is why they are classics and watched by cinephiles) and not see bad film movies, only the cream of the crop, while being exposed to the whole gamut of quality when watching todays movies shot digitally. Would reinforce that grain = good while not being necessarily the case - and their opinion might be heard more than gen pop.

At any rate, it can be a neat tool to lower sharpness!

I have to imagine past glassmakers would have been absolutely enthralled by the ability we now have to make uniform, large sheets of glass, but here we are emulating the compromises they had to make because we are used to how it looks.

this is likely the result of ~100 years of film-based filmmaking and projection. hell, we still call it filmmaking.

Well ackchually -- illumination is inherently random, so all time-bounded captures of a scene (including what your eyes do) are subject to shot noise: https://en.wikipedia.org/wiki/Shot_noise

Does it add any more than modern video compression techniques? What constitutes noise in cinema, is somewhat subjective.

There are two possible advantages for this kind of grain synthesis. For Netflix, they could produce the same perceived quality at lower bitrates, which reduces costs per view and allows customers with marginally slow connections to get a higher quality version. For a consumer, the advantage would be getting more non-grain detail for a fixed bitrate.

You are right that if you subtract the dentists frame from the raw one, showing only the estimated noise, you would get some impression of the scene. I think there’s two reasons for this. Firstly, the places where the denoiser produced a blurry line that should be sharp may show up as faint lines. I don’t think this is ‘hidden information’ so much as it is information lost to lossy compression. In the same way, if you look at the difference between a raw image and one with compression, you may see some emphasized edges due to compression artefacts. Secondly, the less exposed regions of the film will have more noise so noisiness becomes a proxy for darkness, allowing some reproduction of the scene. I would expect this detail to be lost after adjusting for the piecewise linear function for grain intensity at different brightness levels.

Perhaps a third thing is the level of noise in the blacks and the ‘grain size’ or other statistical properties tell you about the kind of film being used, but I think those things are captured in the film grain simulation model.

Possibly there are some other artefacts like evidence of special effects, post processing, etc.

Fake lights, fake shadows, fake sky, ...

People remember the emotions the artwork engendered, and thus the whole work is associated with the feelings, flaws and all. If the work is particularly widely known, the flaws can become a stand-in for the work itself.

I see this in video games - I'm fond of the NES-era "flaws" and limitations (palette limits, sprite limits, sound channel limits), but less connected to the Atari 2600 or SNES/PS1/NDS/etc flaws. Shovel Knight is charming; A Short Hike, while great, doesn't resonate on a style level.

I've read that a 15-perf 65mm IMAX negative shot with slower film stocks is "virtually grainless", even when viewed on a 70ft screen. Grain is apparently noticeable in IMAX films when large/fast stocks are used and pushed toward their limits, and (of course) when smaller-format film stocks have been blown up.

1. You prefer Betamax or VHS to digital media (highly unlikely)

2. You own laserdiscs (limited to 480i)

3. You own 35mm prints of film.

Since all other formats film has been made available on are both digital media and compressed.

All that is 24fps.

That's without audio, which I assume you also want to be uncompressed.