Linux

VP9 is AV1's predecessor and VP8 is VP9's predecessor. Dunno what the “264K 360° Surround sound 3D VR” thing is about, but AV1 is a good general purpose video codec and I recommend using it, with Opus audio.

EDIT: I should add, h264 and h265 are non-free because of software patents, for which there are licensing fees. There are free implementations and there always have been, but the extent to which these implementations can actually freely be used legally is limited by this. Cisco's OpenH264 is an exception, because there is a cap to the licensing fee and Cisco is already paying the max amount. This allows them to freely distribute binaries for their h264 implementation without having to pay additional licensing fees for every user. It's a clever loophole, but there are still limitations, namely that you have to be using Cisco's pre-built binaries. If you want to use the source code, you still need to pay for the licensing fee.

Because patents last twenty years and the initial release of h264 was made in August 2004, the key h264 patents should all expire within the next few years, which will eliminate the problem. h265 however was introduced in 2013 and its patents still have a good decade left in them.

DVD's max out at about 580p (for PAL, NTSC is 480p), resolutions are measured by the number of horizontal lines of pixels (counted from top to bottom of video/screen), not vertical which at 4:3 square aspect ratio on dvds does tend to be 720 pixels (by contrast full resolution HD video's number of vertical lines is 1920 while it's horizontal lines are of course 1080, hence 1080p). You're not the first person to be confused by this.

Professional encoders who fully understand the encoders and the schemes in use and care about not seeing artifacting or low quality would never intentionally go as low as 300mb for a feature length movie of even an hour. Yes there are people who do such things but they're not well regarded and it won't look even passable on anything larger than a phone screen.

Recognized quality groups that seek low sizes might get an animated feature (less bitrate needed due to lack of fine detail in animation vs real film) in SD quality down to around that. But for most live action content the sizes I see from the best of the best concerned with smaller release sizes are in the 900mb to 1.5GB range for 60-90 minute features.

300mb for a 90 minute live action feature even in SD is just not going to look good, some of the groups who get those sizes make them look even half-passable by running pre-filters in virtualdub that smooth, reduce grain and detail, etc before passing to the encoder. That kind of thing is way beyond anything you're going to learn in a few youtube videos though, that's advanced stuff with scripting.

Think about it this way, if you shoot for 1GB encodes with 265 or AV1 you can store over 900 movies on a 1tb drive which can be had for well under a hundred dollars.

I would like the best and fanciest algorithms to have least dataloss.

There is no magic that will get you where you want. If you want detail preserved you need more bitrate which translates to larger sizes. Modern codecs like HEVC and AV1 mean you need as much as 1/5th the bitrate you needed with old MPEG2/4 encoding schemes used on DVDs, that's darn good savings but it has its limits.

Do as you will but anything live action (non-animated) significantly under 1000kbps average bitrate is going to look awful on a 1080p screen and much worse than what it would look like if you popped your dvd in the disc drive and played it from there.

Opus is fine if you're not worried about compatibility and just playing on a computer.

I believe h.265 has particular handling for "film grain". And it has hardware decoding on just about every chip out there. And you probably already have a hardware encoder, so you can do something like QSV in a reasonable time frame.

300MB for a half-hour is a pretty reasonable bitrate, for one and a half hours it is quite dire.

AV1 and VP9 are likely going to be your highest efficiency "free" codecs. AV1 is the way to go if you mean free as in free open source. It's not very likely to be implemented in many TVs or set-top-boxes, but VLC/ffmpeg will be able to decode any of these. Webm uses vp8 or VP9 which are "free"(made by Google) but it's just more specific settings for sharing online/viewing in browser.

H264/H265 has license fees for non-free software and hardware, but they will be your most widely supported option. H265 is approximately twice as efficient as h264 (meaning you can get the same quality of encode from half the file size).

Regardless of preset I think you can get handbrake to encode something reasonable from any of these codecs. Especially with DVD video you'll be able to crank through videos with modern high efficiency codecs

My short opinion:

Video

h264 is the best option for compatibility. There have been free software encoders and decoders forever and IIRC all patents have expired. Basically every device you will encounter and every software system can play h264 videos and the encoders are fairly good.

AV1 is the best option for quality. It is completely free and is becoming widely supported. It will likely be supported for a long time as it is the first widely available high quality free codec. It is significantly better quality than h264 so will result in smaller files for the same quality or better quality for the same file size. Hardware decoding support has only really become common in devices that hit the market in the last few years. But most new devices will have hardware decoding.

Both of these are web-compatible as well which is nice.

Audio

Opus for lossy and FLAC for lossless are both some of the best codecs in their class, completely free and widely available. There are also both web-compatible.

I encode everything with AV1 and Opus, although, as others have pointed out, there is the disadvantage that older (as in "not new") devices won't have hardware acceleration for those. I don't care about that tho, because 1080p runs just fine on decent CPUs and I want to encode for the future, not yesteryear.

For AV1 I use 10bit, VFR, RF25. If you want, you can play around with the performance profile to trade file size for encoding speed.

For Opus I use 320 kbit/s fixed Bitrate and 7.1 downmix. Note that when dealing with audio and subtitles, you absolutely should include tracks of 'unknown' language! Otherwise they are just thrown away. And of course you should include every other language you care about. You can also choose to pass through Opus encoded tracks automatically.

I prefer MKV as container format, because from all I hear it's flexible and robust. Pretty much everything else is set to default. I'd advise to safe your settings to a preset and to briefly verify encoded videos actually work and contain the desired tracks.

I'd say it depends on what your hardware supports for encoding and for playback.

Best quality at small size = AV1

Good with decent support = h265

Best support = h264

If you want hardware decoding, first check what your gpu actually supports. vainfo can be used on Linux for example.

If you have a very new gpu, it might support av1 decoding. This is currently the best codec and alsi free.

Otherwise I'd prefer hevc and then avc, in that order. Vp8 or vp9 might also be supported, but they never caught on much so encoders and decoders are generally worse for them than the equivalents for other codecs.

I don't know any private person who was prosecuted for using hevc or avc. Most likely your hardware manufacturer and other companies already paid for the license to use hevc/avc. If you live in the EU it's even less of a problem as software patents are not a thing here, i.e. you can't patent a way to do things.

Foe audio opus is the best codec at the momenr.

Please make sure to compare the quality before and after decoding. Even using a more efficient codec I'd be wary of a 300mb sized movie.

I’d use AV1, there should be a normal option somewhere. It has higher quality to size ratio, but requires more performance. Webm mainly uses V9, which is not very efficient. Important for the size is also the setting, one codec can have different levels of compression.

Look at the compatibility of the device you are running it on. Older hardware like the PlayStation Vita, will only work with H.264 AAC.

Handbranks is able to convert this for you with no issues (even on Linux flatpak ftw), and web playback on a Apache2 server is great. But if your planning on watching it on more modern devices, then don't worry too much about it.

https://en.m.wikipedia.org/wiki/X265

Don't overcomplicate things.

Well pretty much every DVD maxes at 480p for storage constraint purposes, so that might be your first issue. Ripping and encoding at a higher resolution than the source nets you nothing but wasted space.

That being said, HEVC is going to be the smallest file size for your rips, with reasonable encoding times and compatibility mostly not an issue.

AV1 is new-ish, so depending on your hardware, could take anywhere from 5x-10x longer, and only net you an average of 15% difference in size reduction, and then you have compatibility to worry about.

I usually go for VP9 and opus in a webm container. This is widely supported, because it is Google sponsored, used on YouTube and supported by all browsers. As other people told you here, DVD are not 720p, they are 480 or 576 interlaced or progressive. You can find more details on Wikipedia.

AV1 is the best compressing standard for video, but it is slower in encoding than HEVC (usually). VP9 is the predecessor, which is compresses better than AVC and comes close to HEVC. Speed wise is comparable to x265.

Opus is the best audio standard, the only reason you may want to avoid it is for incompatibilities with older hardware players (in car radios, BluRay players,...).

I'd use HEVC/H265 anyway, always, the best quality for its size. And AAC for audio