ssokolow

The problem with making a custom web server is that you take responsibility for re-solving all the non-obvious security vulnerabilities. I always try to delegate as much network-facing code as possible to a mature implementation someone else wrote for that reason.

Here's how I'd implement it, based on stuff I've done before:

1. Start with either Actix Web or Axum for the server itself.
2. Use std::thread to bring up mpv in a separate thread.
3. Use an async-capable channel implementation like flume as a bridge between the async and sync worlds.
4. If the async side needs to wait on the sync side, include the sending side of a tokio::sync::oneshot in the "job order" object your async code drops into the channel and then have the async task await the receiving side. That way, you can have the async task block on the some kind of completion signal from the sync thread without blocking the thread(s) underlying the task executor.

I'm sure other people have a more teachable way of learning these things but I'm just one of those nerdy guys who's been reading technical materials for pleasure since he was in elementary school and gathered the core "this will tell you how the system is designed so you know what to ask about" knowledge along the way.

For example, I just ran across The TTY Demystified, Things Every Hacker Once Knew, The Art of UNIX Programming, and A Digital Media Primer for Geeks on my own. (Sort of the more general version of "It showed up in the YouTube sidebar one day" or "I landed on it while wandering Wikipedia for fun".)

Beyond that, it's mostly "exposing yourself to things the professionals experience", like running a Linux distro like Archlinux or Gentoo Linux which expect you to tinker under the hood and give you documentation to do so, maybe working through LinuxFromScratch to get exposed to how the pieces of Linux fit together, reading periodicals like LWN (articles become un-paywalled after a week, if you're tight on money or need time to convince yourself it's worthwhile), and watching conference talks on YouTube like code::dive conference 2014 - Scott Meyers: Cpu Caches and Why You Care or "NTFS really isn't that bad" - Robert Collins (LCA 2020).

(I switched to Linux permanently while I was still in high school, several years before YouTube even existed, and I'm only getting back into Windows now that I'm buying used books to start learning hobby-coding for MS-DOS beyond QBasic 1.1, Windows 3.1, Windows 95 beyond Visual Basic 6, and classic Mac OS, so I haven't really picked up much deep knowledge for Windows.)

The best I can suggest for directed learning is to read up on how the relevant system (eg. the terminal, UNIX I/O) works until you start to get a sense for which are the right questions to ask.

What you're running into is that read() does blocking I/O by default and, while you can change that, both approaches (checking for pending data before reading or setting stdin into non-blocking mode so it'll return immediately either way) require different code for Windows and for POSIX, so it's best to let your platform abstraction (i.e. termwiz) handle that.

I have no experience with Bevy or Termwiz, but see if this does what you want once you've fixed any "I wrote this but never tried to compile it" bugs:

use std::time::Duration;

fn flush_stdin(main_terminal: terminal::SystemTerminal) {
    while let Ok(Some(_)) = main_terminal.poll_input(Some(Duration::ZERO)) { }
}

If I've understood the termwiz docs correctly, that'll pull and discard keypresses off the input buffer until none are left and then return.

Note that I'm not sure how it'll behave if you call enter_name a second time and you're still in cooked mode from a previous enter_name. My experience is with raw mode. (And it's generally bad form to have a function change global state as a hidden side-effect and not restore what it was before.)

The answer depends on what's actually going on. I've yet to do this sort of thing in Rust but, when I was doing it in Python and initializing curses for a TUI (i.e. like a GUI but made using text), I remember the curses wrapper's initialization stuff handling that for me.

Because of the way terminal emulation works, there are actually two buffers: the one inside the fake terminal hardware that lets the user accumulate and backspace characters before sending the data over the fake wire when you press Enter and the one on the receiving end of the wire.

Paste your initialization code for your curses wrapper and I'll take a look at it after I've had breakfast.

For an interactive terminal program with the characteristics you want, you need to do two things:

1. Flush stdin before using it, similar to what things like sudo do. (Basically, once your program is all started up and ready to go, read everything that's already in there and throw it away.)
2. Disable the terminal's default echoing behaviour which traces back to when the terminal was a completely separate device possibly half-way around the world from the machine you were logged into on the other side of a slow modem and you didn't want the latency from waiting for the machine on the far end to handle echo. (See this if you want more context.)

Windows and POSIX (Linux, the BSDs, macOS, and basically everything else of note) have different APIs for it. On the Linux side, you want something that wraps the curses library, which can put your terminal in "raw mode" or some other configuration that operates unbuffered and lacking terminal-side echo. On the windows side, it can either be done by wrapping the Windows APIs directly or by using the pdcurses library.

Something like termwiz should do for both... though you'll probably need to reimplement print_typewriter but that should be trivial from what I see of its README.

Wouldn’t you want your SSG to include a dev-server anyways? Zola has zola serve which even does incremental rebuilds, but something less sophisticated should be easy to add to your own (only took me a weekend to add to hinoki including rebuilds, though mostly starting the build from scratch on changes).

I don't want the overhead of looping through an HTTP client and server implementation in places it doesn't need to. I design my tooling based on a test target roughly comparable to the Raspberry Pi 4, performance-wise.

Have you investigated some of the options already now?

A bunch of other things came up, forcing me to put the project on the back burner.

(eg. Most recently (about a week ago), I had my 6-month-old boot drive go bad and it took me several days to rush-order a new NVMe drive, learn ZFSBootMenu, restore my backups, and redesign my backup strategy so that, when the original comes back from RMA, if the ZFS mirroring and snapshotting and the trick to mirror the EFI system partition isn't enough to ensure high availability, a full, bootable backup of the NVMe pool's contents can be restored in 2 hours or less with the sequential read performance of my first tier of backup being the bottleneck.)

missing flexibility for output paths has been an annoyance.

Hmm. We'll see if I wind up using it. Avoiding deadlinks has been non-negotiable to the point where replicating my WordPress blog on a local httpd, spidering it, and logging the URLs I need to preserve has been one of the big hold-ups.

is that I found Zola to be quite hard to hack on

Hmm. Potentially a reason I'll wind up making my own, given that I've written SSGs in Python before (eg. https://vffa.ficfan.org/ is on a homebrew Python SSG) and I've already got a single-page pulldown-cmark frontend I've gone way overboard on the features for and a basic task-specific Rust SSG for my mother's art website that I can merge with it and generalize.

EDIT: Here's a screenshot of what I mean by saying I've gone way overboard.

and Tera (its templating lang) to be a little buggy / much less elegant than minijinja API-wise.

Hmm. Noted. I think i'm using Tera for my mother's SSG.

Re. link checking, have you seen lychee? When I found out about it, the priority of building my own link checker in my SSG (that was only an idea at that point, I think) basically dropped to zero :D

You accidentally re-used the link to the Zola issue tracker there. I have not yet checked out lychee and I'm getting a docs.rs error when clicking the examples link, so all I can say is that it'll depend on how amenable it is to checking a site rooted in a file:// URL so I don't need the overhead and complexity of spinning up an HTTP server to check for broken links.

Maybe a web of trust for audited dependencies would help. This version of this repo under this hash.

It'll probably please you to know that the alternative lib.rs frontend for the crates repository integrates both cargo-vet and cargo-crev data via an "Audit" tab that appears on the pages for crates that have such information.

I'm using the web UI, so I'm assuming whatever broad-spectrum Markdown rendering library it uses has smart quote rendering turned on.

Edit: and just to be snarky: I didn’t type “…” I typed “…”. ;)

*chuckle* I think Lemmy typed those for you, because I typed three periods and got a Unicode ellipsis, and both of those are also unicode ellipses.

It still returns relative paths if the input was relative

False

and it doesn’t resolve “…”

I'll assume you meant .., since ... is an ordinary filename. (Aside from the "who remembers ...?" feature introduced in Windows 95's COMMAND.COM where cd ... was shorthand for doing cd .. twice and you could omit the space after cd if your target was all dots.)

The reason it doesn't do that is that, when symlinks get involved, /foo/bar/.. does not necessarily resolve to /foo and making that assumption could introduce a lurking security vulnerability in programs which use it.

Ahh, yeah. In the beginning, Rust was built around the idea that individual files and invoking rustc are internal details, only relevant for integration into some other build system like Bazel, and that "normal" Rust projects need to be inside a Cargo project structure.

There is in-development work to have official support for something along the lines of rust-script, but it's still just that... in development. If you want to keep an eye on it, here is the tracking issue.