Programming

21332 readers

309 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Follow the programming.dev instance rules
Keep content related to programming in some way
If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities !webdev@programming.dev

founded 2 years ago

MODERATORS

snowe@programming.dev

Ategon@programming.dev

MaungaHikoi@lemmy.nz

UlrikHD@programming.dev

Lukas Atkinso: Net-Negative Cursor (lukasatkinson.de)

submitted 1 month ago* (last edited 1 month ago) by HaraldvonBlauzahn@feddit.org to c/programming@programming.dev

12 comments fedilink hide all child comments

What I think in addition to what Atkinso writes: If you just strip arbitrary bytes that happen to be equal in value to the numeric value of ASCII control characters or whitespace, how can you be sure that you don't destroy valid non-whitespace unicode symbols?

You can't! This will work only of you have actually ASCII input.

you are viewing a single comment's thread
view the rest of the comments

[–] HaraldvonBlauzahn@feddit.org 0 points 1 month ago (8 children)


let mut bytes = vec![0u8; len as usize];
    buf.read_exact(&mut bytes)?;

// Sanitize control characters
let sanitized_bytes: Vec<u8> = bytes.into_iter()
    .filter(|&b| b >= 32 || b == 9 || b == 10 || b == 13) // Allow space, tab, newline, carriage return
    .collect();

This implicitly, and wrongly, swaps the interpretation of the input from UTF8 text to pure ASCII.

[–] setsubyou@lemmy.world 6 points 1 month ago (2 children)

In UTF8, all bytes that are not an ASCII character have the high bit set.

[–] sxan@midwest.social 1 points 1 month ago

This person Unicodes

[–] HaraldvonBlauzahn@feddit.org -2 points 1 month ago (2 children)

You are right with this. But still, in Rust, a vector of u8 is different from a sequence of unicode characters. This would not work in Python3 either, while it'd work in Python2.

[–] brian@programming.dev 2 points 1 month ago* (last edited 1 month ago) (1 children)

~~yeah it's incorrect bc it destroys multibyte characters, but~~ no idea what you're saying about u8 being a different type from unicode. the original code was reading bytes and converting them too? the typing isn't the issue, you can still store utf8 as a series of bytes

[–] Markaos@discuss.tchncs.de 1 points 1 month ago (1 children)

it's incorrect bc it destroys multibyte characters

It doesn't. As the poster two levels up said, all bytes that don't represent an ASCII character have the high bit set, even the follow-up bytes in multibyte sequences. So the condition b >= 32 will match and preserve them.

[–] brian@programming.dev 4 points 1 month ago

yeah fair enough. that wasn't really my point and I wasn't paying attention

[–] FizzyOrange@programming.dev 1 points 1 month ago (1 children)

No, the filter is correct even for UTF-8. Any ASCII character is exactly unchanged in UTF-8 (part of the reason it is popular). Since this code only filters out ASCII characters it works fine with ASCII or UTF-8.

load more comments (5 replies)