Programming

17432 readers

227 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Follow the programming.dev instance rules
Keep content related to programming in some way
If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities !webdev@programming.dev

founded 1 year ago

MODERATORS

snowe@programming.dev

Ategon@programming.dev

MaungaHikoi@lemmy.nz

Project idea for learning (lemmy.ca)

submitted 10 months ago by GreatBlueHeron@lemmy.ca to c/programming@programming.dev

26 comments fedilink hide all child comments

I'm a retired Unix sysadmin. Over the years I've built things in COBOL, FORTAN, C, perl, rexx, PHP, visual basic, various Unix shells and maybe others. Nothing has been a real "application" - mostly just utilities to help me get things done.

Now that I'm retired, and it's cold outside, I'm curious to try some more coding - and I have an idea.

The music communities here seem to post links to YouTube. I generally use Lemmy on my phone but don't use YouTube, or listen to music, on my phone if I can help it. I'd like to scrape a music community here and add the songs posted to a playlist in my musicbrainz account.

Does that sound like a reasonable learner project? Any suggestions for language and libraries appreciated. My preferred IDE is vim on bash and I have a home server running Linux where this could run as a daemon, or be scheduled.

you are viewing a single comment's thread
view the rest of the comments

[–] Diabolo96@lemmy.dbzer0.com 1 points 10 months ago* (last edited 10 months ago) (1 children)

Indeed. while my bash-fu is redimentary at best, I don't think Bash can be used for web scrapping ? But I think he could use RSS to get the posts, then extract youtube links with Regex and use the dump feature of yt-dlp* to get the video category, title,etc by using jq to parse the json. Then, it's probably just a matter of using curl to do the API calls and voilà.

*yt-dlp is better maintained than youtube-dl, or so I heard.

[–] some_guy@lemmy.sdf.org 2 points 10 months ago (1 children)

I built two scrapers for a website that hosts images and videos using bash.

They're educational, I swear! /s

I looked through the html and figured out regexes for their media. The scripts will parse all the links on the thumbnail pages and then load the corresponding primary pages with curl. On those pages, it then uses wget to grab the file. Some additional pattern matching names the file to the name of the post.

It's probably convoluted, but you can accomplish a lot in bash if you want to.

[–] Diabolo96@lemmy.dbzer0.com 2 points 10 months ago (1 children)

Man, there's something really wrong with lemmy lately. I only got the notification for your comment 8 days after you sent it. It's the third time this happens but this must be the longest time before the notification reaches me.

[–] some_guy@lemmy.sdf.org 2 points 10 months ago (1 children)

Yes, there's a discussion about this on my instance. Someone there provided a link to where this was getting addressed. Some aspects of federation have been broken for a bit.

https://github.com/LemmyNet/lemmy/issues/4288#issuecomment-1878442186

[–] Diabolo96@lemmy.dbzer0.com 1 points 10 months ago (1 children)

Hope it get fixed soon.

[–] some_guy@lemmy.sdf.org 1 points 10 months ago

Seems like it. My inbox had five replies yesterday (after >1w of only local replies). Today, even more. Yesterday, the GUI was partially broken. Today looks normal.