technology

23890 readers

414 users here now

On the road to fully automated luxury gay space communism.

Spreading Linux propaganda since 2020

Rules:

1. Obviously abide by the sitewide code of conduct. Bigotry will be met with an immediate ban
2. This community is about technology. Offtopic is permitted as long as it is kept in the comment sections
3. Although this is not /c/libre, FOSS related posting is tolerated, and even welcome in the case of effort posts
4. We believe technology should be liberating. As such, avoid promoting proprietary and/or bourgeois technology
5. Explanatory posts to correct the potential mistakes a comrade made in a post of their own are allowed, as long as they remain respectful
6. No crypto (Bitcoin, NFT, etc.) speculation, unless it is purely informative and not too cringe
7. Absolutely no tech bro shit. If you have a good opinion of Silicon Valley billionaires please manifest yourself so we can ban you.

founded 5 years ago

MODERATORS

context@hexbear.net

EmmaGoldman@hexbear.net

SexUnderSocialism@hexbear.net

gaycomputeruser@hexbear.net

ZoomeristLeninist@hexbear.net

Trying to set up PySpark on Windows. Need help. (hexbear.net)

submitted 1 year ago* (last edited 1 year ago) by Tomorrow_Farewell@hexbear.net to c/technology@hexbear.net

12 comments fedilink hide all child comments

Have been trying to set it up for hours now. Nothing works.

Latest version does not seem to have winutils support, and using it causes errors when using some important methods. (EDIT: this is likely wrong, and the winutils stuff that I have should probably be fine.)
Older versions require to be built with Maven. However, that just gives me a PluginExecutionException.

I need to do this ASAP, preferably within the next 3 hours.

I have nowhere else to ask for help, it seems, especially considering that reddit-logo suspended an account I set up specifically for asking questions after I edited a relevant post.

Highly doubt that anybody will be able to help me.

EDIT2: the issue has, thankfully, been resolved. I was using Python 3.12, and switched to 3.11.8. That made the problem go away.

you are viewing a single comment's thread
view the rest of the comments

[–] bunnygirl@hexbear.net 2 points 1 year ago (1 children)

Sorry for the late response,

I can't help for PySpark specifically cause I have no experience with it. In general tho you'll have to install the tooling you need to compile/run the program in WSL, I haven't used Spark in years so I don't know specifics but you'll want to have at least Java and Python installed here. On Ubuntu, you'll want the packages default-jdk, python3, python3-pip, python3-venv (if you're using venv), as well as python-is-python3 for convenience. If you're using venv, you might want to rerun python -m venv env again to make sure it has the files Bash needs, then do source env/bin/activate to activate the venv. You might also have to install pyspark from the Bash shell in case it needs to build anything platform specific. You can set environment variables in ~/.bashrc (It's the home dir in the Linux VM, not Windows so use the terminal to change this e.g. nano ~/.bashrc or vim ~/.bashrc if you're familiar with vi) with the shape export VARIABLE=VALUE (put quotes around VALUE if it has spaces etc), then start a new shell to load those (do exec bash to replace the currently running shell with a new process)

From there you should be able to just run the code normally but in WSL instead

[–] Tomorrow_Farewell@hexbear.net 2 points 1 year ago

Thankfully, the problem has been resolved, so, after I finish with this project, I will have more time to get an actual programming set up on a non-VM NixOS that I have already installed.