That's kind of how file compression works - by replacing patterns with shorter symbolic representations!
this post was submitted on 29 Aug 2023
63 points (95.7% liked)
No Stupid Questions
35719 readers
2097 users here now
No such thing. Ask away!
!nostupidquestions is a community dedicated to being helpful and answering each others' questions on various topics.
The rules for posting and commenting, besides the rules defined here for lemmy.world, are as follows:
Rules (interactive)
Rule 1- All posts must be legitimate questions. All post titles must include a question.
All posts must be legitimate questions, and all post titles must include a question. Questions that are joke or trolling questions, memes, song lyrics as title, etc. are not allowed here. See Rule 6 for all exceptions.
Rule 2- Your question subject cannot be illegal or NSFW material.
Your question subject cannot be illegal or NSFW material. You will be warned first, banned second.
Rule 3- Do not seek mental, medical and professional help here.
Do not seek mental, medical and professional help here. Breaking this rule will not get you or your post removed, but it will put you at risk, and possibly in danger.
Rule 4- No self promotion or upvote-farming of any kind.
That's it.
Rule 5- No baiting or sealioning or promoting an agenda.
Questions which, instead of being of an innocuous nature, are specifically intended (based on reports and in the opinion of our crack moderation team) to bait users into ideological wars on charged political topics will be removed and the authors warned - or banned - depending on severity.
Rule 6- Regarding META posts and joke questions.
Provided it is about the community itself, you may post non-question posts using the [META] tag on your post title.
On fridays, you are allowed to post meme and troll questions, on the condition that it's in text format only, and conforms with our other rules. These posts MUST include the [NSQ Friday] tag in their title.
If you post a serious question on friday and are looking only for legitimate answers, then please include the [Serious] tag on your post. Irrelevant replies will then be removed by moderators.
Rule 7- You can't intentionally annoy, mock, or harass other members.
If you intentionally annoy, mock, harass, or discriminate against any individual member, you will be removed.
Likewise, if you are a member, sympathiser or a resemblant of a movement that is known to largely hate, mock, discriminate against, and/or want to take lives of a group of people, and you were provably vocal about your hate, then you will be banned on sight.
Rule 8- All comments should try to stay relevant to their parent content.
Rule 9- Reposts from other platforms are not allowed.
Let everyone have their own content.
Rule 10- Majority of bots aren't allowed to participate here.
Credits
Our breathtaking icon was bestowed upon us by @Cevilia!
The greatest banner of all time: by @TheOneWithTheHair!
founded 1 year ago
MODERATORS
If I remember correctly, you can even pre-build dictionaries for some compression algorithms to speed up the process if you're working with compressing lots of similar data.
That’s literally how compression works
I figured that's roughly how JPEGs work but I wasn't sure how exactly text compression would work. Do you have any recommendations for videos on data compression?
Jpeg compression is much more complex than what you described. I recommend this video as a primer: https://youtu.be/0me3guauqOU
its a good idea, you might want to look at huffman codes to start your reading
This technique is called Huffman coding, and it's used in a number of compression algorithms.
Huh that was an interesting read. I'm not sure I understood the entirety of it but it sounds like it would be a lot more efficient than what I was thinking
Yes, it's very efficient and the core of what complession formats like .zip do.
The main difference to your idea is that computers count in binary like 0, 1, 10, 11, 100, 101 and so on and that you don't want to assign these very low codes words directly. Say you'd have assigned 1 the most common word, then that will would be encoded very short, but you'd sort of take the one away from all the other codes, as you don't know if 11 is twice the most common word or once the 11th (3rd in decimal) common word.
Huffman essentially computes the most optimal word assignments mathematically.
The main other difference between your suggestion and most compression algorithms is that you wouldn't use a huge dictionary in real time and loading it and looking into it would be very slow. Most compression algorithms have a rather small dictionary buildin and/or they build one on the fly looking at the data that they want to compress.
That's pretty much what a tokenizer does for Large Language Models like Chat-GPT. You can see how it works here: https://platform.openai.com/tokenizer
Type in the word 'Antidisestablishmentarianism' and you can see it becomes 5 tokens instead of 28 characters.
To get the data, search "frequency list (of English)".
Computers are quick enough nowadays its better to just count how many of each word in a block, then use it as indices. Saves from having a dictionary the size of the English language for decompression
The best compression algorithms are good at this. Look at the size of this Reddit download and compare when you uncompress it.
There's a website, I can't remember the link, but any text you search can be found to have already been written on one of its pages.
It used an algorithm to generate every possible page of text in the English language, including gobbledygook and random keyboard smashing.
You can then share a link to a full page of text that is far smaller than the page itself, given infinite computational resources, it would be possible to parse any program or any bit of software into its text equivalent and then generate the URL that attaches to this algorithm for that entire page reducing a thousand characters to 16.
It would then be possible to make a page consisting of these 16 digit characters and then compress that page to another page, turning 70,000 words of text or so into 16 digits.
Once you had a page full of those, you could compress them again by finding a page that contains a link to the links of all of the pages.
Given that this is perfect compression you could literally reduce any file to the URL of the page that you need to use (and some information about how many pages deep you have to go and any needed information on the final file structure and format at the end of the reconstruction) to regenerate the page that you need to use to regenerate the page that you need to use to regenerate the page and so on until you have fully reconstructed the file.
And, given that these pages that can be summoned by a URL and do not actually exist until they are generated by the algorithm, it is entirely feasible that the algorithm itself could be stored on your local computer, meaning that with a somewhat complicated but still reasonable system, you could send a file of any size to any person in a text.
The problem is the computational resources needed to utilize this algorithm. It would have to crunch out the numbers to rebuild every single file that is sent its way and even on a fairly fast system that's going to take some time.
However, if anyone wants to take this idea and run with it I'm pretty sure the world would appreciate your efforts. Since we have the ability to create a page of text using an algorithm then we probably should use that and then once it becomes commonplace I'm sure people will build chiplet accelerators for the process.
There’s a website, I can’t remember the link, but any text you search can be found to have already been written on one of its pages.
It's The Library of Babel
You can type up to 3200 characters in lower case. With a short sentence though, the "title" and page number may be longer than your original text!
Edit: Fixed url title
This is kind of blowing my mind.
It's the Library of Babel. Sorry, pedantic, I know, but the Tower of Babel is something different. If people go searching for that they're going to find a lot of discussion around the bible story and no links to the text generator.
Edited/Fixed.
You cannot represent everything using english text, and text in known languages can be extremely compressed just because we know details about its structure. (And anyway, it cannot be compressed that extremely, information theory explains this very well, computational power isn't the only limit here).
If you cannot represent everything using valid english text, you cannot compress at high rates without losing information. A big part of digital data is actually noise, and noise cannot be compressed by definition.
There is simply no way to pre-store any arbitrary long text because the possible combination is just tooo huge. It is like draw a bullseye where you arrow landed. You search any paragraph, it search existing index, if not found then save it and assign it a new index. Of course it may use some algorithms to optimize/decrease the space needed.
From it's wiki page, https://en.m.wikipedia.org/wiki/The_Library_of_Babel_(website) ''' The website can generate all possible pages of 3200 characters and allows users to choose among about 10^4677 potential pages of books. '''
PS, I think again, indeed this actually does not need any storage, it can be URL to text encoding/decoding, the URL itself can determine/generate the actual text. And same for the opposite direction
it would be possible to parse any program or any bit of software into its text equivalent and then generate the URL that attaches to this algorithm for that entire page reducing a thousand characters to 16.
This can't work. Let's use a simpler example, instead of 16 characters for the link let's say it's a single digit and let's say the content of the "page" is 4 digits. One digit has 10 possible values: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9. 4 digits have 10,000 possible combinations. With only one digit to index into the 10,000 possible combinations, you can point to only 10 of them.
It's the same thing for pages of text. If you have a 16 character link and the content you're trying to index with it is more than 16 characters then you can only point to some of the possibilities in the larger set.
Then how is it that I was able to link to 800 words with 5 characters, (stripping aside the static portion of the links)?
First just think about the logic of what I said before: if there are finite number of combinations in the link, how can you possibly link to a larger number of items? It's just logically impossible.
Then how is it that I was able to link to 800 words with 5 characters, (stripping aside the static portion of the links)?
The fact that you were able to link to 800 words doesn't really mean anything. somesite.com/a could point to a file that was gigabytes. This doesn't meant the file got compressed to a. Right?
There also might be less combinations for that site than it appears. For an 800 word chunk of grammatical English text, there are a lot less combinations than the equivalent length in arbitrary characters. Instead of representing each character in a word, it could just use an id like dog=1, antidisestablishmentarianism=2 and so on. Even using tricks like that though, it's pretty likely you're only able to link to a subset of all the possible combinations.
Regarding compression in general, it's a rule that you can't compress something independent of its content. If you could do that, even if the compression only reduced the file by the tiniest fraction you could just repeatedly apply the algorithm until you end up where the problem I described is obvious. If you could compress any large file down to a single byte, then that single byte can only represent 256 distinct values. However there are more than 256 distinct files that can exist, so clearly something went wrong. This rule is kind of like breaking the speed of light or perpetual motion: if you get an answer that says you have perpetual motion or FTL travel then you automatically know you did something wrong. Same thing with being able to compress without regard to the content.