Hashing won't fix sloppy typos/grammar.
How much would hash ~~digsets~~ digests be shortened if the whole ~~alphabel~~ alphabet was used ?
this post was submitted on 30 Jan 2024
13 points (100.0% liked)
Technology
37705 readers
82 users here now
A nice place to discuss rumors, happenings, innovations, and challenges in the technology sphere. We also welcome discussions on the intersections of technology and society. If it’s technological news or discussion of technology, it probably belongs here.
Remember the overriding ethos on Beehaw: Be(e) Nice. Each user you encounter here is a person, and should be treated with kindness (even if they’re wrong, or use a Linux distro you don’t like). Personal attacks will not be tolerated.
Subcommunities on Beehaw:
This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.
founded 2 years ago
MODERATORS
It gets subtle when you consider Unicode. But you said latin alphabet, so you can look at just the UTF-8 section of this table, and assume 1byte = 1letter.
https://github.com/qntm/base32768#base32768
HTH
I think we should consider Unicode, I want hashes that look like lau52gj🍀pr18e🍅
Yeah you can always take a hex hash output and convert it to Base64...which does conpress it significantly. Apply LZ Compression and boom.
Apply LZ Compression and boom.
That would produce a binary stream. If that's what OP wants, they could just leave the original hash in binary. And that would be unlikely to compress any further since hashes are, by their nature, high entropy already.
Tried to convert to base 64 and.. it actually makes it longer. Why?
You didn't convert a hex number into Base64, you Base64 encoded the hex string.
TL;DR, you used the wrong tool.
Whats the right tool? Cant seem to find one
If you're using Linux (or macOS or MinGW or CygWin or MSYS), you can do something like this in the terminal:
xxd -r -ps | base64
The first command will read the standard input and decode hex strings back into raw data, and the second one will do base64 to the output.
If I pass the hex string mentioned in your original post through this command, I get:
Z3nFNDK4ut8Em7nYkkpXhd2IckM=
So, hexadecimal uses 16 characters. Each character stores 4 bits of data (2⁴ = 16).
If you use the 10 digits and 26 letters of the Latin alphabet, the resulting encoding is called Base36.
It is a rather impractical format for storing data, though, because for purposes of simple conversion, the number of possibilities should be a power of 2 -- that way a program can do (quick) bit shifts instead of (difficult, especially on big numbers) division to determine which character to use. That's why it's mostly used to encode numbers, and not large sequences of data.
Base32 is a slightly-smaller variant that can fit 5 bits of data into one character. (2⁵ = 32)
If you add up digits, uppercase and lowercase characters together (differentiating between upper and lower case), you get 62. This is also an impractical number for computer purposes. But add two extra characters and you get 64, which is another nice power of two (2⁶ = 64), letting one character store 6 bits. And Base64 is a common encoding scheme for data.
And when you know how many bits a character can fit, you can calculate how "efficient" the encoding will be and how many characters will be needed to store data. A Base32 encoding will need 20% fewer characters than hexadecimal, and Base64 needs 33.3% fewer.
You could try base64 maybe? The above would be: Z3nFNDK4ut8Em7nYkkpXhd2IckM= (28 chrs)
base64 uses A-Z, a-z, 0-9, and the + and / characters to encode 6 bits per character. That means you can encode every 3 bytes (or 6 hex) in 4 characters (since 3 * 8 bits = 4 * 6 bits). If the data are not a perfect multiple of 3 bytes, the last group of 4 characters gets padded out with = signs.
There's also a Base64URL variant that is a little more friendly in the modern world where the +/= often need escape sequences.
The first two are replaced with more sensible characters and the third is just removed entirely - do you really need padding?
- It's a 160-bit hash, so using letters and numbers, it'd be log base (10+26) of 2^160, which is roughly 31. So 31 letters.
- Using upper and lower case, it'd be log base (10+26+26) of 2^160, or 27 letters.
- Don't use SHA-1; use SHA-256
- Upper and lower case to represent SHA-256 would be log base (10+26+26) of 2^256, 43 letters
- Internally, it's represented using 32 "letters" of 8 bits each, effectively using every possible ASCII character. The string representation is only of consequence when you're exchanging it over a medium where it needs to be robust and human-readable, and probably the benefit from squeezing it down to fewer characters for that representation is not worth the cost in terms of making it unclear how you've chosen to squeeze it and making life difficult for people who are trying to convert to and from the format. Hexadecimal is a little bigger but it's very clear and unambiguous what you've done, whereas using the full alphabet doesn't have that property.
https://www.unitconverters.net/numbers/decimal-to-base-36.htm
base 10 = 590741618446309885662238049322513167918815539779
base 16 = 6779C53432B8BADF049BB9D8924A5785DD887243
base 36 = C34WAO39N9K9XWPHW5W9XGRH0AHT0CG