So after reading through the answers...could compilation be used as a form of encryption?
this post was submitted on 09 Jul 2024
68 points (88.6% liked)
No Stupid Questions
35780 readers
1008 users here now
No such thing. Ask away!
!nostupidquestions is a community dedicated to being helpful and answering each others' questions on various topics.
The rules for posting and commenting, besides the rules defined here for lemmy.world, are as follows:
Rules (interactive)
Rule 1- All posts must be legitimate questions. All post titles must include a question.
All posts must be legitimate questions, and all post titles must include a question. Questions that are joke or trolling questions, memes, song lyrics as title, etc. are not allowed here. See Rule 6 for all exceptions.
Rule 2- Your question subject cannot be illegal or NSFW material.
Your question subject cannot be illegal or NSFW material. You will be warned first, banned second.
Rule 3- Do not seek mental, medical and professional help here.
Do not seek mental, medical and professional help here. Breaking this rule will not get you or your post removed, but it will put you at risk, and possibly in danger.
Rule 4- No self promotion or upvote-farming of any kind.
That's it.
Rule 5- No baiting or sealioning or promoting an agenda.
Questions which, instead of being of an innocuous nature, are specifically intended (based on reports and in the opinion of our crack moderation team) to bait users into ideological wars on charged political topics will be removed and the authors warned - or banned - depending on severity.
Rule 6- Regarding META posts and joke questions.
Provided it is about the community itself, you may post non-question posts using the [META] tag on your post title.
On fridays, you are allowed to post meme and troll questions, on the condition that it's in text format only, and conforms with our other rules. These posts MUST include the [NSQ Friday] tag in their title.
If you post a serious question on friday and are looking only for legitimate answers, then please include the [Serious] tag on your post. Irrelevant replies will then be removed by moderators.
Rule 7- You can't intentionally annoy, mock, or harass other members.
If you intentionally annoy, mock, harass, or discriminate against any individual member, you will be removed.
Likewise, if you are a member, sympathiser or a resemblant of a movement that is known to largely hate, mock, discriminate against, and/or want to take lives of a group of people, and you were provably vocal about your hate, then you will be banned on sight.
Rule 8- All comments should try to stay relevant to their parent content.
Rule 9- Reposts from other platforms are not allowed.
Let everyone have their own content.
Rule 10- Majority of bots aren't allowed to participate here.
Credits
Our breathtaking icon was bestowed upon us by @Cevilia!
The greatest banner of all time: by @TheOneWithTheHair!
founded 1 year ago
MODERATORS
Hardest part is its a lossy 'encoder' which the parts lost are the human readable part
No, because it would not be possible to retrieve the original information.
It is not. idk who told you it was.
Disassembling an executable is trivial to do. Everything is open source if you can read assembly. Obfuscation be damned.
The hard part isn't reading assembly. The hard part is figuring out why it's doing what it's doing with no comments or function names or anything useful to help.
This is like saying if you can read English you can understand an advanced math or physics paper written in English without having any knowledge or context of those subjects.
Well decompiling is only one step in the reverse engineering process. I would recommend taking a look at the Legend of Zelda: Ocarina of Time decompile projects. They reversed engineered the whole thing, which took years and was a team effort.
In the end they got perfectly readable source code, fully documented. And the most amazing thing is, when compiled with the right compiler and right flags, it recreates the original rom perfectly.
I would also recommend a YouTuber called Kaze. He's been working on Mario 64 for years, re-writing large parts of the engine to get some pretty cool stuff going.
I've used a decompiler to peek at the source code of an app written in Visual Basic I wanted to recreate as a browser addon. It was mostly successful but some variable and function names were messed up.
Variable names, class names, package structure, method names, etc. won't normally be maintained in the disassembled code. They are meaningless to the CPU, and just a series of memory addresses. In cases where you have method names being mentioned, it's likely a syscall, and it's calling a method from an existing library. I'm not familiar with VB, but at least in .Net and .Net Framework, this would be something like the System.Collections.Generic providing the implementation for List and when .Sort() is called, it makes the syscall to that compiled .dll.
Imagine being presented with an aircraft. You bloody well know what it does and you get permission to disassemble the whole thing to your heart's content. How big of a task do you think it'd still be to be able to work out how the winged metal tube works and why it does what it does when it does it?
Exactly.
To add on "the first programs" written in assembler: You'll find they have some structure to them. As they've been written by humans. You can recognize the conditions, loops, ... And they've grouped similar code together... A compiler does none of that. It'll be happy to make a complete mess, re-organize it, combine chunks, do away with loops and other stuff if it can be done more efficiently another way. It'll be more optimal (ideally) but generally unrecognizable to the human eye what happens in that code. And it might be one big pile of instructions, jumping around arbitrarily without any subdivision into chunks that would be logical to go together.
It's not impossible, just very labour intensive and difficult. Compiling an abstract, high level language into machine code is not a reversible process. Even though there are already automated tools to "decompile" machine code back to a high level language, there is still a huge amount of information loss as nearly everything that made the code readable in the first place was stripped away in compilation. Comments? Gone. Function names? Gone. Class names? Gone. Type information? Probably also gone.
Working through the decompiled code to bring it back into something readable (and thus something that can be worked with) is not something a lone "very smart person" can do in any reasonable time. It takes likely a team of smart people months of work (if not years) to understand the entire structure, as well as every function and piece of logic in the entire program. Once they've done that, they can't even use their work directly, since to publish reconstructed code is copyright infringement. Instead, they need to write extremely detailed documentation about every aspect of the program, to be handed to another, completely isolated person who will then write a new program based off the logic and APIs detailed in the documentation. Only at that point do they have a legally usable reverse engineered program that they can then distribute or modify as needed.
Doing this kind of reverse engineering takes a huge amount of effort and motivation, something that an app for 350 total sneakers is unlikely to warrant. AI can't do it either, because they are incapable of the kind of novel deductive reasoning required for the task. Also, the CarThing has actually always been "open-source", and people have already experimented with flashing custom firmware. You haven't heard about it because people quickly realised there was no point - the CarThing is too underpowered to do much beyond its original use.
To build on/give some example about what you said with the comments and function names (programmers, excuse the sloppy pseudocode that's about to follow, it's been a long time since high school intro to computer science)
Let's say in a video game, you run around collecting coins, and if you get 100 coins you earn an extra life
One small part of that code may look something like:
IF
newGame = TRUE
THEN
coinCount = 0
lifeCount = 3
coinModel.all.visibility = TRUE
//Players start a new game with 3 lives and 0 coins, and all coins are visible in the level
IF
playerModel.isTouching.coinModel.x = TRUE
THEN
coinModel.x.visibility = FALSE
coinCount++
//If the player character model touches one of the coin models, that coin model disappears, and the players coin count is increased by 1
IF
coinCount % 100 = 0
THEN
lifeCount++
//if that coin count is divisible evenly by 100, then the players life count is also increased by 1
Quick notes for people who have even less programming background than me
++ Is used by a lot of programming languages to increase a value by 1
% is often used as the "modulo" operator, which basically returns the remainder from division. So 10 % 2 = 0, because 10 is evenly divisible by 2, 10 % 3 = 1, because 10 is divisible by 3 but not evenly and leaves a remainder of 1
// Are comments, they don't affect the code, they're just there for human readability to make it more understandable, so you can explain why you did what you did for anyone who has to maintain the code after you, etc.
Hopefully, between the simple variable names and comments, those pseudocode blocks all pretty readable for laypeople, but if not
The first block basically detects if you're starting a new game (IF newGame = TRUE)
If it is, then it resets your life counter to a default 3, and you start with 0 coins and sets all of the coins in the level to be visible so you can collect them
Otherwise it would carry over the values from your previous level, or save game, or whatever
The second block detects if you touch a coin (playerModel.isTouching.coinModel.x = TRUE)
If you do, that coin vanishes (coin.x.visibility = FALSE)
It also increases your coin count (coinCount++)
Then if your coin count is divisible evenly by 100 (coinCount % 100 = 0) it increases your life total (lifeCount++)
When the code gets compiled, that gets turned into machine code, basically all 1s and 0s that the computer can understand. The computer doesn't care if you call a coin a coin or if you call it object1, it's going to strip all of those human-readable elements out because it would just be a waste of storage and processing power to keep it in.
So when you recompile that, you don't get any of the explanatory comments or the easy to read variable names, so you might end up with something looking kind of like this
IF
Variable1 = TRUE
THEN
Variable2 = 0
Variable3 = 3
object1.all.condition1 = TRUE
IF
object2.condition2.object1.x = TRUE
THEN
object1.x.condition1 = FALSE
variable2++
IF
variable2 % 100 = 0
THEN
variable3++
Which is a lot harder to understand. The code will still work, you could recompile it and run it, but if you want to make any changes, you'd basically need to comb through it, figure out what all the variables, objects, conditions, etc. are, and try to piece together why the programmers who originally wrote the code did it the way they did
And that's of course a bit of an oversimplification, for various reasons it may not decompile and recompile exactly 1:1 with the original code, it's almost like translating the same sentence back and forth between 2 languages with Google translate.
And even this little snippet of fairly simple and straightforward code would probably going to be backed up by dozens, if not hundreds or thousands of other lines of code just to make this bit work, defining what a coin is, the hit boxes, animations, how it determines if it's a new game or or continuing a previous game, etc.
Thank you for adding this! If people want a real life example of the effect shown in this pseudocode, here is a side-by-side comparison of real production code I wrote and it's decompiled counterpart:
override fun process(event: MapStateEvent) {
when(event) {
is MapStateEvent.LassoButtonClicked -> {
action(
MapStateAction.LassoButtonSelected(false),
MapStateAction.Transition(BrowseMapState::class.java)
)
}
is MapStateEvent.SaveSearchClicked -> {
save(event.name)
}
// Propagated from the previous level
is MapStateEvent.LassoCursorLifted -> {
load(event.line + event.line.first())
}
is MapStateEvent.ClusterClick -> {
when (val action = ClusterHelper.handleClick(event.cluster)) {
is ClusterHelper.Action.OpenBottomDialog ->
action(MapStateAction.OpenBottomDialog(action.items))
is ClusterHelper.Action.AnimateCamera ->
action(MapStateAction.AnimateCamera(action.animation))
}
}
is MapStateEvent.ClusterItemClick -> {
action(
MapStateAction.OpenItem(event.item.proposal)
)
}
else -> {}
}
}
decompiled:
public void c(@l j jVar) {
L.p(jVar, D.f10724I0);
if (jVar instanceof j.c) {
f(new i.h(false), new i.r(c.class, (j) null, 2, (C2498w) null));
} else if (jVar instanceof j.e) {
m(((j.e) jVar).f8620a);
} else if (jVar instanceof j.d) {
List<LatLng> list = ((j.d) jVar).f8619a;
j(I.A4(list, I.w2(list)));
} else if (jVar instanceof j.a) {
d.a a7 = d.f8573a.a(((j.a) jVar).f8616a);
if (a7 instanceof d.a.b) {
f(new i.j(((d.a.b) a7).f8575a));
} else if (a7 instanceof d.a.C0058a) {
f(new i.a(((d.a.C0058a) a7).f8574a));
}
} else if (jVar instanceof j.b) {
f(new i.k(((j.b) jVar).f8617a.f11799a));
}
}
keep in mind, this was buried in hundreds of unlabeled classes and functions. I was only able to find this in a short amount of time because I have the most intimate knowledge of the code possible, having written it myself.
It's always possible to convert binary (machine code) into assembly since they're basically the same thing. Assembly is just human readable binary.
It's not possible to generate high level source code from assembly the same way you can't generate a recipe for a cake by analyzing the composition of the cake. It doesn't matter how smart you are, the temperature the oven was set at and time the cake spent in the oven can't be found in the molecular structure of a cake.
Thanks that's actually a pretty helpful metaphor
It's not impossible, just expensive. How much money do you want to spend?
To your point, the programs are already in code. Machine code. Taking random machine code, and making a human readable, that's the trick
Computer code is very complicated, so when humans write code we write in a way we can understand. We name functions and variables with names that make sense, and we put comments in the code so we can understand how it works.
Compliers don't care about any of those things. Variable names are turned into numbers, and comments are ignored.
You can convert machine code back to source code, it will be missing all those human readable labels and explanations. You can recreate them, but its a major process. Reverse engineering is done sometimes, but there's a reason is not common.
There's also the issue of licensing. An important part of free and/or open source software is that you have permission to modify the source code. You probably don't have a license to use the code if its closed source. There are ways to do this legally but it adds extra hurdles and inconvenience to an already major process.
In addition to the other comments that explained it well… Back in the day, that process was easier in part because executable files had far fewer instructions.
As others have mentioned, it's possible but very complicated. Decompilers produce code that isn't very readable for humans.
I am indeed awaiting the big news headlines that will for some reason catch everyone by surprise when a LLM comes along that's trained to "translate" machine code into a nice easily-comprehensible high-level programming language. It's going to be a really big development, even though it doesn't make programs legally "open source" it'll make it all source available.
I am indeed awaiting the big news headlines that will for some reason catch everyone by surprise when a LLM comes along that's trained to "translate" machine code into a nice easily-comprehensible high-level programming language.
Another commenter dismissed the idea outright. WTF... What is implausible about an LLM that takes decompiled code, deals with the obfuscating bs, recognizes known libraries, and organizes the remaining code. That will totally happen, if it hasn't already been done.
It's easy to say that we should throw AI at a problem and in a few years it will solve it, but most of the time it doesn't actually work that way. If you think about the Turing Test itself, where the history goes back to the 1950s, how many decades did it take for us to get to anything that could reasonably come close to passing it? So anytime you think to yourself that one of these days AI is going to get there, remember that one of these days might actually be a half century from now.
The other aspect to this challenge, or rather specifically with regards to this challenge, is that the setup involves humans organizing code in a certain way according to some kind of reasoning that the authors know about, and then that being compiled away, and then another computer program trying to get back what the original authors might have been thinking when they designed the thing originally. That's a steep hill to climb. Can it be done on a small scale? It certainly can. On a large scale? Don't hold your breath.
There's a lot of outright rejection of the possibilities of AI these days, I think because it's turning out to be so capable. People are getting frightened of it and so jump to denial as a coping mechanism.
I recalled reading about an LLM that had been developed just a couple of weeks ago for translating source code into intermediate representations (a step along the way to full compilation) and when I went hunting for a reference to refresh my memory I found this article from March about exactly what's being discussed here - an LLM that translates assembly language into high-level source code. Looks like this one's just a proof of concept rather than something highly practical, but prove the concept it does.
I wonder if there are research teams out there sitting on more advanced models right now, fretting about how big a bombshell it'll be when this gets out.
I have a bunch of 16-bit applications that I would love to be able to do that with. Mostly dos and windows 3.1 games.
You might actually consider dipping your toes into trying to learn how to analyze/reverse those yourself. Relatively speaking, software that old can sometimes be easier to reverse.
Yeah I'm not unfamiliar (still a novice though) with the process and mostly used it circumvent something obnoxious or tweak save files. Just takes a lot of effort when you're just looking to spend a couple hours playing a game before bed.
I'm currently experiencing a frustrating bug in dolphin and I'm being tempted to learn enough about it. My MIPS buddy won't help me with it because he thinks it's a waste of time.
I like LLMs for the time it saves you to do something laborious or mundane. One day we'll have general ai fingers crossed
~Love the toes pun
My apologies for preaching to the choir. (And I didn't notice your username when I wrote that, LOL. Happy accident.)
Apps are huge and compilers optimize the fuck out of the code. Code optimization doesn't always make sense so you need to have a detailed understanding of which compilers were used. There could be hundreds of libraries involved or even layers of obfuscation in some cases. Loops can be unwrapped, or other bits of code optimized for specific architectures. Some of the logic won't appear logical.
Disassemblers can do a decent job converting code back to C/C++, but even then, you have to go through the code line by line converting function names and variable names back to something that can be referenced later as a meaningful name.
You aren't wrong: All the code is there. It's just a matter of putting all the human readable references back into anything you disassembled.
Waaaay back in the day, we could tear apps apart easily if they were small. There were only a few flavors of assembly and compilers were still fairly basic for what they were. Regardless, it wasn't a small task.
I played around with cracking for a while just to learn about it and honestly, it was kinda easy before everything was offloaded to "the cloud". It's just a matter of tracing execution and finding a few critical comparisons or jumps to alter. Even then, it could take me a day or two just to walk through what was basically one or two functions.
It's not. I believe lowlevellearning has a tutorial on tearing down binaries. If not him, john hammond does for sure. Both are on youtube. That skill set is usually employed in security research since it pays more than reverse engineering old software with problematic licensing and uncertain ownership.
-
most software packages are ridiculously complicated. it's not as simple as just running a decompiler and seeing code. It's labor intensive, and loaded with bugs and errors, many of which you would never catch unless you already had in idea of what was supposed to be there.
-
many applications rely on external services/system packages that may or may not exist on your machine.
-
companies take steps to protect their application from it being reverse engineered, making it that much more difficult to actually pull off.
-
you don't have access to the documentation/commenting that would be in the uncompiled code, turning a lot of the script into incomprehensible jibberish.
-
all the labor involved means it's very likely to not pass the cost/benefit analysis. unless you're able to add something to it; something the other guy doesn't have.... then you're not going to be getting a substantial market share. It won't be profitable.
It's not impossible. It's being done all the time. It's just tedious complicated work. So if you don't have someone willing to invest their time and expertise it won't be done for most stuff.
Yes, and people do do it. It's just incredibly difficult to do it even for relatively simple programs, and the more complex the program is, the more exponentially hard the reverse engineering will be.
The problem is not necessarily turning it into code, since many decompilers do it already for you nowadays. The issue is understanding what in the world the code is supposed to do. Normally, open source code would be commented and there would be documentation, so it's easy to edit or build on the code. Decompiled code comes with no documentation or comments, and all the variable names are virtually illegible.
It's sometimes easier to build something new than to fix what's broken, and this would be one of those cases where it's true
people do do it.
giggles under my breath
We can and have done this, but there's not much gain, which is why it's mostly done by hobbyists to their favorite older software whose parent company went bust. It's especially common for older games.
It's not impossible just difficult.
You could use a tool like ghidra for example to study a program and workout what everything does.
Assuming you have all the source code... it is possible. It's usually a huge pain in the ass though and software is so complicated that it's extremely difficult to get anything useful.