Assignment 2 Part 2: Reverse Engineering

This is an individual assignment. Indeed, it is customized to every student. You may discuss questions and potential solutions with your classmates, but you may not look at their code or their solutions. If in doubt, ask the instructor.

You will need to use the CSUG machines. Your assignment includes binaries that are only guaranteed to work on CSUG machines. They may work elsewhere, but we will ignore any issues that arise from not using the CSUG machines.

If you want to take a slip day, as permitted by your syllabus, your email request must be sent to the instructor before the deadline. The instructor will note it, but will not acknowledge it immediately.

Preliminaries

Download the assignment using the following commands on the CSUG machines:

wget https://cs.rochester.edu/~sree/courses/csc-252-452/fall-2022/a2/a2p2-URID.zip

Replace URID with your URID. Unpack the downloaded file.

Prologue

[This part is not relevant to solving the assignment]

"So, the JWST delivered?"

"Yes. We finally were able to get a better look at that star."

"A civilization able to switch a star on and off to send a message! Imagine what they must be trying to tell us!"

"About that, we decoded the message. It appears ... to be some sort of computer code."

"What?"

"Yes, and it's x86 code!"

"So no primer? No math? So nothing like Contact? The most efficient way to communicate with us is to send us computer code? I guess that makes sense ..."

"We got three assembly files, an object file, and a binary. They appear to be two programs."

"What do the programs do?"

"We don't know. We can't compile one -- the assembly files are corrupted. The other program, the binary, requires a key to unlock. It looks like the assembly files generate the key. We tried brute forcing the key, but the program locked itself after a few attempts."

Setup

The purpose of this assignment to write three C files containing only standard C constructs and no goto statements. The code to be written in the C files must be inferred from looking at incomplete fragments of some supplied assembly language files. In addition to local variables, expressions, conditionals and loops you encountered in Part I, in this part you will also encounter functions, global variables and arrays.

To do a quick test, execute the following commands:

cd a2p2
touch a.c g.c s.c
make

The touch command creates empty files (if they do not exist) or changes the last modified data of the file if it exists. In this case, you should see the following errors from make (some of them will be different for you):

gcc -c -std=c99 -Wall a.c -o a.o
gcc -c -std=c99 -Wall g.c -o g.o
gcc -c -std=c99 -Wall s.c -o s.o
gcc -std=c99 -Wall genkey.o a.o g.o s.o -o genkey
genkey.o: In function `main':
genkey.c:(.text+0x12): undefined reference to `r'
genkey.c:(.text+0x23): undefined reference to `ask'
genkey.c:(.text+0x32): undefined reference to `NP'
genkey.c:(.text+0xc7): undefined reference to `s'
genkey.c:(.text+0xd9): undefined reference to `ncmp'
genkey.c:(.text+0x11b): undefined reference to `ncmp'
collect2: error: ld returned 1 exit status
Makefile:7: recipe for target 'genkey' failed
make: *** [genkey] Error 1

Next, type ./unlock, you should see:

ERROR: Too many attempts.

If you see this, you're all set to begin solving the assignment.

Goal 1: Unlock the unlock program.

The unlock program is obviously locked. Your goal is to defeat the protection built into it. To do this, you must first understand how its protection works. The method I recommend is to trace the program and observe what it is doing.

In general, tracing a program reveals a lot of information about it. Tracing can be done manually, using a debugger like gdb. Alternatively, you can get some partial, but still very useful, information using tools like strace and ltrace. You can also use tools like objdump to obtain disassemblies of the binary and figure out how the protection mechanism operates just like you did in part 1 of this assignment.

However, such full reconstruction is probably unnecessary. Clearly the protection mechanism tracks how many times the program has been called. All you have to figure out is how it does that. Does it contact a server [in a galaxy far far away]? Does it store data in a file?

Simply stepping through the program one instruction at a time using a debugger should give you a lot of information. The unlock program does not include the source code, so you'll have to debug at the assembly level. This should not be very difficult at this point in the course.

The textbook's student website contains some resources for GDB. There is also a good tutorial for debugging assembly language with GDB by Prof. Robert G. Plantz.

Once you figure out what's happening, you may need to use a good hex editor or write your own program to implement a fix.

If you succeed in unlocking the program, you should see this:

0 attempts.
Enter output from genkey for key length 8:

Note the key length (here 8). Now, you can press CTRL+C to exit the program without typing in a key.

Bonus: You can also disable this mechanism if you understand it well enough. Then the count of attempts will stop increasing.

Goal 2: Recreate the C files

As you can see, unlock requires a key generated by genkey. However all you have are three assembly language files a.asm, g.asm, and s.asm. They correspond to the files a.c, g.c and s.c.

You now need to reconstruct the contents of these files by reading the assembly. Note that the assembly language files are mildly corrupted, but you should be able to understand what's happening regardless.

Once you have recreated the files, use the make command to compile genkey. Then run it, using ./genkey. If everything goes well, you should see:

Press CTRL+C to exit.
Enter a positive integer for key size: 8
8 is just right!
Your combination is:
8 55 23 41 23 45 9 3 78

Now run ./unlock again, and type in this combination as the key. If you made a mistake you'll see this:

0 attempts.
Enter output from genkey for key length 8: 8 1 2 3 4 5 6 7 8 9
ERROR: Combination failed.

If not, you'll know you have succeeded.

Run ./package.py and submit to the autograder which will test your submission more thoroughly.

Epilogue

[This part is not relevant to solving the assignment]

"Wow! Who's going to tell the NASA and the ESA?"