University of Texas at El Paso

CS 4390/5390 System Security - Attack and Defense for Binaries

Spring 2026, TR 3:00 pm - 4:20 pm, EDUC 112

General Information

Instructor

Dr. MD Armanuzzaman (Arman)
E-mail: marmanuzzaman@utep.edu
Homepage: https://tomal-kuet.github.io/armanuzzaman/
Office hours will be TR 1:30 pm - 3:00 pm or by appointment
CCSB 3.1008 or Teams at Meeting Link

Overview

This course is designed to provide students with good understanding of the theories, principles, techniques and tools used for software and system hacking and hardening. Students will study, in-depth, binary reverse engineering, vulnerability classes, vulnerability analysis, exploit and shellcode development, defensive solutions, etc. to understand how to crack and protect native software. In particular, this class covers offensive techniques including stack-based buffer overflow, heap security, format string vulnerability, return-oriented programming, etc. This class also covers defensive techniques including canary, shadow stack, address space layout randomization, control-flow integrity, etc. A key part of studying security is putting skills to the test in practice. Hacking challenges known as Capture The Flag (CTF) competitions are a great way to do this. In this class the progress of students are evaluated by lab assignment and in-class Capture-The-Flag (CTF) competitions.

Downloads: Course Syllabus

Class CTF Platform: http://yeast.utep.edu:2223/

UTEP VPN Setup Guide: Link

Tentative Schedule

Date Topic Dues
Week-1 Class-1 1/20 Course Overview, Logistics, Introduction to hacking platform; Slides  
Week-1 Class-2 1/22 Background Knowledge; Slides; HW1  
Week-2 Class-1 1/27 Background Knowledge: Process Map, System Calls & Stack; Slides;
Week-2 Class-2 1/29 Buffer Overflow (Local variables); Slides; HW2 HW-1
Week-3 Class-1 2/3 Buffer Overflow (Return address); Slides;
Week-3 Class-2 2/5 Buffer Overflow (Return to shellcode 1); Slides;
Week-4 Class-1 2/10 Buffer Overflow (Return to shellcode 2); Slides; HW3 HW-2
Week-4 Class-2 2/12 Buffer Overflow (Frame Pointer Attack); Slides;
Week-5 Class-1 2/17 Buffer Overflow defenses 1 (DEP, Shadow Stack, Stack Canary); Slides;
Week-5 Class-2 2/19 Buffer Overflow defenses 2 (DEP, Shadow Stack, Stack Canary); Slides; HW4 HW-3
Week-6 Class-1 2/24 ASLR and Seccomp; Slides
Week-6 Class-2 2/26 Buffer Overflow (Real-world Examples); Slides
Week-7 Class-1 3/3 Shellcoding; Slides; HW5 HW-4
Week-7 Class-2 3/5 Shellcoding (In class Activity); Slides;
Week-8 Class-1 3/10 Mid Term Review;
Week-8 Class-2 3/12 =============Midterm CTF; HW6============ HW-5
Week-9 Class-1 3/17 ===============Spring Break===============  
Week-9 Class-2 3/19 ===============Spring Break===============  
Week-10 Class-1 3/24 Format String (Memory Read); Slides;
Week-10 Class-2 3/26 Format String (Memory Write); Slides; HW7 HW-6
Week-11 Class-1 3/31 Format String defenses and In class hacking; Slides;
Week-11 Class-1 4/2 Return-oriented Programming 1; Slides;
Week-11 Class-2 4/7 Return-oriented Programming 2; Slides; HW8 HW-7
Week-12 Class-1 4/9 Return-oriented Programming Defenses; Slides;
Week-12 Class-2 4/14 Heap Basics; Slides;
Week-13 Class-1 4/16 Heap Exploitation; Slides; HW9 HW-8
Week-13 Class-2 4/21 UAF; Double Free; Slides
Week-14 Class-1 4/23 Cache side channel attacks; Slides
Week-14 Class-2 4/28 Meltdown and Spectre; Slides; Teams; HW10 HW-9
Week-15 Class-2 4/30 Review for Final Exam; Slides
Week-16 Class-1 5/5 Office hours for final exam;
Week-16 Class-2 5/7; 3.00 PM - 6.30 PM; Location: CCSB 1.0410 Final CTF; HW-10

Resources

GDB Cheat Sheet Tmux Cheet Sheet x64 Cheat Sheet x32 Cheat Sheet x32 Cheat Sheet ARM Cheat Sheet ARM Cheat Sheet 32-bit