==Lec1, Lec2 - Intro==resource management + abstractionCPU+RAM+storage(devices)**benefits of abstractions: (p10 and p14)preemptive multitaskingmonolithic vs. microkernels (cf....

Midterm Exam of Operating Systems














==Lec1, Lec2 - Intro== resource management + abstraction CPU+RAM+storage(devices) **benefits of abstractions: (p10 and p14) preemptive multitasking monolithic vs. microkernels (cf. FUSE/udev) command-line, text UI, GUI **kernel mode/space (ring 0) vs. user mode/space processes, PID (cf. program binaries) **lifecycle & symbols address space layout: stack, data, code, heap, arg/env, etc. users, UID (e.g., root UID=0) - just a label files - way of organizing data file systems - way of organizing files **lib calls vs. sys calls the higher the abstraction level is -> the more portable ==Tut1== -environment- variables user account info ==Lec3, Lec4 - Abstraction== execution: direct, indirect, limited direct execution context -> context switch (cost!) -> scheduling scheduling principles (p6) concurrency vs. parallelism **fork(), exec() threads (shared addres space) vs processes pthread? higher level of abstraction, vs. clone() paging (in pages) and swapping (in addres spaces) onto disk roles of kernel in abstraction ==Tut2== memory allocation symbols libcall -> syscall ==Lec5, Lec6 - Facilities== **steps for talking to a computer terminal -> /sbin/getty login -> /usr/bin/login shell -> 3000shell, /bin/bash user account info (/etc/passwd, /etc/shadow) **UID, EUID (determines access), GID/EGID **be able to convert between different notations -> file permission bits workflow of a shell **init (PID 1): before any user process **zombie state (reaping) signals - asynchronous, only predefined, concurrency issues SA_RESTART signal handler (sigaction()), send signals (kill()) **shell pipeline | redirection <,>, >> ways to provide inputs, args, stdio, files, etc. path, pathname, filename absolute pathname vs. relative pathname, ~, / creat(), open(), read()/write(), ioctl() mount a file system -> mountpoint ==Tut3== *shell* (many) /dev/stdout/stdin/stderr ==Tut4== login info in /etc/passwd understanding of the "environment" ==Lec7, Lec8, Lec9, Lec10 - Filesystems== RAM vs. storage (it's I/O!) block device layer - reads/writes can be cached/buffered block size **file types file descriptors (per process) - just an integer stdin (0), stdout(1), stderr (2) are just FDs file = inode (metadata) + data blocks filesystem = superblock (metadata) + inodes + data blocks inode: link count, data block pointers, stat <- inode="" **dentry:="" file="" name="" -=""> inode number readdir() -> getdents() a directory is also an inode + data blocks (dentries) **hard links (just dentry) vs. symbolic links (just pathname) copy/move/remove **special files (VFS only) != special file systems device/special files: major number + minor number char/block devices **physical size vs. logical size file system crash consistency vs. data recovery vs. device repair atomicity **what-ifs fsck special file systems user-space file systems =="connections"== external fragmentation vs. internal fragmentation memory: pages, storage: blocks dynamic linking vs. static linking -> program lifecycle COMP 3000: Operating Systems COMP 3000 Operating Systems Further into the Abstractions Lianying Zhao Viau Typewriter 1. Process -> CPU 2. Address space -> memory What Makes Abstraction/Virtualization Possible? • The abstraction creator/enforcer must be: • “Omnipotent”, that always has a higher privilege than the target • “Omniscient”, that sees everything about the target • All this leads to the requirement for hardware support COMP 3000 (Winter 2023) 2 Viau Typewritten Text We want and currently have "direct execution" but that will lose control as the OS is also software. Inspired from indirect execution (e.g., emulators, interpreters, JVM), we can achieve limited direct execution with some hardware support Viau Typewritten Text Software vs. Software Viau Typewritten Text Direct Execution Viau Typewritten Text Indirect Execution Viau Typewritten Text Interpreted Languages / Java VM (JVM) Viau Typewritten Text Limited Directed Execution Virtualizing the CPU COMP 3000 (Winter 2023) 3 • Recall: the few purposes • Containing faults (reliability) • Security • Simplicity • Also more: • Maximum CPU utilization • Therefore, what’s expected from the illusion: • No need to think about others • No access to others’ resources Viau Typewritten Text To avoid CPU idling The Execution Context • The computer processor is like a state machine • A program (a sequence of instructions), as input, just triggers transitions • What determines the machine state? • Take x86 as an example: • The program counter – (E/R)IP • Status/flag registers, e.g., (E/R)Flags • General-purpose registers, e.g., (E/R)AX, BX, CX, … • Other registers • Memory info • Much more in reality COMP 3000 (Winter 2023) 4 Viau Typewritten Text These are the "machine state" --> "execution context" The Process Abstraction (recap) • Just a running program • Typically loaded into memory from an executable (binary file) • Or in any other less typical forms • What is hidden: • The fact that other processes are also sharing the same CPU • All the (privileged) software/hardware facilities that back it up • New semantics: • Processes with unique identifiers (PIDs) • API COMP 3000 (Winter 2023) 5 We will discuss the specifics later Viau Typewritten Text fork() Viau Typewritten Text you can load anything into the address space for execution Viau Typewritten Text system calls are the main API for the process abstraction Viau Typewritten Text signals are another important interface (via system calls) Viau Typewritten Text RAM Scheduling COMP 3000 (Winter 2023) 6 • Principle 1: To keep the CPU busy! • Principle 2: To finish tasks as soon as possible • Principle 3: To be fair to all processes • Each process gets a time slice • The scheduler of modern operating systems can be very complicated Viau Typewritten Text This course will focus on "Mechanisms", as opposed to "Policies" The actual policies of a modern OS can be way more complicated Saving/restoring the process execution context and switch between them is the mechanism When to switch and which process to pick is the policy Viau Typewritten Text We need to make CPU do something else if the current process is waiting on I/O, for example Viau Typewritten Text An example performance metric is: turnaround time = Tcompletion - Tarrival Viau Typewritten Text For fairness, an example metric is: response time = T1st_execution - Tarrival Viau Typewritten Text This is based on the typical scheduling policy called: Round-Robin (RR), where the idea is to assign equal-sized time slices to each process for execution Context Switch • A fundamental mechanism, simply: • Saving the context of one process and suspending it • Restoring the context of another process and getting it to run • The data structure where a process’s context is stored is (often) called: • Process Control Block (PCB) • There is a cost! COMP 3000 (Winter 2023) 7 Viau Rectangle Viau Typewritten Text In linux, this data structure is called "task_struct" Viau Typewritten Text this is why we cannot make the time slice as small as possible to approximate best fairness Concurrency vs. Parallelism • Concurrency (one of the three “pieces”) • Multiple tasks (contexts) progressing at the same time • In overlapping time periods, but not necessarily exactly at the same time • Posing a host of challenges (we will discuss in this course) • Parallelism • Literally at the same time • E.g., on a multi-core CPU • How to understand the two? • Concurrent → interruptible/divisible Parallel → independent • No true parallelism actually justifies the need for task scheduling COMP 3000 (Winter 2023) 8 Viau Typewritten Text single core Viau Typewritten Text multiple cores Viau Typewritten Text concurrency and parallelism just have different emphases but not mutually exclusive When you have multiple things overlapping the same duration you are faced with a concurrency problem, then you want to maximize concurrency to approximate parallelism (although true parallelism for all tasks is infeasible due to limited number of cores) Working with Processes - Creation • The fork system call • The exec system call • CreateProcess() on Windows COMP 3000 (Winter 2023) 9 Viau Typewritten Text Shell Viau Typewritten Text Tutorial 3 Viau Typewritten Text fork() returns 0 to child, or PID of child to parent Viau Typewritten Text fork.c: https://people.scs.carleton.ca/~lianyingzhao/comp3000/w23/misc/fork.c Processes vs. Threads • Various ways to understand the difference • Multiple points of execution for one program • Threads are light-weight processes • Actually, the kernel does not necessarily see (user) threads, i.e., they are the same at the low level, so • Context switch is also needed for scheduling threads • Thread Control Block (TCB) • Threads’ defining difference: shared address space COMP 3000 (Winter 2023) 10 Viau Typewritten Text execution context Viau Typewritten Text in our Linux, a process is considered as a thread group, which can contain one or multiple threads so, the PID = the thread group ID (TGID) Working with Threads • Pthreads (POSIX threads) • E.g., pthread_create() • Library calls, not system calls • At the OS level, the actual system calls to create threads can be: • Clone – Linux specific (recall: portability) • by specifying CLONE_THREAD (there are also other levels of sharing for processes) • Shared address space causes more concurrency issues COMP 3000 (Winter 2023) 11 Viau Typewritten Text keep in mind the portability discussion Viau Typewritten Text The clone() system call may not exist on other OSes Viau Typewritten Text This is a user-space library -> making system calls to achieve its purpose Virtualizing the Memory COMP 3000 (Winter 2023) 12 • How is memory (RAM) used/organized? • Addressable at the granularity of bytes • Accessed at the granularity of the width of the processor architecture • Data storage + code execution • The main problem to solve: • Space allocation • Mapping between physical and virtual • Shared purposes with virtualizing the CPU: • Security, reliability, simplicity and max efficiency Viau Typewritten Text each byte is addressable at an address, e.g., 0x87af6654 Viau Typewritten Text external, i.e., how address spaces are placed ==> unusable, too small holes outside Viau Typewritten Text External Fragmentation The Memory Hierarchy COMP 3000 (Winter 2023) 13 Original (reality) Abstraction (illusion) CPU registers CPU registers Also in terms of access latency L1 cache L2 cache L3 cache Main memory Hard disk Memory Viau Typewritten Text last-level cache Viau Typewritten Text orders of magnitude slower than the CPU The Memory Layout COMP 3000 (Winter 2023) 14 Original (reality) Abstraction (illusion) Memory Viau Typewritten Text No need to read this The Address Space Abstraction • A (seemingly) independent numbering system starting from 0 • Addresses in an address space are virtual • Address translation happens behind the scene • Other than the space, there are also other meta data, such as • Stack • Heap COMP 3000 (Winter 2023) 15 Viau Typewritten Text layout is another aspect of the abstraction Mechanism: Segmentation • The idea: assign a segment for each purpose • Code – CS • Data – DS • Stack – SS • The start of a segment serves as the base address. An address is formed by applying an offset: • Pure segmentation was in the old days COMP 3000 (Winter 2023) 16 Viau Typewritten Text Just an example, no need to understand this calculation Mechanism: Paging • The need for contiguous allocation in physical memory • External fragmentation • Let’s go for finer granularity! COMP 3000 (Winter 2023) 17 Caution: overloaded term, cf. swapping Viau Typewritten Text a typical page size Viau Typewritten Text 1. allocation with the granularity of pages Viau Typewritten Text 2. mapping: from virtual to physical addresses Mechanism: Swapping • Don’t want to be limited by physical memory? • The swap partition (pagefile.sys on Windows) • The page fault • Paging (the other meaning): moving the pages • Swapping: moving the entire process COMP 3000 (Winter 2023) 18 Abstractions are Provided by the OS (Kernel) • So most of the abstractions are not directly seen in kernel space • E.g., don’t try to find processes in a kernel • BUT the same or similar mechanisms still apply • E.g., paging also happens in kernel space COMP 3000 (Winter 2023) 19 Viau Typewritten Text The kernel manages processes as data structures Viau Typewritten Text The kernel can choose to work with virtual addresses or physical addresses COMP 3000 (Winter 2023) 20 COMP 3000 Operating Systems Misc. Command Arguments and Options • What’s passed to the program int main(int argc, char *argv[]) { • argc: number of arguments • argv: the array of arguments, each being a string • The first one (argv[0]) is always the command name (or… should be) • Subsequent ones are space-delimited strings • Note: these are determined by the shell • Example: ls -lais • Conventions COMP 3000 (Winter 2023) 21 Linking Options and System Calls • Static linking: • To avoid dependency issues at the cost of space (disk + memory) • So, no library calls should be made by design • Dynamic linking: • Reuse of common functions • The expected versions must be ensured • System calls are invoked when any OS service is needed • In general, I/O-intensive functions tend to generate more system calls • Expensive COMP 3000 (Winter 2023) 22 Viau Typewritten Text System calls -> requesting OS services Viau Typewritten Text Library calls -> calling functions in dynamic libraries Viau Oval Viau Oval More about Program Binaries • Relocatable object files (.o) → (archive, ar) → Static libraries (.a) • ↓ (linking, ld) • Shared libraries (.so) - ELF • Executable files - ELF COMP 3000 (Winter 2023) 23 Dynamic Library Dependencies • Symbols • Just variable names and function names • Symbol tables • Various ways to list symbols • readelf • nm • objdump COMP 3000 (Winter 2023) 24 Viau Typewritten Text Understand how external dependencies (references) are resolved using any of these commands Slide 1: COMP 3000 Operating Systems Further into the Abstractions Slide 2: What Makes Abstraction/Virtualization Possible? Slide 3: Virtualizing the CPU Slide 4: The Execution Context Slide 5: The Process Abstraction (recap) Slide 6: Scheduling Slide 7: Context Switch Slide 8: Concurrency vs. Parallelism Slide 9: Working with Processes - Creation Slide 10: Processes vs. Threads Slide 11: Working with Threads Slide 12: Virtualizing the Memory Slide 13: The Memory Hierarchy Slide 14: The Memory Layout Slide 15: The Address Space Abstraction Slide 16: Mechanism: Segmentation Slide 17: Mechanism: Paging Slide