Processes; foreground and background, ps, top, kill, screen, nohup and daemons
A process is simply the instance of a running program. Processes are a fundamental concept of Linux systems and we'll start by discussing what can be termed as the life cycle of processes. This includes the concepts of parent and child processes and while going through this subject, we'll also introduce some essential commands that allow us to view ongoing processes in our terminal. These commands are ps, jobs and top.
Next we discuss foreground and background processes and how processes can be managed practically using UNIX commands and operators. These commands include sleep, fg, bg, jobs and kill. The only new operator we'll be looking at is &, which is used to start a command as a background process. We'll be revising redirectional operators, <, > and taking a look at file descriptors.
Lastly we go a bit more into detail with the 'SIGHUP' signal. In short, this is a signal that is sent out when a terminal session is closed, which results in child processes being terminated. We'll discuss how one can make processes immune to the 'SIGHUP' signal, so that processes can continue even after we've closed the terminal. The commands introduced for this are screen, disown and nohup. Lastly we introduce the concept of daemon processes.
Parent and child processes
Processes are distinguished into parent, child and daemon processes, but for now we'll focus on parent and child processes. All processes have parent processes but not all parent processes have child processes. The relationship between the two is how you'd imagine; a child process is derived/spawned from its parent process. When a process starts the execution of a new program, it first makes a copy process of itself. In Linux we call this 'forking' and it is carried out by the system call fork(). You can think of system calls as an interface between applications and the kernel, where the shell is the intermediary translator between the two. The fork() is usually followed by a exec() system calls, which is what converts the child process into a new process.
You can see the Linux processes currently running in your terminal by using the command ps, which is short for 'process status'.
Prompt$ ps -ef
A process is a child process of another process, if its PPID (Parent process ID) is the same as that of the other processes PID (Process ID). By using different command options you can make ps display a lot of different column statistics on the processes running on your computer. You can often guess what the meaning of these columns are, but if you're in doubt you can always go to the man page for ps. UID is short for 'user ID' and in this case there are only two instances of users; root and goodboy. PID is short 'process IDS' which are unique identification numbers, assigned to processes making it easier to target them. PPID is short for 'parent process IDS' which unique identification number of a process's parent process. Recall, that UNIX commands are in fact small programs, which is why ps -ef appears as a process. Take note that the PPID for ps -ef is 4, which is the PID for -bash. This means that -bash is the parent process of ps -ef. The parent process for -bash is /init ro, which in turn has no parent processes. You can think of the init process as 'super parent process', which is the very first process to be run when you start/boot a unix computer. init should always have the PID 1. If a parent process terminates before it's child process, init will become the new parent of the child. For a more graphical view of this you can use the command pstree,
Prompt$ pstree
which displays a tree of parent and child processes. The init process should always be the starting node of this tree.
Foreground and background processes
Processes can run in the foreground and background. Foreground processes is any command that you enter in the prompt, whereafter you have to wait for its completion before being able to enter a new command. Up till now you've only been executing commands as foreground processes. Unlike foreground processes, when a background process has been executed you don't have to wait for its completion before being able to issue a new command. Any command can be run as a background process by typing a space and '&' after the command,
Prompt$ <COMMAND> &
As an example, we use the command, sleep, which is essentially a pause command that does nothing for a specified amount of time,
Prompt$ sleep 30 &
will create a pause of 30 seconds as a background process, which you can view with the ps command. Now try to run sleep as foreground process,
Prompt$ sleep 60
which will create of pause of 60 seconds. To exit this pause you have two choices; Press Ctrl-c which will send a kill signal to any process running in foreground, terminating it immediately. This is a very effective and often used way to stop programs. You can instead press Ctrl-z to send a suspend signal to any process running in the foreground, pausing it immediately. If you chose the pause option you'll be able to see the process with ps ax, but its status will be T (Stopped). If you've paused the process you can restart it as a background process or foreground process.
Prompt$ bg %<job ID> Prompt$ fg %<job ID>
which will restart the process as a background or foreground process respectively. Do NOT forget your suspended processes, as they use resources, even if they do not do anything. You can see job ID's of currently running background processes by executing,
Prompt$ jobs
which displays a list of active jobs. Jobs can be defined as processes that are initiated in the shell interactively by you, the user. Each job is assigned a job ID which the commands, bg and fg, use for targeting.
You can terminate a background process by sending it to the foreground and then terminating it with Ctrl-c. But a more direct approach to terminating a background process is to use the kill command. It doesn't use job ID's and instead uses PID's,
Prompt$ kill <PID>
will terminate the process with corresponding <PID>. The kill command can actually be used to send a multitude of signals to processes. You can see a list of all these different signals by using -l command line option,
Prompt$ kill -l
By default the signal used by kill is the kill signal, 'SIGKILL', but you can also use kill to send a stop signal, 'SIGSTOP'.
Prompt$ kill -19 <PID>
will send a stop signal to the process with corresponding <PID>.
Lastly, an alternative command to ps is top, the difference being that top provides a continuous representation of ongoing processes with an interactive command mode and ps only provides a snapshot of current processes. By typing,
Prompt$ top
an interactive command mode and columns containing statistics of ongoing processes is displayed. The meaning of the columns is listed herunder.
- PID: Unique process id.
- PR: Priority of the task.
- VIRT: Total virtual memory.
- USER: User name of owner.
- %CPU: CPU usage.
- TIME+: CPU Time, similar to ‘TIME’.
- SHR: Shared Memory size (kb).
- NI: Nice Value. A negative nice value implies high priority and a positive nice value implies a low priority.
- %MEM: Memory usage.
You can exit the interactive command mode of top by pressing q. You can use the kill utility by pressing k while in the interactive command mode. You will then be prompted to enter which PID you want to send a signal to, whereafter you'll be prompted to enter what kind of signal you'd like to send. By default this is the the kill signal, so if you don't enter any signal type and simply press enter, a kill signal will be sent. In figure 8.2 a kill signal is being sent to the process with the PID, 24, from the top interactive command mode.
File descriptors
When you a open a file on your computer, your operating system will create an entry wherein information for that file can be stored. These entries have an entry number, which can be any positive integer (6,7,12,301 etc.) and this number is what's called a file descriptor. Standard input, standard output and standard error are also thought of as files in Unix, and they therefore also have file descriptors. These are by default always set to 0, 1 and 2 for stdin, stout and stderror respectively. You can actually find these files on your system by going to the device directory.
Prompt$ cd /dev
Redirecting outputs and providing necessary stdin for background processes
If a command requires additional stdin from the user or has some output, it will cause problems if it's run as a background process. If a program run in the background requires additional stdin and doesn't receive it, the program will suspend and wait for infinity for the input. Furthermore, stdout and stderror from a background process needs to be redirected to somewhere else than the terminal. For example, when installing packages with apt there's a continuous stream of output regardless of whether it's run in the background or foreground. While this is being outputted we cannot continue working with other stuff in the terminal, which ruins the point of background processes.
So in order to run a program as a background process, you often need to redirect stdout and stderror to somewhere else, which can be done by using file descriptors.
Prompt$ <COMMAND> > <output_file> 2> <error_file> &
In this command, stdout and stderror from Unix command is redirected to two separate files. By default, the > operator redirects stdout, which is why we only have to write >. If we don't care about the stdout and stderror we can redirect to what's called the null device.
Prompt$ <COMMAND> > /dev/null 2>&1 &
In this command, the stdout is used to redirect to '/dev/null' with >, and the stderror is redirected, 2>, to the same place as the stdout, &1. Alternatively, you could have also typed,
Prompt$ <COMMAND> > /dev/null 2> /dev/null &
which will also redirect both stdout and stderror to the null device. If it helps, think of the null device as a black hole where we send data we don't need.
We've dealt with the problem of redirecting stdout and stderror, but we still have to deal with the stdin. For instance, in the last section were we used apt to install packages, we needed to type in 'y' for installation of a package to continue. A solution to this, is simply to use the y option as shown in the last section.
Prompt$ sudo apt -y install <PACKAGE> > /dev/null 2>&1 &
This command, will install <PACKAGE> without asking for the additional 'y' confirmation and will direct all output to '/dev/null'. There might, however, not always be an option like the y option. Therefore, we show another method where we use echo to pipe the stdin you need,
Prompt$ echo 'Your_stdin' | sudo apt install <PACKAGE> > /dev/null 2>&1 &
Also, you could create an <inputfile> and use redirectional operator for stdin, <, to feed its contents.
Prompt$ sudo apt install <PACKAGE> < <inputfile> > /dev/null 2>&1 &
When you use sudo you'll be prompted to enter your password, which will grant you root privileges for 15 minutes. In these 15 minutes you won't be prompted again for your password when executing commands that require root privileges. If you run a command that requires root privileges as a background process while output is redirected to /dev/null and you don't have root privileges, the process will stop. The simplest solution to this problem is to run another command with root privileges prior to running your background process. It doesn't matter what command you use to acquire these root privileges, it could for example be ls,
Prompt$ sudo ls Prompt$ sudo <COMMAND> > dev/null 2>&1
You could also utilize the fact that sudo by default doesn't read passwords from stdin, but directly from your keyboard. This is useful, if you need to run a command that requires both password and some stdin like 'y', because then you don't have to worry about 'y' being fed as your password.
Prompt$ echo 'y' | sudo apt remove <PACKAGE> > /dev/null 2>&1
will ask you for your password which it receives from your keyboard. Subsequently, it's fed 'y' as stdin and all output is redirected to /dev/null. After having initiated the process, you can make it a background process, by first pausing it with Ctrl-z and then making it a background process with bg %<job ID>.
It might be useful to know that you can actually feed your password with stdin by using -S option.
Prompt$ echo "Password" | sudo -S <COMMAND>
Continuing processes after exiting terminal: screen, disown and nohup
Under normal circumstances, any child processes are sent what's called 'SIGHUP' (short for signal hangup) when the terminal session ends, effectively terminating them. In other words, if the computer or server you're working from crashes, or you simply have to go home and bring your laptop with you, ongoing processes will be terminated. This is obviously problematic and are 3 ways to avoid this; screen, disown and nohup.
When people talk about multiple screens in Linux, they're talking about running multiple terminal windows separate from each other. The processes run within these different terminal windows are not affected by the user logging of, and they're all equipped with a shell.
Prompt$ 'sudo screen -S <Screen_name>
will start a new terminal window called <Screen_name>. Within this terminal window you can start a lengthy process in the foreground, then quit the terminal window while keeping its shell running by pressing Ctrl a-d. You don't have to redirect output as it is directed to the terminal where you started the process, which is the screen. Screens are quite useful if you're doing a lengthy proces and you're not sure what stdin it might need, so you want to be able to be able interact with it.
You can read more about screens here:
Additional info on screens 1
Additional info on screens 2
As mentioned, the signal that terminates processes when terminal session ends, is called 'SIGHUP'. You can make processes inmmune to this signal by using the command disown, which will detach the process from the shells job list, so that the process is not sent a 'SIGHUP' signal when the terminal session ends. For example,
Prompt$ sudo bash -c 'apt-get -y install <PACKAGE> > /dev/null 2>&1 & disown'
will install <PACKAGE> as a child process separate from the terminal, and the installation of the package will therefore proceed even if the terminal is shut down.
Lastly, a very useful command that has similar function to disown is nohup. Unlike disown, the command nohup is defined by POSIX, which means that it works within most shells. We haven't really discussed what POSIX is. Essentially, POSIX represents a standard for the syntax of Unix operating systems and command line shells, and ensures compatibility between them. It is, however, rarely the case that an OS is 'POSIX-certified', and most OS's are defined as 'Mostly POSIX-compliant'. For instance, there are some shells where disown doesn't work (tcsh, csh, dash and sh). Any command run with nohup will be immune to SIGHUP signaling, furthermore, nohup ignores all standard input and directs any output bound for the terminal to the file, nohup.out. This means that this command is not ideal for processes that require additional stdin.
Prompt$ nohup <COMMAND>
will execute <COMMAND>, direct all output to nohup.out and ignore all stdin. Also, if the terminal ends, the 'SIGHUP' signal will be ignored and the process will not end.
Daemon processes
Daemon processes are difficult to characterize as they share many of the characteristics that normal background processes have. They run in the background and are detached from the terminal. The parent process of daemon processes is most often the init process, which means that daemons will most often have PPID value of 1. They're usually created by forking a child process, followed by an immediate exit of the process, however, they can also be created directly by the init process. Users will typically have no control over daemons. You can see the daemon processes running on your device by typing,
Prompt$ ps -A
which lists ongoing Unix processes. Except for the init process and a few others, daemon processes typically have the ending, 'd'. The output from executing ps -A is shown in figure 8.3. Here, there are 3 daemon processes which are underlined with red. Two of these are the init process and the third is what's called 'Secure shell server' process. The screen process has been underlined with orange to show that processes with a PPID value of 1, aren't necessarily a daemon. There are other daemons which won't show up unless you're using a Linux computer.
The purpose of daemons is to provide a service. Examples could be printer service, network service, sound, web service, mail service, etc. It is system level services that we often do not have to care about.
Command list
In the table below, we list this sections Unix commands.
| Unix Command | Acronym translation | Description |
|---|---|---|
| <COMMAND> & | - | Runs command as a background process |
| bg %<PID> | bg and PID are short for background and process identification respectively | Continues a stopped job in the background |
| fg %<PID> | fg and PID are short for foreground and process identification | Continues a stopped job in the foreground |
| sleep <NUMBER>[s/m/h/d] | - | Delays for a specified amount of time. This can be specified <NUMBER> and suffixes; s, m, h and d which are short seconds, minuted, hours and days respectively. By default, the suffix is s |
| top [OPTION] | - | Displays all the processes running on your computer |
| ps [OPTION] | Process status | Reports a snapshot of current processes |
| kill [OPTION] <PID> | - | Sends a signal to a process and by default this signal is to terminate the process |
| screen [OPTION] | - | Used to create new terminal windows that are detached from each other. Child processes created within these new terminal windows are not affected if their parent process is terminated |
| disown | - | Dissociates process from current terminal session |
| nohup [OPTION] | No hangup | Used to run commands immune to hangups, ignoring stdin. By default output is redirected to nohup.out. |
| pstree | Process tree | Display a tree of parent and child processes |
Exercise 1: Working with PIDs
1. Take a snapshot of current processes and save the process information to a file. The process information should contain PID's and PPID's as a minimum.
2. In the file, replace the occurrence of bash with 'Hacked'.
3. Extract the process ID of the 'Hacked' process and have the output directed to your terminal (Hint 1).
4. Make a screen and exit it without closing it.
5. Extract its PID and have it redirected to your terminal.
Hint 1: A good way to do this is to use what you learned in 'Filtering and regular expressions' combined with pipelines.