Linux introduction guide¶

This page contains a short introduction to Linux, a Linux chest sheet, and some links to resources for learning more about Linux.

Much of this material is from the Linux introduction part of the “Introduction to HPC2N and Kebnekaise” course that HPC2N usually runs twice yearly.

HPC2N’s supercomputer “Kebnekaise”, as well as most HPC centers supercomputers run Linux.

“Kebnekaise” runs Linux Ubuntu, currently (spring 2024) Ubuntu 20.04.

What is Linux¶

Linux is a family of open-source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991, by Linus Torvalds.

An operating system is the software that sits underneath all of the other software on a computer, managing the computer’s hardware (CPU, GPU, memory, storage…) and taking care of the connections between your other software and the hardware.

Linux is typically packaged as a Linux distribution, which includes the kernel and supporting system software and libraries, many of which are provided by the GNU Project.

There are many Linux distribuitions, including Ubuntu, Debian, Fedora, Gentoo, and many others. Many distributions are free and open source, but there are also commercial distributions, like Red Hat Enterprise and SUSE.

Desktop Linux distributions include a desktop environment, like GNOME, MATE, KDE Plasma, Xfce, Unity, or many others. A window manager together with applications written using a widget toolkit are generally responsible for most of what the user sees.

In addition, a windowing system of some sort (X11, Wayland) interfaces directly with the underlying operating system and libraries, providing support for graphical hardware, pointing devices, and keyboards. The window manager generally runs on top of this windowing system.

While the use of Linux on desktop is only 2-3%, the vast majority of web servers (>96%), most mobile devices (Android is based on the Linux kernel), and all supercomputers on the Top500 list run Linux.

There is much more information about Linux on Wikipedia.

Command line¶

The command line is an interface for typing commands directly to a computer’s operating system. It is a user interface that is navigated only with the keyboard, by typing commands at a prompt, instead of by clicking with a mouse or similar.

The Command Line Interface (CLI) is available in all operating systems, including Windows, Linux and macOS, but it is most commonly associated with Linux.

In Unix or Linux, the prompt may be shown as “%” or “>” depending on the shell.

The opposite of a CLI is a GUI (Graphical User Interface), which generally uses a mouse or similar for navigating.

Command line interfaces often gives access to many more capabilities than the graphical user interface does, and it is also practical and normally faster in situations where you login with a terminal on a remote system, like Kebnekaise.

The picture above shows a terminal window where I am logged into Kebnekaise (from my desktop defiant).

Navigating the File System¶

This section is going to be a high-level talk about the Linux filesystem concepts, not a low-level description of filesystem types.

The Linux filesystem directory structure starts with the top root directory, which is shown as /. Below this are several other standard directories. Of particular interest are usr/bin, home, usr/lib, and usr/lib64. A common directory which you will also often find is usr/local/bin.

The picture on the right shows typical subdirectories under / (note that the command ‘tree’ does not work on Kebnekaise). Some of the directories have a symbolic link to a different name - this is often done to make it quicker to write, but can also be for compatibility reasons since some software have hardcoded paths.

• usr/bin: contains (most) of the system-specific binaries
• usr/local/bin: non-system binaries. often locally compiled/maintained packages
• home: where the home directories of the users of the system are located
• usr/lib: kernel modules and shared library images needed to boot the system and run commands in the root filesystem
• usr/lib64: same as /lib, just for 64-bit libraries

User-installed binaries are often located in /opt.

ls - listing files/directories¶

The ls command is used to list files. If you just give the command “ls” with no flags it will list all files in the current directory except for hidden files.

ls [flags] [directory]

This way you can to list files/subdirectories for any directory, but the default one is the one you are currently standing in.

Some examples:

• ls / lists contents of the root directory
• ls .. lists the contents of the parent directory of the current
• ls ~ lists the contents of your user home directory
• ls * lists contents of current directory and subdirectories

Commonly used flags:

• -d */ lists only directories
• -a lists content including hidden files and directories
• -l lists content in long table format (permissions, owners, size in bytes, modification date/time, file/directory name)
• -lh adds an extra column to above representing size of each file/directory
• -t lists content sorted by last modified date in descending order
• -tr lists content sorted by last modified date in ascending order
• -R lists contents recursively in all subdirectories
• -s list files with their sizes
• -S sort files/directories by size in descending order
• -Sr sort files/directories by size in ascending order

To get more flags, type ls --help or man ls in the terminal to see the manual.

b-an01 [~/mytestdir]$ ls
myfile.txt  myotherfile.dat  testdir1/  testdir2/

b-an01 [~/mytestdir]$ ls -a
./  ../  myfile.txt  myotherfile.dat  testdir1/  testdir2/

b-an01 [~/mytestdir]$ ls -l
total 16
-rw-r--r-- 1 bbrydsoe folk   90 Jun  9 14:19 myfile.txt
-rw-r--r-- 1 bbrydsoe folk  101 Jun  9 14:19 myotherfile.dat
drwxr-xr-x 2 bbrydsoe folk 4096 Jun  9 14:14 testdir1/
drwxr-xr-x 2 bbrydsoe folk 4096 Jun  9 14:19 testdir2/

b-an01 [~/mytestdir]$ ls -la
total 24
drwxr-xr-x  4 bbrydsoe folk  4096 Jun  9 14:19 ./
drwxr-xr-x 49 bbrydsoe staff 4096 Jun  9 14:10 ../
-rw-r--r--  1 bbrydsoe folk    90 Jun  9 14:19 myfile.txt
-rw-r--r--  1 bbrydsoe folk   101 Jun  9 14:19 myotherfile.dat
drwxr-xr-x  2 bbrydsoe folk  4096 Jun  9 14:14 testdir1/
drwxr-xr-x  2 bbrydsoe folk  4096 Jun  9 14:19 testdir2/

b-an01 [~/mytestdir]$ ls -lah
total 24K
drwxr-xr-x  4 bbrydsoe folk  4.0K Jun  9 14:19 ./
drwxr-xr-x 49 bbrydsoe staff 4.0K Jun  9 14:10 ../
-rw-r--r--  1 bbrydsoe folk    90 Jun  9 14:19 myfile.txt
-rw-r--r--  1 bbrydsoe folk   101 Jun  9 14:19 myotherfile.dat
drwxr-xr-x  2 bbrydsoe folk  4.0K Jun  9 14:14 testdir1/
drwxr-xr-x  2 bbrydsoe folk  4.0K Jun  9 14:19 testdir2/

b-an01 [~/mytestdir]$ ls -latr
total 24
drwxr-xr-x 49 bbrydsoe staff 4096 Jun  9 14:10 ../
drwxr-xr-x  2 bbrydsoe folk  4096 Jun  9 14:14 testdir1/
drwxr-xr-x  2 bbrydsoe folk  4096 Jun  9 14:19 testdir2/
-rw-r--r--  1 bbrydsoe folk    90 Jun  9 14:19 myfile.txt
-rw-r--r--  1 bbrydsoe folk   101 Jun  9 14:19 myotherfile.dat
drwxr-xr-x  4 bbrydsoe folk  4096 Jun  9 14:19 ./

b-an01 [~/mytestdir]$ ls *
myfile.txt  myotherfile.dat

testdir1:
file1.txt  file2.sh  file3.c  file4.dat

testdir2:
file1.txt  file2.txt  file3.c

b-an01 [~/mytestdir]$ cd testdir1
b-an01 [~/mytestdir/testdir1]$ ls -l
total 16
-rw-r--r-- 1 bbrydsoe folk 24 Jun  9 14:16 file1.txt
-rw-r--r-- 1 bbrydsoe folk 52 Jun  9 14:16 file2.sh
-rw-r--r-- 1 bbrydsoe folk 82 Jun  9 14:17 file3.c
-rw-r--r-- 1 bbrydsoe folk 40 Jun  9 14:17 file4.dat

b-an01 [~/mytestdir/testdir1]$ ls -ls
total 16
4 -rw-r--r-- 1 bbrydsoe folk 24 Jun  9 14:16 file1.txt
4 -rw-r--r-- 1 bbrydsoe folk 52 Jun  9 14:16 file2.sh
4 -rw-r--r-- 1 bbrydsoe folk 82 Jun  9 14:17 file3.c
4 -rw-r--r-- 1 bbrydsoe folk 40 Jun  9 14:17 file4.dat

chmod - change permissions¶

The command chmod is used to change permissions for files and directories.

The permissions for a file, directory, or symbolic link has 10 “bits” and looks similar to this:

As shown, the first bit can be “-” (a file), “d” (a directory), or “l” (a link).

The following group of 3 bits are for the owner, then the next 3 for the group, and then the last 3 for all users. Each can have the r(ead), w(rite), and (e)x(ecute) permission set.

chown - change ownership¶

To change ownership of a file or directory, use the command chown.

chown [OPTIONS] USER[:GROUP] FILE(s)

Create and remove directories/files¶

This section contains a few examples of how to work with files and directories through command line interface.

• mkdir DIR: Create a directory DIR
• rm -rf DIR: Remove a directory DIR. The flag “-r” means recursively and “-f” means do so without asking for each file and subdirectory. Useful, but dangerous. Be careful!
• cd: Go to your home directory ($HOME)
• cd DIR: Change directory to DIR
• cd ..: Change directory to the parent directory of the current directory
• touch FILE: create an empty file with the name FILE

You also user “rm” to remove files;

rm file.txt

The command pwd tells you the current directory path.

b-an01 [~]$ mkdir mytestdir
b-an01 [~]$ cd mytestdir/
b-an01 [~/mytestdir]$ mkdir testdir1
b-an01 [~/mytestdir]$ mkdir testdir2
b-an01 [~/mytestdir]$ mkdir testdir3
b-an01 [~/mytestdir]$ rm -rf testdir3
b-an01 [~/mytestdir]$ cd testdir1
b-an01 [~/mytestdir/testdir1]$ touch file1.txt
b-an01 [~/mytestdir/testdir1]$ touch file2.sh
b-an01 [~/mytestdir/testdir1]$ touch file3.c
b-an01 [~/mytestdir/testdir1]$ touch file4.dat
b-an01 [~/mytestdir/testdir1]$ touch file5.txt
b-an01 [~/mytestdir/testdir1]$ rm file5.txt 
b-an01 [~/mytestdir/testdir1]$ 
b-an01 [~/mytestdir/testdir1]$ cd ..
b-an01 [~/mytestdir]$ cd testdir2/
b-an01 [~/mytestdir/testdir2]$ 

cp - copy files/directories¶

This command is used to copy files or directories.

• cp myfile.txt DIR/: copy the file “myfile.txt” to the directory DIR
• cp DIR1/ DIR2/: copy the directory DIR1 into the directory DIR2 (Note: overwrites existing files with same name)
• cp -R DIR1/ DIR2/: copy the directory DIR1 and all subdirectories into the directory DIR2.

mv - rename files/directories¶

The command mv is used to rename files and directories.

• mv file1.txt file2.txt: renames file1.txt to file2.txt
• mv DIR1/ DIR2/: renames directory DIR1 to directory DIR2/

Symbolic links¶

Symbolic links are also called soft links, or just symlinks. It is a pointer to another file or directory.

• It is useful both for ease
  • you avoid using a long path each time you change to a directory, like your project directory
  • as well as to avoid changing hard links within other scripts or programs. This is good if you for instance install a program or use a script that assumes the library it uses is called libcoolness.a and not libcoolness.2.0.a. You can then just update the symlink instead of renaming the library or updating potentially many instances where it is mentioned in the program.

Command:

ln -s real-file-or-lib link-name

ln -s /proj/nobackup/hpc2nxxxx-yyy/mydir $HOME/myproj

b-an01 [~]$ ls -l
lrwxrwxrwx  1 bbrydsoe folk    28 Feb  1  2023 myproj -> /proj/nobackup/hpc2nxxxx-yyy/mydir

Redirection¶

Usually, standard input comes from the keyboard etc. and the standard output goes to the screen. There is also standard error. All of these can be redirected with Linux commands.

• > redirects the output of some command
  • Example, output of “ls” to a file: ls > test.dat
• >> concatenate the output of some command to the content of a file
  • Example, adds the output of ls to the end of a file “test.dat”: ls >> test.dat
• < changes the standard input
• 2> redirects the standard error:
  • Example, redirect the error that is thrown from your program named “myprogram” to a file “error.log”: ./myprogram 2> error.log
• 2>&1 redirects both standard output and standard error
  • Example, redirect output and errors from your program to the same file: ./myprogram > logfile 2>&1

cat file >> file2

echo 'text to append_add_here' >> file2

printf "text to append\n" >> fileName

Pipes¶

Pipes are used when you want to take the output of one command and use it as input for another command.

Here follows some examples:

grep -o -i string file.txt | wc -l

grep string file.txt > file.out

grep string file.txt >> file.out

Exporting variables¶

Environment variables store data that is used by the operating system and other programs.

Some are intrinsic to the operating system, some for a specific program/library/programming language, and some are created by the user.

The variables can both be used in scripts and on the command line. Usually you reference them by putting a special symbol in front of or around the variable name. By convention, environment variable names are in UPPER CASE.

Examples:

• $HOME Your home directory
• $PWD This variable points to your current directory
• $LD_LIBRARY_PATH a colon-separated list of directories that the dynamic linker should search for shared objects before searching in any other directories
• $OMP_NUM_THREADS Number of OpenMP threads
• $PYTHONPATH Path to the directory where your Python libraries and packages are installed

echo $ENVIRONMENT-VARIABLE

env

export VARIABLE=value

setenv VARIABLE value

Some examples:

export OMP_NUM_THREADS=8

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/your/custom/path/

echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/your/custom/path/" >> ~/.bashrc

Editors¶

Some editors are more suited for a GUI environment and some are more suited for a command line environment.

Command line¶

These are all good editors for using on the command line:

• nano
• vi, vim
• emacs

They are all installed on Kebnekaise.

Of these, vi/vim as well as emacs are probably the most powerful, though the latter is better in a GUI environment. The easiest editor to use if you are not familiar with any of them is nano.

GUI¶

If you are connecting with ThinLinc, you will be presented with a graphical user interface (GUI). From there you can either open a terminal window/shell (Applications -> System Tools -> MATE Terminal) or you can choose editors from the menu by going to Applications -> Accessories. This gives several editor options, of which these have a graphical interface:

• Text Editor (gedit)
• Pluma - the default editor on the MATE desktop environments (that Thinlinc runs)
• Atom - not just an editor, but an IDE
• Emacs (GUI)
• NEdit “Nirvana Text Editor”

If you are not familiar with any of these, a good recommendation would be to use Text Editor/gedit.

Data Handling¶

This section will briefly cover compressing/decompressing files/directories, transferring files, and logging in. There will be links to other sections which cover the material in more detail.

Compressing and decompressing¶

Compressing files are done with utilities like gzip, bzip2, or zip.

gzip FILE

gunzip FILE.gz

Archiving¶

Archiving is generally done with tar.

A tarball is a commonly used name to refer to an archive file in the tar (Tape Archive) format.

A tarball can be compressed with something like gzip or bzip2.

tar [-options] <name of the tar archive> [files or directories which to add into archive]

Basic options:

    -c, --create — create a new archive;
    -a, --auto-compress — additionally compress the archive with a compressor which will be automatically determined by the file name extension of the archive. If the archive's name ends with *.tar.gz then use gzip, if *.tar.xz then use xz, *.tar.zst for Zstandard etc.;
    -r, --append — append files to the end of an archive;
    -x, --extract, --get — extract files from an archive;
    -f, --file — specify the archive's name;
    -t, --list — show a list of files and folders in the archive;
    -v, --verbose — show a list of processed files.

Here follows some examples:

tar -cvf DIRECTORY.tar DIRECTORY

tar -xvf DIRECTORY.tar

tar -zcvf DIRECTORY.tar.gz DIRECTORY

tar -zxvf DIRECTORY.tar.gz

More information in the main Archiving and compressing section.

File transfer and syncing¶

There are several possible ways to transfer files and data to and from HPC2N’s systems: scp, sftp, rsync…

SCP¶

SCP (Secure CoPy) is a simple way of transferring files between two machines that use the SSH (Secure SHell) protocol.

$ scp sourcefilename user@hostname:somedir/destfilename

$ scp user@hostname:somedir/sourcefilename destfilename

SFTP¶

SFTP (SSH File Transfer Protocol or sometimes called Secure File Transfer Protocol) is a network protocol that provides file transfer over a reliable data stream.

enterprise-d [~]$ sftp user@kebnekaise.hpc2n.umu.se
Connecting to kebnekaise.hpc2n.umu.se...
user@kebnekaise.hpc2n.umu.se's password:
sftp> put file.c C/file.c
Uploading file.c to /home/u/user/C/file.c
file.c                          100%    1    0.0KB/s   00:00
sftp> put -P irf.png pic/
Uploading irf.png to /home/u/user/pic/irf.png
irf.png                         100% 2100    2.1KB/s   00:00
sftp>

sftp> get file2.c C/file2.c
Fetching /home/u/user/file2.c to C/file2.c
/home/u/user/file.txt  100%  1  0.1KB/s 00:00    
sftp> get -P file3.c C/
Fetching /home/u/user/file3.c to C/file3.c
/home/u/user/file.txt  100%  1  0.4KB/s 00:00    
sftp> exit
enterprise-d [~]$ 

rsync¶

rsync is a utility for efficiently transferring and synchronizing files between a computer and a storage drive and across networked computers by comparing the modification times and sizes of files.

rsync -rlpt username@remote_host:sourcedir/ /path/to/localdir

rsync -a /path/to/localdir/ username@remote_host:destination_directory

Much more information and examples in the File transfer section.

Connecting with ssh¶

The ssh command is used for connecting to a remote computer.

Some useful examples:

ssh username@kebnekaise.hpc2n.umu.se

ssh -Y username@kebnekaise.hpc2n.umu.se

More information about accessing Kebnekaise in the section about Accessing.

More advanced topics¶

This section will look at finding patterns (grep, awk, wild cards, regular expressions) and scripting.

Finding patterns¶

Here you will find descptions on how to search for files with specific patterns.

grep¶

This command searches for patterns in text files.

grep ’word’ FILE

grep -rine ’word’ path/to/dir

awk¶

This command finds patterns in a file and can perform arithmetic/string operations.

awk ’/snow/ {print$1}’ FILE

Wild cards¶

Wild cards are useful ‘stand-ins’ for one or more character or number, that you can use for instance when finding patterns or when removing/listing all files of a certain type.

Wild cards are also called globbing patterns.

• ? represents a single character
• * represents a string of characters (0 or more)
• [ ] represents a range
• { } the terms are separated by commas and each term must be a wildcard or exact name
• [!] matches any character that is NOT listed between the [ and ]. This is a logical NOT.
• ** specifies an “escape” character, when using a subsequent special character.

myfile?.txt

r*d

r[a,i,o]ck

a[d-j]a

[0-9]

cp {*.dat,*.c,*.pdf} ~

rm thisfile[!8]*

Regular Expressions¶

Regular Expressions are a type of globbing patterns that are used when you are working with text.

Regular Expressions can be used with programs like grep, find and many others.

Some common examples of regular expressions:

• . matches any single character. Same as ? in standard wildcard expressions.
• \ is used as an “escape” character for a subsequent special character.
• .* is used to match any string, equivalent to * in standard wildcards.
• * the proceeding item is matched zero or more times. ie. n* will match n, nn, nnnn, nnnnnnn but not na or any other character.
• ^ means “the beginning of the line”. So “^a” means find a line starting with an “a”.
• $ means “the end of the line”. So “a$” means find a line ending with an “a”.
• [ ] specifies a range. Same as for normal wildcards. This is an ‘or’ relationship (you only need one to match).
• | This wildcard makes a logical OR relationship between wildcards. You can thus search something or something else. You may need to add a ‘' before this command to avoid the shell thinking you want a pipe.
• [^] This is the equivalent of [!] in standard wildcards, i.e. it is a logical “not” and will match anything not listed within the square brackets.

$ cat myfile | grep '^s.*n$'

Scripting¶

Scripting is used to perform complex or repetitive tasks without user intervention. All Linux commands can be used in a script including wild cards.

The most common reason for making a script is probably to avoid writing the same command again and again.

#!/bin/bash
grep ’ABCD’ file.pdb > file_filtered.pdb
program < file_filtered.pdb > output.dat

To change the permissions to execute a script (here named analysis.sh), for just the user, you could do:

$ chmod u+x analysis.sh

The above script can then be executed with

$ ./analysis.se

For more examples of (more useful) scripts, see for instance this list of 25 Easy Bash Script Examples.

Alias¶

You will often have to write the same command again and again. If it is a longer command, it is reducing your productivity having to repeat it. Then you can use the alias command to create an ‘alias’ for your command.

To see the currently definted aliases, execute the ‘alias’ command:

$ alias

b-an01 [~]$ alias
alias cdn='cat >/dev/null'
alias dir='ls -lAF'
alias l='dir'
alias ls='ls -F'

In order to create a new alias, you could write:

$ alias shortName="your custom command here"

Hints and tricks¶

This section contains some hints that might make working with Linux easier.

Short-cuts on the CLI¶

CTRL-SOMEKEY refers to pressing down the CTRL key and then another key while continuing to hold down CTRL.

Examples:

• CTRL-a: Go to the beginning of the line
• CTRL-e: Go to the end of the line
• CTRL-l: Clear the terminal
• TAB: Auto-complete (i.e. start write a command or file name and then press TAB to auto-complete, if possible)
• ARROW-UP: Pressing the arrow-up key repeatedly will let you cycle through recent commands
• CTRL-r: you will get a prompt to write text to search in the list of recent commands. The list is saved in .bash.history in your $HOME.

Misc¶

• Write ‘clear’ to clear the terminal
• write ‘history’ to see a list of the most recent commands written in the terminal
  • You can change the number of saved commands by setting the environment variable HISTSIZE in your .bashrc file in your home directory.
  • Example: Open .bashrc with nano. Somewhere (at the end for instance) add: export HISTSIZE=NUMBER” where NUMBER is the number of commands to save, for instance 10000.
• man PROGRAM will give you the manual for a specific program or command, if it exists
  • Example: man gcc will give open manual/help for the compiler gcc, containing flags to the compiler etc. Note that you need to first load a module that has gcc in.

Linux Cheat Sheet¶

written by P. Ojeda.

PDF version

The below is the same content as in the PDF file, but with minor changes/updates by B. Brydsö.

Miscellaneous

Wildcards

grep

find

pipes

Secure copy protocol

Note: vim commands are in ESC mode

awk

Batch jobs

sshfs