Linux Command Line and Shell Scripting Bible

Acknowledgments xiPart I: The Linux Command Line 1Chapter 1: Starting with Linux Shells 3Investigating Linux 3Looking into the Linux kernel 4System memory management 5Software program management 6Hardware management 8Filesystem management 8The GNU utilities 9The core GNU utilities 10The shell 10The Linux desktop environment 11The X Window software 12The KDE Plasma desktop 12The GNOME desktop 13Other desktops 15Examining Linux Distributions 17Core Linux distributions 18Specialized Linux distributions 19Summary 20Chapter 2: Getting to the Shell 21Reaching the Command Line 21Console terminals 22Graphical terminals 22Accessing CLI via a Linux Console Terminal 23Accessing CLI via Graphical Terminal Emulation 26Using the GNOME Terminal Emulator 27Accessing GNOME Terminal 27The menu bar 32Using the Konsole Terminal Emulator 36Accessing Konsole 36The menu bar 38Using the xterm Terminal Emulator 42Accessing xterm 43Command-line parameters 44Summary 45Chapter 3: Basic Bash Shell Commands 47Starting the Shell 47Using the Shell Prompt 48Interacting with the Bash Manual 49Navigating the Filesystem 53Looking at the Linux filesystem 53Traversing directories 57Using absolute directory references 57Using relative directory references 59Listing Files and Directories 60Displaying a basic listing 61Displaying a long listing 63Filtering listing output 64Handling Files 66Creating files 66Copying files 66Using command-line completion 69Linking files 69Renaming files 71Deleting files 73Managing Directories 74Creating directories 74Deleting directories 75Viewing File Contents 77Viewing the file type 77Viewing the whole file 78Using the cat command 78Using the more command 78Using the less command 80Viewing parts of a file 80Using the tail command 80Using the head command 82Summary 82Chapter 4: More Bash Shell Commands 85Monitoring Programs 85Peeking at the processes 85Unix-style parameters 86BSD-style parameters 89The GNU long parameters 91Real-time process monitoring 93Stopping processes 95The kill command 95The pkill command 96Monitoring Disk Space 96Mounting media 97The mount command 97The umount command 99Using the df command 100Using the du command 101Working with Data Files 102Sorting data 102Searching for data 106Compressing data 108Archiving data 109Summary 111Chapter 5: Understanding the Shell 113Investigating Shell Types 113Exploring Parent and Child Shell Relationships 117Looking at process lists 121Creatively using subshells 123Investigating background mode 123Putting process lists into the background 125Looking at co-processing 126Understanding External and Built-In Commands 127Looking at external commands 128Looking at built-in commands 129Using the history command 130Using command aliases 134Summary 135Chapter 6: Using Linux Environment Variables 137Exploring Environment Variables 137Looking at global environment variables 138Looking at local environment variables 140Setting User-Defined Variables 141Setting local user-defined variables 141Setting global environment variables 142Removing Environment Variables 144Uncovering Default Shell Environment Variables 145Setting the PATH Environment Variable 150Locating System Environment Variables 152Understanding the login shell process 152Viewing the /etc/profi le file 152Viewing the $HOME startup files 156Understanding the interactive shell process 157Understanding the noninteractive shell process 158Making environment variables persistent 159Learning about Variable Arrays 159Summary 161Chapter 7: Understanding Linux File Permissions 163Exploring Linux Security 163The /etc/passwd file 164The /etc/shadow file 165Adding a new user 166Removing a user 169Modifying a user 169usermod 170passwd and chpasswd 170chsh, chfn, and chage 171Using Linux Groups 173The /etc/group file 173Creating new groups 174Modifying groups 175Decoding File Permissions 176Using file permission symbols 176Default file permissions 178Changing Security Settings 179Changing permissions 180Changing ownership 181Sharing Files 182Access Control Lists 184Summary 186Chapter 8: Managing Filesystems 189Exploring Linux Filesystems 189Exploring the Linux filesystem evolution 190Looking at the ext filesystem 190Looking at the ext2 filesystem 190Digging into journaling filesystems 191Looking at the ext3 filesystem 191Looking at the ext4 filesystem 191Looking at the JFS filesystem 192Looking at ReiserFS 192Looking at XFS 192Understanding the volume-managing filesystems 192Looking at the ZFS filesystem 193Looking at the Btrfs filesystem 193Looking at the Stratis filesystem 193Working with Filesystems 194Creating partitions 194Looking at the fdisk utility 194Working with gdisk 197The GNU parted command 198Creating a filesystem 199Checking and repairing a filesystem 201Managing Logical Volumes 202Exploring LVM layout 203Physical volume 203Volume group 203Logical volume 203Understanding the LVM in Linux 203Create the PVs 204Create a VG 204Create an LV 205Using the Linux LVM 206Format and mount an LV 206Growing or shrinking your VGs and LVs 206Summary 207Chapter 9: Installing Software 209Exploring Package Management 209Inspecting the Debian-Based Systems 210Managing packages with apt 211Installing software packages with apt 213Upgrading software with apt 215Uninstalling software with apt 216The apt repositories 218The Red Hat-Based Systems 220Listing installed packages 220Installing software with dnf 221Upgrading software with dnf 223Uninstalling software with dnf 223Dealing with broken dependencies 223RPM repositories 224Managing Software Using Containers 225Using snap containers 225Using flatpak containers 227Installing from Source Code 229Summary 232Chapter 10: Working with Editors 235Visiting the vim Editor 235Checking your vim package 236Exploring vim basics 237Editing data 240Copying and pasting 240Searching and substituting 241Navigating the nano Editor 242Exploring the Emacs Editor 244Checking your Emacs package 245Using Emacs on the console 246Exploring the basics of Emacs 247Editing data 248Copying and pasting 249Searching and replacing 249Using buffers in Emacs 250Using windows in console mode Emacs 251Using Emacs in a GUI 252Exploring the KDE Family of Editors 253Looking at the KWrite editor 253Looking at the Kate editor 259Exploring the GNOME Editor 263Starting gedit 264Understanding basic gedit features 265Managing plugins 268Summary 271Part II: Shell Scripting Basics 273Chapter 11: Basic Script Building 275Using Multiple Commands 275Creating a Script File 276Displaying Messages 278Using Variables 279Environment variables 280User variables 281Command substitution 283Redirecting Input and Output 284Output redirection 285Input redirection 285Employing Pipes 287Performing Math 289The expr command 290Using brackets 292A floating-point solution 293The basics of bc 293Using bc in scripts 295Exiting the Script 297Checking the exit status 297The exit command 298Working through a Practical Example 300Summary 301Chapter 12: Using Structured Commands 303Working with the if-then Statement 303Exploring the if-then-else Statement 306Nesting ifs 307Trying the test Command 311Using numeric comparisons 313Using string comparisons 314Looking at string equality 315Looking at string order 316Looking at string size 318Using file comparisons 320Checking directories 320Checking whether an object exists 321Checking for a file 322Checking for read access 324Checking for empty files 325Checking whether you can write to a file 326Checking whether you can run a file 327Checking ownership 328Checking default group membership 329Checking file date 330Considering Compound Testing 331Working with Advanced if-then Features 332Using single parentheses 332Using double parentheses 333Using double brackets 335Considering the case Command 335Working through a Practical Example 337Summary 343Chapter 13: More Structured Commands 345Looking at the for Command 345Reading values in a list 346Reading complex values in a list 347Reading a list from a variable 349Reading values from a command 350Changing the field separator 351Reading a directory using wildcards 352Trying the C-Style for Command 354The C language for command 355Using multiple variables 356Exploring the while Command 357Basic while format 357Using multiple test commands 358Using the until Command 359Nesting Loops 361Looping on File Data 364Controlling the Loop 365The break command 365Breaking out of a single loop 365Breaking out of an inner loop 366Breaking out of an outer loop 367The continue command 368Processing the Output of a Loop 371Working through a Few Practical Examples 372Finding executable files 373Creating multiple user accounts 374Summary 375Chapter 14: Handling User Input 377Passing Parameters 377Reading parameters 377Reading the script name 380Testing parameters 381Using Special Parameter Variables 382Counting parameters 382Grabbing all the data 384Being Shifty 386Working with Options 388Finding your options 388Processing simple options 388Separating options from parameters 389Processing options with values 391Using the getopt command 392Looking at the command format 392Using getopt in your scripts 393Advancing to getopts 395Standardizing Options 398Getting User Input 399Reading basics 399Timing Out 401Reading with no display 402Reading from a file 403Working through a Practical Example 404Summary 408Chapter 15: Presenting Data 411Understanding Input and Output 411Standard file descriptors 412STDIN 412STDOUT 413STDERR 414Redirecting errors 414Redirecting errors only 414Redirecting errors and data 415Redirecting Output in Scripts 416Temporary redirections 416Permanent redirections 417Redirecting Input in Scripts 418Creating Your Own Redirection 419Creating output file descriptors 419Redirecting file descriptors 420Creating input file descriptors 421Creating a read/write file descriptor 422Closing file descriptors 423Listing Open File Descriptors 424Suppressing Command Output 426Using Temporary Files 427Creating a local temporary file 427Creating a temporary file in /tmp 428Creating a temporary directory 429Logging Messages 430Working through a Practical Example 431Summary 433Chapter 16: Script Control 435Handling Signals 435Signaling the Bash shell 435Generating signals 436Interrupting a process 436Pausing a process 437Trapping signals 438Trapping a script exit 440Modifying or removing a trap 441Running Scripts in Background Mode 443Running in the background 443Running multiple background jobs 445Running Scripts without a Hang-up 447Controlling the Job 448Viewing jobs 448Restarting stopped jobs 450Being Nice 451Using the nice command 452Using the renice command 453Running like Clockwork 454Scheduling a job using the at command 454Understanding the at command format 454Retrieving job output 455Listing pending jobs 457Removing jobs 457Scheduling regular scripts 458Looking at the cron table 458Building the cron table 459Viewing cron directories 460Looking at the anacron program 460Starting scripts with a new shell 462Working through a Practical Example 463Summary 469Part III: Advanced Shell Scripting 471Chapter 17: Creating Functions 473Exploring Basic Script Functions 473Creating a function 474Using functions 474Returning a Value from a Function 477The default exit status 477Using the return command 478Using function output 479Using Variables in Functions 480Passing parameters to a function 480Handling variables in a function 482Global variables 483Local variables 484Investigating Array Variables and Functions 485Passing arrays to functions 485Returning arrays from functions 487Considering Function Recursion 488Creating a Library 489Using Functions on the Command Line 491Creating functions on the command line 491Defining functions in the bashrc file 492Directly defining functions 492Sourcing function files 493Working Through a Practical Example 494Downloading and installing 494Building the library 495The shtool library functions 496Using the library 497Summary 497Chapter 18: Writing Scripts for Graphical Desktops 499Creating Text Menus 499Create the menu layout 499Create the menu functions 501Add the menu logic 502Putting it all together 502Using the select command 504Doing Windows 505The dialog package 506The msgbox widget 507The yesno widget 508The inputbox widget 508The textbox widget 510The menu widget 511The fselect widget 512The dialog options 513Using the dialog command in a script 515Getting Graphic 516The KDE environment 517kdialog widgets 517Using kdialog 519The GNOME environment 521zenity Widgets 521Using zenity in scripts 523Working Through a Practical Example 525Summary 529Chapter 19: Introducing sed and gawk 531Manipulating Text 531Getting to know the sed editor 532Defining an editor command in the command line 533Using multiple editor commands in the command line 534Reading editor commands from a file 534Getting to know the gawk program 535Visiting the gawk command format 536Reading the program script from the command line 536Using data field variables 537Using multiple commands in the program script 538Reading the program from a file 539Running scripts before processing data 539Running scripts after processing data 540Looking at the sed Editor Basic Commands 542Introducing more substitution options 542Substituting flags 542Replacing characters 543Using addresses 544Addressing the numeric line 544Using text pattern filters 545Grouping commands 546Deleting lines 547Inserting and appending text 549Changing lines 551Transforming characters 552Printing revisited 553Printing lines 554Printing line numbers 555Listing lines 555Using files with sed 556Writing to a file 556Reading data from a file 557Working Through a Practical Example 559Summary 564Chapter 20: Regular Expressions 565Exploring Regular Expressions 565A definition 565Types of regular expressions 567Defining BRE Patterns 567Plain text 567Special characters 569Anchor characters 570Starting at the beginning 570Looking for the ending 571Combining anchors 572The dot character 572Character classes 573Negating character classes 575Using ranges 576Special character classes 577The asterisk 578Trying Out Extended Regular Expressions 579The question mark 579The plus sign 580Using braces 581The pipe symbol 582Grouping expressions 583Working Through Some Practical Examples 584Counting directory files 584Validating a phone number 585Parsing an email address 587Summary 589Chapter 21: Advanced sed 591Looking at Multiline Commands 591Navigating the next command 592Using the single-line next command 592Combining lines of text 593Navigating the multiline delete command 595Navigating the multiline print command 596Holding Space 598Negating a Command 599Changing the Flow 602Branching 603Testing 605Replacing via a Pattern 606Using the ampersand 606Replacing individual words 607Placing sed Commands in Scripts 608Using wrappers 608Redirecting sed output 609Creating sed Utilities 610Spacing with double lines 610Spacing files that may have blanks 611Numbering lines in a file 612Printing last lines 613Deleting lines 614Deleting consecutive blank lines 615Deleting leading blank lines 616Deleting trailing blank lines 616Removing HTML tags 617Working Through a Practical Example 619Summary 624Chapter 22: Advanced gawk k 627Using Variables 627Built-in variables 628The field and record separator variables 628Data variables 631User-defined variables 634Assigning variables in scripts 634Assigning variables on the command line 635Working with Arrays 636Defining array variables 636Iterating through array variables 637Deleting array variables 638Considering Patterns 638Regular expressions 639The matching operator 639Mathematical expressions 640Structured Commands 641The if statement 641The while statement 643The do-while statement 644The for statement 645Printing with Formats 645Using Built-in Functions 648Mathematical functions 649String functions 650Time functions 652Trying Out User-Defined Functions 653Defining a function 653Using your functions 654Creating a function library 654Working Through a Practical Example 655Summary 657Chapter 23: Working with Alternative Shells 659Considering the Dash Shell 659Looking at the Dash Shell Features 661The Dash command-line parameters 661The Dash environment variables 662Default environment variables 662Positional parameters 663User-defined environment variables 663The Dash built-in commands 664Scripting in Dash 665Creating Dash scripts 665Things that won't work 665Using arithmetic 666The test command 666The function command 667Exploring the zsh Shell 668Viewing Parts of the zsh shell 668Shell options 669Built-in commands 670Core built-in commands 670Add-in modules 673Viewing, adding, and removing modules 673Scripting with zsh 674Mathematical operations 674Performing calculations 674Mathematical functions 675Structured commands 676Functions 677Working Through a Practical Example 677Summary 678Part IV: Creating and Managing Practical Scripts 679Chapter 24: Writing Simple Script Utilities 681Performing Backups 681Backing up files daily 682Obtaining the required functions 682Creating a daily archive location 685Creating a daily backup script 686Running the daily archive script 688Creating an hourly archive script 689Running the hourly archive script 692Managing Account Deletion 693Obtaining the required functions 693Getting the correct account name 693Creating a function to get the correct account name 695Verifying the entered account name 696Determining whether the account exists 698Removing any account processes 699Finding account files 702Removing the account 702Creating the script 703Running the script 708Monitoring Your System 710Obtaining the default shell audit functions 710Obtaining the permission audit functions 714Creating the script 716Running the script 719Summary 721Chapter 25: Getting Organized 723Understanding Version Control 723Working directory 724Staging area 725Looking at the local repository 725Exploring a remote repository 726Branching 726Cloning 726Using Git for VCS 727Setting Up Your Git Environment 727Committing with Git 732Summary 739Appendix A: Quick Guide to Bash Commands 741Appendix B: Quick Guide to sed and gawk 755Index 767