Parallel ssh or pssh is a group of commands that allow you to perform operations on multiple hosts simultaneously. Included in the package are the commands pnuke, prsync, pscp, pslurp and pssh.
· pnuke is for killing processes in parallel on a number of hosts.
· prsync is for copying directories in parallel to a number of hosts.
· pscp is for copying files in parallel to a number of hosts.
· pslurp is a program for copying files in parallel from a number of hosts.
· pssh is for executing ssh in parallel on a number of hosts.
While some people are more familiar with using Ansible for running ad-hock commands, I find that pssh is much more efficient and yields better results. With pssh, I can capture the output of the commands I’ve executed which gives me the opportunity to do analysis on the returns. For instance, I can run uptime on all of my hosts. The output will be captured into a directory where I can then find servers that have been online more then some number of days, or I can find hosts in my environment that have a load average of more than some threshold.
Development:
The parallel-ssh suite of tools is mature and has been around for about 11 years. While there isn’t a need for active development, version 2.3.1-7 has been updated to support Python 3. Previous version of the pssh suite of tools will only work out of the box on Python 2. For this discussion, I will only focus on the 2.3.1-7 version of the software.
Downloading and Installing Pssh- 2.3.1-7:
CentOS 7 and RHEL 7:
The rpm package is available here: https://centos.pkgs.org/7/nux-dextop-x86_64/pssh-2.3.1-7.el7.nux.noarch.rpm.html, Or you can:
# sudo yum install https://li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-5.el7.nux.noarch.rpm
# sudo yum install pssh
For all other UNIX or Linux distributions:
# sudo mkdir -p /usr/local 2> /dev/null # cd /usr/local # sudo git clone https://github.com/nplanel/parallel-ssh # echo “PATH=$PATH:/usr/local/parallel-ssh/bin” >> ~/.bashrc # echo “MANPATH=$MANPATH:/usr/local/parallel-ssh/man” >> ~/.bashrc # source ~/.bashrc
Using the pssh Suite of Tools By Example:
While I generally use the pssh suite of tools with ssh keys, you do have the ability to use a different username and you can make it ask for a password. To use a different username with any of the tools, you can use the -l <username> switch and to make it ask for a password you can use the -A switch.
The host list:
The pssh suite of tools is generally used in conjunction with a list of hosts. The list of hosts is a file containing host names, one host per line; standard UNIX/Linux comments beginning with ‘#’ are honored.
###################################### # this is an example of a host file ###################################### drop.biffsocko.com # somehost.domain.com ? will be skipped sdf.freeshell.org www.greenstarsystems.com
Using pssh:
pssh -O StrictHostKeyChecking=no -h <hostfile> -t <timeout time> -o <output directory> “command here”
I generally use pssh with the above syntax.
-O StrictHostKeyChecking=no with the -O switch, I can pass ANY of the ssh options to pssh. In this case I’m asking for the host key checking to be disabled. This will disable getting this message:
biff@new-host-3 ]$ ssh host1
The authenticity of host ‘host1 (11.11.11.11)’ can’t be established.
ECDSA key fingerprint is SHA256:m1WFCXq+Z7+eW7EOIKYZI5Oov0+XjDa1BktBsYacoAE.
Are you sure you want to continue connecting (yes/no)?
-h <hostfile> This indicates the host file containing the list of servers to perform the command on
-t <timeout> This option gives a timeout for ssh.
-o <output directory> with this option, a directory will be created (if there isn’t one already there) that will contain a file for each server (listed in the host list) that the command was performed on. The file will contain the output of the command. For instance, if I “cat /etc/passwd”, the file would contain the contents of the password file for that host.
A typical scenario would look like this:
# pssh -O StrictHostKeyChecking=no -h listOfHosts -o psshout -t 20 “uptime”
The output of the command will look like this:
[biff@new-host-3 ~]$ pssh -O StrictHostKeyChecking=no -h list -o psshout -t 20 "uptime" [1] 10:58:04 [SUCCESS] drop.biffsocko.com [2] 10:58:04 [SUCCESS] www.greenstarsystems.com [3] 10:58:22 [FAILURE] sdf.freeshell.org Timed out, Killed by signal 9
As you can see, I was successful on two of the three servers (my account has expired at freeshell.org). Now, to see the output I move to the psshout directory that was created.
biff@new-host-3 ~]$ cd psshout [biff@new-host-3 psshout]$ ls -l total 8 -rw-rw-r--. 1 biff biff 73 Dec 31 10:58 drop.biffsocko.com -rw-rw-r--. 1 biff biff 0 Dec 31 10:58 sdf.freeshell.org -rw-rw-r--. 1 biff biff 73 Dec 31 10:58 www.greenstarsystems.com
See the output above and note the 0 byte file on the server that the command failed on. Below, you can check out the contents of one of the files:
[biff@new-host-3 psshout]$ cat drop.biffsocko.com
10:58:04 up 173 days, 37 min, 0 users, load average: 0.03, 0.13, 0.11
Using pscp to copy files TO multiple hosts
Once you’ve learned the syntax for pssh, the rest of the commands come easily. For pscp, the general syntax that I use is as follows:
pscp -O StrictHostKeyChecking=no -h <hostfile> -t <timeout> localfile remotefile
for example:
[biff@new-host-3 ~]$ pscp -O StrictHostKeyChecking=no -h list -t 5 testfile /tmp/testfile [1] 11:11:05 [SUCCESS] drop.biffsocko.com [2] 11:11:05 [SUCCESS] www.greenstarsystems.com [3] 11:11:09 [FAILURE] sdf.freeshell.org Timed out, Killed by signal 9
As you can see, I was able to copy my file to two of the remote servers, however since my account lapsed on the third server, a FAILURE was indicated.
In this case (below), I incorporated the same command using a test user account and required a password input:
[biff@new-host-3 ~]$ pscp -O StrictHostKeyChecking=no -l testuser -A -h list -t 5 testfile /tmp/testfile Warning: do not enter your password if anyone else has superuse privileges or access to your account. Password: [1] 11:13:53 [SUCCESS] www.greenstarsystems.com [2] 11:13:53 [SUCCESS] drop.biffsocko.com [3] 11:13:57 [FAILURE] sdf.freeshell.org Timed out, Killed by signal 9
Note the -l testuser for the username testuser and the -A to ask for the password of testuser.
Using pslurp to copy files FROM multiple hosts
Similar to pscp, I use pslurp along these lines:
pslurp -O StrictHostKeyChecking=no -h <hostfile> -t <timeout> remotefile localfile
In an effort to check local accounts on each host, I’d want to download the /etc/passwd file for each host. For this purpose I can run the following commands:
By default, pslurp will make the output directory the current working directory. So, I find it’s better to create a new directory and run it from there
[biff@new-host-3 ~]$ mkdir pslurp-out[biff@new-host-3 ~]$ cd pslurp-out/ [biff@new-host-3 pslurp-out]$ pslurp -O StrictHostKeyChecking=no -h ../list -t 5 /etc/passwd passwd [1] 11:30:21 [SUCCESS] drop.biffsocko.com [2] 11:30:21 [SUCCESS] www.greenstarsystems.com [3] 11:30:25 [FAILURE] sdf.freeshell.org Timed out, Killed by signal 9
Instead of creating a file for each host like pssh does, pslurp creates a directrory for each host containing the download file(s),
[biff@new-host-3 pslurp-out]$ ls -l total 0 drwxrwxr-x. 2 biff biff 20 Dec 31 11:30 drop.biffsocko.com drwxrwxr-x. 2 biff biff 6 Dec 31 11:30 sdf.freeshell.or drwxrwxr-x. 2 biff biff 20 Dec 31 11:30 www.greenstarsystems.com [biff@new-host-3 pslurp-out]$ cd drop.biffsocko.com [biff@new-host-3 drop.biffsocko.com]$ ls passwd
Using prsync to sync directories TO multiple hosts
While pscp is great for distributing a file, prsync is good for distributing entire directories. Consider the following syntax.
prsync -O StrictHostKeyChecking=no -h ../list -t 20 -a /home/biff/pslurp-out/ /tmp/rsyncstuff
Note that I have used the same syntax as the other pssh commands, with the addition of the “-a” switch. This is the rsync archive option. The prsync command also supports -r for recursive, -z for compression and -v for verbose.
Here is an example of how I might use it:
[biff@new-host-3 pslurp-out]$ prsync -O StrictHostKeyChecking=no -h ../list -t 20 -a /home/biff/pslurp-out/ /tmp/rsyncstuff [2] 11:50:18 [SUCCESS] www.greenstarsystems.com [3] 11:50:37 [FAILURE] sdf.freeshell.org Timed out, Killed by signal 9
Using pnuke to kill processes on multiple hosts
You can use pnuke to kill a process on multiple hosts. Here is the common syntax:
pnuke -O StrictHostKeyChecking=no -h list -t 20 <pattern>
where pattern is a regex to match a process. For example, to kill the httpd process on all of our hosts, we’d run the following:
pnuke -O StrictHostKeyChecking=no -h list -t 20 httpd
A Word of Caution
The pssh suite of tools are incredibly powerful, but with great power comes great responsibility. Pssh and it’s accompanying tools can be used to run ad-hock commands and configuration changes across dozens, hundreds and even thousands of servers within a few seconds. Given the correct permissions, a mistake can also be pushed out. I recommend running each command on a small subset of test servers prior to running against production servers.
Notes:
pssh can be installed using the pip command (pip install pssh), however the version that is currently supported on pypi.org is incompatible with python 3. I have contacted the maintainers and have asked them to update to version 2.3.1-7 which does work with python 3. Additionally, I’ve asked Redhat to support pssh under its EPEL repository.
