Wednesday, September 9, 2009

Unix Xargs Piping Toolkit Utility

Xargs constructs an argument list for an arbitrary Unix command using
standard input and executes this command. xargs [options] [command]

The xargs command creates an argument list for command from standard
input. It is typically used with a pipe getting its input from
commands like ls and find The latter is probably the most common
source of xargs input and is covered in examples below.

One of the most common xargs applications in pipes is to execute a
command once for each piped record:

find . -name '*050106*' print | xargs -n2 grep 'From: Ralph'
cat iplist | xargs -n1 nmap -sV
The find command searches the entire directory structure for filenames
that contain 050106 . The xargs command executes grep command for each
two argument (so that filename was visible). In the second example cat
command supplies the list of IPs for map to scan.

In many Unix shells there is a limit to the number of arguments
allowed on a single command line. This is often a problem when you
analyze spam blocked by a spam filter. Here xargs can help: if the
argument list read by xargs is larger than the maximum allowed by the
shell, xargs will bundle the arguments into smaller groups and execute
command separately for each argument bundle.

If no command is specified, xargs works similar to the echo command
and prints the argument list to standard output.

Options
Option Description
-n# Execute command once for every # argument. For example, -n2
bundles arguments into groups of two or less and executes command on
each argument bundle.
-l# Execute command once for every # lines of input. For example, -l1
creates a bundle of arguments for every one line of input and executes
command on each argument bundle.
-i Normally xargs places input arguments at the end of command. Used
with the -i option, xargs will replace all instances of {} with input
arguments. You need to put them in single brackets or use a backslash
(\) before each bracket to keep the shell from interpreting the
special characters.
-t Echo each command before executing. Nice for debugging
-p Prompts the user before executing each command. Useful for debugging.

Examples
To use a command on files whose names are listed in a file, enter:
xargs lint -a < cfiles
If the cfiles file contains the following text:

main.c readit.c
gettoken.c
putobj.c
the xargs command constructs and runs the following command:

lint -a main.c readit.c gettoken.c putobj.c
If the cfiles file contains more file names than fit on a single shell
command line (up to LINE_MAX), the xargs command runs the lint command
with the file names that fit. It then constructs and runs another lint
command using the remaining file names. Depending on the names listed
in the cfiles file, the commands might look like the following:

lint -a main.c readit.c gettoken.c . . .
lint -a getisx.c getprp.c getpid.c . . .
lint -a fltadd.c fltmult.c fltdiv.c . . .

This command sequence is not quite the same as running the lint
command once with all the file names. The lint command checks
cross-references between files. However, in this example, it cannot
check between the main.c and the fltadd.c files, or between any two
files listed on separate command lines.

For this reason you may want to run the command only if all the file
names fit on one line. To specify this to the xargs command use the -x
flag by entering:
xargs -x lint -a <cfiles
If all the file names in the cfiles file do not fit on one command
line, the xargs command displays an error message.


To construct commands that contain a certain number of file names, enter:
xargs -t -n 2 diff <<EOF
starting chap1 concepts chap2 writing
chap3
EOF
This command sequence constructs and runs diff commands that contain
two file names each (-n 2):
diff starting chap1
diff concepts chap2
diff writing chap3
The -t flag causes the xargs command to display each command before
running it, so you can see what is happening. The <<EOF and EOF
pattern-matching characters define a here document, which uses the
text entered before the end line as standard input for the xargs
command.


To insert file names into the middle of command lines, enter:
ls | xargs -t -I {} mv {} {}.old
This command sequence renames all files in the current directory by
adding .old to the end of each name. The -I flag tells the xargs
command to insert each line of the ls directory listing where {}
(braces) appear. If the current directory contains the files chap1,
chap2, and chap3, this constructs the following commands:
mv chap1 chap1.old
mv chap2 chap2.old
mv chap3 chap3.old

To run a command on files that you select individually, enter:
ls | xargs -p -n 1 ar r lib.a
This command sequence allows you to select files to add to the lib.a
library. The -p flag tells the xargs command to display each ar
command it constructs and to ask if you want to run it. Enter y to run
the command. Press the any other key if you do not want to run the
command.
Something similar to the following displays:

ar r lib.a chap1 ?...
ar r lib.a chap2 ?...
ar r lib.a chap3 ?...

To construct a command that contains a specific number of arguments
and to insert those arguments into the middle of a command line,
enter:
ls | xargs -n6 | xargs -I{} echo {} - some files in the directory
If the current directory contains files chap1 through chap10, the
output constructed will be the following:

chap1 chap2 chap3 chap4 chap5 chap6 - some files in the directory
chap7 chap8 chap9 chap10 - some file in the directory
Typically arguments are lists of filenames passed to xargs via a pipe.
Please compare:

$ ls 050106*
$ ls 050106* | xargs -n2 grep "From: Ralph"
In the first example list of files that starts with 050106 is printed.
In the second for each two such files grep is executed.
Additional Examples
John Meister's UNIX Notes

Change permissions on all regular files in a directory subtree to mode
444, and permissions on all directories to 555:

find -type f -print | xargs chmod 444

$ ls * | xargs -n2 head -10
line 1 of f1 line 2 of f1 line 3 of f1

ls * } xarg -n1 wc -1

(date +%D ; du -s ~) | xargs >> log

ls *.txt | xargs -i basename \{\} .ascii \
| xargs -i mv \{\}.ascii \{\}.ask

(Note that the backslash usage)


Another example. Let's "cat" the Contents of Files Listed in a File,
in That Order.

$ cat file_of_files
file1
file2

$ cat file1
This is the data in file1

$ cat file 2
This is the data in file2
So there are 3 files here "file_of_files" which contains the name of
other files. In this case "file1" and "file2". And the contents of
file1" and "file2" is shown above.


$ cat file_of_files | xargs cat
This is the data in file1
This is the data in file2

What if you want to find a string in all finds in the current
directory and below. Well the following script will do it.

#!/bin/sh
SrchStr=$1
shift
for i in $*; do
find . -name "$i" -type f -print | xargs egrep -n "$SrchStr"/dev/null
done
Another quite nice thing, used for updating CVS/Root files on a Zaurus:
find . -name Root | xargs cp newRoot
Just copies the contents of newRoot into every Root file. I think this
works too:

find . -name Root | xargs 'echo user@machine.dom:/dir/root >'

as long as the quote are used to avoid the initial interpretation of the >.

These pieces of randomness will look for all .sh files in PWD and
print the 41st line of each - don't ask me why I wanted to know.
Thanks to Brian R for these.

for f in *.sh; do sed -n '41p' $f; done
or

ls *.sh | xargs -l sed -n '41p' Remove all the files in otherdir that
exist in thisdir.
ls -1d ./* | xargs -i rm otherdir/{}

No comments: