Fun with bash shell brace expansion

September 9, 2009

Today, I learned something new about bash scripting – the power of brace expansion.  Brace expansion in bash is a way to quickly generate variations of strings used for command line arguments.  I won’t explain this since  this Linux Journal article and the Bash Reference Manual already do an excellent job of it.

First, let me describe my problem.  Some years ago, during a construction project, I set up a cheap web cam in a shed at the site and wrote a quick script that grabbed an image every minute and stored it.  I did this so that I could later create a stop motion video animation of the whole project.  If you’re interested, I posted the videos I made to Flickr.  If you want to look at the one with the most action, just look at this video.

For the duration of the project (about 5 weeks) I wound up with over 60,000 images, all consistently named with timestamps and in directories by day.  I browsed through the thumbnails of each day, and wrote a small text file containing the date and time ranges I was interested in for the video (about 12,000 images).  So I wound up with a manually created text file that had lines that looked like this:

8/25 - 8:56 -> 4:49 - smoke, jackhammer

I planned on using Quicktime Pro to create the stop motion video, by using its feature to open an ‘Image Sequence’.  An image sequence is a directory full of images that are named in a sequence (like picture1.jpg, picture2.jpg, picture3.jpg, etc).  Using ‘Open Image Sequence’, select picture1.jpg and QTPro will find all the sibling images in that directory and create a stop action movie for you in the order the files are named.

This is great, but I had 60,000 files to sort through.  I needed a way to copy (and rename) the images of the time ranges I wanted and put them into a directory for quicktimepro to use.  I decided to specify my time ranges using bash brace expression arguments.  So the images, which were consistently named like this:

for example: 2004-08-25/cam_2004-08-25_13_59_00.jpg

I then changed my text file with human readable date ranges, from above, into something that had lines like this:

2004-08-25/cam_2004-08-25_{08_5{6..9},09,{10..15},16_0,16_{10..49}}*.jpg #8/25-8:56->4:49-smoke,jackhammer

I just needed to write a script that kept a counter, processed each line of my date range file by cp’g each matching file into a new directory and renaming that file based on a counter.  Sounds easy.  First though, I had to learn something about the brace expansion.

Brace expansion only happens once, right after the command line is tokenized.  So this works:

$ echo foo{1,2,3}
foo1 foo2 foo3

That’s great, but this does not work:

$ myline=foo{1,2,3}
$ echo $myline

This does not work since the brace expansion (foo{1,2,3} -> foo1 foo2 foo3) happens prior to the shell parameter expansion ($myline -> foo{1,2,3}).  To examplify the order, try the reverse experiment.  It should work out the same way:

$ one=1
$ two=2
$ echo values_{$one,$two}
values_1 values_2

So it makes sense, but if we still want to force our $myline variable value through brace expansion, we’ll need to have bash evaluate it twice.  for this, we’ll need the bash builtin ‘eval’ command.  eval evaluates a command line for you, and using it with echo and backtics you can get it to double evaluate our variable, like this:

$ myline=foo{1,2,3}
$ evaluatedline=`eval echo $myline`
$ echo $evaluatedline
foo1 foo2 foo3

And there you go.  Now, using a similar mechanism, we can iterate through our file based list of brace expressions, expand each one and copy the files I want to a new location and sequenced name.  The full script to do this is here:


# make the output directory if it doesn't exist
if [ ! -e $OUTPUT_DIR ]
 mkdir -p $OUTPUT_DIR

# clear out previous run's output
# use this construct since passing such a huge amount (12,000) of
#   files to rm (via rm *.jpg)
#    will fail otherwise (due to 'too many args' error)
ls -tr $OUTPUT_DIR | xargs -t -I{} rm -f ${OUTPUT_DIR}/{}

# $FILE_OF_PATTERNS is the path to a file that contains path specifiers,
#   one per line, like this:
# 2004-10-26/cam_2004-10-26_{08,09_00}*.jpg
# each line specifies some files we want to include in the animation
# do this for each pattern spec I want to grab
for i in `egrep -o '^[^#]*'  $FILE_OF_PATTERNS`
 echo =========== PROCESSING: $i ================
 # force the brace expression through another evaluation, like so:
 foo=`eval echo $i`
 # for every file that matches this sub pattern, copy that image
 #   to a file named with a more simple sequence (p1.jpg, p2.jpg, ...)
 for file in `ls $foo 2> /dev/null`
   let COUNT=$COUNT+1

2 Responses to “Fun with bash shell brace expansion”

  1. jearsh Says:

    I was trying to use brace expansion in one of my rsync/backup scripts. I had the expansion in a variable, and I couldn’t figure out why it wasn’t working. All I needed was eval, thanks.

  2. Matteo Says:

    Me too! I’ve been trying to solve this problem for 48 hrs….
    Thank you so much!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: