Compressor error: "Unable to submit to queue"

If you're getting this error message when trying to submit a batch to Compressor, follow the steps listed here:

Compressor error: "Unable to connect to background process"

Pro Applications Update 2010-01

Apple released the second Pro Apps update for Final Cut Studio 3 yesterday. It includes Final Cut Pro 7.0.2, Motion 4.0.2, Color 1.5.2, Cinema Tools 4.5.1, Compressor 3.5.2 and Qmaster 3.5.2. There are a lot of fixes there, some for critical issues, and also some performance updates too.

Here's the complete list of changes:

Final Cut Pro 7.0.2
* Drag and drop of files from iTunes works as expected with supported file types.
* Fixes the "Preview Unsupported For This Format" message that would occur occasionally with P2 media recorded to a Firestore FS100 recording device from Panasonic cameras.
* Fixes an issue ingesting from P2 cards with spanned clips which contain more than 80 individual elements.
* Fixes a performance issue with transcoding AVCHD to ProRes on Mac systems using the GeForce 9400M graphics card.
* Fixes an issue with some third-party filters that would cause Final Cut Pro to quit unexpectedly on Mac OS X v10.6 Snow Leopard.
* Markers on multi-cam clips are now preserved when edited into a sequence.
* Log and Transfer function is no longer disabled while using the Share function to export media.
* Media Manager will now copy ProRes media created by AJA Ki Pro.
* Multiple fixes to Change Speed functionality including:
* The term "Duration" in the Change Speed interface is replaced with "Smoothness" which describes the function better.
* Speed keyframes no longer snap to other speed key frames. This improves ease of use when dragging to change speed in the timeline.
* Fixes an issue in which removing a reverse speed change with Ripple Sequence enabled would cause a clip to become half its length.

Motion 4.0.2
* Fixes rendering and performance issues that could occur when using Motion projects in a Final Cut Pro timeline.
* Motion is now more stable when operating in low system memory or low graphic card memory situations.
* The default project setting is now set to PAL or NTSC depending on the operating system localization.
* Addresses an issue that impacted the reliability of the Undo command.
* Corrects the rendering of the Checkerboard Generator to the proper orientation when rendering from a 3D camera.
* Fixes an image usability issue that could be encountered on complex projects when rendered in 32-bit float with Motion Blur turned on.
* Exclusively exporting the alpha channel of a Motion project using the Share feature now returns the proper alpha channel.
* An issue with importing files named with dual-byte characters has been addressed.
* An issue with the importing of Japanese language text has been fixed.
* Fixes aspect ratio issues encountered with imported shapes and circle masks.
* A correction to the selection tool now makes it easier to select text.
* Fixes an issue related to the the rendering of soft shadows.
* Addresses issues that could produce significant artifacts when objects would intersect in 3D space in certain ways.
* Addresses issues that could be encountered with images of large resolution.
* A performance related fix was applied to the caustics generator.
* Adjusting paint stroke control points in 3D space is now more responsive.
* Corrects the rendering of anamorphic projects to the digital cinema desktop output.
* Changes were made to certain filters to match the output they produced in Motion 3.
* To address a user interaction issue, the selected animation curve in the keyframe editor now displays on top of other curves.
* Fixes were made that benefit some filters applied to images that are cropped.
* A performance issue with text flattened to a single plane was addressed.
* Fixes an issue that could have affected some third-party FxPlug plug-ins that perform temporal manipulations like retiming.
* Addresses issues which could appear to soften images, or flattened groups of images, of odd-sized resolutions.
* Manipulation issues that could occur with multiple duplicated and cloned objects or groups within a project have been addressed.

Color 1.5.2
* 3D Color Space Scope now allows zooming.
* Resolves an issue in which media would sometimes become disconnected after a save and restart.
* Fixes an issue with CFX nodes not being imported when choosing File > Import > Color Corrections.
* Addresses several issues with the loading and processing of motion trackers.
* Fixes an issue with the green channel on video scopes displaying an incorrect value.
* RED 2K renders from 4K media are no longer softened.

Cinema Tools 4.5.1
* Trying to save a Cinema Tools database to a read-only volume no longer closes the database.

Compressor 3.5.2
* Corrects disk size limitation messages when burning a project to an AVCHD dual-layer disc.
* Fixes issues with long chapter names not appearing correctly in disc templates.
* Fixes issues when inserting a DVD-R or BD-RE disc with data already present.
* Provides warnings before a DVD or Blu-ray Disc is burned when the source will not fit on the media.
* Provides an option to cancel a disc burn when quitting Compressor.
* Fixes issues with burning a DVD using DTS audio.
* Fixes an issue with automatic bit rate so that the appropriate bit rate for a single or dual layer DVD-R is applied.
* Corrects an issue with the encoding of HD video when burning a Blu-ray disc with surround sound.
* Addresses issues with saving and changing audio channels and video using surround sound.
* Corrects a problem that caused Share to not work when switching to another user on the same machine.
* Fixes a MobileMe sharing issue when entering a single character in the "Viewable by Name" field.
* Fixes MobileMe upload for file names that contain the "+", "?", or "%" characters.
* Corrects an issue with uploading an IFF image sequence with an uppercase extension.
* Corrects the default bit rate for the iPod encoding setting from the Share menu.
* Corrects color shift when transcoding image sequences.
* Correct timecode is now displayed in the Preview window when sending a Final Cut Pro sequence to Compressor and applying a timecode generator.
* Job Action "Open Application" now works with multiple targets.
* Fixes an issue with creating a new job with Target Output from an image sequence to a cluster.
* Fixes an issue with source field order for Compressor plugins.
* Fixes a pixel aspect ratio issue with transcoding a duplicated image sequence to a custom TIFF setting.
* Corrects an issue that stopped a WAV file being associated with with an image sequence for output.
* Compressor preferences are now honored when submitting jobs from Final Cut Server.
* Corrects a field order issue with certain sources.
* Fixes an issue with the audio tab becoming inactive when moving between the video and audio tab on MPEG-2 targets.
* Addresses an issue with upload to MobileMe with files containing reserved characters.
* Improves file size estimation for H264 files produced for Blu-ray.
* Corrects an issue when burning a Blu-ray or DVD disc from a Final Cut Pro sequence with 5.1 Surround Sound.
* Fixes an issue with display of password-protected MobileMe movies on iPhone.
* Addresses a pixel aspect ratio compatibility issue.
* Improves stability when processing large batches.

Qmaster 3.5.2
* Fixes a memory leak associated with transcoding to a YouTube destination.
* Fixes unexpected quitting when copying large numbers of image sequences to a destination.
* Corrects issues when copying AVI and reference movie media to a Wide Area Network cluster.
* Corrects an issue with Apple Qmaster command line preferences utility when a user is only logged in remotely.

Be aware, however, that people are already reporting issues with the new update.

Apple also posted the following known issues:
* Final Cut Studio (2009), Final Cut Pro 7.0.1, & 7.0.2 with Mac OS X v10.6: Final Cut may stop responding when capturing DVCPRO HD via FireWire
* Final Cut Studio (2009), Color 1.5: Upgrading a Color 1.0.x project to 1.5.x changes any existing Trackers
* Final Cut Studio (2009), Color 1.5.2: Video Output of REDCODE material clips at 97 IRE
* Final Cut Studio (2009), Color 1.5.2: Unrendered freeze frames in Color displays an unexpected frame

Combining odd and even frames in Shake

I've just finished helping a guy with a Shake issue via email and I think the solution would be useful to others. He had a situation where he had one movie clip with all the odd frames from a sequence and another clip with all the even frames, and he wanted to combine them into a single file.

At first glance, it would appear to be as simple as using the Select node to switch between them, however you would miss out half of the frames by doing this. Here's the solution that worked for him:

1. Import the two files into Shake via the FileIn node.

2. Add a TimeX node to the first clip and set it to 0.5*time (this doubles every frame) and do the same for the second clip.

3. Add a Select node and plug the output of the two TimeX nodes into it. So that it's not confusing later on, make sure the output of clip 1 goes to the first input and clip 2 goes to the second.

4. In the Branch parameter of the Select node, set a keyframe at frame 1 with a value of 1 (i.e. the first input), then step to frame 2 and keyframe the value to 2 (the second input).

5. Load the parameter into the Curve Editor by clicking the clock icon next to the Branch field so that a tick icon appears, and then click the Curve Editor tab.

6. In the Curve Editor, change the Cycle parameter to MirrorValue.

7. When you render the FileOut, make sure to double the frame range as the sequence is now twice as long.

Speeding up Compressor

Is Compressor taking too long to encode? Here are some tips to speed it up.

Don't export directly to Compressor from Final Cut Pro

Although sending your timeline directly to Compressor may seem faster because it cuts out the step of exporting as a QuickTime movie, it is much slower overall because Compressor needs to request each frame one-by-one from Final Cut Pro instead of just reading the data out of the movie clip. It gets even slower if you're using Frame Controls or doing multi-pass encoding.

Just go to File > Export > QuickTime Movie (not QuickTime Conversion) and make sure Setting is set to Current Settings and Recompress All Frames is switched off. This ensures that the movie will not be recompressed unnecessarily.


Here's a tip - if most of your timeline is already rendered, deselect Make Movie Self-Contained. This creates a small reference movie that links to the render files on disk instead of writing the data into the file. This will be much quicker to export. If your timeline is not rendered, however, this setting is unlikely to offer any speed advantages over a self-contained movie.

Once the file is exported, drag it into Compressor and set up your batch as normal.

Only use Frame Controls where necessary

Frame Controls allow you to improve the quality of resizing, retiming and deinterlacing operations, however they should only be switched on when you are actually performing one of these operations.


In addition, you should use Better quality instead of Best, as Best is usually reserved for extreme circumstances and in most situations gives you a massive performance hit with no benefit.

Perform heaving-lifting in a separate pass to the encoding

Using Frame Controls with a multi-pass or long-GOP encoder can really slow things down because the Frame Controls processing will need to be reapplied to every frame each time it is read. Even if you're not using Frame Controls, compressing from a processor-intensive codec such as HDV or H.264 can seriously slow things down.

It makes sense, therefore, to perform all these processor-intensive operations on an intermediate movie clip (ProRes would be a suitable codec for this) and then drag in the intermediate clip and encode it to the desired format.

But there is an easier way. Job chaining is a little-used yet very powerful feature of Compressor that allows you to plug the output of one job into the input of another for additional processing.

Here's how to do it:
1. Drag your movie clip into Compressor.
2. Drag either the ProRes 422 for Interlaced Material or ProRes 422 for Progressive Material (depending on your source media) setting onto the job in the batch window.
3. Go to Job > New Job with Target Output. This will add a new job to the batch with a chain symbol to show it is linked to the job above it.
4. Drag your desired output setting onto the chained job and set up the destination as normal.
5. Submit the batch

Use a cluster

A cluster is a group of computers where each machine processes a portion of the movie simultaneously, potentially providing a massive speed boost. This service is provided by Qmaster and can be set up in the Qmaster section of System Preferences or the Apple Qadministrator utility.

You will need Qmaster set up on each computer in the cluster, and all computers should have the required codecs installed and have full access to the source media and cluster storage area.

If you have a multi-core computer, Qmaster can emulate the effects of a cluster by launching multiple copies of Compressor side-by-side to process the movie in a "virtual cluster". This can make a dramatic difference to encoding times. I wrote a tutorial on this a while back.

Conclusion

Compressor is not the fastest encoder in the world but with these tricks you can make it a lot more bearable. One important trend to note is that the simplest method isn't always the fastest - with a little extra setup beforehand you can save a significant amount of time overall.

How to run Final Cut Studio 3 on a MacBook

I've been meaning to post this for a while but haven't had the chance until now. I had to be an early adopter of Final Cut Studio 3 because I develop software that interfaces with it, but I had no desire to edit on a x.0 release so I decided to install it on my MacBook solely for testing.

The only problem is that, unlike FCS 2, the FCS 3 installer refuses to continue because it requires a 128 MB graphics card. Only Motion and Color are heavy on the graphics card, so the other applications in the suite will work just fine on a lesser machine. Luckily the block is easy to work around.

Here's how I got it working on my MacBook:

1. You will need to change one of the files on the DVD. As you cannot write directly to the DVD itself, you will need to create a writable copy of the disc. Put in the Final Cut Studio DVD and open up Disk Utility (located in /Applications/Utilities).

2. Select the disc in the left-hand pane and click New Image in the toolbar.



3. Set Image Format to Read/write and Encryption to None.



4. Type a relevant filename and click Save. It will take several minutes to process.

5. Open up the disk image and ctrl-click on Install Final Cut Studio. Select Show Original.



6. A new window will appear with the FinalCutStudio.mpkg file selected. Ctrl-click on this file and select Show Package Contents.



7. Navigate to Contents/Resources and ctrl-click on the Requirements Checker bundle. Select Show Package Contents.



8. Navigate to Contents/Resources and open up minsys.plist in Property List Editor (if you have the Apple developer tools installed) or TextEdit.



9. Under the heading AELMinimumVRAM change the number from 128 to a number lower than or equal to your current video memory. I changed mine to 32. Alternatively you could change block to warn and the installer will warn you but let you continue installation.



10. Save the file, close all the folders that appeared and double-click Install Final Cut Studio on the disk image. You should now be able to install Final Cut Studio without problems.

Some of the applications in the studio also contain a minsys.plist file that will need to be modified. These applications are Final Cut Pro, Motion, Color, Soundtrack Pro and DVD Studio Pro (even though the latter apparently has no changes from FCS 2).

If you have FCS Maintenance Pack installed, you can do this quickly and easily by checking the Low Minimum Requirements option in System Toolkit. Otherwise follow the steps below.

11. Ctrl-click on the application in question and select Show Package Contents.



12. Navigate to Contents/Resources and open minsys.plist in either Property List Editor (if you have the developer tools installed) or TextEdit.



13. Under the heading AELMinimumVRAM change the number from 128 to a number lower than or equal to your current video memory. I changed mine to 32. Alternatively you could change block to warn.



14. Color will still warn you about your screen resolution - however you can just select Never show again and continue.



That's it! You will need to repeat steps 11-14 whenever you update Final Cut Studio.

Note: Some people have suggested simply removing the Requirements Checker application but I do not advocate deleting files.

Mac OS X 10.5.7 released

Apple has finally released the much-delayed 10.5.7 update to Leopard. There was much speculation that it was delayed to coincide with the full release of Safari 4 which is currently in beta, however this is clearly not the case as Apple has just released Safari 3.2.3 and an updated beta of Safari 4 (you won't see this in Software Update until you update to 10.5.7 though).

The release notes are quite brief, however it is possible to get a much better overview by looking at World of Apple's seed notes for the 10.5.7 beta. This is not a comprehensive list but gives a good indication of the final changes.

There's some good stuff in there - lots of networking fixes and several changes aimed at improving sleep/wake reliability (I've actually stopped putting my MacBook to sleep because it either wakes up constantly every 10 mins or never wakes up at all). It also contains updated graphics drivers. There's really nothing serious in there so you can update at your leisure. It's very much a polish update.

If you are thinking of updating though, remember the Golden Rules:
1. Never update in the middle of a project
2. Wait a week or two for problems to surface
3. Perform a clone of your system before installing
4. You will experience greater reliability from the larger combo updater

10.5.7 Delta Update (10.5.6 -> 10.5.7)
10.5.7 Combo Update (10.5.x -> 10.5.7)
10.5.7 Server Delta Update (10.5.6 -> 10.5.7)
10.5.7 Server Combo Update (10.5.x -> 10.5.7)

The bundled security update is also available separately for Tiger users as Security Update 2009-02.

Update: Mac Mini users are reporting problems already. It seems like certain DVI displays are not being recognized correctly on Mac Minis.

Guide to Final Cut Pro Internal Tools

Did you know Final Cut Pro has a hidden menu? Just hold down Cmd + Option + Shift and click the Tools menu. You should now see a new menu item called Internal Tools.



This is a debug menu used by the developers of FCP, but it actually has some uses for us mere mortals as well. It's especially useful for plugin developers and system troubleshooters, but some features benefit all users. Here is a rundown of what's in the menu.

Adjust Windows

When you open the Audio Mixer, Frame Viewer and several other tools in the Tools menu, they open in a tabbed window called the Toolbench. This normally appears on top of the Viewer and is hidden behind it when you select the Viewer again.

The Adjust Windows menu item adjusts the window layout so the Toolbench can fit in without being obscured by other windows. If you don't have the Toolbench open when you click this item, it will leave a gap in your window layout.

Before


After


To restore the default window configuration again, press Ctrl+U.

App / Perf Info

This generates a text file containing just about everything you could ever wish to know about your FCP configuration.



Probably the most useful part is the Project / Sequence Information section. This gives various statistics about your project including the total number of clips placed in your timeline and the proportion of those clips that were unique. It will even tell you how many edits you performed per minute (don't let your boss see this!).

Log OpenGL Profile Costs

This function displays timing information about OpenGL Profiling in the Console. It will do nothing until you start profiling (see the Statistics section below), at which point it tells you how long (in ms) it took your machine to process the given frame sizes and codecs.



One thing I noticed about this is that it takes a fair while for the data to be posted to the Console - around 1-2 minutes.

Log VM Footprint

This outputs the current memory usage to the Console.



Resident refers to data in physical memory, Virtual refers to data on the disk ready to be copied into main memory as required. Note that the system often assigns more virtual memory space than it needs, so not all of that space will actually be used. So if you see a ridiculous number like 4 GB, it is not necessarily a cause for alarm.

Movie Dumpster

This one crashes Final Cut Pro for me, so I wasn't able to ascertain exactly what it does. However, Apple offers a tool called Dumpster on their developer site which gives you a large amount of technical information about a specific movie clip. I would imagine this would be similar if it worked.

Playback Meter

This will show you how your media drives are performing, and is probably the most useful tool in the menu.



If you are not using a RAID, only the audio bars will work. Everything is fine while the bar is green, but if it turns yellow or red, this is an indication that your computer cannot keep up.

Screen Size Override

This allows you to fool Final Cut Pro into thinking that you have a different-sized monitor.

Just select the desired resolution from the pull down menus and press Ctrl + U to automatically resize the windows to the new dimensions.



If you mess things up, just restart Final Cut Pro and press Ctrl + U to restore the windows to the correct position for your screen resolution.

Statistics

This displays a lot of technical information about what's going on under the hood of FCP.



For instance, select the Render check box, drag the playhead across the timeline and then click the Average button. This will average the times taken to perform the various function calls required to render the image to the screen. If your system is getting sluggish, you may find it useful to use the Averages function to see which areas are taking a long time to process.

Most of the checkboxes are self-explanatory so I won't elaborate on them here. However, I would advise selecting them one at a time otherwise you'll end up with an overflow of information.

If you want to clear the text box, click Reset. Copy copies the entire data to the clipboard and Copy Avg copies only the averages. Unset All deselects all of the checkboxes.

Both Auto Profile and RT Verify go through each item in the Browser (video clips, sequences, images, etc), playing each one four times and averaging the data. Sequences take longer because the profiler initially tests with only the V1 track enabled, and then gradually enables V2 and V3 to perform compositing tests (if such tracks exist of course).

It's worth mentioning that since it goes through everything four times, it will take a while to finish processing, depending on the size of the project. You cannot switch to another application either or the profiling will fail. I would advise creating a brand new project with short media clips and sequences. If you truly have serious bottlenecking, this will still be apparent in the shorter sequence.

All the timing information is stored inside the Profile.csv log. Although you can view this in TextEdit, it will be difficult to read because it will not be formatted correctly. A spreadsheet application, such as MS Excel or the free NeoOffice or OpenOffice packages, is recommended.



If you are experiencing poor performance, look at the timings in this file to see where the major bottlenecks are. For instance, a slow FXScript filter could be indicative of a CPU bottleneck or an inefficient script. A slow Flop operation could be indicative of a GPU bottleneck.

Auto Profile and RT Verify appear, on the surface, to perform exactly the same task. However, it is likely that Auto Profiling actually updates your RT information whereas RT Verify merely performs the test without changing anything. This is difficult to verify without official documentation though.



Most of the functions in the Internal Tools menu will be too technical for a lot of users, and so the menu will likely remain something of a novelty. But if you are troubleshooting performance issues or developing applications and plugins for FCP, it is an invaluable source of information on the inner workings of Final Cut Pro.

The only downside is that the menu must be reactivated every time you start Final Cut Pro. If anyone knows of a way to enable it permanently through a preference file, I'd love to hear about it.

Compressor error: "Unable to connect to background process"

The "Unable to connect to background process" and "Unable to submit to queue" messages are common errors that unfortunately have many causes.

Before you do anything else, check that your version of Compressor is compatible with your operating system. Only the versions included with FCP 5.1.x and 6.0.x are compatible with OS X 10.5. If you have an incompatible earlier version, you will need to either upgrade Final Cut Studio or downgrade your operating system.

Try the following steps:

1. In Compressor, go to Compressor > Reset Background Processing.



2. Run Compressor Repair.

3. Trash Compressor preferences (can also be done in Compressor Repair).

4. Try these steps (use at your own risk).


Do the following if nothing else works:

5. Disable your internet / network connections.

6. Reinstall Compressor and Qmaster.

7. Perform a full erase and install of the operating system and reinstall from scratch.

Getting to Know the Terminal Part 2: More File Operations

In the previous article, I showed you how to launch the Terminal and perform basic tasks like navigating through folders and dealing with files. Today I will build on this with more advanced file operations, working with directories, and wildcards.

Working with Directories

To create a new directory, navigate to the relevant parent directory using the cd command and type the following:

mkdir MyNewFolder

Remember: as discussed last time, if you want a file name with spaces, you must do either of the following:

mkdir "My New Folder"

or:

mkdir My\ New\ Folder

What if you need to create several new directories? Simple:

mkdir Dir1 Dir2 Dir3

A little-known feature of the mkdir command is that you can create multiple directories with just one command. Just separate them with spaces.

If you want to create a long chain of directories (e.g. Renders/Video/TIFF) you can use the -p parameter like so:

mkdir -p Renders/Video/TIFF

Press Enter and it's all done, instantly. Imagine how long it would have taken to do that within the Finder GUI! This is why the command-line is still used in the 21st Century even though we have pretty GUIs to look at.

To remove a directory, use the rmdir command:

rmdir Renders/Video/TIFF

The above command assumes that all directories are empty. If they are not, an error will be returned. To remove a directory and all its subfiles and folders, use the following command:

rm -dR Renders

(the d parameter tells it to include directories, R tells it to be recursive)

Wildcards

Last time I mentioned file operations such as cp for copying and mv for moving files. But what if you need to perform an operation on a large number of files at once?

Wildcards can be used to substitute characters. So if you want to copy, say, 100 files that are named Image.001.jpg to Image.100.jpg, you would use wildcards to substitute for the numbers. This will allow you to copy all of the files with one command instead of a hundred like so:

cp Image.*.jpg Renders

Question mark (?) substitutes a single character. So for instance, if you had files called render_v1.iff, render_v2.iff and render_v10.iff and you typed ls render_v?.iff, it would return only renders 1 and 2 because they are a single digit whereas render 10 is two digits.

Asterisk (*) substitutes a potentially infinite length of characters (including no characters at all). Use this if you don't know the precise length of the substitution. In the example above, if you type ls render_v*.iff, it will return all three files because it matches both the 1- and 2-digit numbers. You can also type, for example, *. jpg to return all JPEG images or *.* to return all files in the directory.

So, going back to the 100-frame image sequence previously mentioned, you would type ls Image.*.jpg to return a list of all of the images. That's all well and good, but what if you don't want to return all of the images - what if you only want a specific range?

Braces ([ ]) can be used to be more specific. In the above example, type ls Image.[20-30].jpg to restrict the results to frames 20-30. You can also restrict characters in the same way, such as [b-f] (remember that everything is case-sensitive).

If your numbers or letters don't fall into a sequential range, you can pick a set of non-sequential numbers or letters such as [brz] or [179] to match any of these characters. You can also combine them with ranges like so: ls Image.[1-10,12,14,20-40].jpg. This will return frames 1-10, skip frame 11, return 12, skip 13, return 14 and display frames 20-40.

And if you don't want specific characters to be returned, use the following syntax: [^bgv].

Combining the examples above: if you want to match a file starting with the letters a-c or x-z, with three miscellaneous characters in the middle, ending with a three-digit number at the end, you would type ls [a-cx-z]???[0-9][0-9][0-9].*. As you can see, it gets complicated pretty quickly.

But imagine if you had to search through and move or copy those files manually. It would be time-consuming, tedious and prone to human error. The command line comes into its own when you want to perform operations on a large number of files at once.

Editors can get away with not knowing this (although it can be very useful for them to know) but it is required knowledge for anyone who wants to get into visual effects.

I had originally planned to cover permissions here but I'm going to move it to the next article because I went into a lot of detail and it's too large to add on the end of this one. I'd much rather delay it than miss out information.

So next time I will be covering file permissions, symbolic links and opening, viewing and saving text files.

Getting to know the Terminal Part 1: Basic File Operations

The Terminal is an application that drives fear into the heart of a lot of Mac users - an application they only dream of using in their worst possible nightmares.

It's really not that scary - in fact, it can actually be a very useful timesaving device. You can perform operations on a large number of files at once in a fraction of a second, saving a lot of time.

Although the GUI is prettier and more intuitive, constantly moving, clicking and dragging the mouse around the screen wastes time - not to mention that rendering the GUI takes away precious processing cycles from the operation you are trying to perform. And you don't even need to constantly type things either - you can write a shell script to perform a task and set it to run automatically.

If you are thinking of entering the visual effects industry, most employers will expect basic command-line knowledge and shell scripting abilities. While most VFX houses use some form of Linux, Mac OS X's Terminal is almost identical with the exception of a small number of proprietary commands.

So let's get started.

Setting up

First of all, open up the Terminal by navigating to /Applications/Utilities and double-clicking the Terminal application. I find it useful to ctrl-click the dock icon and select Keep in Dock so it is always there for convenience, but this is up to you.

A window like this will appear:

The title bar of my window says "Terminal - bash - 80x24".

bash refers to the shell I am using. There are various different shells available, each with minor differences. We will just concentrate on bash. Incidentally, if your title bar doesn't say "bash", type bash and press enter.

80x24 refers to the number of characters that can be displayed on screen at once - 80 horizontally and 24 vertically. If you resize the window these numbers will change. It doesn't matter if yours has a different value to mine.

Directory navigation

You will see something like this on screen:
Last login: Mon Jan 5 15:27:17 on ttys000
MacBook:~ Jon$

MacBook is the name of my machine, Jon is my username. The ~ indicates that the current directory is my user directory. To view the full path of the current directory, type:

pwd

This stands for Print Working Directory and on my machine outputs the path /Users/Jon. The working directory is the directory the shell will perform all commands in unless specifically told otherwise.

To change to a different working directory use the cd command like so:
cd Documents

The prompt changes to MacBook:Documents Jon$.

It's worth noting at this point that all commands are case-sensitive, so cd Documents is not the same as cd documents, and cd is not the same as CD.

There are some specific shortcuts you can use with the cd command that are summarized below.



It's worth mentioning here that shell commands are easily confused by spaces in a file path. If you must type a path with spaces, you must either:

a) Surround it in quotes
cd "~/Desktop/My Spaced Filename.doc"

or b) Use escape characters
Escape characters are characters placed before a potentially conflicting character (a space in this case) so that the shell knows to treat it as text and not as a command. The escape character for the Unix shell is \ (backslash).

cd ~/Desktop/My\ Spaced\ Filename.doc

Viewing directory contents

If you want to view the contents of a directory inside the Terminal window, use the ls command. ls is short for "list" (you'll notice that shell commands are generally quite short as they need to be typed often).

cd /
ls

Returns:


Notice that ls returns system files and folders that would normally be hidden by OS X, except for files and folders that have a dot at the front of their name such as .DS_Store. You can show these by typing ls -a (short for "all").

You can also specify a directory as a parameter such as ls /usr/bin and it will list the contents of this directory instead of the current one.

You can set several options when you call ls such as:



You can combine as many of these parameters as you like such as:
ls -l -R

Or more simply:
ls -lR

(note: some parameters automatically override others)

These are just a few parameters of many. For exhaustive details of the parameters available, type:

man ls

This is an important Unix concept that also translates to OS X. Man (manual) pages are documentation files easily accessible from the shell using the man command. They will tell you everything you could possibly want to know about the command such as its syntax, parameters, compatibility and return values (more on that in the upcoming scripting tutorial).

You can use man with any command such as man cd, man ls, man echo. With the man page open, press Space to go to the next page and press q to quit and return to the shell.

This gives you documentation instantly at your fingertips, even if the machine you are using doesn't have an internet connection.

Basic File Operations

Use the cp command to copy files like so:

cp [source file] [destination path]

Here's an example on my system:
cd ~/Desktop
cp UntitledDoc.txt ../Documents

This will set the working directory to my desktop. It will then copy UntitledDoc.txt to my Documents folder (remember that .. refers to the parent directory).

You can also create a duplicate of a file within a directory by changing the destination filename:
cp UntitledDoc.txt UntitledDocDupe.bak

To copy a folder and all its subfiles and subfolders, use cp -R. Type man cp to learn about the other parameters.

To move a file, use the mv command with the exact same syntax as the cp command.

To rename a file, you can either use the rename command (same basic syntax as cp) or use mv like so:
mv myfile.txt mynewfile.txt

To delete a file, use rm:
cd ~/Desktop
rm myfile.txt

Some useful rm parameters are listed below:



That's it for the first tutorial. In Part 2 I will be covering more file operations, working with directories, wildcards and permissions.