Chapter 3: Getting Exact with Advanced Masking Techniques In This Chapter ✓ Creating and editing layer masks ✓ Using vector masks ✓ Working with channel masks I f you haven’t already checked out Book VI, Chapter 2, which covers Photoshop’s quick-and-easy masking tools, you might want to breeze through that chapter first, especially if the word mask brings to mind an image from Halloween rather than a selection technique. In this chapter, I dive into some manual masking techniques. Layer masks are tremendously useful, and if you’re like me, you’ll find your- self addicted to them. They can be fantastic for blending layers and making multiple images dissolve into one another. Vector masks create shapes defined by vector paths and produce clean, smooth-edged graphic elements. Channel masks are probably the most time-consuming of the masking lot, but they’re powerful and accurate. Like anything in life, the more you practice using them, the faster and better you get. But Photoshop CS5 includes a feature to help in your masking tasks — the Masks panel. This handy panel enables users to add, control, and refine masks of all sorts. After you get through this final chapter of Book VI, you’ll be familiar with every masking technique Photoshop has to offer. By then, you’ll be prepared to use masks to select a very hairy orangutan, dyed green, perched in a tree in a lush rainforest. And how many people can say that? Working with the Masks Panel The Masks panel, shown in Figure 3-1, enables you to add, control, and refine your layer, vector, and filter masks. I explain layer and vector masks in the sections “Working with Layer Masks” and “Creating and Editing 490 Working with the Masks Panel Vector Masks,” later in this chapter. Details on filter masks can be found in Book VII, Chapter 1. For now, I cover just the features in this panel because they may assist you in getting your mask exactly the way you want it: ✓ Thumbnail: Shows what layer or mask is currently selected in your file. ✓ Add Pixel Mask/Select Pixel Mask: Click this icon to add a layer mask. If you added a layer mask by another means, such as via the Layers panel or Layer menu, the icon is automatically selected. Remember that when working with masks, black hides areas of your image, white shows areas of your image, and any gray areas are partially hidden at varying per- centages, depending how dark or light the gray is. ✓ Add Vector Mask/Select Vector Mask: Click this icon to add a vector mask. If you added a vector mask by another means, the icon is automatically selected. ✓ Select Filter Mask: This icon appears only when a Smart Filter has been applied. For more on Smart Filters, see Book VII, Chapter 1. ✓ Density: Think of this option as a kind of opacity control for your mask. To lessen the transparency of the masked (or hidden) area, select the mask and drag the slider to the left. ✓ Feather: The Feather option softens the edges of the mask, creating more of a dissolve between your layers when creating a composited image. To see the effect on the actual mask itself, hold down Alt (Option on the Mac) and click the mask thumbnail in the Layers panel, which temporarily hides the layer and shows only the mask. Hold down Alt (Option on the Mac) and click to display the layer again. ✓ Mask Edge: Click this button to bring up the Refine dialog box. In this dialog box, you can fine-tune the edges of your masks to your liking. For an explanation of the Refine Edge dialog box, see Book III, Chapter 1. ✓ Color Range: Click this button to open the Color Range dialog box. Color range is yet another way to create a selection or mask. For details on this feature, see Book VI, Chapter 2. ✓ Invert: This option reverses the colors of the mask. Therefore, black areas become white, white becomes black, dark gray converts to light gray, and so on. ✓ Load Selection from Mask: Click this option to load your mask as a selection. Note that the black areas of your mask are unselected, the white areas are selected, and the gray areas are partially selected. For more on selections, see Book III. ✓ Apply Mask: Click this icon to have your mask permanently applied to the layer. The mask is then deleted. Be careful when applying this option because you can no longer edit the mask and refine your visible areas. ✓ Disable/Enable Mask: Click this icon to show or hide your mask. ✓ Delete Mask: Click the trash can icon to delete the mask. Working with Layer Masks 491 Filter mask Layer mask Book VI Chapter 3 Advanced Masking Getting Exact with Techniques Add Select Vector Mask Load Selection from Mask Add/Select Pixel Mask Apply Mask Disable/Enable Mask Corbis Digital Stock, Photodisc/ Getty Images Figure 3-1: The Masks panel aids in adding and refining various types of masks. ✓ Mask Options: This first command in the panel pop-up menu enables you to change the color and opacity of your mask overlay. ✓ Add Mask to Selection, Subtract Mask from Selection, Intersect Mask with Selection: These commands, also in the panel pop-up menu, allow you to add to, delete from, or intersect with existing selections made from your masks. ✓ Close/Close Tab Group: The final commands in the panel pop-up menu close the Masks panel and the group that the Mask panel belongs to, respectively. Working with Layer Masks Like any other mask, a layer mask is a grayscale image that you can edit to your heart’s content. Layer masks are excellent for blending layers of images together and creating soft transitions between elements. For versatility, layer masks are unparalleled. They allow you to gradually brush in transparency and opacity on a selective pixel basis. Paint with black to hide 492 Working with Layer Masks portions of the layer; paint with white to display portions; and paint in varying shades of gray to partially show elements. You can even apply gradients, image adjustments, and filters to your layer masks to create interesting special effects. After you get the concept of layer masks, you’ll never use the eraser tools (covered in Book VI, Chapter 2) again. You won’t have to because one of the great things about layer masks is that you can forever edit, or even delete them, with no permanent harm whatsoever to the image. Creating layer masks To create a layer mask, select your desired layer and choose Layer➪Layer Mask➪Reveal All or Hide All. ✓ Reveal All: Creates a mask filled with white, which shows the layer. ✓ Hide All: Creates a mask filled with black, which hides, or masks, the layer and shows nothing but transparency. You can also click the Add Layer Mask icon at the bottom of the Layers panel (which, by default, selects Reveal All). Or you can click the Add a Pixel Mask icon in the Masks panel. You can’t add a layer mask to a background layer. You must convert the background layer to a regular layer if you want to use a layer mask on the background of an image. You can also use an existing selection to create a layer mask. Select your desired layer and make a selection by using one of the selection tools. Choose Layer➪Layer Mask➪Reveal Selection or Hide Selection. You can also click the Add Layer Mask button in the Layers panel to create a mask that reveals the selection. After you create the layer mask, you can grab the painting tool of your choice and apply your grayscale color. Remember: Add white to the mask to display the image. Add black to hide the image. Add gray to make the layer semitransparent. Using the Gradient and Brush tools on a layer mask I must confess: I use two of the layer masking tools more than the others: ✓ The Gradient tool: Setting this tool to a linear gradient of black to white or white to black is truly awesome. Select the layer mask in the Layers panel and drag with the Gradient tool on the layer mask to create the gradient. The darker areas of the gradient gradually hide the image, whereas the lighter areas gradually show the image. Working with Layer Masks 493 ✓ The Brush tool: With a large, feathered tip, using the Airbrush option and the Flow set to around 10%, this tool is amazing. With these settings in place, you can create feathered edges that blend one layer into another without any harsh lines. Again, select the layer mask in the Layers panel and drag with Brush tool on the layer mask. In Figure 3-2, which is an image with two layers (the flag on the bottom and the girl on top), I used a combination of both these tools. I started with the black-to-white linear gradient, which I dragged from the left edge of my image through to the right edge. I then took the Brush tool with a large feath- ered tip (265 pixels), selected the Airbrush option, set the Flow to 10%, set my foreground color to black, and worked my way around the profile of the Book VI girl’s face to get rid of some more of the background behind her. Chapter 3 To edit a layer mask, click the Layer Mask thumbnail in the Layers panel. Advanced Masking Getting Exact with Select your desired painting or editing tool, and paint or edit the mask to Techniques perfection. Just be sure that you’re working on the layer mask, rather than editing the image itself. Otherwise, you apply paint directly to your image. You can tell because you see brackets around the layer mask thumbnail in the Layers panel. You also see the Pixel Mask icon highlighted and the mask thumbnail displayed in the Masks panel. No layer mask With layer mask Figure 3-2: Layer masks enable you to seamlessly blend two layers. 494 Working with Layer Masks Managing layer masks Here are some tips to help you work with your layer masks. You can do the following: ✓ Load a layer mask. Loading a layer mask means getting a selection out- line based on the layer mask. Simply Ctrl-click (Ô-click on the Mac) the Layer Mask thumbnail. You can also click the Load Selection from Mask icon in the Masks panel. ✓ View the mask without viewing the image. Sometimes when you’re editing a layer mask, you may find it helpful to see the mask itself with- out having to view the image, too. For example, in Figure 3-3, I hid the image of the little girl and the flag to see the layer masks. Simply Alt-click (Option-click on the Mac) the Layer Mask thumbnail to view the mask and hide the image on the layer. To redisplay the image, Alt-click (Option-click on the Mac) again or click the eye icon in the far left column. ✓ View the layer mask as a red over- lay. If you prefer to see your layer mask as a red overlay, Alt+Shift- click (Option+Shift-click on the Mac) the Layer Mask thumbnail. You can also click the eyeball icon on the layer mask in the Channels panel. ✓ Click again with the same keys to remove the overlay. You can Figure 3-3: Check out just your layer mask change the opacity and color of the without the image. overlay in the Layer Mask Display Options dialog box, which you access by double-clicking the layer mask channel in the Channels panel. You can also change it by selecting Mask Options from the Masks panel pop-up menu. ✓ Paste a copied selection into a layer mask. Simply Alt-click (Option- click on the Mac) the Layer Mask thumbnail. Choose Edit➪Paste and then choose Select➪Deselect. Click the image thumbnail in the Layers panel to return to the image. The copied selection can consist of anything, but this technique comes in particularly handy when you’re copying one layer mask into another. ✓ Create a mask from the transparent areas of a layer. Select the layer in the Layers panel and choose Layer➪Layer Mask➪From Transparency. ✓ Disable (temporarily hide) or enable a layer mask. Just Shift-click the Layer Mask thumbnail or choose Layer➪Layer Mask➪Disable or Enable. Or click the Disable/Enable Mask icon in the Masks panel. Working with Layer Masks 495 ✓ Unlink a layer from its layer mask. By default, Photoshop links a layer mask to the contents of the layer. This link allows them to move together. To unlink a layer from its layer mask, click the link icon in the Layers panel. Click the icon again to re-establish the link. You can also choose Layer➪Layer Mask➪Unlink or Link. ✓ Delete a layer mask. Just drag the Layer Mask thumbnail to the trash can icon in the Layers panel. Click Delete in the dialog box. Or you can choose Layer➪Layer Mask➪Delete. Finally, click the Delete (trash can) icon in the Masks panel. ✓ Apply a layer mask. When you apply a layer mask, you essentially fuse the mask to the layer, so your mask is permanently applied to the layer. Book VI Photoshop replaces all black areas in the mask with transparent pixels Chapter 3 and all gray areas with partially transparent pixels; all white areas are unaffected. Drag the thumbnail to the trash can icon in the Layers panel. Advanced Masking Getting Exact with Click Apply in the dialog box. You can choose Layer➪Layer Mask➪Apply Techniques or click the Apply Mask icon in the Masks panel. Putting It Together Making a Photo Gradually Fade from Color to Grayscale Layer masks are extremely powerful when it comes to blending multiple images so that one seems to dissolve into the others. In this project, I don’t show you how to use a layer mask to blend different images, but rather the same image — one in color and one in grayscale. Follow these steps: 1. Open a copy of your favorite color image, like the one shown in the figure. The subject matter isn’t critical here, so feel free to whip out that old prom picture. 2. Choose Image➪Duplicate. Accept the default name and click OK in the dialog box that appears. 3. With the duplicate image active, choose Window➪Channels. View each of the channels to find the one that gives you the best grayscale image. 4. Select your desired channel in the Channels panel. If you need a refresher on channels, see Book VI, Chapter 1. The Red channel gives the best contrast for the portrait in my example. Because skin tones tend to have a lot of red in them, the Red channel usually provides the best grayscale image of a person. continued 496 Working with Layer Masks continued 5. Choose Image➪Mode➪Grayscale and click OK in the Discard Other Channels alert box. Photoshop has now stripped the color from the image, shown in the figure. This is only one way to convert a color image to grayscale. There are a lot of others. Check them out in Book II, Chapter 2. 6. Choose Window➪Layers. The Layers panel appears. 7. Holding down the Shift key with the Move tool selected, drag and drop your grayscale image onto your color image. This action automatically creates a new layer from the grayscale image. (See Book V for more information about layers.) By holding down the Shift key, you keep the grayscale image centered over the color image. 8. Close your duplicate image. 9. Press D to access the default colors. This step gives you a black foreground swatch and a white background swatch in your Tools panel. Now, I show you how to use one of my favorite techniques, a black-to-white gradi- ent, on the layer mask. 10. Make sure that the grayscale layer is the active layer in the Layers panel and click the Add Layer Mask icon at the bottom of the panel. It’s the icon that looks like a dark square with a white circle on top. Photoshop adds a second thumbnail on your layer, indicating that a layer mask has been applied. A layer mask acts like a piece of clear acetate over your layer. 11. Select the Gradient tool and then select the default gradient of Foreground to Background. To get the default gradient, click the Gradient Picker on the Options bar and select the first gradient. It should be a gradient of white to black because it’s based on the current fore- ground and background colors, which reverse when you select the Layer Mask thumbnail. For a gradient refresher, see Book IV, Chapter 2. 12. Drag the gradient from the top of your image to the bottom. Or from left to right. Or at a diagonal. It’s your call. Creating and Editing Vector Masks 497 13. Experiment with long drags and short drags. The angle and length of your mouse movement determine how the layer mask reveals the underlying image. In my example, I dragged from the bottom of the image to the top and stopped about two-thirds of the way up. Book VI Chapter 3 Advanced Masking Getting Exact with Techniques Where black appears on the layer mask, the grayscale image is hidden. Where white appears on the mask, the grayscale image shows through and everything in between allows the grayscale image to partially show. Although I used a linear gradient in my example, you can experiment with the other types, as well. Radial gradients can provide some interesting effects. 14. When you complete your mask, save and close the file. Creating and Editing Vector Masks Whereas layer masks let you create soft-edged masks, vector masks create hard-edged masks defined by shapes created by a vector path on a layer. Vector-based shapes produce clean, smooth, and well-defined edges that are 498 Creating and Editing Vector Masks never jagged. And you can size and transform vector shapes without ever degrading the appearance of the element. (For an introduction to vector images, see Book IV, Chapter 1.) You create a vector mask when you create a shape with any one of the shape tools. You can also create a vector mask when you convert type to a shape. (Choose Layer➪Type➪Convert to Shape.) See Book IV, Chapter 3. Adding a vector mask to a layer To add a vector mask to layer, follow these steps: 1. Select the layer in the Layers panel and choose Layer➪Vector Mask➪Reveal All or Hide All. I describe Reveal All and Hide All in the “Creating layer masks” section, earlier in this chapter. Remember that you can’t add a vector mask to a background layer. You can also click the Add a Vector Mask icon in the Masks panel. 2. Select the Paths icon in the Options bar. On the vector mask, create a path with the Pen tool or grab any shape tool and create a shape. See Book III, Chapter 2 for more on paths and Book IV, Chapter 1 for more on shapes. I selected the Custom Shape tool and dragged a sunburst shape on my vector mask. Notice how everything inside the path (represented by the white area on the vector mask thumbnail) is visible and everything out- side the path (represented by the gray areas on the vector mask thumb- nail) is hidden, or masked, as shown in Figure 3-4. Like layer masks, you can add vector masks only to layers, not back- grounds. If necessary, simply convert your background to a layer by double-clicking the background in the Layers panel. 3. If your vector mask is satisfactory, save your file and then close it. Without vector mask With vector mask Figure 3-4: Everything outside of a vector mask is hidden, or masked, from display.