This tutorial will explain how to randomize a selection to obtain irregular edges.
A free Photoshop Action file is included for download.

1. Make a Selection that will be the base for your "random" shape

2. Make a new layer

3. Set the foreground and background color respectively to black and white (you can do this by pressing 'D')

4. With the Selection Tool ('M') click on the image to clear the selection

6. Fill the layer with white (Ctrl+Canc)

7. Reload the previous selection (Select > Reselect or Ctrl+Shift+'D') and fill it with black (by pressing 'X' and Ctrl+Canc)

5. Select all the layer (Ctrl+'A') (see image)

6. Go to Filter > Brush Strokes > Spatter and apply the filter with the following settings:

- Spray Radius: 22
- Smoothness: 15

You can modify these settings to obtain more or less irregular edges.

7. Now you need to smooth these edges. Go to Filter > Sketch > Stamp and use the following settings:

- Light/Dark Balance: 10
- Smoothness: 6

As said before, you can modify these settigs to obtain different results.

4. Now select the black region with the Magic Wand Tool

5. Delete the layer created at step 2.

6. Now you have obtained a randomized selection, ready for use.

NOTE: This process may seem quite time-expensive, however it can be automated into a Photoshop Action.

CLICK HERE to download the "Randomize Selection" action.