Steps to make Irregular Shaped Border:
1. Create a design with a border.
2. Make two duplicate layers of design.
3. Make one of the duplicate layers transparent (Double click on the selected layer and minimize the opacity level, Set the the opacity level up to 150).
4. Now, work on the topmost layer.
5. Go to Image >> Rotate Layer.
6. Set the Angle value to 45.
7. Move the layer from the Pan panel or change the X Pan and Y Pan co-ordinates to move in x axis and y axis respectively.
8. Once it is done, click OK.
9. Now select the area which is needed for border area, using Lasso tool.
10. Make the layer fully visible by decreasing the Opacity.
11. Invert the selection [ Edit >> Invert Selection or press CTRL + I ] and delete the selected area.
Do some manual work with the help of tools from Quick toolbar.
12. Go to Art to graph and reduce the color in the image to 3 colors.
13. Copy it back to the same layer.
Select all the unwanted area and delete it. Also delete the unwanted area from the main design (background layer).
Repeat the same procedure with the middle layer (another duplicate layer) with different angle value while rotating the layer.
Merge the layer into one and do some manual refinement to get the final result.