You can do something similar to an if/then/else in a regular expression, the syntax is (?if then|else). The else part is optional and could be removed with the vertical bar. The then and else part are subexpression of any kind. One of those will be evaluated depending on the if condition results. The if part is a condition and depnding on the implementation, it could be:
/«
(?{local $nest=0})
(?>
(?:
[^«»]+
| « (?{$nest++})
| » (?(?{$nest != 0}) (?{$nest--}) | (?!) ))
)*
)
(?(?{$nest!=0}) (?!))
»/x
For your conveninence, I have used the x mode for a better presentation. This is also the occasion to introduce the special syntax (?!) which litteraly says "fail now".
Finaly, using perl, there is another way to match nested contruct, which is called dynamic regex.