You want to find conditionally matching brackets of three different types. That is either impossible or very verbose within a single regular expression.
If you wanted to find pairs of matching simple quotation marks ' and ", that is possible in many engines: (["'])[^"']*\1 or ("|').*?$1, or some other combination of matching the start character, optionally advancing and then matching that start character as end character.
Your current start character is a left parenthesis ( which needs to be escaped \( but is not yet in a matching group (\(). Likewise, your current end character is a right parenthesis (\)). You could add square brackets and curly braces to these groups: either (\(\[\{) and (\)\]\}) or ([([{]) and ([)\[}]). This would, however also match unequal pairs like (], {) and [}. If that is good enough because you will never encounter nested brackets, your solution (with non-capturing groups – otherwise $2 must become $3) could be:
$regexp(%title%,(.*)(?:[([{])(.*?[mM]ix|.*?[eE]dit|.*?[vV]ersion|.*?[aA]coustic)(?:[)\[}]),$1'['$2']')
PS: You had vertical bars | inside character classes [aA], but they only separate alternatives as or within groups (a|A).