The two dimensional representation in the C3 character table is actually reducible, but you must use complex numbers. Because complex numbers are generally undesirable in chemistry applications, the two dimensional representation is usually treated as basically irreducible and often expressed by combining the complex numbers with their complex conjugates to form a "real valued" representation.
The C3 character table is rigorously expressed as:
[tex]
\begin{array}{c|ccc}
C_4&\hat{E}&\hat{C_3}&\hat{C_3}^2 \\ \hline
A&1&1&1\\
E&\lbrace\begin{array}{l}1\\1\end{array}&\begin{array}{l}\epsilon\\ \epsilon^*\end{array}&\begin{array}{l}\epsilon^*\\\epsilon\end{array}\rbrace \\ \end{array}
[/tex]
Where ε = exp(2πi/3).
Because these representations are actually reducible, you run into some issues when trying to use the various formulas and rules for manipulating character tables, including the reduction formula, that are intended for use with irreducible representations. You can go through and derive the results long hand, but if you replace h by 2h in the reduction formula it will give you the right answer when working with these pseudo-reducible 2-dimensional representations.