In the end, I tried it and eventually got 2A1' + E' + A2" as my irreducible representation but I don't think it's right. I also looked at a previously (really old) post here and that person got this same irreducible representation for ML5 with D3h symmetry.
I don't think 2A1' + E' + A2" is right, as it would mean you have 5 bonds (one from each A and two from E). But for 3 ligands, we expect only 3 bonds.
In case you don't have any point group table at hand, you can have a look at
http://www.webqc.org/symmetrypointgroup-d3h.html. There, you see that 3d
z2 belongs to group A
1'. Orbitals 3d
x2-y2 and 3d
xy belong to E'. And so on. It is all tabulated.
Now, let's imagine hydrogen as a ligand. The simplest way to construct the molecular diagram is first to combine the three 1s orbitals from hydrogens together to form a H
3 subunit. Afterwards, you combine this H
3 with the metal to form MH
3. It may look artificial to proceed like that in two steps, but it is easier to do.
So, the three 1s orbitals will combine to give three hybrid H
3 orbitals.
Look at the right side of this diagram for D
3h symmetry.
The lowest orbital symmetry belongs to group A
1'. The two degenerate orbitals above belong to group E'. OK?
Now, we bring H
3 and the transition metal M together. Only orbitals within same group will interact. Orbitals from different groups are orthogonal and do not interaction with one another. So, the lowest orbital from H
3 will interact with 3d
z2 (same group A
1'). The two degenerate orbitals from H
3 will interact with 3d
x2-y2 and 3d
xy (same group E'). Orbitals 3d
xz and 3d
yz remain unchanged, as no H
3 orbital belong to group E''.
Once this is done, you add the electrons and find the representation A
1'+E'.
Also, if my (big) negative-charged ligands were really good pi donors, how can I find out which d-orbitals will be doing the pi-bondings?
It depends on the pi orbital symmetry. For example, let's consider on each ligand a 2p
z orbital perpendicular to the plan of the molecule. When you take the three ligands together to form L
3, the lowest resulting orbital is in the group A
2" and the two other degenerate orbitals are in E". In ML
3 diagram, the orbital in A
2" will not interact, but the ones in E" will interact with 3d
xz and 3d
yz.
And one more thing about the geometry, if I added 2 more ligands above and below this triangular complex, then I know I'd get a triangle-bipyramid, I don't understand how this can change the d-orbital diagram nor the SALC (symmetry adapted linear combination).
With two more ligands above and below, the orbital 3d
z2 which is pointing directly at them will interact with their orbitals. It will change its energy. And as well, you bring two new orbitals from the new ligands into the diagram.