If the colors are symmetrical, then it's just "my pieces" and "opponent's pieces". You could encode who has the move by how the board is ordered, and predict it from the king and queen position, then just encode the differences.