Was there something inherently wrong with my suggested edit?
I agree with David's take on this. En dash to em dash conversion is technically correct but rather pedantic and usually not done or recommended online. Use of a single em dash in-line to separate a phrase is also rather controversial and the edit doesn't improve the use of parentheses, in fact making it worse by starting the next phrase with a conjunction.
Formatting changes like this are controversial as we typically leave it up to the author to pick a style as long as it's consistent and doesn't make for awkward reading. Further more, as Draken commented, it's usually the result of trying to hit the minimum edit count.
If it was just the em dash or just the superfluous edits most people would probably choose to Improve, but add them together and the right call is probable to reject. The point after all is to give you feedback on your edits.
All that being said, I do understand where your edits are coming from and believe they were made in good faith. Please don't be discouraged by this. It's good that you're asking feedback.
The reason for rejection does not really resonate with me. [...] Is there no better option to chose for rejections in a case like mine? I certainly did fix the issues despite the reason chosen stating otherwise.
As David stated, users aren't prompted to select a reason when opting for Reject and Edit. The general idea is that the post did need edit work but the reviewed edit didn't change enough or changed too much.
As my main edits were adopted I feel this could have been resolved with an "Accept and Edit".
The problem is that part of your edit was considered harmful or too minor. At that point standard practice is to reject to send a clear signal to the user that the edit wasn't correct. Part of the reason behind the Suggested Edit review system is that users get feedback on their initial changes. It's only once they have enough site experience they are left to their own devices.
Note that the fact that this prevents well-intended minor edits from people with less than 2k site rep is an unfortunate side-effect. But it prevents more problems than it creates.