I feel your pain. I installed that rack a year or so ago, and all I recall was the amount of sweat dripping off me after about 4 hours of shredding my hands trying to fit the bolts up into the frame. I don't recall what panel needed to be removed, I've tried to forget the torment. I, too, had to modify my plate frame and bent the reflector stays in.
Once installed, I haven't had any problems, but I remember sending H&B a nasty note about the install process. Wouldn't wish it upon my worst enemy.
Oskar, yes the rack sides can be removed with an allen wrench and a little effort, but I've never bothered. I really didn't care about that functionality when I bought the rack, but like everything else with the CT/VFRX, there are little to no bike specific aftermarket parts here in the US, including racks. That happens when you only produce 2 model years of a bike before pulling it off the market.