I don’t have too much conclusive to say other than that I think what you’re seeing is just a difference in the renderings.
I did process the R3D to ACES and took a sampling of a region of the raspberry. Below is a graph of those values plotted on top of the AP1 primaries and DC2 primaries.
There are values outside of both AP1 and DC2. I’m not sure the inner workings of DC2/RG4 rendering, but I know in the RRT the values outside of AP1 get clipped in to AP1.
To compare what happens through each rendering in a different manner than the vectorscope, I sampled the same region of interest from your supplied output .jpgs. (Side note: my ACES processed image to 709 doesn’t exactly match yours, but that’s probably attributable to a number of different factors, including we used different IDTs. I’m gonna leave that difference out of this for now, because my result also showed the “line” clumping on the vectorscope similar to yours)
Treated as 709 code values, the DC2/RG4 result plots like this:
The same region of interest sampled from ACES_709.jpg, you can see a bit more values slammed up against the 709 boundary. This is probably the same clumping you’re seeing on the vectorscope - indicating values clipped to the 709 boundary.
I believe this is just a difference in the renderings that you are seeing, and the clumping is a side effect of the AP1 rendering primaries.