In that case it might be driver overhead. Only way for you to know for sure is to test different scenarios and HW configurations.
It sounds like you have narrowed at least a part of it to the GPU, I still suspect it has more to do with the drivers in that case though, but if you have access to two systems with different gpu skus of the same manufacturer and same generation(say nvidia 1050 and 1080) that might be a good way to test.
Just remember that rest of the HW has to be exactly the same to get any meaningful data from the test towards either validating or invalidating the theory.
