zaphara Working on this I realized our artists have switched pretty much all of our meshes to multiple surfaces with different albedo colors instead of using textures (makes it much easier for me).
So using the forum post above - I was able to get it all working fairly easily - I can scan the surfaces, find which one has a face matching the collision point, and read off the color.
In that case it may be better to copy mesh data into colliders and let the engine do the rest.