xyz hey! thank you for the formula (and help)!
the results now look hundreds times better! Here is the code I am using in C#:
public void SmoothCurve(Curve curve)
{
// Adjust the tangents to smooth the curve
for (int i = 0; i < curve.PointCount; i++)
{
// Set the tangent mode to free for all points
curve.SetPointLeftMode(i, 0);
curve.SetPointRightMode(i, 0);
if (i == 0)
{
Vector2 curPoint = curve.GetPointPosition(i);
Vector2 nextPoint = curve.GetPointPosition(i + 1);
/*
float angle = Mathf.Atan2(nextPoint.Y - curPoint.Y, nextPoint.X - curPoint.X);
float tangent = Mathf.Tan(angle);*/
var tangentAtP = ((nextPoint - curPoint).Normalized()) * 0.5f;
var angleAtP = Mathf.Atan2(tangentAtP.Y, tangentAtP.X);
float tangent = angleAtP * 2;
curve.SetPointRightTangent(i, tangent);
}
else if (i == curve.PointCount - 1)
{
Vector2 prevPoint = curve.GetPointPosition(i - 1);
Vector2 curPoint = curve.GetPointPosition(i);
float angle = Mathf.Atan2(curPoint.Y- prevPoint.Y, curPoint.X - prevPoint.X);
float tangent = Mathf.Tan(angle);
curve.SetPointLeftTangent(i, tangent);
}
else
{
// Compute the tangents for the current point
Vector2 prevPoint = curve.GetPointPosition(i - 1);
Vector2 curPoint = curve.GetPointPosition(i);
Vector2 nextPoint = curve.GetPointPosition(i + 1);
var tangentAtP = ((curPoint - prevPoint).Normalized() + (nextPoint - curPoint).Normalized()) * 0.5f;
var angleAtP = Mathf.Atan2(tangentAtP.Y, tangentAtP.X);
float inTangent = angleAtP * 2;
float outTangent = angleAtP * 2;
/*
// also works
float angle = Mathf.Atan2(curPoint.Y- prevPoint.Y + nextPoint.Y - curPoint.Y, curPoint.X - prevPoint.X + nextPoint.X - curPoint.X);
float tangent = Mathf.Tan(angle);
float inTangent = tangent;
float outTangent = tangent;*/
// Set the tangents for the current point
curve.SetPointLeftTangent(i, inTangent);
curve.SetPointRightTangent(i, outTangent);
}
}
}
Before:

After:

If there is one thing to complain about, is, I believe that starting and ending points are not having the correct tangents. If you check the curve:

See that the angle for the first point a bit under the curve. Per my understanding a correct angle would look like this:

either touching or slightly above the curve.. Is there something I am missing still or is the result works as expected?
I tried alternative solution that I came up with
float angle = Mathf.Atan2(nextPoint.Y - curPoint.Y, nextPoint.X - curPoint.X);
float tangent = Mathf.Tan(angle);
but it does not work as intended either..