You're fine! I've drank spoiled milk LOADS of time and have never had a problem. Like someone said, it's the same idea as yogurt or cheese. Think of coffee shops that burn the milk to make specialty drinks. You'll be FINE, worry not.