If you wrote the answer, try to integrate your reply to the answer.
See Why do I need 50 reputation to comment? What can I do instead?:
- Comments are second class citizens on the Stack Exchange network, not
designed to hold information for all eternity. They may get cleaned up
at any time. Generally, truly important information should be
incorporated into an answer anyway (either by posting a new answer, if
the information answers the question at least partially, or by editing
an existing answer, if the information is a minor complement or
clarification of that answer).
It's not a perfect system because users can in general only clean up (delete) their own comments, and what's left is orphaned replies to those comments.
It's also good to know that (How does comment voting and flagging work?)
No reputation of any kind is earned or lost from comment votes.
So deleting one's "popular comment" should not be an issue if the information can be moved elsewhere.