When you click on the flag icon (aka follow-up) it sets a default flag. Clicking a second time should IMHO remove it. Currently it asks if you want to update it. Removing would work much faster in handling the flags.
So clicking the dropdown arrow and sliding to remove is inappropriate? Using your premise, what do you want to have occur when you select multiple documents, say four, and one is QuickFlag enabled, one is a custom flag with a description entered, and two lack any follow-up designation.
I suspect I understand what you would like to occur, but would rather not guess.
My concerns is that if you are off a document, click the flag in thinking you are adding a new flag, then remove the flag for a custom follow-up and lose all of your prose on what you needed to accomplish. That would be brutal. How do you handle that?
But outside of this case, I suspect your idea could save people some time.