I think you can call someone saying “it’s unfair to judge people by race” a racist when they’re using that line to applaud the removal of protections against institutional racism. We can argue the merits of AA as a form of protection, but it was protection nonetheless. To say that it was unfair is to entirely ignore the unfairness which necessitated its existence.
…and nothing will be done about it.