It is fundamentally dishonest to have a machine […] using first-person pronouns
I would argue that it is fundamentally dishonest to have a machine emit many categories of sentences, including any sentences using first-person pronouns.
Ted Chiang arguing that No, Artificial Intelligence Is Not Conscious, but…
Generative AI is harmful enough when we understand it as a conventional technology, but if we confuse fluency at generating text with consciousness or moral agency, we’re at risk of assigning responsibility to entirely the wrong parties whenever anyone uses a chatbot.