It is fundamentally dishonest to have a machine […] using first-person pronouns

I would argue that it is fundamentally dishonest to have a machine emit many categories of sentences, including any sentences using first-person pronouns.

Ted Chiang arguing that No, Artificial Intelligence Is Not Conscious, but…

Generative AI is harmful enough when we understand it as a conventional technology, but if we confuse fluency at generating text with consciousness or moral agency, we’re at risk of assigning responsibility to entirely the wrong parties whenever anyone uses a chatbot.