LLMs are really good at repeating text that they've just seen. Very occasionally they'll mix up a word or two, but it's not at all challenging for them to regurgitate text from a previous section of input.
I have yet to see a documented example of a system prompt leak that was NOT the real system prompt. Have you seen one?
Loosely, LLMs give plausible responses. And LLMs are really good at writing confident-sounding responses.
LLM output is as if someone is replying with the sole purpose of appearing helpful and knowledgeable.
I wouldn't trust opinions on LLMs from people who are entirely positive or entirely negative: the technology is just too mixed for that. I'd say it's useful for someone to have had a bad experience with LLMs (e.g. LLMs being confidently wrong), as well as making use of LLMs for things they're powerful at. (e.g. "small" programming tasks).
I wonder if there's formatting that's been stripped; because when I tried to read it, it looked like I was hitting headings and had to guess at possible line breaks.
I have connected yt music app then asked about what playlists do i have then it dumped that and continued with an explanation about it couldn't list the playlist but have a idea about what type of musics i listen.
Since the content was irrelevant, i called it as "randomly".
I had something similar w gemini in gmail. I asked it a question and it just dumped out the instructions. Oddly, it didn't give me an answer - just the dumped instructions
Hey this context is more importante than the prompt itself, make it more clear in the post! As this hints to a way to reproduce the output and likely estimate if it's an hallucination or not
reply