Hi! I am writing feedback about your blog post, "Lyric Prank on Gemeni". Your prompting strategy was creative and I like that you chose an LLM that was not ChatGPT. The prompts were short and sequential which was a great way to try to get the LLM to recognize song lyrics out of a conversational context. This seemed like a really effective strategy!
I also thought it was a great idea to test how an LLM responds as a human would in a social context. Since this is a niche trend, the LLM was probably also less aware of it, making it a better test. However, I can think of a few refinements or ideas. It could have been good to regenerate the LLM's answer or vary prompt phrasing rather than to accept the first response each time. Trying mid song or alternative openings could have also tested the sensitivity of the LLM to wording.
In the future, you could do the following prompt variations: "act like you are having a natural conversation with a friend" or "act as if you are a college student of the current generation". I wonder if that context would affect the detection of song lyrics, even with the niche songs you chose.
Your expectation that the model might miss obscure lyrics was reasonable, but the surprise at Gemini’s strong recognition of mainstream songs shows a good instinct. The model’s failure to understand and catch on to the Daniel Candiotta lyrics was expected because its domain knowledge probably excludes such niche content. The model uses statistical likelihood and prior exposure rather than human-like contextual and social knowledge. This would make an LLM perform worse than a human :)
Overall, this exploration/blog was really engaging and was a great idea for prompt-based experimentation. I liked the focus on LLM behavior for cultural and trend familiarity. This is a really fun social experiment in general!
Hi! I am writing feedback about your blog post, "Lyric Prank on Gemeni". Your prompting strategy was creative and I like that you chose an LLM that was not ChatGPT. The prompts were short and sequential which was a great way to try to get the LLM to recognize song lyrics out of a conversational context. This seemed like a really effective strategy!
I also thought it was a great idea to test how an LLM responds as a human would in a social context. Since this is a niche trend, the LLM was probably also less aware of it, making it a better test. However, I can think of a few refinements or ideas. It could have been good to regenerate the LLM's answer or vary prompt phrasing rather than to accept the first response each time. Trying mid song or alternative openings could have also tested the sensitivity of the LLM to wording.
In the future, you could do the following prompt variations: "act like you are having a natural conversation with a friend" or "act as if you are a college student of the current generation". I wonder if that context would affect the detection of song lyrics, even with the niche songs you chose.
Your expectation that the model might miss obscure lyrics was reasonable, but the surprise at Gemini’s strong recognition of mainstream songs shows a good instinct. The model’s failure to understand and catch on to the Daniel Candiotta lyrics was expected because its domain knowledge probably excludes such niche content. The model uses statistical likelihood and prior exposure rather than human-like contextual and social knowledge. This would make an LLM perform worse than a human :)
Overall, this exploration/blog was really engaging and was a great idea for prompt-based experimentation. I liked the focus on LLM behavior for cultural and trend familiarity. This is a really fun social experiment in general!