Accept this assignment: GitHub Classroom Link
Due: January 16, 2026 at 11:59 PM EST
Click the link above to create your private repository for this assignment. Complete your work in Google Colab, then push your notebook to the repository before the deadline.
In this assignment, you will build a simplified version of ELIZA, one of the earliest programs to mimic human conversation. Created by Joseph Weizenbaum at MIT in 1964-1966, ELIZA was designed to simulate a session with a non-directive (Rogerian) psychotherapist using simple pattern matching and string manipulation.
But this assignment goes beyond just implementing a chatbot. You'll also analyze what makes ELIZA feel conversational despite its simplicity, compare it to modern chatbots, and reflect on the psychological phenomenon known as the "ELIZA effect"—the tendency for people to attribute human-like understanding to computer programs.
This is your first assignment in the course, and it's designed to introduce you to fundamental concepts in natural language processing: pattern matching, text manipulation, rule-based systems, and the critical distinction between appearing intelligent and actually understanding language.
For this assignment, it is possible to code everything from scratch in the allotted assignment time (1 week). However, I strongly recommend that you use a coding agent to help you implement the core components. Some suggestions are provided below. If you're already familiar with vibe coding, then feel free to use whatever approach you're comfortable with. If you're new to vibe coding, or if you'd like to try something different, what works well for me is:
-
Start with a highly detailed description of exactly what you want your coding agent to make. I would suggest including the assignment as a reference file to provide additional instructions and context. But to increase the chances of working code, hallucinations, and other quality issues, you should explain as clearly as possible precisely what the core algorithms are, how the work should be approached, and so on. In order to create this document, it is critical that you have a detailed and comprehensive understanding of what the solution should look like. You don't necessarily need to know how to code it yourself (LLMs are fantastic at writing code), but you do need to know exactly what each function will do (i.e., what are the inputs, what are the outputs, and how can you convince yourself that it's working-- including tricky edge cases). It helps to include examples to illustrate how each component should work.
-
Next, pass your description to your coding agent, and ask it to come up with a detailed technical design document. You should review this in detail, edit carefully, and iterate (with help from the coding agent, other LLMs, web searches, and your own intuitions) until you are 100% happy with the design. Ideally the design should contain skeleton code and/or code snippets showing exactly how each component of your project will be implemented.
-
Once you have your technical design document, the next step is to construct a detailed implementation plan. Given your target (i.e., your technical design document) ask your coding agent to draft a plan for how to implement it. You can use both the assignment instructions and your technical design document as context. Since implementation plans can get lengthy, you may be pushing up against the context limits of your coding agent of choice. A nice "trick" is to break your task into smaller sub-tasks, and then use agents to do each sub-task. Then no single instance of the coding agent needs to store the full code base and plan in its context. As with the technical design document, you should iterate until you are 100% happy with the plan. Importantly, you should include in your plan a way of verifying that everything is working correctly.
-
Then let your model "loose" on the problem and have it draft a solution. Provide the assignment, technical design document, and implementation plan as context. It's highly likely that the first solution your coding agent comes up with will be wrong in important ways-- it might not run, it might not do what you asked, you might have had a conceptual bug in your understanding that propagated to the model's solution, and so on. That's where you should turn to the verification checks from your implementation plan. In addition to making sure those checks "pass," you should also (when you're creating chatbots or other interactive applications, like in this assignment) use it yourself. Pretend you're using your chatbot (e.g., pretend you're me, and you're trying to stress test the implementation). Does the code behave like you expect? Does the code break down in unexpected ways? Does it break down in expected ways? Are there any pieces that you don't understand fully?
By completing this assignment, you will:
- Implement a rule-based chatbot using pattern matching and text transformations
- Understand the mechanics of ELIZA including pre/post-substitutions, synonym handling, and decomposition/reassembly
- Analyze conversational patterns by testing ELIZA on diverse inputs
- Compare simple and complex approaches to conversational AI
- Explore psychological aspects of human-computer interaction (ELIZA effect, anthropomorphization)
- Reflect critically on what conversation actually requires vs. what creates the illusion of understanding
- Appreciate historical context of early AI and its relevance to modern systems
When Weizenbaum first demonstrated ELIZA, he was shocked by people's reactions. His secretary, who watched him build the program, asked him to leave the room so she could talk to ELIZA privately. Users shared intimate details of their lives. Some refused to believe it was "just" a program following simple rules.
This phenomenon—the tendency to attribute human-like understanding to computer systems based on their outputs—became known as the ELIZA effect. It's particularly relevant today as we interact with increasingly sophisticated chatbots and AI systems.
Unlike modern chatbots that use machine learning, ELIZA operates purely through:
- Pattern Matching: Identifying keywords and patterns in user input
- Text Transformations: Applying substitutions to normalize and manipulate text
- Template Responses: Using predefined templates to generate replies
- Context-Free Operation: No memory, no learning, no real understanding
Despite this simplicity, ELIZA can create surprisingly convincing conversational experiences, especially when mimicking a Rogerian psychotherapist who reflects questions back to the patient.
Understanding ELIZA is crucial for understanding modern AI:
- It demonstrates the difference between appearing intelligent and being intelligent
- It shows how simple rules can create compelling illusions
- It raises questions about what we mean by "understanding" and "conversation"
- It provides historical context for today's language models
You will follow these steps to complete the assignment. Each part corresponds to specific functionality that the chatbot must have.
The chatbot will use a predefined file (instructions.txt) containing patterns, synonyms, and rules for text manipulations. You will need to write code to read and parse this file into an appropriate data structure that your program can use for:
- Greeting the user at the start and end of the conversation.
- Substituting text before and after pattern matching.
- Handling synonym substitutions.
- Matching user inputs to patterns and responding appropriately.
Once the chatbot starts, it should display one of the pre-defined greeting lines from the instructions.txt file. For example:
Welcome. What brings you here today?
Your chatbot should repeatedly ask for user input and respond based on pattern matching until the user types a quit command. The conversation ends when the user enters one of the quit keywords specified in the instructions file (e.g., bye, quit).
Before matching patterns, the chatbot should perform pre-substitutions. These substitutions map certain words to alternatives that are easier for the chatbot to handle. For example:
dont→don'trecollect→remember
You should implement a function that performs these substitutions on the user’s input before further processing.
Your chatbot should also handle synonym substitution. The rules file provides sets of synonyms that the chatbot should treat as equivalent for pattern matching purposes. For example, the words "sad", "unhappy", and "depressed" are considered synonymous.
You will need to implement synonym substitution, where the input is normalized before attempting to match any patterns. For example, if a user says, "I am unhappy," the chatbot should recognize that "unhappy" is synonymous with "sad."
Your chatbot needs to match user input against a set of predefined patterns (as specified in the instructions.txt file). Patterns contain:
- Decomposition rules: These define how to break down a sentence into components.
- Reassembly rules: These dictate how to form a response based on the decomposed parts of the sentence.
For example, if the user says, "I feel sad," a decomposition rule might match I feel *, and a reassembly rule might respond with, "Why do you feel sad?"
You should:
- Match user input to the most relevant pattern.
- Decompose the input according to the matched pattern.
- Reassemble the parts into a response.
Once the chatbot has generated a response, it should perform post-substitutions to finalize the output. This step swaps certain words to ensure grammatical correctness. For example:
i→youyour→my
For instance, if the user says, "I feel sad," and the chatbot's response is based on a pattern that reflects the user’s words back, the chatbot might generate, "Why do you feel sad?" It will then substitute "I" with "you" to respond with, "Why do you feel sad?"
Once the chatbot generates a response, it should display the response and continue the conversation until the user types a quit command. The chatbot should have default responses if no patterns match the user input, such as:
I'm not sure I understand you fully. Can you elaborate?
When the user inputs one of the quit keywords (e.g., bye or quit), the chatbot should print a final message, like:
That will be $200. See you again next week.
Your chatbot should handle a conversation like the following:
Chatbot: Welcome. What brings you here today?
User: I am feeling sad.
Chatbot: Why do you feel sad?
User: I can't stop thinking about my family.
Chatbot: Tell me more about your family.
User: I have to go now.
Chatbot: That will be $200. See you again next week.
- No hard-coded responses: You're provided with an example
instructions.txtfile. However you should read from the file to generate all of the messages, rules, and patterns when you initialize a new instance of your chatbot. Your code should be able to support otherinstructions.txtfiles as long as they follow the same syntax and formatting (even if the numbers of rules are different, the specific words are different, and so on). - Initial and final messages: These must be drawn from the predefined options in
instructions.txt - Pre-substitutions, synonym handling, and post-substitutions: Must be implemented according to the rules in
instructions.txt - Pattern matching: Must use the decomposition and reassembly rules from
instructions.txtto generate responses - Conversation loop: The chatbot must continue the conversation until the user types a quit word
- Code quality: Well-organized, commented code with clear function/class structure
Once your ELIZA implementation is working, conduct the following analyses to understand what makes it tick and where it breaks down.
Test your ELIZA on at least 5 different conversation scenarios:
a) Typical therapy session: Discuss feelings, family, dreams (what ELIZA was designed for)
b) Technical/factual questions: Ask about facts, math problems, or technical information
c) Casual conversation: Try small talk about weather, hobbies, or daily life
d) Adversarial testing: Try to "break" ELIZA with unusual inputs, edge cases, or nonsense
e) Emotional depth: Explore a single topic in depth across multiple turns
For each conversation:
- Include the full transcript in your notebook
- Describe what worked and what didn't
- Identify specific patterns that matched (or failed to match)
- Describe where ELIZA seems "intelligent" vs. where the illusion breaks
Reflect on the psychological aspects of interacting with ELIZA (500-1000 words in total:
a) When does ELIZA feel human?
- What specific responses create the illusion of understanding?
- Which conversational techniques (reflection, questions) are most effective?
- How does the psychotherapist framing affect perception?
b) When does the illusion break?
- What inputs expose ELIZA's limitations?
- What types of context does ELIZA fail to maintain?
- How do repetitive responses affect the experience?
c) Modern parallels
- How do modern chatbots create similar illusions?
- What techniques do they use beyond pattern matching?
- Where do modern systems still exhibit "ELIZA-like" behaviors?
Analyze ELIZA's rule system (500-1000 words in total):
a) Pattern effectiveness
- Which patterns in
instructions.txtare most useful? - Which keywords have priority (higher rank numbers)?
- Why might certain patterns be prioritized over others?
b) Coverage gaps
- What topics or conversation types are not covered?
- What patterns would improve ELIZA's performance?
- How would you extend the rule set?
c) Substitution impact
- How do pre- and post-substitutions affect responses?
- Find examples where substitutions are critical
- What happens if substitutions are applied incorrectly?
Compare your ELIZA implementation with a modern chatbot (500-1000 words in total, excluding conversation excerpts):
a) Choose a comparison system
- ChatGPT (or similar LLM-based chatbot)
- A simple chatbot library (ChatterBot, Rasa, etc.)
- Another student's enhanced ELIZA (if available)
- Another demo model from this course
b) Side-by-side testing
- Use the same prompts with both systems
- Compare response quality, coherence, and relevance
- Document at least 3 example conversations (10+ turns each). Try to really find the limits of each model and also find what each model does especially well.
c) Analysis
- What capabilities does the modern system have that ELIZA lacks?
- Are there any scenarios where ELIZA performs comparably well?
- What's needed to bridge the gap from ELIZA to modern chatbots? E.g., if we could just be a bit more clever, or if we had enough time, could we (in principle) write down enough ELIZA-style rules to achieve performance comparable to today's best chatbots? Or is something "more" needed? What do you think that something is?
Write a thoughtful reflection (500-1000 words in total) addressing:
-
What is conversation?
- What does ELIZA reveal about the nature of conversation?
- Can pattern matching alone constitute "conversation"?
- What's missing from ELIZA that humans have?
-
Understanding vs. simulation
- Does ELIZA "understand" anything? Why or why not?
- What would it take for a system to truly understand language?
- How do we know if modern LLMs "understand" vs. just simulate?
-
The gap to modern AI
- What are the key limitations of rule-based approaches?
- What fundamental advances enabled modern chatbots?
- What problems remain unsolved even with modern systems?
-
Ethical implications
- Should users be informed they're talking to a bot?
- What are the risks of systems that simulate understanding?
- How do Weizenbaum's concerns apply to today's AI?
Choose one or more extensions to enhance your ELIZA:
Implement improved pattern matching using:
- Regular expressions instead of simple wildcards
- Part-of-speech tagging to match grammatical patterns
- Named entity recognition to identify people, places, etc.
Compare the enhanced version with your original implementation.
Add a simple emotion tracking system:
- Detect emotional words (happy, sad, angry, etc.)
- Track emotional state across conversation turns
- Adjust responses based on detected emotion
- Visualize emotional trajectory of the conversation
Build analytics for ELIZA conversations:
- Track which patterns match most frequently
- Measure conversation length and depth
- Identify common conversation topics
- Detect when users disengage or get frustrated
Create visualizations of these metrics across multiple conversations.
Implement a minimal transformer-based chatbot:
- Fine-tune a small model (GPT-2, DistilGPT-2) on therapy conversations
- Compare with ELIZA on the same test conversations
- Analyze where each system excels
- Measure computational costs vs. performance gains
Create a hybrid system that combines:
- ELIZA's pattern matching for structure
- Modern NLU for understanding
- Template responses with learned variations
- Memory of conversation context
Document design decisions and performance improvements.
Submit a single Jupyter notebook that includes:
- Complete, working ELIZA chatbot implementation following the template notebook
- Clean, well-commented code
- Proper handling of all rule types from
instructions.txt - Markdown cells explaining your approach
- High quality and diverse content covered in conversation transcripts
- Analysis (reflection) about each conversation
- Pattern matching explanations
- Examples (and discussions) of successes and failures
- ELIZA effect analysis
- Pattern effectiveness analysis
- Comparison with modern chatbot (with examples)
- Insights into why ELIZA works/fails
- Thoughtful reflection
- Addresses all required questions
- Shows critical thinking about conversation and understanding
- Connects to modern AI systems
- Well-organized notebook structure
- Clear markdown explanations throughout
- Proper formatting and readability
- Runs without errors in Google Colab
- Any extensions you implement (+5-15% extra credit depending on depth)
-
Core ELIZA functionality (25 pts): Correct implementation of all required components
- Pattern matching with decomposition/reassembly (8 pts)
- Pre- and post-substitutions (6 pts)
- Synonym handling (6 pts)
- Conversation loop with greeting/quit (5 pts)
-
Code quality (10 pts): Organization, comments, clarity
-
Completeness (5 pts): All features working correctly
- Diversity of tests (8 pts): Cover required scenarios thoroughly
- Documentation (7 pts): Clear transcripts with analysis
- Insights (5 pts): Thoughtful observations about pattern matching
- ELIZA effect analysis (8 pts): Insightful examination of psychological aspects
- Pattern analysis (7 pts): Understanding of rule system effectiveness
- Modern comparison (10 pts): Fair, thorough comparison with analysis
- Depth of thought (5 pts): Thoughtful engagement with big questions
- Connections (3 pts): Links ELIZA to modern AI systems
- Clarity (2 pts): Well-written and organized
- Organization (2 pts): Logical structure and flow
- Formatting (2 pts): Readable, well-formatted notebook
- Completeness (1 pt): Runs without errors
Total: 100 points (plus potential bonus)
- A (90-100): Excellent implementation with thorough analysis and insightful reflections
- B (80-89): Complete working implementation with good analysis
- C (70-79): Working ELIZA with basic analysis and some gaps
- D (60-69): Partial implementation or weak analysis
- F (<60): Incomplete or non-functional submission
- Read Weizenbaum's paper first: Understanding the original design will help immensely
- Start with simple patterns: Get basic matching working before tackling complex cases
- Test incrementally: Don't write everything at once—test each component separately
- Use print statements: Debug by printing what patterns match and why
- Study
instructions.txt: Understand the format before parsing it - Learn from a reference implementation: Use the reference implementation from our course, including the rule breakdown and rule editor tabs, to see how each component should behave.
- Incorrect pattern priority: Patterns with higher rank numbers should be checked first
- Substitution order: Pre-substitutions before pattern matching, post-substitutions after
- Wildcard matching: The
*should match any sequence of words - Synonym expansion: Remember to treat all synonyms as equivalent during matching
- Reassembly: Captured groups should be properly reinserted in responses
- Case sensitivity: Normalize case for matching but preserve it appropriately in output
- Start tiny: Test with just 2-3 patterns before using the full
instructions.txt - Print everything: Show which pattern matched and which reassembly was selected
- Manual trace: Walk through a simple example by hand to verify logic
- Edge cases: Test empty input, very long input, special characters
- Compare outputs: Check your responses against online ELIZA implementations
- Be specific: Use concrete examples from your conversations
- Think critically: Don't just describe—analyze why things work or fail
- Make connections: Link ELIZA to modern AI systems and concepts
- Be honest: It's okay to note limitations and struggles
- Show creativity: Test ELIZA in interesting or unexpected ways
This assignment is designed to be completed in one week (7 days). While substantial, it is achievable within a week with focused effort. Here's a suggested daily breakdown:
- Days 1-2: Implement core ELIZA functionality (pattern matching, pre/post-substitutions, synonym handling, conversation loop)
- Days 3-4: Conduct conversation testing and analysis (at least 5 diverse conversation scenarios, ELIZA effect analysis, pattern effectiveness analysis)
- Days 5-6: Complete comparison with modern chatbots and write your reflections and discussions
- Day 7: Polish your notebook, test it in a fresh Colab session, and prepare for submission
Key tip: These phases can overlap! While implementing, you can begin testing. While analyzing, you can refine your implementation. The daily breakdown above represents the primary focus for each phase, but iterating and moving between phases is normal and expected. Start early in the week and test incrementally as you build each component.
- Weizenbaum, J. (1966). "ELIZA—A Computer Program For the Study of Natural Language Communication Between Man and Machine"
- PDF Link
- Read this first! It will help you understand the design and implementation
-
Weizenbaum, J. (1976). "Computer Power and Human Reason: From Judgment to Calculation"
- PDF Link
- Weizenbaum's later reflections on ELIZA and AI ethics
- Highly relevant to your reflection section
-
Hofstadter, D. (1995). "Fluid Concepts and Creative Analogies: Computer Models of the Fundamental Mechanisms of Thought"
- PDF Link
- Chapter on the copycat program and understanding vs. simulation
-
Turkle, S. (2011). "Alone Together: Why We Expect More from Technology and Less from Each Other"
- PDF Link
- Modern examination of human-computer emotional connections
-
ELIZA Online Implementations: Helpful for seeing expected behavior
- [context-lab.com/llm-course/demos/eliza]](https://context-lab.com/llm-course/demos/eliza/)
-
Google Colaboratory: Setup-free resource for writing and running code on Google Cloud machines
- Link
- Your assignment must run in Google Colaboratory: to test it, press the "Run all" button, or select "Runtime" > "Run all" from the menu. Make sure that all parts of the notebook run without crashing before you submit!!
-
Dartmouth GenAI: Free access to powerful LLMs that can help with coding (only for Dartmouth students, faculty, and staff)
-
OpenCode: Command line interface for vibe coding
- Link
- Suggested plugin: oh-my-opencode
- Supports many providers and models, including local models
-
LM Studio: Run open models on your own computer
-
Antigravity: Code development environment designed to faciliate interactions with AI agents
-
Python String Methods: Essential for text manipulation
-
Regular Expressions (for bonus extensions):
- ChatGPT: chat.openai.com - For comparison testing
- Claude: claude.ai - Another modern system to compare
- Replika: replika.com - Example of modern "therapeutic" chatbot
- Evergreen: evergreenai.dartmouth.edu - Dartmouth's version of AI-powered therapy!
- Week 1 Lecture Slides: Review pattern matching and ELIZA discussion
- Course Readings: Weizenbaum (1966), along with Fedorenko et al. (2024) and Lupyan et al. (2020) on language and thought
- Searle, J. (1980). "Minds, Brains, and Programs" - The Chinese Room argument
- Turing, A. (1950). "Computing Machinery and Intelligence" - The original Turing Test
- Bender & Koller (2020). "Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data"
This assignment is submitted via GitHub Classroom. Follow these steps:
-
Accept the assignment: Click the accept assignment link
- This creates your own private repository for the assignment (note the repository name!)
- Template repository (your private version will be based on this template): github.com/ContextLab/eliza-llm-course
-
Click the notebook (
.ipynbfile) to view it in GitHub -
Click the "Open in Colab" badge at the top. This will open a new Colaboratory session.
-
Click the Copy to Drive button (at the top) to create an editable copy that you can work on.
-
Make your edits to the notebook file in Google Colaboratory.
-
Save your changes back to GitHub using File > Save a copy in GitHub:
- Select the repository for your assignment from the dropdown menu
- Change the File path to "Assignment1_ELIZA.ipynb" (i.e., remove the "Copy_of_" text at the beginning)
- Add a note to the Commit message field, or leave it as the default
- Make sure the "Include a link to Colab" box is checked
- Press the "OK" button. This will sync your changes back to your GitHub repository.
- Verify submission: Check that your latest commit appears in your GitHub repository before the deadline
Deadline: January 16, 2026 at 11:59 PM EST
- Runtime: The notebook must run from start to finish without errors in a fresh Colab session
- Dependencies: Include all imports and installations in the notebook
- Data: The
instructions.txtfile should be loaded in your notebook (code for doing this is in the template notebook provided with the assignment) - Output: Keep cell outputs visible in your submission
- Deadline: January 16, 2026 at 11:59PM EST
- ELIZA implementation is complete and working
- All conversation examples are included with analysis
- ELIZA effect analysis is thorough and insightful
- Modern chatbot comparison is complete with examples
- Reflection addresses all required questions
- Code is well-commented and organized
- All markdown cells provide clear explanations
- Notebook runs without errors in fresh Colab session
- Cell outputs are visible and meaningful
- Formatting is clean and professional
You are encouraged to:
- Use generative AI tools (ChatGPT, Claude, Copilot) to help write and debug code
- Discuss concepts and approaches with classmates
- Search for online tutorials and ELIZA resources
- Ask questions in class, office hours, or discussion forums
You must:
- Write your own code and analysis (even if AI-assisted)
- Understand every line of code you submit
- Write your own reflection in your own words (do not use generative AI to write your reflections)
- Cite any significant code snippets adapted from external sources
- Submit original conversation examples and analysis (do not use generative AI to come up with your conversation examples)
Violations of academic integrity will result in a failing grade for the assignment.
If you have questions about the assignment:
- Review this README and the Weizenbaum paper
- Check the Week 1 lecture materials
- Post in the course discussion forum on Discord
- Attend office hours
- Email me with specific questions
Have fun exploring one of the most important programs in AI history!
ELIZA may be nearly 60 years old, but it raises questions about intelligence, understanding, and human-computer interaction that remain deeply relevant today. By building and analyzing ELIZA, you're engaging with fundamental questions that will resurface throughout this course as we explore modern language models.