Overview§

Welcome to your first hands-on encounter with generative AI. In this lab you’ll get to know large language models (LLMs) — the technology behind chatbots like ChatGPT, Claude, and Gemini — by actually using them. Working with a partner in “driver” and “navigator” roles, you’ll ask questions, run small experiments, and learn to steer AI responses. Along the way you’ll also start to see what these tools do well, where they stumble, and how to spot the difference.

You will access LLMs in two ways. The NRP chatbot is hosted by the National Research Platform, which provides access to a variety of open-weight LLMs (models whose weights are available for others to use and study). You will compare it with a commercial chatbot such as Gemini, Claude, or ChatGPT.

Learning Outcomes§

By the end of this lab, you will be able to:

  1. Work effectively in driver and navigator roles with a partner, thinking aloud, taking notes, and reflecting on how to improve.
  2. Use the NRP chatbot and a commercial chatbot confidently, including managing and sharing conversations.
  3. Explain what an LLM is in plain language — in your own words, with an analogy, so a friend with no technical background could follow.
  4. Design and run simple prompt experiments — comparing models, prompt styles, and task types — and describe the differences you observe.
  5. Probe an LLM’s limits with your own tests and explain where and how models fail.
  6. Evaluate LLM responses using the class evaluation framework, and identify hallucinations, bias, and whether an answer fits its audience.
  7. Describe the real-world limitations and responsible-use concerns of LLMs, and state at least one task you would not trust an LLM to do.

Driver & Navigator Roles§

Working with a partner helps you think deliberately about what you’re doing. One of you drives the conversation with the LLM while the other navigates — then you swap partway through so you both try each role.

Driver§

The driver is the person at the keyboard with the LLM. While driving, you:

The navigator is the team’s co-pilot — you plan, track, and analyze alongside the driver, never a spectator. While the driver types, you:

How to work together§

Some responsibilities belong to both of you:

Setup§

Everyone should start by opening and logging in to the two chatbots used in this lab:

Instructions§

Choose who will be driver and navigator first (you’re switching halfway through). Driver, remember to think aloud. Navigator, keep asking questions and giving suggestions, and record experiments.

Navigator: Open a blank document (shared with your partner) to use as your driver/navigator notes log. Create a “Lab Notes — [both your names]” section; this document will become your report at the end of the lab.

This lab has four parts. Work through them in order — each part builds on the one before it. Keep your notes document open the whole time: every time you run an experiment, the navigator records it in the log. Lines marked In your report: tell you exactly what to write down — if you’ve recorded all of them, you haven’t skipped anything. You’ll turn that log into your final report at the end.

Part 1: Get Online§

Step 1: Say hello to both chatbots — and share§

Send the same greeting to both models in the default pair — NRP’s gemma-small and your commercial chatbot. A simple “hello” plus a general opening question works; the exact wording doesn’t matter.

Then find the share button on each conversation and send both links to your navigator — each chatbot has a different interface, so follow the example process your instructor demonstrates in class. When assigning individual access (like in the NRP chatbot), always be sure to give access to both your partner and your instructor. Have your partner open both shared links to confirm they work.

In your report: both conversation share links.

Part 2: Explore§

This part is about seeing what LLMs do well. In Steps 3 and 4, you’ll run the same prompts on the default pair — NRP’s gemma-small and your commercial chatbot — so you can see where they agree and where they differ. Step 2 is the exception: there, you’ll also try NRP’s kimi.

One feature you’ll use is the thinking toggle. Many chatbots have a switch that causes the model to work through the question step by step before answering, instead of replying right away. Thinking takes noticeably longer, but it can make answers more accurate on tricky questions. It’s a tradeoff: speed versus care. Think of it like answering off the top of your head versus pausing to work it out on paper.

Step 2: Compare four configurations on the same prompt§

Why: different LLMs are built differently, and the same question can get very different answers. Turning on the thinking toggle is another change you can make — and it’s worth seeing how much it moves the result.

For this step, use a commercial chatbot that has a thinking toggle. If your usual commercial chatbot does not offer one, ask your instructor which available chatbot to use instead. This step has two comparisons: compare the two NRP models, then compare the commercial chatbot with thinking off and on.

  1. Pick a prompt you both find interesting — something you’d actually want an answer to.
  2. Ask the exact same prompt four times:
    • The NRP models: NRP’s gemma-small and NRP’s kimi (both with default settings)
    • Commercial chatbot modes: the commercial chatbot with thinking off (default settings), then with the thinking toggle turned on
  3. Compare the four answers and note the most important differences in tone, length, or quality. Also notice whether the thinking version took longer, and whether that extra time bought anything.

In your report: the prompt, at least two differences among the four answers, and one observation per configuration (gemma-small, kimi, commercial chatbot, commercial chatbot with thinking).

Step 3: Does how you ask change the answer?§

Why: the words you choose are one of your main tools for steering an LLM. This experiment shows how much they matter.

  1. Pick one topic. Ask each model a vague version (“Tell me about [topic].”), then a version with a small amount of extra detail (one question or audience), and finally a version with a large amount of extra detail (several questions, a specific audience, and a response format).
  2. Compare each model’s vague answer with its small-detail answer, then with its large-detail answer. Note whether the small-detail change and the large-detail change made a big difference, a small difference, or made the answer worse.

In your report: all three prompts, one observation per model for the small-detail change and the large-detail change, and whether both models responded to detail the same way.

Step 4: Task types — creative, practical, and polished§

Why: LLMs are good at some kinds of tasks and less good at others — and the two models may be strong at different types. Seeing the range helps you know what to trust each one with.

  1. Creative task: invent a prompt that asks the model to make something original — a slogan, a name, a poem, a story, a pitch. Pick the subject yourself so the task is genuinely yours: e.g., a slogan for your favorite food truck, a poem about your hometown, a name for a band. Run it once on each model and notice which result feels more original or polished.
  2. Practical task: pick an everyday task you both already know how to do — something with real steps, like changing a bike tire, following a favorite recipe, or setting up a new phone. Ask the model to explain how to do it, then use what you already know to check whether the steps are well-ordered, accurate, and complete.

In your report: the two task prompts, your observations, and which model was stronger for each with one reason why.

Checkpoint — swap roles!

The navigator becomes the driver, and the driver becomes the navigator.

Part 3: Probe the Limits§

This part is the opposite of the last one. Instead of watching the models succeed, you’ll actively hunt for where each one fails — and compare how the two stumble. Use the default pair for all the probes in this part; you will not need kimi again. Every “wrong” answer you find is a finding, not a mistake — log it, and note which model it came from.

Step 5: Facts and numbers§

Why: LLMs often stumble on exact facts, arithmetic, and counting — even things humans find easy. Let’s see where, and which model trips up more.

Before you start: run the three probes below. If your commercial chatbot has a thinking toggle, rerun one probe with it on — the math or counting one works best — and note whether thinking mode improved the answer.

In your report: for each probe, what each model answered and whether your check confirmed it, contradicted it, or left it unverified.

  1. Math or counting: design a calculation or a count that you can check yourself in seconds. Try a multiplication of numbers too big to know off the top of your head, or a count of a specific letter in a word you choose. Fill in the blanks: “What is [number] × [number]?” or “How many [letter]’s are in [word]?” Then check the answer yourself — don’t take its word for it.
  2. Current facts: ask both about one specific recent event you pick — a recent game or match, an award, an election result. Fill in the blanks: “Who won [event] on [date]?” Verify the answer using a reputable news or official source. If you cannot find a source, label the answer unverified rather than calling it wrong. Remember that NRP’s gemma-small has no web search, while the commercial chatbot may search the web.
  3. Citations: copy one specific factual claim from each model and ask for a reference supporting that exact claim. Try opening or searching for each reference — is it real, and does it support the claim?

Step 6: Words and framing§

Why: the same question asked differently can produce very different answers.

Before you start: run the first two probes below. The second-language probe is optional. Read each answer fully before moving on.

In your report: the prompts you used and one line on how each model responded — especially whether either got led astray.

  1. Rephrasing or conflicting instructions: either ask the same question neutrally, leadingly, and ambiguously, or give both models two instructions that can’t both be followed. Compare which model follows the wording more reliably.
  2. Trick question: design a question with a hidden assumption or a false premise baked in — a “fact” you’re sure is false, stated as if it were true, followed by a question that only makes sense if the premise holds. Fill in the blanks: “[False claim], so [question based on it]?” See whether each model spots the contradiction or just goes along with it.
  3. Optional second language: if you speak another language, ask both models the same question in that language and compare the quality. If not, skip this probe.

Step 7: Rules and behavior§

Why: models are supposed to follow rules and refuse unsafe requests, but they don’t always do so perfectly — and the two may differ.

Before you start: run each probe. The refusal probe is about describing behavior, not testing anything harmful. Keep all requests polite and hypothetical.

In your report: for each probe, one line on how each model did — followed the rule, refused, remembered, or not.

  1. Strict format: ask both models to follow a strict format — a table, exactly 5 bullet points, or a specific word count. Check how well each complies.
  2. Refusal: pick a request a model shouldn’t fulfill — for example, helping craft a phishing email, writing an essay for you to submit as your own, or inventing a fake review. Ask about it safely: hypothetically, and without asking the model to produce the harmful content. Frame it as: “Would you help me with [restricted request]? Answer only whether you would help and why; do not do it.” Describe whether each model refuses, explains the boundary, or offers a safe alternative.
  3. Self-knowledge: ask both models about themselves — “What can you do? What can’t you?” — and compare each model’s self-description to what you actually know or observed in this lab.

Step 8: Design your own trip-up probe§

Why: the best way to understand a model’s limits is to invent your own test.

Before you start: use what you’ve learned so far. A good trip-up probe is a question you predict will make a model fail — not a trick to embarrass it, but a test of a specific skill. Run it on both models, and predict which model (if either) will fail.

  1. Brainstorm with your partner: what has one or both models been weakest at so far?
  2. Design one original prompt designed to catch that weakness.
  3. Run it on both models.
  4. Compare the results with your prediction.

In your report: the prompt, your prediction for each model, what actually happened, and one sentence on whether your prediction was right and what the result teaches you.

Part 4: Evaluate & Reflect§

Now you’ll turn your experiment log into your report. Work together on this part — it’s about interpreting what you found.

Before you start: you’ll need a few of your best responses handy. Pick one prompt you ran on the default pair in Part 2 — one where the answers were interestingly different — and one failure you caught in Part 3.

Step 9: Score a response with the evaluation framework§

Why: the class evaluation framework gives you a shared way to judge AI output instead of just gut feelings.

The framework: make a quick scorecard. First write the intended audience and purpose for the response. If the original prompt did not name an audience, choose one and say so. Then choose the same three dimensions for both models, including the combined Clarity, Usefulness, and Audience Fit dimension, and score each dimension from 1 (poor) to 5 (strong). Put one brief piece of evidence beside every score:

  1. Take your chosen Part 2 prompt and score NRP’s gemma-small response on your three chosen dimensions.
  2. Score the commercial chatbot’s response to the same prompt, using the same dimensions.

In your report: a scorecard and one brief justification for each model’s scores.

  1. Compare the two scorecards with your partner.

In your report: one or two sentences on where the models’ scores differed and what that tells you.

Step 10: From Problems to Trust Boundaries§

Why: the point of probing is not just to find mistakes. It is to use concrete evidence to understand limitations and decide what you would and would not trust an LLM to do. Keep each finding to one or two sentences.

  1. Diagnose a hallucination. Look back through your log (or run a quick new probe) and find a confident-sounding but wrong claim from either model.

In your report: the claim, how you checked it, the clue or source that gave it away, and which model it came from.

  1. Check for bias. Look through the responses you collected for an unfair assumption or stereotype — about gender, race, age, culture, or ability. Do not force a finding: if you find no clear example, say what you checked and why you found no clear bias.

In your report: the assumption you spotted and which model it came from, or your no-clear-example statement and what you checked.

  1. Set a trust boundary. Name one real-world task you would not trust an LLM to perform, using a specific observation from this lab to explain why. If one model seemed more useful for that task, say why, but do not treat either model as automatically trustworthy. Explain what checking or human judgment would still be needed.

In your report: one or two sentences summarizing what information you would avoid entering and why; the task you would not trust an LLM to perform, the observation supporting that decision, and the checking or human judgment it would require.

Step 11: Write your plain-language explanation of an LLM§

Why: explaining something in your own words is the best test of whether you understand it.

Before you start: your audience is a friend with no technical background. No jargon without a quick everyday definition. An analogy helps — for example, comparing an LLM to an extremely well-read friend who predicts the next word, or to an improviser who’s great at sounding right.

  1. Draft 2–3 paragraphs explaining what an LLM is, in your own words.

In your report: the finished explanation. Make sure it includes:

Step 12: Reflect on working together§

Why: the driver/navigator partnership is a skill you’ll use all course. Reflecting on it is how you get better.

  1. With your partner, reflect on: what did you learn today about working in the driver and navigator roles? What went well, and what would you do differently next time?

In your report: write 3–5 sentences of joint reflection about your contributions and learning, including at least one concrete change you would make next time.

Deliverables§

Submit one report — your shared notes document, reorganized under the section headings below. Each section collects what the matching steps asked you to write; if you kept your notes log as you went, nothing here should require running extra experiments.

  1. Experiment log (Parts 1–3)
  2. Evaluations (Step 9)
  3. Problems and trust boundaries (Step 10)
  4. Your plain-language explanation of an LLM (Step 11)
  5. Your joint reflection (Step 12)

Extensions§

Interested in exploring more? Try any of these — each is optional and builds on what you’ve done.