Agentic Step 4: Chain-of-Thought Reasoning

Chain-of-thought (CoT) prompting makes the AI "think step by step" before answering, which dramatically improves reasoning.

What You'll Learn

• What chain-of-thought is and why it works
• Adding CoT to your system prompt
• Parsing reasoning from responses

Step 1: The Problem

Without CoT, AI models often jump to conclusions:

Q: "A bat and a ball cost $1.10. The bat costs $1.00 more than the ball.
    How much does the ball cost?"
AI: "$0.10"  ← Wrong! (The correct answer is $0.05)

With CoT:

Q: "A bat and a ball cost $1.10. The bat costs $1.00 more than the ball.
    How much does the ball cost?"
AI: "Let's think step by step:
    1. Let the ball cost x dollars.
    2. The bat costs x + $1.00.
    3. Together: x + (x + $1.00) = $1.10
    4. 2x + $1.00 = $1.10
    5. 2x = $0.10
    6. x = $0.05
    So the ball costs $0.05."  ← Correct!

Step 2: Adding CoT to the System Prompt

system_prompt = """You are a helpful assistant.
When solving problems, think step by step.
Before giving your final answer, show your reasoning.
Wrap your reasoning in 思考...<｜end▁of▁thinking｜> tags.
"""

Step 3: Structured CoT with Function Calling

An alternative approach — make the AI return its reasoning and answer separately:

cot_tools = [
    {
        "type": "function",
        "function": {
            "name": "answer_with_reasoning",
            "description": "Provide a final answer with step-by-step reasoning",
            "parameters": {
                "type": "object",
                "properties": {
                    "reasoning": {
                        "type": "string",
                        "description": "Step-by-step reasoning process"
                    },
                    "answer": {
                        "type": "string",
                        "description": "The final answer"
                    }
                },
                "required": ["reasoning", "answer"]
            }
        }
    }
]

Step 4: CoT Agent

def run_cot_agent(user_message: str) -> str:
    messages = [
        {"role": "system", "content": "Solve problems step by step. Show your reasoning."},
        {"role": "user", "content": user_message},
    ]

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=messages,
        temperature=0,  # 0 = deterministic, best for reasoning
    )

    return response.choices[0].message.content

Step 5: Compare With and Without CoT

def run_without_cot(user_message: str) -> str:
    messages = [
        {"role": "system", "content": "Answer concisely and directly."},
        {"role": "user", "content": user_message},
    ]
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=messages,
        temperature=0,
    )
    return response.choices[0].message.content


if __name__ == "__main__":
    problems = [
        "A bat and a ball cost $1.10. The bat costs $1.00 more than the ball. How much does the ball cost?",
        "If there are 3 apples and you take away 2, how many do you have?",
        "I have two coins that add up to 30 cents, and one of them is not a nickel. What are the coins?",
    ]

    for problem in problems:
        print(f"\n{'='*60}")
        print(f"Problem: {problem}")
        print("-" * 60)
        print(f"Without CoT: {run_without_cot(problem)}")
        print("-" * 60)
        print(f"With CoT: {run_cot_agent(problem)}")

When to Use Chain-of-Thought

Pro Tips

1. Lower temperature (0-0.3) for reasoning tasks — you want consistency
2. Higher temperature (0.7-1.0) for creative tasks
3. Add "think step by step" to any prompt to improve reasoning by 10-30%
4. Combine CoT with tools: "Think step by step about what tools you need to call"