Token

# Ground Token in your agent

## Objective
Use “Token” precisely in code, documentation, and conversations: A unit representing content processed by a model; in text, it may correspond to a word or a fragment.

## Prerequisites
- Inspect how the term is used today in the repository, documentation, and prompts.
- Identify the neighboring notions already named: Context window, Prompt.
- Preserve existing correct usages.

## Working definition
A unit representing content processed by a model; in text, it may correspond to a word or a fragment.

## Decisive difference
A token is not always a word. Segmentation varies by model and language; use the appropriate counter rather than a page count.

## Example that counts as Token
You measure an excerpt with the model’s counter before deciding which passages to supply.

## Counter-example
Ten pages do not represent a fixed number of tokens regardless of content.

## Acceptance criteria
- Every use of the term matches the working definition.
- Edge cases are settled with the decisive difference, not by intuition.
- Neighboring notions (Context window, Prompt) are named with their own term.

## Guardrails
- Show the proposed changes before any external action.
- Do not publish, send, delete, pay for, or change remote state without explicit authorization.
- Preserve unrelated changes and stop if the scope becomes ambiguous.

## Output format
- Outcome or verdict.
- Files or actions involved.
- Checks run and observable evidence.
- Remaining blockers or limitations.

• Requires · The agentic vocabulary already used in the repository and documentation

# Map the neighboring notions

## Objective
Extend the same rigor to the related notions (Context window, Prompt) so the project vocabulary stays coherent.

## Checks
- List every neighboring notion used without a definition.
- Apply its documented decisive difference.
- Record the still-ambiguous cases for the next entry.

## Guardrails
- Show the proposed changes before any external action.
- Do not publish, send, delete, pay for, or change remote state without explicit authorization.
- Preserve unrelated changes and stop if the scope becomes ambiguous.

## Output format
- Outcome or verdict.
- Files or actions involved.
- Checks run and observable evidence.
- Remaining blockers or limitations.

sha256:a3a5af5540b8d2f6d6416c3e0c744b266d2a2764427341ce7af43b893fc196bf

# Fix a mix-up around Token

## Observed symptom
[DESCRIBE THE SYMPTOM HERE]

## Observable checks
- Compare the disputed usage with the working definition: A unit representing content processed by a model; in text, it may correspond to a word or a fragment.
- Apply the decisive difference: A token is not always a word. Segmentation varies by model and language; use the appropriate counter rather than a page count.
- Confront the case with the documented counter-example: Ten pages do not represent a fixed number of tokens regardless of content.

## Possible causes
- The term is used for a neighboring notion (Context window, Prompt).
- The observed form (interface, script, service) is confused with the actual behavior.
- An alias (text token) carries a different meaning in the project.

## Bounded fixes
- Rename or reclassify only the usages that contradict the working definition.
- Document each choice by citing the decisive difference.

## Final verification
- Reread every corrected occurrence against the documented example and counter-example.

## Guardrails
- Show the proposed changes before any external action.
- Do not publish, send, delete, pay for, or change remote state without explicit authorization.
- Preserve unrelated changes and stop if the scope becomes ambiguous.

## Output format
- Outcome or verdict.
- Files or actions involved.
- Checks run and observable evidence.
- Remaining blockers or limitations.

• Requires · The agentic vocabulary already used in the repository and documentation

# Anchor the vocabulary in the project

## Objective
Turn the clarification into a reusable rule so the mix-up does not return.

## Checks
- Add the working definition to the project glossary.
- Link every neighboring term to its own entry.
- Check new documents against the decisive difference.

## Guardrails
- Show the proposed changes before any external action.
- Do not publish, send, delete, pay for, or change remote state without explicit authorization.
- Preserve unrelated changes and stop if the scope becomes ambiguous.

## Output format
- Outcome or verdict.
- Files or actions involved.
- Checks run and observable evidence.
- Remaining blockers or limitations.

sha256:58929e3d910afc95cf55f7e62d4eefd8df652de283e23c9f2c7c5f772b88c859