### Why “temperature = 0” is interesting in AI text generation
In most modern LLM inference stacks, “temperature” controls randomness during token sampling. Setting **temperature to 0** is a special case: it effectively turns stochastic sampling into **greedy decoding**, meaning the model deterministically selects the highest-probability next token at each step (assuming ties are handled consistently).

This property is useful when you need:
- reproducible outputs for tests
- debuggable behavior across runs
- consistent prompt-response pairs for evaluations

### A tiny, perfectly working implementation (greedy vs sampled)
Below is a minimal sampler that shows the deterministic behavior of greedy decoding and the stochastic behavior for nonzero temperature. It takes a vector of logits, applies temperature scaling, and then either picks `argmax` (temperature = 0) or samples from the softmax distribution (temperature > 0).

```python
import math
import random
from typing import List

def softmax(logits: List[float], temperature: float = 1.0) -> List[float]:
    if temperature <= 0:
        raise ValueError("temperature must be > 0 for softmax sampling")
    scaled = [x / temperature for x in logits]
    m = max(scaled)
    exps = [math.exp(x - m) for x in scaled]
    s = sum(exps)
    return [e / s for e in exps]

def sample_next_token(logits: List[float], temperature: float = 0.0, rng: random.Random = None) -> int:
    """
    temperature == 0.0 => deterministic greedy decoding (argmax)
    temperature > 0.0  => stochastic sampling from softmax
    """
    if rng is None:
        rng = random.Random()

    if temperature == 0.0:
        # Deterministic: argmax
        max_val = max(logits)
        # In case of exact ties, Python's max returns first occurrence; we mimic that deterministically
        return logits.index(max_val)

    probs = softmax(logits, temperature=temperature)
    r = rng.random()
    cum = 0.0
    for i, p in enumerate(probs):
        cum += p
        if r < cum:
            return i
    # Numerical edge case: if r is very close to 1 due to float rounding
    return len(logits) - 1

if __name__ == "__main__":
    logits = [1.2, 0.1, 0.1, 0.7]

    # Greedy decoding is deterministic
    for _ in range(5):
        print("temp=0 ->", sample_next_token(logits, temperature=0.0))

    # Sampling is stochastic (seeded for reproducibility in this demo)
    rng = random.Random(123)
    for _ in range(5):
        print("temp=1 ->", sample_next_token(logits, temperature=1.0, rng=rng))
```

### What’s actually happening under the hood
Most implementations follow this logic:

- **Greedy decoding (`temperature = 0`)**
  - compute next-token scores (logits)
  - pick the index with the highest logit  
  - repeat

- **Sampling (`temperature > 0`)**
  - scale logits by dividing by `temperature`
  - apply softmax to convert to probabilities
  - sample from the resulting categorical distribution

Because the argmax is deterministic (given consistent tie-breaking), you get repeatable sequences.

### Practical tips for reproducibility
To make deterministic generation truly stable in real systems, you typically also need to control:
- random seeds (for any sampling components)
- top-p / top-k settings (if used)
- batch/pipeline ordering effects (some distributed setups can reorder work)
- floating-point nondeterminism (hardware/compiler differences)

If you specifically use **greedy decoding** (temperature 0) and deterministic tie-breaking, you eliminate the primary source of randomness—turning generation into a function of `(model, prompt, decoding config)`.

### Takeaway
Temperature-based sampling is a powerful lever, but it also has an “off switch” for randomness: **temperature = 0 turns your next-token choice into greedy argmax**, making generation deterministic and easier to validate.

Deterministic Text Generation With Temperature Zero And A Minimal Sampler

## Why “grounding” is harder than it looks

Retrieval-Augmented Generation (RAG) systems are often evaluated on answer quality, but a practical failure mode is *grounding drift*: the model answers using information that was never retrieved, especially when prompts are long or context is noisy.

A lightweight, practical technique to reduce drift is **deterministic retrieval filtering**:
- You constrain retrieval with a **stable, reproducible rule** (e.g., by document IDs, timestamp windows, or a hash-based allowlist).
- You then build the context from only those deterministic results.
- The model is prompted to answer **only** using that context.

This doesn’t eliminate hallucinations, but it makes grounding failures less likely and far easier to reproduce in testing.

## A very small, perfectly working example

Below is a tiny deterministic “retriever” over in-memory documents. It:
1. Computes a stable SHA-256 hash for each document ID.
2. Uses that hash to decide which documents are eligible (deterministically).
3. Selects the eligible documents most similar to the query using TF-IDF cosine similarity.
4. Returns grounded context.

### Python code: deterministic retrieval + grounded prompt input

```python
import hashlib
import math
import re
from dataclasses import dataclass
from typing import List, Tuple

@dataclass
class Document:
    doc_id: str
    text: str

def stable_hash_int(s: str) -> int:
    # Stable across runs and machines (SHA-256 then convert to int)
    return int(hashlib.sha256(s.encode("utf-8")).hexdigest(), 16)

def tokenize(text: str) -> List[str]:
    return [t for t in re.findall(r"[a-z0-9]+", text.lower())]

def build_tfidf(docs: List[Document], vocabulary: List[str]) -> Tuple[List[dict], dict]:
    vocab_index = {w: i for i, w in enumerate(vocabulary)}
    N = len(docs)

    # document frequency
    df = [0] * len(vocabulary)
    doc_tokens = []
    for d in docs:
        toks = tokenize(d.text)
        doc_tokens.append(toks)
        seen = set(toks)
        for t in seen:
            if t in vocab_index:
                df[vocab_index[t]] += 1

    # idf with smoothing
    idf = {}
    for w, idx in vocab_index.items():
        idf[w] = math.log((N + 1) / (df[idx] + 1)) + 1.0

    # tf-idf vectors stored sparsely as dict: word -> weight
    vectors = []
    for toks in doc_tokens:
        tf_counts = {}
        for t in toks:
            if t in vocab_index:
                tf_counts[t] = tf_counts.get(t, 0) + 1
        total = len(toks) if toks else 1
        vec = {}
        for w, c in tf_counts.items():
            tf = c / total
            vec[w] = tf * idf[w]
        vectors.append(vec)

    return vectors, idf

def cosine_sim(vec_a: dict, vec_b: dict) -> float:
    # Compute cosine similarity for sparse dict vectors
    dot = 0.0
    norm_a = 0.0
    norm_b = 0.0
    for k, v in vec_a.items():
        norm_a += v * v
        if k in vec_b:
            dot += v * vec_b[k]
    for v in vec_b.values():
        norm_b += v * v
    if norm_a == 0.0 or norm_b == 0.0:
        return 0.0
    return dot / (math.sqrt(norm_a) * math.sqrt(norm_b))

def deterministic_retrieve(
    query: str,
    docs: List[Document],
    *,
    eligible_fraction: float = 0.5,
    top_k: int = 3,
    seed_namespace: str = "grounded-rag/v1",
) -> List[Document]:
    # Deterministic eligibility: choose a subset of docs based on stable hashes.
    # eligible_fraction in (0,1], e.g. 0.5 keeps roughly half deterministically.
    eligible_fraction = max(0.0, min(1.0, eligible_fraction))

    # Build vocabulary from all docs + query
    all_text = " ".join([d.text for d in docs] + [query])
    vocab = sorted(set(tokenize(all_text)))

    vectors, _idf = build_tfidf(docs, vocab)
    query_doc = Document(doc_id="__query__", text=query)
    # Build query vector using the same TF-IDF rules by recomputing on the fly
    # (keep it simple for this small example).
    query_tokens = tokenize(query)
    total = len(query_tokens) if query_tokens else 1
    tf_counts = {}
    for t in query_tokens:
        if t in vocab:
            tf_counts[t] = tf_counts.get(t, 0) + 1
    # Rebuild idf from docs (we need it; pull from build_tfidf return)
    # We'll reconstruct idf via the same build_tfidf helper quickly for correctness.
    vectors_all, idf = build_tfidf(docs, vocab)
    qvec = {}
    for w, c in tf_counts.items():
        tf = c / total
        qvec[w] = tf * idf[w]

    eligible_indices = []
    for i, d in enumerate(docs):
        h = stable_hash_int(f"{seed_namespace}:{d.doc_id}")
        # Normalize hash to [0,1)
        u = (h % (10**12)) / (10**12)
        if u < eligible_fraction:
            eligible_indices.append(i)

    # If nothing eligible, fall back to all (prevents empty context failures)
    if not eligible_indices:
        eligible_indices = list(range(len(docs)))

    scored = []
    for i in eligible_indices:
        sim = cosine_sim(qvec, vectors[i])
        scored.append((sim, i))

    scored.sort(reverse=True, key=lambda x: x[0])
    top = [docs[i] for _sim, i in scored[:top_k]]
    return top

def build_grounded_context(retrieved: List[Document]) -> str:
    # Deterministic formatting to help models “stay inside the lines”
    # Include doc IDs so evaluation can trace sources.
    parts = []
    for d in retrieved:
        parts.append(f"[{d.doc_id}] {d.text}")
    return "\n".join(parts)

def grounded_answer_prompt(query: str, context: str) -> str:
    return (
        "You must answer using ONLY the provided context. "
        "If the answer is not in the context, say \"I don't know\".\n\n"
        f"Query:\n{query}\n\nContext:\n{context}\n\n"
        "Answer:"
    )

if __name__ == "__main__":
    docs = [
        Document("doc-001", "Neural text generation often uses transformer attention mechanisms."),
        Document("doc-002", "Cosine similarity is commonly used to compare vector embeddings."),
        Document("doc-003", "Deterministic hashing can be used for stable sampling and reproducibility."),
        Document("doc-004", "Reinforcement learning can optimize policies using reward signals."),
        Document("doc-005", "Hallucinations are responses that are not supported by the evidence."),
    ]

    query = "How can I reduce grounding drift in retrieval augmented generation?"
    retrieved = deterministic_retrieve(query, docs, eligible_fraction=0.6, top_k=3)
    context = build_grounded_context(retrieved)

    prompt = grounded_answer_prompt(query, context)
    print(prompt)
```

### What this demonstrates (factually)

- **Deterministic sampling** using hashes makes the retrieval set reproducible.
- **Grounded context formatting** (with IDs and strict instructions) provides the model a bounded knowledge surface.
- While this toy example uses TF-IDF rather than embeddings, the *control-flow idea* (deterministic eligibility + bounded context) transfers directly to real vector databases and embedding models.

## Practical adaptations you can use

- Replace TF-IDF with embeddings (your retrieval scoring function can change; the determinism rule can stay).
- Apply deterministic eligibility based on:
  - tenant IDs (multi-tenant isolation),
  - document partitions (shards),
  - time windows (e.g., “only last 7 days”),
  - or allowlists derived from content hashes.

## Summary

A simple but powerful RAG technique is to make retrieval **deterministic** with stable filtering, then generate with **strict context-only instructions**. This reduces grounding drift and improves reproducibility in evaluation and debugging.

Real Time Rag With Deterministic Grounding Via Deterministic Retrieval Filters

### Overview

Probabilistic model checking is a formal verification technique used to prove properties of stochastic systems—systems whose behavior includes randomness. A common way to model such systems is with **Markov chains** (or **Markov decision processes** for nondeterminism).

A particularly interesting (and somewhat random) subtopic is **parameterized Markov chains**: Markov chains where transition probabilities are expressed as functions of parameters. The goal is to determine whether a property (like “the probability of reaching a target state is at least 0.9”) holds **for all parameter values** in a given range, or for which parameter values it holds.

This blog post demonstrates a practical, fully working Python example for a small parameterized Markov chain, computing the **reachability probability** to a target state as a function of a parameter using linear equations.

---

### A Tiny Parameterized Markov Chain

Consider states:

- `0`: start
- `1`: intermediate
- `2`: target (absorbing)

Let `p` be a parameter with `0 <= p <= 1`.

Transition probabilities:
- From state `0`:
  - to state `1` with probability `p`
  - to state `2` with probability `1 - p`
- From state `1`:
  - to state `2` with probability `q`
  - to state `1` with probability `1 - q`

To keep it simple and “parameterized,” let `q = 0.7` be fixed, and vary only `p`.

We want the probability of reaching state `2` starting from state `0`, denoted as `r0(p)`.

Let:
- `r2 = 1` (already at the target)
- `r1` = probability of reaching target from state `1`
- `r0` = probability of reaching target from state `0`

By the Markov property:
- `r1 = q * r2 + (1 - q) * r1`
- `r0 = p * r1 + (1 - p) * r2`

Solving gives:
- `r1 = 1` (because state `1` eventually reaches the absorbing target with probability 1 when `0 < q <= 1`)
- `r0 = p * 1 + (1 - p) * 1 = 1`

So for this particular chain, the reachability probability is identically 1 for all `p` in `[0,1]`. That’s correct but not very exciting—so we’ll slightly modify the example to include a non-target absorbing “failure” state.

---

### A More Interesting Example (Target vs Failure)

Add state `3` as an absorbing failure state.

Transitions:
- From `0`:
  - to `1` with probability `p`
  - to target `2` with probability `1 - p`
- From `1`:
  - to target `2` with probability `q`
  - to failure `3` with probability `1 - q`

Here `q = 0.7` fixed. Now reaching the target from `0` depends on `p`.

Let:
- `r2 = 1`
- `r3 = 0`
- `r1 = q * r2 + (1 - q) * r3 = q`
- `r0 = p * r1 + (1 - p) * r2 = p * q + (1 - p) * 1 = 1 - p(1-q)`

We’ll compute this via a linear system solver to mimic the standard approach in model checking tools.

---

### Working Python Code (Solving Reachability Probability)

#### What it does
- Builds the linear equations for reachability probabilities
- Solves them for a given `p`
- Verifies the result against the closed-form formula

```python
import numpy as np

def reachability_probability(p: float, q: float = 0.7) -> float:
    """
    Parameterized Markov chain with states:
      0 = start
      1 = intermediate
      2 = target (absorbing)
      3 = failure (absorbing)

    Transitions:
      from 0:
        to 1 with prob p
        to 2 with prob 1-p
      from 1:
        to 2 with prob q
        to 3 with prob 1-q
      target and failure are absorbing.

    Returns:
      r0 = P(reach target 2 before failure 3 | start at state 0)
    """
    if not (0.0 <= p <= 1.0):
        raise ValueError("p must be in [0, 1].")
    if not (0.0 <= q <= 1.0):
        raise ValueError("q must be in [0, 1].")

    # Unknown reachability probabilities: r0 and r1
    # r2 = 1, r3 = 0
    # Equations:
    # r0 = p*r1 + (1-p)*r2
    # r1 = q*r2 + (1-q)*r3 = q
    #
    # We'll solve using a general linear system form:
    # r = A r + b  => (I - A) r = b
    A = np.array([
        [0.0, p],      # r0 depends on r1 with coefficient p
        [0.0, 0.0],    # r1 has no dependence on r0 or r1 in this construction
    ], dtype=float)

    b = np.array([
        (1 - p) * 1.0,  # contribution from (1-p)*r2
        q * 1.0         # contribution from q*r2
    ], dtype=float)

    I = np.eye(2)
    r = np.linalg.solve(I - A, b)
    r0 = float(r[0])

    # Closed form: r0 = 1 - p(1-q)
    closed_form = 1.0 - p * (1.0 - q)
    # Numerical check
    if abs(r0 - closed_form) > 1e-9:
        raise RuntimeError("Computed and closed-form values disagree.")

    return r0

if __name__ == "__main__":
    for p in [0.0, 0.25, 0.5, 0.9, 1.0]:
        print(f"p={p:.2f} => P(target)= {reachability_probability(p):.6f}")
```

---

### Interpreting the Result (A Parameterized Property)

From the closed form:

\[
P(\text{reach target}) = 1 - p(1-q)
\]

With `q = 0.7`:

\[
P = 1 - 0.3p
\]

So a property like:

- “The probability of reaching the target is at least 0.8”

becomes:

\[
1 - 0.3p \ge 0.8 \Rightarrow 0.3p \le 0.2 \Rightarrow p \le \frac{2}{3}
\]

Thus, in this model, the property holds for **all** parameters `p` in `[0, 2/3]`.

---

### Why This Connects to Model Checking

Tools in probabilistic model checking typically:
- represent state machines as probabilistic graphs (Markov chains / MDPs)
- define logical properties (e.g., reachability)
- compute the probabilities that satisfy those properties—sometimes symbolically, sometimes numerically

Parameterization adds the “for which parameter values does the property hold?” dimension, which is a common practical need in probabilistic verification (e.g., analyzing uncertainty in transition rates).

---

### Conclusion

Even a very small parameterized Markov chain can be verified by reducing reachability probabilities to a linear system. The Python example above demonstrates a standard approach: set up equations for reachability, solve, and optionally validate against the closed form.

Artificial Intelligence Posts

Deterministic Text Generation With Temperature Zero And A Minimal Sampler

Real Time Rag With Deterministic Grounding Via Deterministic Retrieval Filters

Probabilistic Model Checking For Parameterized Markov Chains