Line-by-line search

You have GitHub’s Terms of Service and a plain-language question about it. You need the lines that answer the question and a way to detect when the document has no answer. You end up with find(), which returns the exists probability and one relevance score per line.

A query scans a document and reveals an answer attached to the matching
line

The search has three parts:

Tag each line with an ID so TypeSafe can point to it.
Use a Choice question to rank those line IDs by how well they answer the query.
In the same request, use a Noul question to check whether the document contains an answer at all.

Setup

Get a TypeSafe API key

Create a key in the TypeSafe console and export it:

export TYPESAFE_API_KEY="your-key-here"

Install the dependencies

pip install 'cooksafe>=0.2.0,<0.3.0'

JsonCache replays the API responses in json_cache.json, so the steps below run without an API key or any spend. Download it from typesafe-public-examples into the folder you run the script from. To make the requests live instead, set TYPESAFE_API_KEY and leave it out.

Create the script

Start semantic_search.py with the imports and the client:

import os
import urllib.request
from pathlib import Path

from cooksafe import JsonCache
from typesafe_sdk import Choice, Noul, NoulCriteria, TypeSafeClient

TYPESAFE_MODEL = "jev-1.12"

client = TypeSafeClient(
    api_key=os.environ.get("TYPESAFE_API_KEY", "cache-only"), timeout=120.0
)
json_cache = JsonCache(Path("json_cache.json"))

Step 1: tag every line with an ID

The test document is GitHub’s Terms of Service, split into 218 clauses, so every search result points to one quotable line.

DOCUMENT_URL = (
    "https://raw.githubusercontent.com/typesafe-ai/typesafe-public-examples/main/cookbooks/semantic_find/github-tos.txt"
)


@json_cache
def fetch_document(url: str) -> str:
    request = urllib.request.Request(
        url, headers={"User-Agent": "typesafe-cookbook/1.0"}
    )
    with urllib.request.urlopen(request) as response:
        return response.read().decode()


LINES = fetch_document(DOCUMENT_URL).splitlines()

The cache prevents repeated downloads. Now prefix each line with a short ID and join the lines back into one document. The model uses these IDs to point to its answer.

def line_id(i: int) -> str:
    return f"L{i:03d}"


DOCUMENT = "\n".join(f"{line_id(i)}| {line}" for i, line in enumerate(LINES))

DOCUMENT now looks like this:

L052| You own Your Content. If you post Content you did not create, you are responsible for...
L053| You grant us and other Users the licenses in Sections D.4–D.8. These licenses apply...
L054| 4. License Grant to Us

Step 2: ask where the answer is

A Choice question returns a probability for every option. Use the line IDs as the options.

def where_question(query: str) -> Choice:
    return Choice(
        instructions=f'Which line of the document contains the answer to: "{query}"?',
        criteria={line_id(i): None for i in range(len(LINES))},
    )

The option descriptions are None because the document already contains the text for each ID. The query goes in instructions; the state stays unchanged between searches.

A Choice question accepts up to 255 options, so this recipe searches documents of up to 255 lines in one request. Past that, search in two passes: one Choice question picks a window of lines, and a second ranks the lines inside it.

Step 3: check whether an answer exists

Choice probabilities always add up to 1, so some line ranks first even when the document doesn’t answer the question.

def exists_question(query: str) -> Noul:
    return Noul(
        instructions=f'Does any line of the document address or answer: "{query}"?',
        criteria=NoulCriteria(
            true="At least one line of the document states or directly implies the answer",
            false="No line of the document addresses this",
        ),
    )

Unlike the Choice probabilities, the Noul probability doesn’t depend on the other options, so it can fall near zero when the document has no answer.

Step 4: send both questions in one request

Both questions go in one system_one call, so the document is sent once.

@json_cache
def _find(
    model: str,
    state: str,
    where: Choice,
    exists: Noul,
) -> dict:
    response = client.system_one(
        state=state,
        questions={"where": where, "exists": exists},
        model=model,
    )
    probabilities = response.answers["where"].probabilities
    return {
        "exists": response.answers["exists"].noul,
        "relevance": [probabilities.get(line_id(i), 0.0) for i in range(len(LINES))],
    }


def find(query: str) -> dict:
    return _find(
        TYPESAFE_MODEL,
        DOCUMENT,
        where_question(query),
        exists_question(query),
    )

The relevance list keeps one score per line, in document order.

Step 5: read the result

The exists probability becomes one of three verdicts, the middle one for partial answers. The ranking prints as a bar chart.

FOUND, ABSENT = 0.7, 0.35  # present answers typically read >=0.9, absent <=0.05


def verdict(exists: float) -> str:
    if exists >= FOUND:
        return "answered in this document"
    return "not in this document" if exists < ABSENT else "partially addressed"


def show(query: str, top: int = 4) -> dict:
    result = find(query)
    print(f'"{query}"')
    print(f"  exists {result['exists']:.2f} -> {verdict(result['exists'])}")
    ranked = sorted(
        range(len(LINES)), key=lambda i: result["relevance"][i], reverse=True
    )
    for i in ranked[:top]:
        bar = "#" * max(1, round(result["relevance"][i] * 12))
        preview = LINES[i][:58].rstrip()
        print(f"  {line_id(i)}  {result['relevance'][i]:.2f}  {bar:<12}  {preview}")
    return result

Tune these thresholds on your own documents before using them in production.

Step 6: run the search

Ask two questions that have direct answers, one that has no answer, and one that has a partial answer.

print(f"{len(LINES)} lines, {len(DOCUMENT):,} characters\n")
show("who owns the code I upload?")
print()
show("can GitHub kick me off the platform without warning?")
print()
show("do I have to take disputes to arbitration?", top=2)
print()
show("can minors use GitHub with parental permission?", top=2)

218 lines, 43,980 characters

"who owns the code I upload?"
  exists 0.98 -> answered in this document
  L052  0.95  ###########   You own Your Content. If you post Content you did not crea
  L046  0.02  #             Short version: You own content you create, but you allow u
  L051  0.02  #             3. Ownership and License Grants
  L217  0.01  #             Questions about the Terms of Service? Contact us through t

"can GitHub kick me off the platform without warning?"
  exists 0.97 -> answered in this document
  L168  0.97  ############  GitHub has the right to suspend or terminate your access t
  L167  0.03  #             3. GitHub May Terminate
  L000  0.00  #             Effective date: April 27, 2026 · A. Definitions
  L001  0.00  #             Short version: We use these basic terms throughout the agr

"do I have to take disputes to arbitration?"
  exists 0.14 -> not in this document
  L205  0.86  ##########    Except to the extent applicable law provides otherwise, th
  L168  0.02  #             GitHub has the right to suspend or terminate your access t

"can minors use GitHub with parental permission?"
  exists 0.46 -> partially addressed
  L029  0.90  ###########   You must be age 13 or older. While we are thrilled to see
  L012  0.07  #             “User,” “You,” and “Your” refer to the individual person,

What the scores mean

The first two queries return direct answers and the source lines needed to verify them.

Arbitration: The ranking gives the closest line a score of 0.86, but exists is only 0.14. The answer is not in the document.
Parental permission: The age rule ranks first, but it doesn’t answer whether parental permission changes the rule. The result is partially addressed.

The ranking tells you where to look; the exists score tells you whether the result answers the question.

Try it on your own document

Open the tagged contract in the TypeSafe playground to edit the questions against the same text. To search your own, swap DOCUMENT_URL; every other line of the script works off LINES.

​Setup

​Get a TypeSafe API key

​Install the dependencies

​Create the script

​Step 1: tag every line with an ID

​Step 2: ask where the answer is

​Step 3: check whether an answer exists

​Step 4: send both questions in one request

​Step 5: read the result

​Step 6: run the search

​What the scores mean

​Try it on your own document

Setup

Get a TypeSafe API key

Install the dependencies

Create the script

Step 1: tag every line with an ID

Step 2: ask where the answer is

Step 3: check whether an answer exists

Step 4: send both questions in one request

Step 5: read the result

Step 6: run the search

What the scores mean

Try it on your own document