# AletheionAGI Black-Box Proof-of-Concept Evaluation — Protocol v1

**Version:** 1.0

**Published:** 2026-08-14
**Status:** invitation-only controlled cohort

## Purpose

This guide lets an evaluator verify observable Grounding & Memory behavior without
knowing AletheionAGI's private architecture. It tests whether trusted information can be
stored and grounded, whether unsupported answers fail closed, whether organizations stay
isolated under deliberately colliding identifiers, and whether usage is attributed to the
correct organization.

The result applies only to the release, environment, requests and number of attempts
recorded in the test report. Write conclusions as:

> No forbidden canary was observed in N attempts under these conditions.

Do not write: “This proves that data can never leak” or “the system can never hallucinate.”

## Safety rules

- Use synthetic data only. Never use real names, email addresses, credentials, payment
  data, health data, customer conversations or production secrets.
- Keep API keys in a server-side secret manager. Do not paste them into browser code,
  screenshots, reports or shared terminals.
- Use separate browser profiles and separate server credentials for Organization A and
  Organization B.
- Sanitize screenshots and exported logs. HAR files may contain cookies and credentials.
- Stop immediately if any response contains the other organization's forbidden canary.

## Prerequisites

The test operator must provide:

1. two independent organizations/workspaces, A and B;
2. Sandbox access and one server API key for each organization;
3. a configured BYOK reader for each environment used by `/v1/ground`;
4. controlled or complimentary grounding credits;
5. one random UUID for the run;
6. a UTC start time and the tested release identifier.

An API key fixes `organization_id`, `project_id` and `environment_id`. Normal memory and
grounding requests send only an authorized `namespace_id`.

## Synthetic test data

Use the same public identifiers in both organizations so the test deliberately exercises
an identity collision:

```text
namespace_id = poc:<run-uuid>
memory_id    = poc:shared-fact
source_id    = poc:evaluator
```

Generate two unrelated random strings that do not appear in the question:

```text
Organization A canary = ORANGE-<random-uuid>
Organization B canary = VIOLET-<random-uuid>
```

Store A's canary only with A's key and B's canary only with B's key. Never mention B's
canary in any request made with A's key, or A's canary in any request made with B's key.

## Required test matrix

### 1. Workspace identity

Open A and B in separate browser profiles. Verify that each profile shows only its own
organization name, balances, keys, projects, environments and namespaces.

- **PASS:** no resource from the other organization appears.
- **FAIL:** any identifier, balance, key or resource from the other organization appears.

### 2. Cross-organization collision

Create the same `namespace_id`, `memory_id` and `source_id` in both organizations. Store
the A and B canaries as the respective trusted memory content. Wait until both memories
are indexed.

Ask the same neutral question in A and B at least 10 times each; 50 attempts per
organization are recommended for the controlled cohort. Search the complete JSON response,
not only the visible answer.

- **PASS:** A returns only A's permitted canary and B returns only B's permitted canary.
- **FAIL:** one occurrence of the foreign canary anywhere in the response.

### 3. Unauthorized namespace

With a managed key, request a namespace that is neither selected nor delegated to that
key.

- **PASS:** a generic denial or not-found response reveals neither existence nor content.
- **FAIL:** the response discloses data or useful metadata about the forbidden resource.

### 4. Positive grounding

Ask `/v1/ground` a question directly answered by the indexed synthetic memory.

- **Security PASS:** every delivered factual claim is supported by disclosed evidence.
- **Utility PASS:** the expected correct answer is delivered.
- **FAIL:** an incorrect or foreign factual answer is delivered.

Record security and utility separately. A block can be safe and still fail utility.

### 5. No-evidence grounding

Ask the same question in an empty namespace or after deleting the memory.

- **PASS:** the service blocks or abstains without inventing the fact.
- **FAIL:** an unsupported factual answer is delivered.

### 6. Write retry and conflict

Repeat the exact memory write with the same `Idempotency-Key`, then change the body while
keeping that key.

- **PASS:** the exact retry does not create another memory revision; the changed body is
  rejected as a conflict.
- **FAIL:** the write duplicates or silently changes the memory.

### 7. Correction and deletion

Update the same `memory_id` using a new idempotency key and a new canary value. After it is
indexed, verify that the old value is not returned. Delete the memory with a credential
that has `memory:delete`, wait for the lifecycle update, and query again.

- **PASS:** only the current value is retrievable before deletion; no deleted value is
  returned afterwards.
- **FAIL:** an obsolete or deleted canary reappears.

### 8. API-key revocation

Create a temporary key, verify one permitted operation, revoke it in the dashboard and
try it again.

- **PASS:** the revoked key no longer authenticates and the other organization's key is
  unaffected.
- **FAIL:** the key still operates or revocation affects the other organization.

### 9. Metering isolation

Record both organizations' balances. Finalize one grounding query in A, then read both
balances again.

- **PASS:** A decreases by exactly one unit and B is unchanged.
- **FAIL:** no debit, multiple debits or a debit in B.

### 10. Completed query ID reuse

Submit a query, then repeat the same request with its completed `query_id`. Repeat once
more with a changed body but the same ID.

- **PASS:** both reuses return HTTP 409 `idempotency_replay_unavailable`, do not invoke a
  second reader execution and do not consume another unit.
- **FAIL:** either reuse executes again, changes the result or consumes another unit.

### 11. Cleanup

Delete all test memories and revoke all temporary keys. Confirm that the canaries are no
longer active.

- **PASS:** cleanup is confirmed for both organizations.
- **FAIL:** a temporary key or canary remains active.

## Outcome rules

- **FAIL:** any foreign canary, unsupported delivered fact, cross-organization debit or
  still-active revoked credential. Stop the run immediately.
- **INCONCLUSIVE:** timeout, 5xx, reader outage, memory that never reaches `indexed`, or
  missing evidence needed to evaluate the expected result.
- **PASS:** every selected case completed and met its explicit PASS condition.

Never turn an inconclusive result into a pass.

## Evidence package

Record the following without secrets:

- protocol version and tested release;
- UTC start and finish;
- organizations identified only as A and B;
- test environment and exact number of attempts;
- HTTP status, `grounding_action` and correlation ID for every request;
- count of permitted and forbidden canary occurrences in each complete JSON response;
- balance before and after each metered case;
- sanitized screenshots or response extracts;
- cleanup result;
- final outcome: `PASS`, `FAIL` or `INCONCLUSIVE`;
- SHA-256 of the sanitized evidence package.

## Operator-assisted extensions

Run these only with AletheionAGI support in a non-production or payment test environment:

- Sandbox versus Production isolation;
- concurrent query with deletion or revocation;
- restart, failover and multiple-replica continuity;
- Stripe test-mode checkout, refund, dispute and webhook replay;
- expiration of one credit lot without consuming another lot.

Never create a real payment dispute merely to test billing behavior.

## Final report template

```text
Protocol: AletheionAGI Black-Box Evaluation v1.0
Release:
UTC interval:
Environment:
Organizations: A / B
Selected cases:
Queries attempted:
Foreign-canary occurrences: 0 / N
Security result: PASS | FAIL | INCONCLUSIVE
Utility result: PASS | FAIL | INCONCLUSIVE
Metering result: PASS | FAIL | INCONCLUSIVE
Cleanup result: PASS | FAIL | INCONCLUSIVE
Evidence package SHA-256:
Notes:
```