Watch it
One question, one ceiling, both screens.
On the left, the chat window sends a short coding question to Ornith-1.5-35B-A3B-OptiQ-4bit. On the right, Observatory shows the same moment: no model loaded, then about 10.6 GB in use under a 12 GB ceiling, with the line "This reply has read experts from the SSD." The clip runs at real speed and shows the model's reasoning as it streams. It was recorded on a development build.
Measured
Ten thousand tokens in, five hundred out.
Each row is one request of 10,077 prompt tokens and 500 written tokens, after a small warmup, with the prompt cache off.
| Memory ceiling | Experts | Peak model memory | Reading the prompt | Writing the reply |
|---|---|---|---|---|
| 36 GB | All in memory | 20.9 GB | 4.8 s · 2,085 tokens/s | 66.5 tokens/s |
| 16 GB | Read from the SSD | 15.4 GB | 4.5 s · 2,228 tokens/s | 40.3 tokens/s |
| 12 GB | Read from the SSD | 11.1 GB | 4.5 s · 2,235 tokens/s | 30.2 tokens/s |
The largest model in the catalog, an experimental build, ran under the same two ceilings:
| Memory ceiling | Experts | Peak model memory | Reading the prompt | Writing the reply |
|---|---|---|---|---|
| 16 GB | Read from the SSD | 15.1 GB | 17.7 s · 570 tokens/s | 16.1 tokens/s |
| 12 GB | Read from the SSD | 10.8 GB | 12.8 s · 789 tokens/s | 14.2 tokens/s |
Peak model memory stayed under the ceiling in every row. On the 122B model the prompt was read faster under the lower ceiling; with one run per row, that difference is within what a second run could reverse.
How it was measured
The workload, the build, and the clock.
- Mac
Apple M5 Pro with 48 GB of memory.
- Build
An optimized development build of version 0.2.103, serving through the local API the way any client would.
- Prompt
The opening of Romeo and Juliet followed by a request for a detailed summary: 10,077 prompt tokens. The reply was capped at 500 tokens.
- Warmup
One request of 602 prompt tokens and 50 written tokens before each measured request, which also loads the model.
- Cache
The persistent prompt cache was turned off, so no saved prompt work counted as speed.
- Clock
Reading the prompt is the time from sending the request to the first streamed token. Writing speed is 500 tokens divided by the time after that first token.
- Memory
Peak model memory is the peak the app reports for the model's own memory, read after each request.
What it does not show
A ceiling on a large Mac is not a small Mac.
The ceiling limits the memory the model itself may use. This Mac had 48 GB, so macOS could keep recently read model files in its own file cache, outside that ceiling. A Mac with 16 GB has far less room for that, and its SSD reads will be slower than the ones measured here. These rows show how the app behaves under a ceiling. They do not predict speed on a Mac with that much memory.
The model files had been read earlier in the session, so every row is a warm run. Cold reads from the SSD were not measured. Each row is a single run, so small differences between rows are not meaningful.
A 21.9 GB model stayed under a 12 GB ceiling and kept writing at a usable speed. Not as a promise for any other Mac, model, or prompt.
Repeat it
Four steps on your own Mac.
- In Observatory, download the model from the Models page.
- On the Settings page, set Maximum model RAM to the ceiling you want and press Apply. Turn the prompt cache off and save.
- Send a long prompt to
/v1/chat/completionson the local API with"stream": trueand"max_tokens": 500, after one short warmup request. - Time the first streamed token and the last, and read peak memory on Observatory's Overview page.
Your numbers will differ with your Mac, your SSD, the model and the prompt.