What an AI Agent Sees on Your Trading Chart

Three Ways to Hand a Chart to a Model, Ranked

The weakest input is a screenshot. A vision model reading a chart image has to infer the price axis from tick labels, map a pixel row to a price, and estimate where a line sits between two gridlines. It will produce a number, and the number will read confidently, but it was measured off an image at whatever resolution you happened to capture. Nothing in that pipeline can be checked.

The middle option is pasting a table of candles into the chat. The values are now exact, which removes the estimation problem, but you have lost everything around them: which indicator settings were on, where your own levels sit, what timeframe this is, which provider the data came from. The model answers about a column of numbers rather than about your chart.

The strongest input is structured context: the same numbers, plus the state that makes them mean something. Symbol, timeframe, the provider and its connection status, every indicator with its actual parameters, every drawing with its real coordinates, and any notes attached to them.

• Screenshot: prices estimated from pixels, unverifiable.
• Pasted candles: exact values, no surrounding state.
• Structured context: exact values plus the state that interprets them.

What Structured Chart Context Actually Contains

A direct connection between your chart and an AI assistant exposes a small, readable set of operations rather than one opaque blob. In MyLinedChart those are get_chart_context for the current symbol, timeframe and view, get_candles for the OHLCV series, get_indicators for every active indicator with its parameters, get_drawings and get_all_drawing_symbols for your own levels and annotations, get_trades for executions, get_provider_status for which data source is connected and live, and get_chart_screenshot when a visual genuinely helps.

The detail that matters most is indicator parameters. A model told there is a moving average on the chart can say almost nothing useful. A model told it is a 21-period EMA on the 5-minute series can reason about it. The same applies to drawings: a trendline described as going from the left to the upper right is decoration, while the same trendline given as two timestamped price coordinates is data.

There is one write operation, propose_chart_changes, and the word proposes is doing real work. It covers annotation-class changes such as drawing a level or adding a note. It does not place orders, and no read tool in that list touches an order either.

• Indicator parameters, not indicator names.
• Drawing coordinates, not drawing descriptions.
• Writes are annotation-class only, never orders.

The Measured Part: Parsing Is Not Comparing

Giving a model good data does not make it good at arithmetic, and the gap is larger than most people assume. We measured this directly on Apple's on-device Foundation Model in August 2026, on macOS 26.6, using our shipped assistant prompt and tools mirroring the real ones. The task was deliberately trivial: over eight candles, with the true maximum planted as a forty-point outlier that is impossible to miss by eye, find the highest high and draw a line there.

The tool loop worked ten times out of ten. It called for the candles, then called to draw the line, every single run. It got the right number zero times out of ten. A minimal prompt scored one out of ten. Every wrong answer was a real value from the table, so parsing was never the problem. It read the rows correctly and then failed to compare them. The failure was identical at eight, fifteen and forty bars, so it was not a context-window limit either.

The lesson generalises well beyond one vendor's model. The plumbing looks flawless while the reasoning is not, and a demo passes because the tools fire and the sentence reads confidently. Only checking the number against ground truth reveals it.

The practical consequence: compute on the deterministic side and let the model interpret. Send the session high as a value rather than asking a model to find it in a table. Send the indicator's computed series rather than asking for it to be derived. Reserve the model for the part it is genuinely good at, which is reading context and explaining it back.

• Tool loop 10/10, correct value 0/10, over eight candles.
• Every wrong answer was a real row, so parsing worked and comparison did not.
• Compute the number yourself; ask the model to interpret it.

Why This Runs Locally

Chart context is account-adjacent data. It carries what you hold, where you drew your levels, what you wrote next to them and which broker you are connected to. MyLinedChart runs on your machine, against your own broker credentials, and the connection to your AI assistant is local as well.

That also removes an entire category of staleness. The context a model receives is the chart as it exists right now, not a snapshot uploaded earlier, because the read happens against the running application.

For a worked example with a specific broker, see Using Codex or Claude Code With IBKR Chart Data. For what a broker-provided connector exposes by comparison, see What Does IBKR's Certified Claude and ChatGPT Connector Actually See, and What Is the Chart Context Gap?.

• Your keys, your machine, no chart data on someone else's server.
• The model reads current state rather than an uploaded snapshot.
• Read tools for market data, annotation-class writes, no order path.

Can an AI assistant read prices off a chart screenshot accurately?

Not reliably. It has to infer the price axis from tick labels and map pixel rows to prices, so every value is an estimate from an image. Structured context supplies the same values exactly and can be checked.

What does structured chart context include that a candle table does not?

The state that interprets the numbers: symbol, timeframe, which data provider is connected, every active indicator with its real parameters, and your own drawings as timestamped price coordinates rather than descriptions.

If the model has the data, why compute values before sending them?

Because parsing and comparing are different skills. In our own August 2026 measurement a model ran the tool loop ten times out of ten and still picked the wrong maximum from eight rows every time. Send the computed value and let the model interpret it.

Can the AI assistant place trades?

No. The read operations cover market data, indicators, drawings and executions. The single write operation is annotation-class, covering things like drawing a level or adding a note. There is no order path.