Joy Zhang

My Portrait

A local-first AI memory system for macOS.

The goal is a proactive agent that knows me well enough to act for me. It starts with a memory built from what I see, hear and type.

Swift · macOS · GitHub · Download

Overview

Capturescreenshots with Vision OCRaudio: VAD, transcription, speakerIDkeyboard inputshippedEventSessionQwen3.5-9B vision-language model,QAD 4-bit, about 6 GBwrites up each app session from itsframes and keysshippedEvent MergeEmbeddingGemma retrieval; the 9Bmodel answers narrow questionssame-day events, then a cross-daylibraryv1 in trainingPortraitsix sections plus Patternevery entry cites its eventspipeline in testingNeural Graphevents as a force-directed graphshippedProactive agentacts for me from what it knows aboutmegoalCapturescreenshots with Vision OCRaudio: VAD, transcription, speaker IDkeyboard inputshippedEventSessionQwen3.5-9B vision-language model, QAD 4-bit, about 6GBwrites up each app session from its frames and keysshippedEvent MergeEmbeddingGemma retrieval; the 9B model answers narrowquestionssame-day events, then a cross-day libraryv1 in trainingPortraitsix sections plusPatternevery entry cites itseventspipeline in testingNeural Graphevents as aforce-directed graphshippedProactive agentacts for me from what it knows about megoal
The pipeline as it is being built. Solid boxes are shipped; dashed boxes are in training, in testing or planned.

Capture

Triggersapp switch, click, Return, typingburst, scroll stop, clipboard change,30 s idle tick200 ms debounce; waits for the screento settlePrivacy gatespassword fields, private windows andDRM content are skippedDeduplication4× downscale to grayscaleFNV-1a hash, then histogram distance(Hellinger < 0.02)one frame kept every 10 s regardlessOCRApple Vision, accurate modeChinese and English, word-level boxescan wait for idle or charging timeStorageJPEG first, HEVC chunks after 10 minTriggersapp switch, click, Return, typing burst, scroll stop,clipboard change, 30 s idle tick200 ms debounce; waits for the screen to settlePrivacy gatespassword fields, private windows and DRM content areskippedDeduplication4× downscale to grayscaleFNV-1a hash, then histogram distance (Hellinger <0.02)one frame kept every 10 s regardlessOCRApple Vision, accurate modeChinese and English, word-level boxescan wait for idle or charging timeStorageJPEG first, HEVC chunks after 10 min
Screen capture. Keystrokes come separately from a listen-only event tap, with input-method composition folded so that Chinese text is kept as typed.
MicrophoneAVAudioEngineSystem audioScreenCaptureKitVoice activity16 kHz monoSilero VAD v5, segments up to 60 snoise and music filtering on vDSPTranscriptionWhisper large-v3 or Qwen3-ASR on MLXbatched while idle or chargingmic and system duplicates mergedDiarizationpyannote segmentationKaldi-style fbankCAM++ speaker embeddingsSpeaker matchcosine ≥ 0.45 to a voiceprint0.10 margin over the next speakera 30 s recording enrolls a voiceMicrophoneAVAudioEngineSystem audioScreenCaptureKitVoice activity16 kHz monoSilero VAD v5, segments up to 60 snoise and music filtering on vDSPTranscriptionWhisper large-v3 or Qwen3-ASR on MLXbatched while idle or chargingmic and system duplicates mergedDiarizationpyannote segmentationKaldi-style fbankCAM++ speaker embeddingsSpeaker matchcosine ≥ 0.45 to a voiceprint0.10 margin over the next speakera 30 s recording enrolls a voice
Audio, from two sources to named speakers.

EventSession

App sessionframes and keystrokes from one appand windowQwen3.5-9B VLMLoRA-distilled from a larger teacherQAD to 4-bit MLX, about 6 GBruns while plugged in or idleWrite-upwhat I did in the session, in plaintextApp sessionframes and keystrokes from one app and windowQwen3.5-9B VLMLoRA-distilled from a larger teacherQAD to 4-bit MLX, about 6 GBruns while plugged in or idleWrite-upwhat I did in the session, in plain text
Trained on 13,265 frames plus 1,288 synthetic ones. QAD (quantization-aware distillation) trains the 4-bit model against its full-precision version while simulating MLX quantization.
ModelBlind score, 1–5
BF163.46
Plain 4-bit2.89
QAD 4-bit3.35

Mean over 313 test frames. For each session, a larger model graded the three versions' write-ups against the original screenshots without knowing which was which. A 5 means the write-up matches the screen with nothing missing; a 1 means it is unusable. QAD recovers about 81% of the score that plain 4-bit quantization loses.

Event Merge

Session write-upsfrom EventSessionRecallEmbeddingGemma 2, 768-d, on MLXtop 4 events by vector, plus thelatest oneNarrow questionsdid this session do anything?does it join this event?are these two events the same?Narrativesummaries of every 10 sessions, thenan overviewCross-day librarycontinues a library event? (top 5 byvector, tags and the last 48 h)rewrites the entry's title andoverviewSession write-upsfrom EventSessionRecallEmbeddingGemma 2, 768-d, on MLXtop 4 events by vector, plus the latest oneNarrow questionsdid this session do anything?does it join this event?are these two events the same?Narrativesummaries of every 10 sessions, then an overviewCross-day librarycontinues a library event? (top 5 by vector, tags andthe last 48 h)rewrites the entry's title and overview
Event Merge v1. Qwen3.5-9B learns each question from a larger teacher: 11,946 training examples and a 1,958-question exam on held-out days. The v1 target is to merge at least 80% of same-event sessions, with at least 90% of merges correct.

Portrait

Eventsfrom the event libraryRecallvectors, BM25 over evidence, titleoverlap94% recall at about 14 candidatesNarrow questionsany material for this section?attach to this entry?Writecreate, deduplicate, rewritea review agent checks each changePortraitentries with a title, a detailparagraph and evidence event idsEventsfrom the event libraryRecallvectors, BM25 over evidence, title overlap94% recall at about 14 candidatesNarrow questionsany material for this section?attach to this entry?Writecreate, deduplicate, rewritea review agent checks each changePortraitentries with a title, a detail paragraph and evidenceevent ids
The portrait pipeline, in testing. A seven-day reference of 107 entries, written and verified by a larger model, is the benchmark for the local pipeline.
SectionWhat it holds
Decision & thinking"Situation → how I tend to decide", with my own stated reasons, scope and counter-examples. Drawn from decisions, vetoes, corrections of the AI and replies to its proposals.
PatternCountable regularities in time, rhythm, sequence and phase changes, computed by code from session times, with no language model.
ExperiencesRésumé-style cards for projects, courses and errands, with their status.
InterestsRecurring attention without a deliverable.
BackgroundExact personal facts, with an expiry date.
SocialOne entry per person.
SkillsGeneralized abilities, marked when AI was involved.

Graph

Sceneme at the center, foldersas hubs, events as leavesradius from event weightPhysics thread, 60 HzBarnes-Hut repulsion (θ²= 0.81)springs, collision,bubble constraintsparks once the layoutsettlesShadow engineruns ahead in thebackground to predict thefinal layoutSnapshotlock-protected handoff tothe rendererSwiftUI Canvaspre-rasterized labelsoff-screen cullinghash-grid hit testsSceneme at the center, folders as hubs, events as leavesradius from event weightPhysics thread, 60 HzBarnes-Hut repulsion (θ²= 0.81)springs, collision,bubble constraintsparks once the layoutsettlesShadow engineruns ahead in thebackground to predictthe final layoutSnapshotlock-protected handoff to the rendererSwiftUI Canvaspre-rasterized labelsoff-screen cullinghash-grid hit tests
The engine behind the memory graph, drawn with SwiftUI.
Interactive memory graph
A JavaScript port of the same engine, on a snapshot of my data from Sep 21, 2026.

Releases

All projects