Personal Projects of andrasp

Going beyond simple memory retention into true experience building for AI agents. By extending the notion of a self-organizing memory layer (such as MemPad) into an experience builder that periodically distills learnings from agent sessions, we can create agents that identify root causes faster and reduce trial-and-error not just on familiar problems, but on novel ones that share underlying fundamentals.

Rather than just memorizing facts or relationships between entities, the system continually reshapes its knowledge by merging new learnings with existing ones. Over time, this ideally converges toward a well-compressed set of core problem-solving primitives for the domain. This isn't true continual learning for generic tasks, but it attempts to approximate that capability for agents working in specific domains like debugging, medical diagnosis, or legal research.

The entire cycle runs automatically with no human intervention: each session's learnings feed back into a richer, more capable agent for the next run.

Definition: An evolving model of any entity built from continuous observation and expressed in natural language.

Narrative Twins are queryable through AI agents to surface human insight about the entity. Think of it like having a conversation with the entity itself.

Narrative Twins emerge from a feedback loop that continuously refines the representation as new observations arrive. The approach mirrors Experience-Driven Agents, but learnings come from real-world signals instead of agent transcripts.

Example: A Customer's Narrative Twin

Every touchpoint (browsing, support calls, purchases, store visits) feeds into a living narrative. A greatly simplified example:

"Sarah researches extensively, pays for quality she trusts, and is frustrated by recent shipping delays."

In practice, these narratives grow into rich, structured characterizations built from thousands of observations, capturing preferences, patterns, contradictions, and evolution over time. Humans or agents can use LLMs to query the customer's Narrative Twin: "Would Sarah respond to this promotion?" At scale, your entire customer base can be queried in parallel.

Risks

Incomplete data produces plausible-sounding narratives that are wrong. Acting on hallucinated understanding erodes exactly the trust you're trying to build.

Customer privacy and safety is a critical concern. People deserve agency over how their data shapes the systems that serve them; opt-out by default should be the standard across the industry in my opinion. When utilized, companies should focus on maximizing the customer experience as the direct measure, with revenue as a secondary benefit.

Beyond Customers

The pattern extends wherever understanding context matters more than raw prediction: patient narratives in healthcare that capture treatment responses and lifestyle factors; student profiles that track learning styles and breakthroughs; employee narratives for talent matching. Any domain where you'd naturally ask "what would this entity do?" benefits from a queryable characterization rather than a statistical model.

Digital Twins vs. Narrative Twins

Digital Twins run simulations and output probabilities: "73% churn risk." Narrative Twins answer questions with reasoning: "Sarah's been frustrated since the pricing change and her usage dropped after we removed the export feature she relied on. She's mentioned in support tickets that she'd stay if bulk export came back. High churn risk, but a personal outreach about the new bulk export might re-engage her."

The two are complementary: digital twin predictions should feed into narrative twin queries to ground narratives in model based predictions.

My home runs on Home Assistant. Lights, blinds, speakers, thermostats: each physical device maps to an entity in HA that can be controlled programmatically. The traditional interface is a mobile app with dashboards, room views, and entity lists.

Here's an alternative: point my phone's camera at the light above the kitchen island and say "turn this on." The app identifies which HA entity I'm looking at and executes the command.

The Flow

Camera captures what I'm pointing at. A vision model identifies the physical object and maps it to a Home Assistant entity. Voice command specifies the action.

The Tricky Bit

Can a fine-tuned local model reliably map "that light fixture in frame" to light.kitchen_island in my HA config? That's what I want to find out.

Training data would be 5-6 photos from different angles of each controllable device in my home, labeled with their HA entity IDs. The model needs to handle different angles, lighting conditions, and partial visibility.

Interfaces that materialize on demand rather than existing as static screens. Instead of navigating pre-built apps or configuring templates, describe what you need and the system generates a purpose-fit interface on the spot.

I see a lot of potential here. The building blocks are maturing: LLMs that understand intent, component libraries that can be composed programmatically, and rendering pipelines fast enough that generation feels instant. Frontier labs and startups are almost certainly experimenting with this already.

A Small Experiment

To explore the edges of this, I want to build a dynamic form generator backed by Postgres. Describe what information you want to collect: "I'm planning a dinner party. I need to know who's coming, dietary restrictions, and whether they can bring a dish."

The system generates a schema, builds a form UI with appropriate input types, and gives you a shareable link. Responses hit the database. Later, ask "show me who's coming and what they're bringing" and get a generated report view.

Why This Scope

Form generation is constrained enough that LLMs do it well. Schema inference from natural language is a solved problem. Report generation is just SQL from natural language. No training data needed, just good prompting and a well-defined component library.

Open Questions

How gracefully can such a solution handle schema evolution? "Actually I also need to know if they need a ride." What about versioning? Even if you solve latency when generating, what about latency of deployment?