I Built My Own AI Memory System. Here's Why It's Better Than ChatGPT's.

Mike Taylor wrote a piece this week about turning off ChatGPT’s memory. His argument: memory sounds great, but it quietly degrades your results over time. Old preferences contradict new ones. Stale context bleeds into fresh conversations. A Kanye quote in his custom instructions had ChatGPT trying to make everything “as dope as possible” for months.

He calls this context rot. And he’s right. But his solution is to just turn it off. Mine was to build something better.

For the past few months, I’ve been running my life through an AI operating system I call Jarvis. It manages my projects, tracks my habits, triages my email, plans my day, and coaches me when I’m slacking. It runs on Claude Code, it has access to my calendar, Gmail, Google Drive, CRM, and SEO tools. And it remembers everything about me, perfectly, every single time.

Not because it has “memory” turned on. Because I built the memory myself.

The Problem With AI Memory

Let me explain what ChatGPT’s memory actually does. Every time you chat, it extracts facts it thinks are important and saves them to a growing list. Your job title. Your dog’s name. That you prefer Python over JavaScript. That you like concise responses.

Sounds helpful. The problem is threefold:

1. You can’t see what’s driving the output. When ChatGPT gives you a weird response, you don’t know if it’s the prompt, the model, or some random fact it memorized six months ago. Taylor’s example: ChatGPT kept recommending BBQ rib restaurants because it had memorized his zip code. Not because he asked for ribs. Because it decided he needed ribs.

2. Context accumulates but doesn’t expire. You told ChatGPT you were learning React in January. By July you’ve moved on to Svelte. But that React preference is still in memory, subtly biasing every coding conversation. The system has no mechanism for knowing what’s stale.

3. Contradictions pile up silently. You tell ChatGPT you want verbose explanations. Three months later you tell it you want concise responses. Now it has both instructions in memory, and you’ll get inconsistent behavior depending on which one it weighs more heavily on any given day.

This is context rot. It’s the AI equivalent of a messy desk: everything you’ve ever touched is still there, piled on top of each other, and you can’t find what you need.

My Approach: Structured Context Files

Here’s the thing. AI models don’t actually have persistent memory. Not really. Every conversation starts from zero. What ChatGPT calls “memory” is just an invisible system prompt that gets prepended to every conversation. It’s a text file you can’t easily see or edit.

So I thought: what if I just built that text file myself? And instead of one big pile of facts, I structured it into layers that I control completely?

That’s what Jarvis is. It’s a set of markdown files organized into a clear hierarchy, with a master instruction file (CLAUDE.md) that tells the AI how to navigate everything. When I start a conversation, Claude reads the CLAUDE.md file and instantly knows who I am, what I’m working on, and how to help me.

No context rot. No stale preferences. No mystery about what’s driving the output.

The system has three layers:

Layer 1: The System Layer (Who I Am)

This is the strategic layer. It answers: what are my goals, what are my life priorities, what projects am I running, and what are the rules of engagement?

system/
├── goals.md          # Annual goals with quarterly milestones
├── projects.md       # Active projects (max 5) with status and next actions
├── areas.md          # 6 life domains with scores and standards
├── contacts.md       # Key people linked to projects
├── follow-ups.md     # Commitments I've made and others have made to me
└── someday.md        # Future ideas on hold

Here’s a snippet from my projects.md:

### Venture5 AI Audit + Build
**Area:** Work
**Context:** notes/venture5/
**Next Action:** Score & map operations, create AI opportunities list
**Status:** Interviews complete. Now analyzing.
**Webinar Date:** Feb 28, 2026 (rehearsal Feb 25)

Every project has a clear status, a next action, and a pointer to its detailed context folder. When Claude reads this file, it immediately knows what I’m working on and what needs attention.

The key rule: maximum 5 active projects. This isn’t just a productivity hack. It’s a memory management strategy. Five projects means five context files. Claude can hold all of them in working memory without anything getting lost.

Layer 2: The Project Layer (What’s Been Done)

Each active project has a _context.md file that acts as its institutional memory. This is the file that prevents the “Groundhog Day” problem where you have to re-explain everything at the start of each conversation.

notes/
├── venture5/
│   └── _context.md
├── funk-coffee/
│   └── _context.md
├── refound-ai/
│   └── _context.md
└── no-dirt-golf/
    └── _context.md

Here’s what a real _context.md looks like:

# Personal Site Rebuild - Project Context

Read this file first when working on this project.

## Current State

Decision made: Repositioning siddharthbharath.com from enterprise
consulting site to technical thought leadership blog.

Tech stack: Astro + Cloudflare Pages
Codebase: `/Users/sid/Projects/siddharthbharath`

## Key Decisions Made

- Positioning: Technical blog with subtle Refound AI cross-promotion
- Design direction: Warm aesthetic, "builder/technical" feel
- Homepage headline: "I write about building with AI."

## Files in This Folder

| File | Contents | Status |
|------|----------|--------|
| homepage-copy-v2.md | Homepage copy | Complete |
| case-studies.md | All 4 case studies | Complete |
| linkedin-launch-posts.md | 5-post launch sequence | In progress |

Last updated: 2025-12-17

A few things to notice:

It’s written for re-reading, not first-reading. I assume the AI has zero memory of previous sessions. Everything Claude needs to pick up where we left off is right here.

It tracks decisions, not just tasks. “Decision made: repositioning from consulting site to blog” tells Claude why things are the way they are. Without this, the AI might suggest consulting-oriented copy because it doesn’t know we already decided against that.

It has an inventory of deliverables. The table of files and their status means Claude never asks “what files exist?” or “what’s been done?” It already knows.

Layer 3: The Daily Layer (What’s Happening Now)

This is the operational layer. Each day gets its own markdown file that connects today’s tasks to the strategic layers above:

# Friday, February 20, 2026

> "Today I run four audit interviews and open the HexClad door."

## MITs
- [ ] Run Funk Coffee AI Audit interviews (1-5pm) → [[Funk Coffee]] (Work)
- [ ] HexClad discovery call with Jeff Faber (10am) → [[Refound AI GTM]] (Work)
- [ ] Reply to No Dirt Golf — confirm Wed demo → [[No Dirt Golf]] (Work)

## Habits (0/9)

**Health**
- [ ] Workout
- [ ] Protein target
- [ ] 7+ hours sleep

**Work**
- [ ] Outreach activity
- [ ] Client progress

**Mind**
- [ ] 30 min learning

**Spirit**
- [ ] Morning meditation (5 min)
- [ ] Evening gratitude (3 things)

## Schedule
- 08:00 - 09:15 — Morning admin block
- 09:30 - 09:45 — Stef Hutka debrief
- 10:00 - 10:30 — Jeff Faber / HexClad call
- 11:00 - 12:00 — Workout
- 13:00 - 17:00 — Funk Coffee interviews (Alison, Luis, Aren, Kurtis)
- 17:00 - 18:00 — Capture interview notes

See those → [[Funk Coffee]] links? That’s the chain. When I ask Jarvis to help me with an MIT, it follows the link to the project in projects.md, which points to the context folder in notes/funk-coffee/, which has the _context.md with everything Claude needs.

MIT → Project → Context. Three hops, and the AI has full situational awareness. No memory feature required.

The Context Chain In Practice

Let me show you what this actually looks like when I use it.

Every morning, I run /plan-day. This is a slash command that triggers a multi-step planning ritual. Claude:

1. Checks yesterday’s daily file for an end-of-day review (and nags me if it’s missing)
2. Reads projects.md and flags anything stale (no progress for 2+ weeks)
3. Checks my Google Calendar for today’s meetings
4. Pulls my unread emails from Gmail
5. Reviews pending follow-ups
6. Checks relationship cadence (have I gone too long without contacting key people?)
7. Proposes 1-3 MITs based on project priorities and calendar availability
8. Creates a time-blocked daily plan

All of this context comes from the file system. None of it comes from “memory.”

When I ask “what should I work on right now?”, Claude doesn’t guess based on vibes. It reads projects.md, checks the daily plan, considers the time of day (morning for deep work, afternoon for collaborative tasks), and gives me one clear action with a time estimate.

When I triage email, Claude looks up the sender in my contacts, checks which project they’re linked to, reads the project context, checks for pending follow-ups, and drafts a response with full awareness of our history and current status.

The AI isn’t remembering any of this. It’s reading it fresh every time. And that’s exactly why it works.

Why This Beats “Memory”

Let me make the case directly.

1. Total Transparency

With structured context files, I can see everything the AI knows about me. I can open projects.md and read exactly what Claude will read. If something is wrong, I fix it. If something is outdated, I delete it.

With ChatGPT’s memory, you get a list of extracted facts with no hierarchy, no structure, and no easy way to understand how they interact. Good luck debugging why it keeps recommending BBQ ribs.

2. Intentional Forgetting

Context rot happens because memory systems don’t know when to forget. My system has built-in expiration:

• Projects with no progress for 2 weeks get moved to “Someday” or dropped
• Daily files are only relevant for that day (plus a few days of lookback for habit tracking)
• Context files get updated when decisions change
• The max 5 project rule prevents context sprawl

I’m not fighting an accumulation problem because there’s nothing accumulating. The system is always current because I actively maintain it.

3. Layered Relevance

ChatGPT’s memory is flat: a list of facts with no hierarchy. My system is layered: strategic (goals, projects), tactical (project context), and operational (daily plan). Claude reads the right layer for the right task. When I’m planning my day, it reads the strategic layer. When I’m working on a specific project, it reads the tactical layer. When I’m deciding what to do right now, it reads the operational layer.

This means Claude never pollutes a project conversation with random facts about my zip code or music preferences.

4. It Works Across Models

Because my context is in plain markdown files, I’m not locked into any specific AI provider. ChatGPT’s memory only works with ChatGPT. My system works with Claude Code, Claude in the browser, or any future model that can read files. The context is mine. I own it. I can move it anywhere.

How to Build Your Own

You don’t need to build something as extensive as Jarvis. Start with these three files:

1. A System File (CLAUDE.md or similar)

This is your master instruction file. It tells the AI who you are, what you’re working on, and how to behave. If you use Claude Code, create a CLAUDE.md in your project root. Claude reads it automatically at the start of every session.

Include:

• Your current role and priorities
• Active projects (max 5) with status and next actions
• Rules of engagement (how you want the AI to communicate)
• Links to context files for each project

# CLAUDE.md

## About Me
I'm a freelance designer working on 3 active client projects.

## Active Projects
1. **Acme Rebrand** - In review phase. Context: notes/acme/
2. **Personal Portfolio** - Building in Framer. Context: notes/portfolio/
3. **Side Project: Type Tool** - Early prototype. Context: notes/type-tool/

## Rules
- Be direct. Skip the caveats.
- When I ask about a project, read its context file first.
- Max 3 priorities per day.

2. Context Files Per Project

For each active project, create a _context.md that captures:

• Current state (what’s done, what’s in progress)
• Key decisions made (so the AI doesn’t re-litigate them)
• Files and their status
• What the AI needs to know to be helpful

Update these as you go. It takes 2 minutes after a work session to jot down what changed.

3. A Daily File

Even a simple daily file helps enormously:

# 2026-02-23

## Top 3 Today
- [ ] Finish Acme presentation deck → [[Acme Rebrand]]
- [ ] Review portfolio layout options → [[Portfolio]]
- [ ] 30 min learning (typography course)

## Notes
[capture as you go]

The magic is in the links. When you ask the AI to help with “Acme presentation deck,” it knows to read notes/acme/_context.md first. No memory feature needed. Just files.

The Bigger Insight

Here’s what I think Taylor’s article gets right at a deeper level: the best AI context is context you curate intentionally.

ChatGPT’s memory is like letting someone follow you around with a notebook, writing down whatever they think is important. Sometimes they capture gold. Sometimes they write down that you ordered ribs once and assume it’s your personality.

A structured context system is like writing your own briefing document. You decide what matters. You decide when it expires. You decide what the AI sees.

It’s more work upfront. But the payoff is an AI that actually knows what you need, every single time, with zero context rot.

I’ve been running Jarvis for months now. It plans my mornings, triages my inbox, tracks my commitments, and pushes back when I’m slacking on my health goals. It never forgets a follow-up. It never confuses one client project with another. It never recommends BBQ ribs when I asked for help with an email.

And if I ever switch AI providers, my entire system comes with me. It’s just markdown files in a folder.

That’s the kind of memory I want.

P.S. If you want to see how I set up the full Jarvis system in Claude Code, I recorded a video walkthrough. And if you’re interested in the technical architecture of building AI agents with structured context, check out my guide on context engineering.