ADR-002

2025-02-21//ACCEPTED

Cache AI generations: staleness is a feature, not a bug

CONTEXT //: Tweetprenuer went mildly viral. Every card generation cost two metered API calls (tweet retrieval + LLM), and people kept re-running the same usernames.
OPTIONS //: Regenerate every time // cache with a TTL // cache permanently and reuse
DECISION //: Persist every generated result in MongoDB keyed by username. Repeat requests return the stored card instead of erroring or regenerating.
TRADEOFF //: Users who want a fresh card for the same handle don't get one. The result is frozen at first generation.
OUTCOME //: Repeat lookups cost zero API spend during the traffic spike, and the stored results powered a new feature: the public carousel of past cards.
REVISIT //: The original implementation returned an error on duplicate usernames; reusing the stored result came one iteration later and should have been the v1 behavior.

The unit economics of a free AI toy are unforgiving: every click costs real money, and a traffic spike is indistinguishable from a billing incident. When Tweetprenuer started getting shared on X, the same handful of usernames were being submitted over and over, friends checking each other's cards.

The shape of the decision

Regeneration produced a different card each time anyway (LLMs are non-deterministic), so "freshness" wasn't even a coherent promise. That reframed the question: not "how long should the cache live?" but "is there any reason for this cache to expire at all?" For a novelty card generator, the honest answer was no. Permanent cache, zero TTL machinery, no invalidation logic to get wrong.

The compounding win

Stored results turned out to be content. The landing page gained a scrolling carousel of previously generated cards, social proof that doubled as a feature, built entirely from what the cache already held. A cost-control measure became the product's best onboarding tool.

The rule

Before reaching for TTLs and invalidation, ask whether the cached thing has any meaningful freshness requirement at all. For generative output it often doesn't, and a permanent store of past generations is frequently worth more as product surface than as a cache.