Every clever shortcut is tomorrow’s bottleneck
When AI model get faster and better, any implementation details are just a form of premature optimisation. They constrain AI to our understanding of a domain and limit its ability to operate independently. Tell the system what to look for and what doing that well means, and then get out of the way.
When building AI products, the goal is to be able to swap out all of your models and retrieval techniques. Everything (other than the system's core task definitions) should be flexible because those elements will improve faster than your domain understanding.
According to Omar Khattab, the guy who invented DSPy, your core task definitions (at the application layer) include:
Your natural language definitions of what your system handles - The most essential parts of your prompts. Separate out all model appeasement chaff like "I will tip you $1000 if you give me the best answer."
Your Evaluations, or your definition of the outcome/s you actually care about.
Your Code - tools, any essential control flow, or function composition. If your system can charge someone's bank account, you don't want any interpretation on what "a bank charge" means.
These are your core operators.
The fundamentals – The evergreen components that define your AI system. This is the answer to how we work with AI systems that will eventually outperform any type of prompt optimisation or specialised domain knowledge.
By keeping things as decoupled as possible and maintaining a clear separation of concerns, you set your AI engineering efforts up for future success. This allows you to maximise AI's intelligence rather than limiting it with a premature understanding of your own domain.