Package vs Service: Architecting Query Transformation for LLMs

Recently, my team had an interesting architectural debate. We are building a query transformation feature to help other teams build great LLM-based applications. Query transformation optimizes an initial user input for efficient information retrieval or further processing. For example, rewriting a query for clarification (who is "he"?), decomposing it into smaller pieces, or utilizing conversation history to re-examine the context.

The core dilemma we faced was whether to implement this feature as a library or a service. Here are the main trade-offs:

• The Service Approach: Deploying this as an independent service means instant bug fixes and updates across all clients. However, it introduces network hops, requires API management, and demands active effort for discovery.

• The Package Approach: An embedded package integrates smoothly and remains incredibly lightweight. The downside is maintainability, i.e., fixing a bug requires notifying every single user to update their version.

The question of where the responsibility lies for bug fixes and logic updates shifted the balance. If a feature relies on custom business logic (e.g., specific prompt structuring), it belongs on the client side. A centralized server should not try to trace every unique business requirement. Instead, our service exposes minimal, generalized utilities.

The final tipping point toward the service approach was the requirement for programming language interoperability. In an organization with long-running legacy systems, a centralized service seamlessly bridges the gap across different languages, allowing us to manage operational updates centrally while clients maintain their unique logic.

Did you implement a query transformation service? What were your considerations for this case?