GXtract Architecture

This document outlines the architecture of the GXtract MCP server.

System Overview

The high-level architecture of GXtract illustrates how different components interact:

        graph TB
    subgraph "Client"
        VSC[VS Code / Editor]
    end

    subgraph "GXtract MCP Server"
        MCP[MCP Interface<br>stdio/http]
        Server[GXtract Server]
        Cache[Metadata Cache]
        Tools[Tool Implementations]
    end

    subgraph "External Services"
        GXAPI[GroundX API]
    end

    VSC -->|MCP Protocol| MCP
    MCP --> Server
    Server --> Tools
    Tools -->|Query| GXAPI
    Tools -->|Read/Write| Cache
    Cache -.->|Refresh| GXAPI

Detailed Component View

For a more detailed look at how the components interact internally:

        flowchart TD
    CLI[CLI Entry Point<br>main.py] --> Config[Config<br>config.py]
    CLI --> ServerInit[Server Initialization<br>server.py]
    Config --> ServerInit
    
    ServerInit --> FastMCP[FastMCP<br>Instance]
    ServerInit --> Transport[Transport<br>stdio/http]
    ServerInit --> ToolRegistry[Tool<br>Registration]
    
    subgraph "Tool Modules"
        GroundX[GroundX Tools<br>groundx.py]
        CacheMgmt[Cache Management<br>cache_management.py]
        OtherTools[Other Tools<br>...]
    end
    
    ToolRegistry --> GroundX
    ToolRegistry --> CacheMgmt
    ToolRegistry --> OtherTools
    
    GroundX --> CacheModule[Cache<br>cache.py]
    CacheMgmt --> CacheModule
    
    CacheModule --> GXClient[GroundX Client<br>groundx SDK]
    GroundX --> GXClient
    
    Transport --> FastMCP
    FastMCP --> ToolRegistry
    
    GXClient -->|API Calls| GXAPI[GroundX API]

The diagram above shows the main components of the GXtract system:

  1. Client Layer: VS Code or other compatible editors that communicate with GXtract

  2. MCP Interface: Handles MCP protocol communication (via stdio or HTTP)

  3. GXtract Server: Core server component managing tool registration and requests

  4. Tools: Specialized implementations for various GroundX operations

  5. Cache: In-memory storage for GroundX metadata to improve performance

  6. GroundX API: External service providing document understanding capabilities

Core Components

GXtract is built upon several key components and libraries:

  1. Python 3.12: The core programming language.

  2. FastMCP v2: A Python library providing the foundation for creating MCP servers. GXtract uses FastMCP to handle MCP communication (JSON-RPC 2.0) and tool registration.

    • Transports: FastMCP supports different transport layers. GXtract allows selection between:

      • StdioTransport: Communication over standard input/output, typically used for direct integration with client applications like VS Code extensions.

      • HttpTransport: Communication over HTTP, with Server-Sent Events (SSE) for streaming responses. This allows GXtract to act as a web server accessible over a network.

  3. asyncio: Python’s library for asynchronous programming. GXtract is heavily asynchronous to ensure high performance and responsiveness, especially for I/O-bound operations like network requests to GroundX or file system access.

  4. UV: The package manager used for dependency management and virtual environments, configured via pyproject.toml.

  5. GroundX Python SDK (groundx): The official Python library for interacting with the GroundX API. Tools within GXtract use this SDK to perform actions like searching documents, querying, and explaining semantic objects.

Project Structure Overview

(Refer to the Development Guide for a detailed file structure.)

The main application logic resides in the src/gxtract directory:

  • main.py:

    • Parses command-line arguments (using argparse).

    • Loads configuration settings from environment variables and CLI arguments (managed by config.py).

    • Initializes the logging system.

    • Creates an instance of FastMCP from the fastmcp library.

    • Selects and configures the appropriate MCP transport (StdioTransport or HttpTransport).

    • Starts the server and manages its lifecycle, including the periodic cache refresh task.

  • server.py:

    • Defines the run_server function which instantiates FastMCP.

    • Handles discovery and registration of all available MCP tools (e.g., GroundX tools, cache management tools) with the server instance.

    • Uses the tool discovery mechanism from tools/__init__.py to dynamically load all tool modules.

    • Tool methods are typically defined in separate modules within the tools/ directory.

  • config.py:

    • Defines the ServerConfig dataclass to hold all server configuration (e.g., log level, API keys, transport settings, cache TTL).

    • Provides logic to load these settings from environment variables and merge them with CLI arguments.

  • cache.py:

    • Implements the caching mechanism for GroundX metadata (e.g., project IDs, bucket IDs).

    • Uses an in-memory dictionary for the cache.

    • Provides structured data types (TypedDict) for cache entries and statistics.

    • Includes logic for Time-To-Live (TTL) based cache entries and a mechanism to refresh the cache, ensuring data doesn’t become too stale while minimizing API calls to GroundX.

  • tools/ directory:

    • Contains individual Python modules for each logical group of tools.

    • groundx.py: Implements tools for interacting with the GroundX API (e.g., searchDocuments, queryDocument, explainSemanticObject). These tools use the groundx SDK and interact with the cache.py module to retrieve necessary IDs.

    • cache_management.py: Implements tools for inspecting and managing the cache (e.g., getCacheStatistics, listCachedResources, refreshMetadataCache).

    • __init__.py: Provides functions for discovering and loading tools dynamically.

    • _example.py: Contains an example tool implementation that can be used as a template for new tools.

Workflow

  1. Server Startup (main.py):

    • Application starts, CLI arguments are parsed.

    • Configuration is loaded (environment variables + CLI args).

    • Logging is initialized.

    • The GroundX metadata cache is initialized, and an initial population attempt might occur.

    • A background task for periodic cache refresh is started if configured.

    • A FastMCP instance is created.

    • Tools are discovered and registered with the server.

    • The chosen transport (STDIO or HTTP) is initialized and started.

    • The server begins listening for incoming MCP requests.

  2. MCP Request Handling (FastMCP & server.py):

    • The transport layer receives an incoming request (JSON-RPC 2.0 format).

    • FastMCP parses the request and identifies the target tool method (e.g., groundx/searchDocuments).

    • FastMCP invokes the corresponding registered handler function.

  3. Tool Execution (e.g., tools/groundx.py):

    • The tool handler function receives parameters from the request.

    • It may interact with the cache.py to get cached data (like GroundX project/bucket IDs).

    • It uses the groundx Python SDK to make API calls to the GroundX service, using the configured API key.

    • It processes the response from GroundX.

    • It returns a result (or an error) that FastMCP will format into a JSON-RPC 2.0 response.

  4. MCP Response (FastMCP):

    • FastMCP sends the JSON-RPC response back to the client via the active transport.

  5. Cache Refresh (cache.py):

    • Periodically or upon manual request, the cache refresh function is called.

    • This function makes API calls to GroundX (e.g., to list projects and buckets) and updates the in-memory cache.

    • This ensures that tools have access to relatively up-to-date metadata without querying GroundX on every request.

Type Annotations

GXtract makes extensive use of Python’s type annotation features:

  • Using TypedDict to define structured response and parameter types

  • Leveraging Python 3.12’s pipe operator (|) for Union types (e.g., str | None)

  • Importing from collections.abc instead of collections for abstract types

  • Strict type checking with mypy to ensure type safety

Error Handling

  • FastMCP provides mechanisms for returning JSON-RPC errors.

  • Tools should catch exceptions (e.g., from the groundx SDK or internal errors) and can return appropriate MCP error responses.

  • Configuration errors (e.g., missing API key) are typically checked at startup, and the server may exit with an informative message.

  • Logging is used to record errors and operational details.

Scalability and Maintainability

  • Modularity: Tools are separated into their own modules, making it easier to add new tools or modify existing ones.

  • Configuration: Centralized configuration in config.py and main.py makes it easy to manage settings.

  • Asynchronous Design: Improves scalability by allowing efficient handling of concurrent operations.

  • Caching: Reduces load on external services and improves response times for common metadata lookups.

  • Dependency Management: UV and pyproject.toml ensure reproducible builds and clear dependency tracking.

This architecture aims to provide a robust, performant, and maintainable MCP server for interacting with GroundX and potentially other future services.