Alex Chen

Lead AI platform infrastructure for enterprise ML serving, responsible for model deployment pipelines, inference optimization, and platform observability across 50+ production models.

Hybrid Retrieval Platform with Reranking

• Built a production retrieval system combining dense embeddings, sparse lexical search, and learned reranking for enterprise knowledge base queries. System handles 10M+ documents with sub-100ms P99 latency.
• Microservices on Kubernetes with dedicated inference nodes, Redis cache tier, and PostgreSQL for metadata. Async indexing pipeline using Kafka for document updates.
• Designed architecture, implemented reranking service, established evaluation datasets with regression testing, and built tracing pipeline for retrieval quality monitoring.
• Technical Stack: Hybrid retrieval (dense + sparse) Cross-encoder reranking Redis caching Kafka streaming Kubernetes
• Challenges: Latency Reduction: Reduced P99 query latency from 450ms to 85ms through multi-tier caching and query optimization.
• Challenges: Quality Metrics: Established NDCG@10 evaluation with automated regression tests preventing 3 production degradations.

Model Serving Infrastructure with Cost-Latency Routing

• Developed model serving platform supporting dynamic routing between on-demand and batch inference, with automatic fallback and cost-aware scheduling across AWS and GCP.
• Custom inference gateway in Rust, Envoy proxy layer, Prometheus/Grafana observability, and Terraform-managed multi-cloud infrastructure.
• Built inference gateway, implemented routing logic with cost and latency budgets, established SLO monitoring and automated rollback procedures.
• Technical Stack: Rust Envoy Prometheus Terraform Multi-cloud routing
• Challenges: Cost Optimization: Reduced inference spend by 35% through intelligent batching and spot instance utilization.
• Challenges: Reliability: Achieved 99.99% availability with automatic fallback and circuit breaker patterns.

Developed ML pipeline infrastructure and model deployment systems for computer vision and NLP products serving 2M+ daily active users.

Structured Output API with Schema Validation

• Built API service guaranteeing schema-constrained structured outputs from LLMs for downstream consumer applications, with runtime validation and fallback chains.
• FastAPI service with Pydantic validation, custom output parsers, and Redis for response caching. Deployed on AWS ECS with auto-scaling.
• Implemented core validation engine, designed fallback chains for schema failures, and built monitoring for output quality and constraint satisfaction rates.
• Technical Stack: FastAPI Pydantic Schema-constrained generation Redis caching AWS ECS
• Challenges: Constraint Satisfaction: Achieved 99.7% schema validity rate with automatic fallback to stricter generation parameters.
• Challenges: Throughput: Scaled to 5K RPS with caching and connection pooling optimizations.

MCP Integration Platform

• Implemented Model Context Protocol integrations with scoped permissions for enterprise tool use, enabling secure LLM access to internal APIs and databases.
• Go-based MCP server with OAuth2 scope enforcement, audit logging, and least-privilege access controls. Integrated with existing identity provider.
• Developed MCP server implementation, designed permission scoping model, and built compliance audit pipeline for tool invocation traces.
• Technical Stack: Go MCP protocol OAuth2 Audit logging Zero-trust architecture
• Challenges: Security: Zero privilege escalation incidents with granular scope enforcement and comprehensive audit trails.