About on Hitesh Pattanayak

Index and Search Petabytes of Data

Mon, 01 Jan 0001 00:00:00 +0000

Project Overview

Architected and implemented comprehensive end-to-end data pipeline infrastructure for processing M365 workload events using modern data engineering technologies. Built scalable real-time processing systems handling high-volume streaming data with advanced analytics and search capabilities.

Key Achievements

Data Pipeline Infrastructure

Architected end-to-end data pipeline processing M365 workload events using Databricks, Apache Spark, and Delta Lake
Implemented high-volume streaming data processing with Event Hubs and Azure Container Apps
Developed complex Databricks jobs for event processing, indexing, and content enrichment
Implemented SCD2 (Slowly Changing Dimension Type 2) algorithms for historical data tracking and temporal data management

Pipeline Services

Data Ingestion Service: Event processing service publishing to multiple Event Hubs (backup, backfill, retention, recovery points, threats) with tenant resolution and validation
Data Discovery Service: Go-based REST API for browsing indexed metadata and data discovery with filtering capabilities and AI-enhanced natural language search

AI-Powered Semantic Search

Implemented RAG (Retrieval Augmented Generation) pipeline for semantic search over backed-up M365 data
Backup data stored in embedded form in CosmosDB using hybrid vector + keyword search capabilities
Natural language queries translated to structured metadata filters using Azure OpenAI Chat Completions
Filter generation prompt uses system prompt combined with few-shot examples for reliable, structured output
Enables users to search petabytes of backup data using plain English queries without knowledge of underlying schema

AI Developer Tooling

Elastic Dashboard Changelog: Python script that diffs .ndjson Kibana dashboard files (unreadable in standard GitHub diffs) and feeds the structured diff to Anthropic API to generate a human-readable changelog
Security Fix Automation: Local AI skill that ingests Cycode security findings and applies targeted fixes using LLM-assisted code correction with full finding context

Observability & Monitoring

Implemented comprehensive monitoring solutions using Elastic Stack (Elasticsearch, Kibana)
Created custom dashboards for service metrics, performance monitoring, and operational insights
Automated alerting and incident management via Incident.io integration
Designed operational runbooks and failure models for production systems

Infrastructure & Operations

Designed detailed troubleshooting procedures and alert response protocols
Created escalation paths for critical data pipeline components
Infrastructure as Code implementation using Pulumi
Integration with OpenAI for enhanced search capabilities

Technologies Used

Data Processing: Databricks, Apache Spark, Delta Lake, PySpark
Streaming: Azure Event Hubs, real-time processing
Backend Development: Go, Python, TypeScript/Node.js
Databases: CosmosDB, Azure SQL Warehouse
Infrastructure: Azure Container Apps, Pulumi (IaC)
Monitoring: Elasticsearch, Kibana, Elastic Stack
AI/ML: Azure OpenAI, RAG pipeline, CosmosDB hybrid vector search, few-shot prompting, semantic metadata filter generation
DevOps: Incident.io, automated alerting, operational runbooks

Enterprise Authentication, Authorization and User Management

Mon, 01 Jan 0001 00:00:00 +0000

Project Overview

Developed and maintained a comprehensive user management microservice handling authentication, authorization, and user lifecycle management for enterprise-scale multi-tenant SaaS platform. Built to serve thousands of users across multiple organizations with complex authorization requirements.

Key Achievements

Authentication & Authorization System

Built robust REST APIs for user management, role-based access control (RBAC), and organization onboarding using Go, Chi router, and OpenAPI specifications
Integrated Auth0 authentication platform with custom role and permission management
Supported social connections, machine-to-machine authentication, and enterprise identity providers
Developed sophisticated authorization system with hierarchical permissions and policy-based access control

Database & Data Architecture

Designed and implemented CosmosDB data layer with optimized queries for user, role, and organization management
Implemented proper partitioning strategies across multiple containers for sub-second response times
Built efficient data models with optimized partition keys and query patterns

Enterprise Features

Implemented fine-grained workload tenant permissions for Azure and Kubernetes services
Built organization lifecycle management including automated onboarding and user invitations
Developed group management and complete organization deletion workflows
Created service account management with client credentials flow for machine-to-machine authentication

Platform Integration

Integrated Azure Key Vault for secrets management
Built automated organization onboarding with Auth0 integration and custom domain validation
Implemented Microsoft tenant discovery capabilities

Quality & Observability

Implemented comprehensive testing suite including unit tests, integration tests with CosmosDB
Developed table-driven test patterns achieving high code coverage
Established observability patterns using OpenTelemetry and structured logging with Clues
Built comprehensive error handling with context propagation

Technical Highlights

Authorization Matrix: Supporting 100+ permissions across different workload types
Multi-tenant Architecture: Scalable design serving thousands of users across multiple organizations
Testing Excellence: Comprehensive unit and integration testing with GoMock

Technologies Used

Backend: Go, Chi Router, OpenAPI/Swagger code generation
Authentication: Auth0, RBAC, policy-based authorization
Database: Azure CosmosDB
Cloud Services: Azure Key Vault, EventHub
Architecture: Microservices, multi-tenant SaaS, event-driven architecture
Observability: OpenTelemetry, structured logging, comprehensive error handling
Testing: Unit tests, integration tests, table-driven patterns, GoMock
DevOps: Containerization, auto-generated Dockerfiles

Canario (Corso) - Microsoft 365 Backup Solution

Mon, 01 Jan 0001 00:00:00 +0000

Project Overview

Developed an enterprise-grade, open-source data protection engine for Microsoft 365 environments, creating the first comprehensive backup solution addressing critical M365 data protection needs for IT administrators.

Key Technical Achievements

Backend Development

Developed enterprise-grade backup and restore system for Microsoft 365 services (Exchange, OneDrive, SharePoint, Teams)
Implemented CLI interface with comprehensive command structure supporting backup, restore, export, and debug operations
Built modular architecture with clear separation between API layer (/pkg), CLI controller, and internal services
Designed repository abstraction layer supporting multiple storage backends (S3, filesystem)

Microsoft 365 Integration

Integrated with Microsoft Graph API for seamless access to M365 data
Implemented service-specific backup handlers for:
- Exchange: Email backup and restore
- OneDrive: File backup and synchronization
- SharePoint: Site and document library protection
- Teams: Conversation and channel data backup
Built robust authentication and authorization flows for Microsoft 365 environments

Enterprise Features

Developed comprehensive backup lifecycle management (create, list, delete, restore)
Implemented data export functionality with multiple format support
Built advanced debugging tools for troubleshooting backup operations
Created extensive test coverage including end-to-end testing infrastructure
Designed for enterprise scalability and security requirements

Production Readiness

Currently in Beta with active community engagement
Built production-ready architecture with enterprise security standards
Implemented comprehensive error handling and logging
Created CI/CD pipeline with automated dependency management

Technical Skills Demonstrated

Languages: Go (advanced CLI applications, microservices architecture)
Cloud Integration: Microsoft 365, Azure, AWS S3
API Integration: Microsoft Graph API, RESTful services
Architecture: CLI applications, microservices, repository pattern
Testing: Unit testing, end-to-end testing, test-driven development
DevOps: Git, Docker, CI/CD, automated dependency management

Project Impact

Created the first open-source solution addressing critical M365 data protection needs
Enabled IT administrators to have full control over Microsoft 365 data backup strategies
Built for enterprise scalability supporting large-scale M365 deployments
Active community engagement with ongoing development and feature requests

Technologies Used

Core Language: Go
Microsoft Integration: Microsoft Graph API, Microsoft 365 services
Storage Backends: AWS S3, filesystem abstraction
Architecture: CLI-first design, modular microservices
Testing Framework: Comprehensive unit and end-to-end testing
DevOps: Docker containerization, CI/CD pipelines

Kubernetes Infrastructure & Platform Engineering

Mon, 01 Jan 0001 00:00:00 +0000

Project Overview

Built a comprehensive bare metal Kubernetes cluster provisioning platform, developing custom controllers and reconciliation logic to manage cluster lifecycle on bare metal infrastructure.

Key Achievements

Kubernetes Components: Built custom Kubernetes controller, API server, and scheduler components for bare metal cluster management
Reconciliation Logic: Developed state management system to ensure desired cluster configuration and handle cluster drift
Bootstrap Service: Created service deployed on private bootstrap machines in client networks that pulled commands from centralized SaaS platform and reported status
Bare Metal Provisioning: Implemented bare metal readiness detection using DHCP and TFTP protocols
Agent-based Architecture: Developed agents installed on bare metal machines to handle cluster operations
Infrastructure Automation: Leveraged Tinkerbell framework for automated bare metal provisioning workflows

Technologies Used

Container Orchestration: Kubernetes (custom controllers, API server, scheduler)
Programming: Go, Python
Infrastructure: Terraform, bare metal provisioning
Protocols: DHCP, TFTP for network boot and discovery
Provisioning: Tinkerbell framework
Architecture: SaaS platform with distributed agent-based management

High-Performance Financial Trading Platform

Mon, 01 Jan 0001 00:00:00 +0000

Project Overview

Architected and delivered microservices-based cryptocurrency trading platform for Voyager Inc., building scalable backend infrastructure to handle high-frequency trading operations and real-time market data processing.

Key Achievements

Architected cryptocurrency trading platform with microservices
Engineered time-series data management with TimescaleDB
Built event-driven architecture with Apache Kafka for real-time market data
Implemented sub-millisecond query performance for trading analytics
Deployed production-grade observability with 99.9% uptime

Technologies Used

Backend: Golang, gRPC, REST APIs
Database: TimescaleDB, PostgreSQL
Messaging: Apache Kafka
Infrastructure: Kubernetes, Microservices
Monitoring: Production observability

Visual Platform Development

Mon, 01 Jan 0001 00:00:00 +0000

Project Overview

Developed a visual configuration platform for insurance workflows, enabling non-technical users to create and manage complex business processes through an intuitive interface.

Key Achievements

Developed visual configuration platform for insurance workflows
Built responsive frontend interfaces with modern React patterns
Created robust backend APIs for workflow management
Implemented microservices architecture for scalability
Enhanced user experience with drag-and-drop functionality

Technologies Used

Frontend: React, modern JavaScript
Backend: Node.js, REST APIs
Architecture: Microservices
UI/UX: Responsive design, visual workflows
Integration: Insurance domain APIs

Process Automation Solutions

Mon, 01 Jan 0001 00:00:00 +0000

Project Overview

Implemented comprehensive process automation solutions across various business domains, focusing on reducing manual effort and improving operational efficiency.

Key Achievements

Implemented RPA solutions for business process automation
Developed web applications for client requirements
Built automated workflows for business process optimization
Reduced manual processing time by significant margins
Created maintainable and scalable automation frameworks

Technologies Used

Automation: RPA tools and frameworks
Web Development: Full-stack development
Process Design: Business workflow optimization
Integration: Legacy system integration
Quality Assurance: Automated testing

Retrieval Pipelines, Re-Ranking, and Grounding: Building Production RAG

Sun, 29 Mar 2026 21:14:59 -0700

A practical guide for software engineers on building production-grade RAG systems using hybrid retrieval, re-ranking, and grounding techniques to reduce hallucinations and improve answer quality.

Vector Embeddings & Similarity: The Foundation of RAG

Sun, 29 Mar 2026 21:14:51 -0700

A practical deep-dive into vector embeddings and cosine similarity — the mathematical foundation that makes retrieval in RAG systems actually work.

Vector Databases, ANN, and Chunking: Storing Knowledge for Retrieval

Sun, 29 Mar 2026 21:11:47 -0700

A practical guide for software engineers covering how vector databases use Approximate Nearest Neighbor algorithms to search millions of embeddings efficiently, and how to chunk documents intelligently so your RAG pipeline actually retrieves useful, precise context.

Page-Aware AI Chat: Floating Widget and Per-Page Context

Fri, 27 Mar 2026 14:48:36 -0700

A practical walkthrough of adding per-page context awareness to a floating AI chat widget built with Hugo and Netlify Functions, covering layout overrides, slug injection, priority chunk labeling, and the prompt engineering fix that made summarise-this-post actually work.

Building an AI Chat Assistant for a Static Blog — No Vector DB Required

Fri, 27 Mar 2026 12:54:33 -0700

A practical walkthrough of building a conversational AI assistant for a Hugo static site using TF-IDF retrieval over a flat JSON knowledge base — no vector database, no backend server, no embeddings infrastructure required.

TCP/IP, DNS, and Data Transmission Protocols Explained

Sun, 22 Mar 2026 17:22:27 -0700

A practical, code-illustrated guide to how TCP/IP, DNS, and modern data transmission protocols work under the hood — from handshakes and packet routing to WebSockets, gRPC, and QUIC.

AI Prompting Techniques: System Prompts, Few-Shot, CoT, and Structured Output

Sun, 22 Mar 2026 12:32:36 -0700

A practical engineering guide to four core LLM prompting techniques—system prompts, few-shot examples, chain-of-thought reasoning, and structured output—covering real failure modes and production-ready patterns.

The Evolution: Beyond Transformers

Sun, 22 Mar 2026 11:43:37 -0700

A practical walkthrough of how the Transformer architecture evolved from encoder-decoder to decoder-only models, why attention’s quadratic scaling became a hard wall, and how Mamba’s state space machines are being absorbed into hybrid architectures that dominate production today.

Training for Greatness: Speed, BLEU Records, and the Multimodal Vision

Sat, 21 Mar 2026 12:20:48 -0700

A practical deep-dive into how the original Transformer model shattered translation benchmarks, slashed training costs, and laid the architectural foundation for every major LLM that followed.

Inside the Machine: Encoders, Decoders, and Masking

Sat, 21 Mar 2026 12:19:38 -0700

A practical deep-dive into how the Transformer’s encoder and decoder stacks work, covering residual connections, positional encoding, masked self-attention, and cross-attention with code examples throughout.

The End of the RNN Era & The Query, Key, Value Revolution

Sat, 21 Mar 2026 12:06:03 -0700

A practical walkthrough of why RNNs hit a fundamental wall with sequential processing and long-range dependencies, and how the Query-Key-Value attention mechanism solves both problems in one elegant step.

Gradient Descent in Neural Networks: Understanding How Machines Learn

Sun, 20 Oct 2024 00:00:00 +0000

Learn how Gradient Descent helps neural networks improve predictions through gradual optimization of weights and biases. Discover the core mechanics of machine learning.

Understanding Neural Networks: Weights, Biases, and Activations

Sat, 12 Oct 2024 00:00:00 +0000

This article breaks down the key mathematical concepts behind neural networks, including weights, biases, and activations, with an example of handwritten digit recognition.

Orchestrating workflows in the Cloud

Thu, 10 Oct 2024 00:00:00 +0000

AWS Step Functions vs Azure Logic Apps vs Azure Durable Functions vs Temporal

Sub-Word Tokenization: Breaking Words Like a Pro

Wed, 02 Oct 2024 00:00:00 +0000

Take a detour before diving into transformers and explore sub-word tokenization techniques like Byte-Pair Encoding, WordPiece, and Unigram models. Learn how they handle rare words, reduce vocabulary size, and make models more efficient!

N-Grams Uncovered: A Key Component of Large Language Models

Sun, 29 Sep 2024 00:00:00 +0000

Decoding N-Grams: The Heart of Large Language Models

Beginner’s Guide to AI: Diving Into My First AI Blog Post

Sat, 28 Sep 2024 00:00:00 +0000

Why am I writing these AI blogs? Discover my journey into AI and LLMs!

Tricky gRPC load balancing

Wed, 24 May 2023 00:00:00 +0000

Emulates and resolves load balancing problems with gRPC