- ACID transaction log for Parquet
- Schema enforcement and evolution
- Time Travel between table versions
- Supports upserts, deletes, and vacuum retention
- Backbone of reliable Lakehouse architecture
- External Tables mapped to MinIO (S3 emulator)
- Parquet reads via PolyBase / OPENROWSET
- Warehouse ↔ Lakehouse bridge for Time Travel
- Executes analytic queries on Delta snapshots
- Automates ETL refresh via SQL Agent or SSIS
- Orchestrates ingestion and Delta snapshot creation
- Automates daily pipeline DAGs for repeatability
- Handles retries, dependencies, and scheduling logic
- Triggers Delta-RS scripts for Time Travel validation
- Integrates with Databricks, MinIO, and Azurite for local orchestration
- Primary compute engine for Delta-RS Time Travel
- Executes MERGE, UPDATE, and snapshot reads
- Integrates with Unity Catalog for PHI control
- Photon engine accelerates vectorized queries
- Bridges notebook POCs to governed production layers
- Columnar storage backing for all lakehouse layers
- Optimized for compression and fast analytical scans
- Foundation for Delta Lake transaction files
- Supports predicate pushdown for efficient filtering
- Compact footprints keep Time Travel snapshots lightweight
- Drives local Delta-RS scripts via PyArrow
- Implements Time Travel query filters
- Bridges CLI automation and VS Code workflows
- Transforms Parquet partitions before snapshot writes
- Integrates with Jupyter and Databricks notebooks
- S3-compatible storage for local Delta tables
- Backs External Tables from SQL Server
- HTTPS + signed URLs for reproducible Time Travel
- Enables versioned object retention and audit
- Mimics enterprise S3 deployments for zero-cloud cost
- Azure Blob emulator for ADLS-style URIs
- Delta-RS snapshots stored & versioned locally
- Tests connectivity for Azure SQL MI Link workflows
- Ensures parity with production Azure pipelines
- HTTPS + signed URLs for reproducible Time Travel; same URIs [you]’d use in cloud
- Foundation for Spark and Delta APIs
- Type-safe batch and ETL patterns
- Bridges notebook POCs to production jobs
- Integrates with Delta-RS and Spark connectors
- Future path for strongly typed data pipelines
- Cloud data warehouse known for elastic compute and secure sharing
- External tables query Parquet or Delta data in object storage
- Complements Delta-RS for cross-platform benchmarking
- Time Travel feature parallels Delta Lake versioning
- Integrates with Power BI, Databricks, and Airflow pipelines
- C++ vectorized query engine inside Databricks Runtime
- Accelerates Delta Lake SQL and DataFrame workloads
- Reduces compute cost with SIMD-optimized execution
- Handles scans, joins, and aggregations near “bare-metal” speed
- Key enabler of near-real-time Time Travel analytics
- Central governance layer across all Databricks workspaces
- RBAC — manage access, masking, and lineage for PHI and PII data.
- Provides fine-grained RBAC down to column and row level
- Tracks lineage for Delta-RS Time Travel datasets
- Foundation for compliant healthcare and finance pipelines
Quarterly Lab Environment Rebuilds
Windows Server 2022 Failover Cluster (WSFC)
SQL Server 2022 Failover Cluster Instances (FCIs) — Availability Groups (AGs)
PowerShell scripting keeps my WSFC and SQL FCI lab rebuilds clean, consistent, and easily reproducible.
Hyper-V & Oracle VirtualBox
openSUSE-Leap & Debian
PostgreSQL
Docker Desktop
Cloud & Modern Data Stack
Azurite — local Azure Blob emulator for Delta Lake-style labs
MinIO — local AWS S3 Blob emulator for Delta Lake-style labs
Databricks SQL & Dashboards (DBX CE)
Unity Catalog & Notebook (DBX CE)
Delta snapshots & lineage tracking
Apache Airflow orchestration
Delta Lake + Time Travel labs
Power BI integration via SQL Server External Tables & Parquet Format using PolyBase and OPENROWSET
Data Engineering & Cost Controls
Local emulation over cloud spend
Performance optimization first
Python for file conversion, merging, and reducing Parquet files via Bernoulli probabilistic sampling
Documentation & Reproducibility
Markdown + Liquid for structured content
GitHub Pages is my portfolio & library
Runbooks & proofs of milestones
SANE / SNAP methodology
Evidence-first engineering
Professional Manifesto
Almost every day, I dive deeper into modern data engineering technologies through my Data Engineering Project Roadmap, where I carry Parquet files from start to finish — using Azurite and MinIO for local blob storage that emulates both Azure and AWS S3, and extending into Delta Lake emulation to demonstrate Time Travel using Python, Databricks SQL, and Medallion Architecture. I also explore Time Travel through DBX SQL and Python within Databricks notebooks, and design Databricks dashboards as well as Power BI dashboards to compare the two platforms — Databricks Dashboards seem to have some catching up, but it works. In parallel, I process Parquet files using SQL Server (via PolyBase) and PostgreSQL to query and analyze data read from blob storage, including spatial functions — publishing interactive NYC Taxi trip maps in my blog posts.
I’m also starting a blog series, HIPAA & SOC 2 Compliance for Data Engineers, where I turn my lab work into practical guardrails: how PHI/PII flows through a Medallion Architecture, how to lock it down with Unity Catalog RBAC, and how to encrypt and prove everything with SQL Server TDE/EKM and PostgreSQL equivalents. It’s a builder’s guide, not a policy memo—real configs, real trade-offs, and clear “how we’d pass an audit” checklists.
Every quarter, I rebuild my lab environments — Hyper-V and Oracle VirtualBox VMs, Windows Server Failover Clusters (WSFC), SQL Server Failover Cluster Instances (FCIs), Always On Availability Groups (AGs), openSUSE Leap, and Debian PostgreSQL nodes — pushing further than the prior quarter across full technology stacks, data lineage, and development cycles. This ensures my skills continue to evolve, my solutions remain articulable, reproducible, and resilient, and that I stay fully prepared to ensure business continuity and meet the challenges of today’s data-driven enterprise landscape.
My GitHub Pages site serves as the journal where I document, articulate, and publish my achievements, guides, and runbooks.
Professional Manifesto
— Inspired by the book, “The Checklist Manifesto”
Thesis: Through continuous repetition, my core strengths, skills and knowdledge evolve, turning multiple platforms, cloud technologies, and programming languages into lasting elements of my Professional Répertoire.
On a recurring quarterly cycle, I push boundaries and stress-test across the full data stack.
Rebuild my environments using Hyper-V and Oracle VirtualBox (WSFC/SQL Server FCI and Linux/PostgreSQL), scripting for repeatability and documenting every step.
Move and transform data with SSIS, Python, and PowerShell—leveraging SSIS for deep ETL and workflow control in the Microsoft stack, while Python and PowerShell provide flexible scripting, integration, and automation.
Orchestrate end-to-end pipelines with Apache Airflow.
Deliver analysis and visuals through Power BI and D3.js.
Expose data as JSON via .NET APIs (REST/GraphQL) to the consumption layer—web clients (Angular, React), Analytics & BI (Power BI), and custom visualizations (D3.js).
Migrate workloads to cloud targets—Azure and AWS S3—staying aligned with evolving technology shifts.
Prioritize performance, transparency, and reproducibility, and always ensure outcomes are explainable and verifiable—almost never quick-and-dirty.1
Document and publish everything from steps to reconstruct to lessons learned in order to accelerate future builds and strengthen architecural knowledge.
HIPAA & SOC 2 Compliance for Data Engineers
Series headlines (working outline)
- Medallion, PHI & PII: Where sensitive data lives; de-identification vs. pseudonymization; safe Bronze patterns
- Unity Catalog, Groups & RBAC: Role design, inheritance, service principals, table/row/column controls, masking
- Delta Lake Controls: Time Travel for forensics, VACUUM/retention policy, lineage, change data capture notes
Encryption Everywhere
- SQL Server: TDE, EKM (HSM/Vault), cell-level encryption, key rotation playbook
- PostgreSQL: pgcrypto, TLS at rest/in transit options, KMS patterns, key rotation
- Secrets & Keys: Centralized key management, rotation cadences, break-glass access
- Network & Perimeter: Private endpoints, firewall allow-lists, workspace/cluster policies
- Data Classification & Tagging: Column tagging, propagation to dashboards and exports
- Row/Column Security & Masking: Patterns that actually scale in lakehouse + SQL engines
- Audit, Logging & Lineage: Unity Catalog events, Delta logs, who-did-what-when; evidence capture
- Monitoring & SIEM: What to forward, useful alerts, anomaly patterns (privilege changes, exfil signals)
- Backups, DR & BCP: RPO/RTO by layer, restore drills, proof artifacts for auditors
- Sharing & Egress Controls: Delta Sharing/UC shares, export gateways, watermarking strategy
- CI/CD & Change Control: Promotion paths, approvals, drift detection, reproducible deployments
- Evidence Packages for Audits: Runbooks, screenshots, configs, and attestation templates mapped to HIPAA §164 and SOC 2 TSC (Security, Availability, Confidentiality)
-
In DevOps and production work, urgent business needs sometimes require quick fixes. The key is to recognize these as exceptions, document them, and follow up with proper regression and stress testing so long-term quality isn’t compromised. ↩