Senior Platform Engineer - Data (GoSpotCheck, Denver, CO)
July 2018 - Present
Hecka - built and maintained to automatically transform Postgres data on an hourly/weekly/daily cadence from half a dozen Postgres databases. Uploaded to S3 and Snowflake.
Ultraviolet - Incrementally upload data to Snowflake using custom SQL for transformations. Overwrite functionally transferred over to Hecka once that project was online. Updated to latest Spark version and continued to maintain. Built Go tooling to ease the addition of tables via code generation.
Evaluated FiveTran, Stitch and a variety of other ETL-as-a-service offerings to determine whether or not they would be a good fit for our data pipeline needs.
Wrote GoSpotKafka, a tool for deserializing Protocol Buffers (Protobufs) off of Kafka as JSON and display them to the screen.
Built Spark Structured Streaming examples to produce/consume Protobufs to/from Kafka for two of our highest volume tables.
Demoed Go Cloud Functions while they were in alpha to the Go guild and provided a number of examples.
Senior Platform Engineer/Threat Engineering Manager (ProtectWise, Denver, CO)
August 2015 - July 2018
Broski - Took over and expanded a stream processor which consumes observations and netflows and produces events and observations after applying rules defined using either a custom DSL or classes containing the logic.
Odin - Built a data warehouse built on Spark (2.x), Kafka, Parquet and Amazon EMR. The system processes all of our netflow, observation and event messages and stores the data in S3 in Parquet w/Snappy compression. Currently storing over 12 billion netflows per day. Deployed in production using a combination of shell scripts and AWS EMR (Spark 2.x, EMR 5.x). Data is retrieved via Spark Shell using Spark SQL. As files finish processing their metadata is sent to SQS for consumption by downstream reporting and data science processes.
Canary - Built a netflow, observation, and event emitter consisting of two parts, a loader and a publisher. The loader scans through cages and produces at regular intervals defined in the cages netflows, observations and/or events into redis, which is then read from by publishers to publish messages onto kafka. This allows for the easy prototyping of messages for consumption by the UI. UI developers can start their development much sooner in the process while the platform figures out how to actually produce the messages.
Arbiter - A customer/sensor policy management service written in Scala using Finagle and Thrift. Kicked this project off and continue to work with another developer on maintaining it in production. The project seeks to unify customer/sensor policy information in the platform.
Senior Software Engineer (CJ Affiliate, Santa Monica, CA)
December 2014 - August 2015
Refactored and extended search API service for elasticsearch written in Scala
Built and styled React components (Table, Money, etc.) as well as introduced the usage of off the shelf UI components such as amCharts
Improved and maintained back-end web services, worked on both front-end and back-end, written in React (front-end) and Java/Hibernate/Spring MVC (back-end)
Extensive experience with Test Driven Development (TDD), pair programming, continuous integration
Built a monitoring solution for builds using a Raspberry Pi to provide immediate feedback on the effects of commits on the build and the status of various teams’ branches
Assisted in the migration away from Perforce to Git by writing documentation for developers
Senior Software Engineer (eHarmony, Santa Monica, CA)
July 2013 - November 2014
Lead back-end developer on Elevated, a job searching platform, which consists of a service-oriented architecture written in Java with Dropwizard, Kafka, Elasticsearch and memcached as key technologies
Improved and maintained Scorer Service, a Java based service that computed scores based on predefined models
Built a next-generation Scorer Service that could take arbitrary Protocol Buffer or JSON input and produce scores. By passing the data into the service as opposed to looking the data up from inside the service, the number of requests per second a server could handle rose dramatically
Software Developer (SteelHouse, Culver City, CA)
January 2012 - July 2013
Designed, implemented, and maintained the back-end for a social network (Honeycomb) as a service-oriented architecture using Scala, Finagle and Cassie
Designed and implemented a load balanced CDN origin with image processing capabilities (image data written from three data centers around the globe). Built on Java, Jetty, Cassandra and Hector
Implemented new features for an ad server based on Java and Jetty
Software Developer (NISC, Lake Saint Louis, MO)
May 2009 - December 2011
Support and development programming for accounting and utility billing software written in Java designed around a three-tier architecture with an Oracle database
Mentored new hires on Ant, Subversion, IDEA and the general structure of the Java code
- APIs and Libraries: Guava, Guice, Spark, Spring
- Application Servers/Platforms: Dropwizard, Grizzly/Jersey, Finagle, Node.JS, Spring Boot
- Build Systems: Ant, Cargo, Maven, Sbt
- Cloud Platforms: Amazon Web Services (AWS), Digital Ocean, Google Cloud Platform (GCP)
- Databases/NoSQL Datastores: Elasticsearch, PostgreSQL, Oracle, Cassandra, Redis
- Front-end: React
- IDEs: Atom, IntelliJ IDEA, Visual Studio Code
- Operating Systems: Linux, Mac OS X, Windows
- Programming Languages: Scala, Java, Python, Go, Rust
- Serialization: Protocol Buffers, Thrift (Scrooge)
- Source Control and Documentation: Git, SVN
Missouri University of Science and Technology, Rolla, MO
- MS CS
- August 2006 – May 2009
Missouri University of Science and Technology, Rolla, MO
- BS CS
- August 2002 – May 2006