Justin Miller

Contact Info:

E-mail: justinrmiller@gmail.com

GitHub: justinrmiller

Experience

Senior Platform Engineer - Data (GoSpotCheck, Denver, CO)
July 2018 - Present

  • Hecka - built and maintained to automatically transform Postgres data on an hourly/weekly/daily cadence from half a dozen Postgres databases. Uploaded to S3 and Snowflake.

  • Ultraviolet - Incrementally upload data to Snowflake using custom SQL for transformations. Overwrite functionally transferred over to Hecka once that project was online. Updated to latest Spark version and continued to maintain. Built Go tooling to ease the addition of tables via code generation.

  • Evaluated FiveTran, Stitch and a variety of other ETL-as-a-service offerings to determine whether or not they would be a good fit for our data pipeline needs.

  • Wrote GoSpotKafka, a tool for deserializing Protocol Buffers (Protobufs) off of Kafka as JSON and display them to the screen.

  • Built Spark Structured Streaming examples to produce/consume Protobufs to/from Kafka for two of our highest volume tables.

  • Demoed Go Cloud Functions while they were in alpha to the Go guild and provided a number of examples.

Senior Platform Engineer/Threat Engineering Manager (ProtectWise, Denver, CO)
August 2015 - July 2018

  • Broski - Took over and expanded a stream processor which consumes observations and netflows and produces events and observations after applying rules defined using either a custom DSL or classes containing the logic.

  • Odin - Built a data warehouse built on Spark (2.x), Kafka, Parquet and Amazon EMR. The system processes all of our netflow, observation and event messages and stores the data in S3 in Parquet w/Snappy compression. Currently storing over 12 billion netflows per day. Deployed in production using a combination of shell scripts and AWS EMR (Spark 2.x, EMR 5.x). Data is retrieved via Spark Shell using Spark SQL. As files finish processing their metadata is sent to SQS for consumption by downstream reporting and data science processes.

  • Canary - Built a netflow, observation, and event emitter consisting of two parts, a loader and a publisher. The loader scans through cages and produces at regular intervals defined in the cages netflows, observations and/or events into redis, which is then read from by publishers to publish messages onto kafka. This allows for the easy prototyping of messages for consumption by the UI. UI developers can start their development much sooner in the process while the platform figures out how to actually produce the messages.

  • Arbiter - A customer/sensor policy management service written in Scala using Finagle and Thrift. Kicked this project off and continue to work with another developer on maintaining it in production. The project seeks to unify customer/sensor policy information in the platform.

Senior Software Engineer (CJ Affiliate, Santa Monica, CA)
December 2014 - August 2015

  • Refactored and extended search API service for elasticsearch written in Scala

  • Built and styled React components (Table, Money, etc.) as well as introduced the usage of off the shelf UI components such as amCharts

  • Improved and maintained back-end web services, worked on both front-end and back-end, written in React (front-end) and Java/Hibernate/Spring MVC (back-end)

  • Extensive experience with Test Driven Development (TDD), pair programming, continuous integration

  • Built a monitoring solution for builds using a Raspberry Pi to provide immediate feedback on the effects of commits on the build and the status of various teams’ branches

  • Assisted in the migration away from Perforce to Git by writing documentation for developers

Senior Software Engineer (eHarmony, Santa Monica, CA)
July 2013 - November 2014

  • Lead back-end developer on Elevated, a job searching platform, which consists of a service-oriented architecture written in Java with Dropwizard, Kafka, Elasticsearch and memcached as key technologies

  • Improved and maintained Scorer Service, a Java based service that computed scores based on predefined models

  • Built a next-generation Scorer Service that could take arbitrary Protocol Buffer or JSON input and produce scores. By passing the data into the service as opposed to looking the data up from inside the service, the number of requests per second a server could handle rose dramatically

Software Developer (SteelHouse, Culver City, CA)
January 2012 - July 2013

  • Designed, implemented, and maintained the back-end for a social network (Honeycomb) as a service-oriented architecture using Scala, Finagle and Cassie

  • Designed and implemented a load balanced CDN origin with image processing capabilities (image data written from three data centers around the globe). Built on Java, Jetty, Cassandra and Hector

  • Implemented new features for an ad server based on Java and Jetty

Software Developer (NISC, Lake Saint Louis, MO)
May 2009 - December 2011

  • Support and development programming for accounting and utility billing software written in Java designed around a three-tier architecture with an Oracle database

  • Mentored new hires on Ant, Subversion, IDEA and the general structure of the Java code

Skills

  • APIs and Libraries: Guava, Guice, Spark, Spring
  • Application Servers/Platforms: Dropwizard, Grizzly/Jersey, Finagle, Node.JS, Spring Boot
  • Build Systems: Ant, Cargo, Maven, Sbt
  • Cloud Platforms: Amazon Web Services (AWS), Digital Ocean, Google Cloud Platform (GCP)
  • Databases/NoSQL Datastores: Elasticsearch, PostgreSQL, Oracle, Cassandra, Redis
  • Front-end: React
  • IDEs: Atom, IntelliJ IDEA, Visual Studio Code
  • Operating Systems: Linux, Mac OS X, Windows
  • Programming Languages: Scala, Java, Python, Go, Rust
  • Serialization: Protocol Buffers, Thrift (Scrooge)
  • Source Control and Documentation: Git, SVN

Education

Missouri University of Science and Technology, Rolla, MO

  • MS CS
  • August 2006 – May 2009

Missouri University of Science and Technology, Rolla, MO

  • BS CS
  • August 2002 – May 2006