Sr. Data Engineer - Analytics & Data Science
Location: New York, NY
Job Type: Engineering
IQ Workforce is a leading recruiting firm for the engineering, analytics, and data science communities.
One of our new clients is a SaaS marketplace start-up that provides licensed cannabis dispensaries the ability to order from their favorite brands, as well as a suite of software tools for those brands to manage their operations and scale. With 4,400+ dispensaries and 1,400+ leading brands in 25 states and territories, the company is setting the industry standard for how cannabis brands and retailers work together.
The company is currently seeking a Sr Data Engineer to join their Data & Analytics team. As a senior member of the data engineering and analytics team, you will be able to have a direct impact on how the company harnesses its first-party data from various sources to generate business value. You are deeply passionate about organizing and managing data. You believe and understand the value that powerful reporting and analytics can drive for the business. The ideal candidate will have a structured and detail-oriented approach to solving problems using a diverse technical toolkit. The ideal candidate should be personable, efficient, rooted in an experimentative and fact-based mindset. Bringing people along, communicating and gathering feedback on plans with internal and external stakeholders and collaborating cross-functionally should come easily to the candidate.
Assist in building a high performing data platform which will power various reporting and analytics applications
Responsible for building and maintaining processes for ingestion of data to the data lake
Implementing ELT / ETL procedures to pipe data from ingestion to data warehouse
Maintain the data dictionary and schema of the data warehouse and data marts serving all company functions and business divisions
Partner with data scientists and business analysts to modify, add, remove fields to relevant schemas and tables
Partner with DevOps in Core Engineering for provisioning and standing up database clusters
Putting in place process for monitoring health of database infrastructure
Addressing data quality issues originating at source and working with vendors on solving quality issues and simplifying ingestion processes
Handle large volumes of data and integrate our platform with a range of internal and external systems.
Be a thought leader; understand new tech and recommend how it can be applied to data management
Be a technical expert and enable other members on the team by providing mentoring and code reviews when required
Troubleshoot and diagnose issues quickly and effectively when they arise, bringing calm and rationality to tense situations
Maintain and evaluate quality of documentation, code, and business logic for data management
Expertise and hands-on experience building a modern data stack using AWS especially s3 and Redshift;
Expertise in developing and maintaining relational database structures and relationships;
Experience monitoring and managing Redshift db Clusters;
Comfortable in diagramming and documenting processes, relational structures using tools like Visio, Lucidchart, Confluence
Expertise writing processing jobs to ingest a variety of structured and unstructured data received from various sources & formats such as Rest APIs, Flat Files, Logs
Expert level skills in using Python for data processing coupled with AWS offerings like Lambda, Fargate, Kubernetes
Expert level skills in writing & managing optimized SQL for creating, updating and querying source of truth tables
Hands-on experience with deployment using CI/CD, Docker; experience with ECS good to have
Hands-on experience to with DAG-based workflow orchestration frameworks like dbt, Luigi, Airflow, AWS Pipeline
Experienced in working within an integration environment with testers to ensure end to end performance and resilience can be achieved
Well-versed in version control systems (Git)
Experience working in a team with data scientists and analysts as clients is a plus
Experience with platforming ML & using Spark is a plus but not required
Comfortable working in a fast-paced growth business with many collaborators and quickly evolving business needs