EMPLOYMENT

The key duties of the position include The Role The Data Engineering and Operations Directorate is looking to fill an APS 5 vacancy for a Data Engineer. The position will be responsible for data engineering and data operations to load data from Defence data source systems and deliver complex data products on a modern data platform leveraging a data lakehouse architecture. Duties will include the following: Leveraging your experience and training to extract, load and transform data using key open source data engineering technologies; Apache NiFi, Spark and Airflow; Development will be done on the Cloudera Data Platform and involve some data (and metadata) integration work on Neo4j, Alation and OpenText InfoArchive. Provide direct data engineering operations support and assistance for other Defence Groups and Services data specialists to develop and publish their domain specific data products within a multi-tenanted operating model within the Defence strategic data platform. Work with other data specialists to extract, load and transform Defence key data sources and integrate data to create data products of value to the Defence mission. Assist with the implementation of the Data Division Data Engineering standards, practices and patterns and Data Access and Security framework as implemented on the strategic data platform. This role will allow you to build your technical expertise through the mentorship from senior data engineering specialists and from the multi-disciplined data and analytics coaching available from the specialists throughout Data Division and from within the Defence data domains you will be supporting and enabling. About our Team The Data Engineering and Operations team have an exciting opportunity to play a key role in Data Division's implementation of the Defence data strategy. The team is assisting Defence to transition from traditional, often siloed, data warehousing to a federated modern data platform where data products are discoverable and readily shared between data domains. There are three challenging and highly technical roles for the Data Engineering and Operations Directorate. The operations component of the Directorate's mission is the administration, configuration and access management for each of the strategic data platform applications. The second vital role is to provide assurance to Data Custodians by effectively implementing Data Division's data access and security framework. The final role is the provision of data engineering services on the strategic data platform and this includes the extraction, loading and transformation of data and leading the management of best practice data engineering standards, patterns and practice. Our Ideal Candidate The successful candidate will have the proven ability or foundational knowledge and aptitude to: Communicate effectively, both verbally and in writing, including drafting technical documentation and providing input into the Directorate's management and status reports; Support the selection and development of data engineering methods, tools and techniques under the Software Development Lifecycle (SDLC) for on premise and cloud based data engineering solutions; Develop data solutions for advanced structure and data storing for usage in analytics, machine learning, data mining and sharing with applications; Support the design, development and implementation of structured and unstructured data pipelines ensuring data is fit for purpose; Contribute to the integration, consolidation, curation, migration, and transformation of data, aligning to relevant guidelines; Collaborate with other data professionals and business stakeholders to determine the required data sets and data structures; Identify process improvements, apply data quality frameworks and assist the governance of data engineering design patterns and practices;