Talk Abstract: Creating Data Pipelines: Build Framework not Pipelines.
Data pipelines are necessary for the flow of information from its source to its consumers, typically data scientists, analysts and software developers. Managing data flow from many sources is a complex task where the maintenance cost limits scale of being able to build a large reliable data warehouse. This presentation proposes a number of applied data engineering principles that can be used to build robust easily manageable data pipelines and data products.
Bio: Gatis specialises in Big Data technology on-premise and on AWS. He has designed and built data warehouses in small fintechs and large multi-national organisations. He is a senior data engineer at Schroders and has put into effect his data engineering frameworks when building data pipelines. He is also the head of Python and plays an active role in mentoring and teaching data engineers, data analysts and business managers on how to use programming well. He has spoken at PyData events on Data Engineering.