The Data Science Hierarchy of Needs
Collect
The collection level is where it all begins — it involves locating data from various sources such as IoT devices, external services, on-premises applications and SaaS platforms. This stage is crucial for breaking down silos and provisioning the infrastructure necessary for data integration and management activities.
Move/Store
The second level is dedicated to the production of reliable data pipelines where governance and compliance needs are met. At this stage, it’s vital to implement best practices for standardizing data. The data is then stored in a data warehouse, data lake, lakehouse or other data models. Executing this level correctly enables data lineage tracking, which is key for building data trust.
Explore/Transform
The third level is where data analysts and scientists become more hands-on. Data integration at this level often involves cleaning and pre-processing data from various sources for consistency and compatibility. This stage is essential for maintaining the quality and accuracy of subsequent analyses.
Aggregate/Label
The fourth level involves bringing data together from various sources to get a complete understanding of the problems you’re trying to solve. This stage is important for tasks like aggregation where you need to combine data to create meaningful summaries. With a comprehensive view of data, enhanced data quality and easy data labeling, you can get the most value out of your data science projects.