Why is Data Sourcing and Collection a Challenge?

The volume, format and specificity of data required for machine learning projects is problematic, especially when a model requires a large sample of high-variance, high-dimensionality data.
‍
In addition, collecting quality data while navigating privacy and use laws such as GDPR is tricky, especially when dealing with potentially personally identifiable information (PII). Using novel data collection techniques to obtain high-quality, compliant data is imperative to training the next generations of machine learning models.

Aya Data uses three primary strategies to help clients build and prepare high-quality datasets.

Data Collection
Data collection involves retrieving data from a business's internal systems and databases, pulling data from open and public datasets, scraping web data, collecting physical data from the environment, and creating entirely new, unique data.

Our data collection techniques include:

• Collecting appropriate data from a business's pre-existing cloud and relational databases.
• Collecting data from public and open-source datasets.
  
• Using compliant data scraping techniques to extract public data from the internet.
• Collecting image, video or sensor data from the real world.

Data Procurement
Aya Data can procure specialist data from a curated list of partners, providing you with the data you need to build case-specific models, for instance:

• Multi-language call centres
• Anonymous and compliant medical diagnostic images
  
• Textual healthcare datasets
• Agricultural Drone Data

Data Curation
Data Cleaning
Data cleaning requirements vary from case to case. Aya Data will identify and correct errors and inconsistencies. This process involves fixing typos, handling missing values, removing duplicates, and standardizing formats. It’s necessary to convert raw data into an easily consumed format through encoding, scaling, and normalization.

Data Splitting and Sampling
Training sets are split into training, validation, and testing sets. Sampling techniques ensure the model is trained and evaluated on a representative sample.

Data Augmentation and Feature Engineering 
In cases where data is limited, we can augment datasets to artificially increase their size and/or dimensionality and variance. New data can be generated through rotation, flipping, scaling, noise injection, pitch shifting, etc. 

Privacy and Compliance
Aya Data is dedicated to collecting ethical and legally compliant data. When dealing with potentially personally identifiable or sensitive data, we take every step to ensure that any participating individuals provide full consent and usage rights, while fully anonymizing their data. Aya Data are GDPR and SOC 2 compliant and provide dedicated high-security delivery centers for sensitive data. 

Proven Track Record
Aya Data has a proven track record when it comes to collecting high-quality data for our clients and their projects. We understand the ethical and legal nuance of data collection and work alongside partners across Africa to obtain complex use case or domain-specific data.
‍
Data Sourcing Africa
Aya Data's diverse team of data labelers are skilled in the remit of data collection as well as data labeling. We understand how one task connects to another across the ML lifecycle, and work closely with our clients to discern what data they need and the best approach to labeling that data to train a high-performing model. ‍