Data Modelling Data modeling is the process of creating visual representations of information to draw connections between data points that illustrate relationships. Data modeling comes with a goal to produce greater quality, structured and consistent data for running business applications, and achieving consistent outcomes. Data modeling in data science can be termed as a mechanism designed for defining and ordering data for the use and analysis by certain business processes. One of the objectives of modeling in data science is to create the most efficient method of storing information while still providing for complete access and reporting. The modeling in data science can include symbols, text, or diagrams to represent the data and the way that it interrelates. The process of data modeling subsequently increases consistency in naming, semantics, rules, and security, while also improving data analytics, mainly because of the structure that data modeling enforces upon data. Steps Involved in Data Science Modeling Key steps in building data science models are as follows: Data Extraction Steps Involved in Data Science Modeling Key steps in building data science models are as follows: Set the Objectives To start with, you need to have an idea about the problem at hand. This may be the most important and uncertain step. What are the goals of the model? What’s in the scope and outside the scope of the model? Asking the right question will determine what data to collect later. This also determines if the cost to collect the data can be justified by the impact of the model. Also, what are the risk factors known at the beginning of the process? Data Extraction Steps Involved in Data Science Modeling Key steps in building data science models are as follows: Set the Objectives To start with, you need to have an idea about the problem at hand. This may be the most important and uncertain step. What are the goals of the model? What’s in the scope and outside the scope of the model? Asking the right question will determine what data to collect later. This also determines if the cost to collect the data can be justified by the impact of the model. Also, what are the risk factors known at the beginning of the process? Data Extraction Not any data, but the collected chunks of unstructured data should be relevant to the business problem you are about to solve. You would be surprised to know how the World Wide Web proves to be a boon for data discovery. Note that not all data is relevant and updated. To make sense out of the gathered data sets, use web scraping. It is a simplified and automated process for extracting relevant data from the websites. Data Cleaning You are required to clean the data while you are collecting it. The sooner you get rid of the redundancies, the better! Some of the common sources of the data errors include duplicated entries gathered from across many databases and missing values in variables across databases etc. Techniques to eliminate these errors include filtering out the duplicates by referring to the common IDs and filling in the missing data entries with the mean value etc. Exploratory Data Analysis (EDA) Data collection is time-consuming, often iterative, and quite often under-estimated. Data can be messy, and need to be curated in order to start the data exploratory analysis (EDA). Learning the data is a critical part of the research. If you observe missing values, you will research what the right values should be to fill in the missing values. One can build an interactive dashboard and see how your data becomes a mirror to important insights. The picture would be clear and now you would know what is driving the variable features of your business. For example, if it is the pricing attribute, you would know when the price fluctuates and why. Feature Engineering When seeking to get hold of key patterns in business, feature engineering can be deployed. This step can’t be ignored as it forms the prerequisite for finalizing a suitable machine learning algorithm. In short, if the features are strong, the machine learning algorithm would produce awesome results. Modeling/Incorporating Machine Learning Algorithms This makes for one of the most important steps as the machine learning algorithm helps build a workable data model. There are many algorithms to choose from. In the words of data scientists, machine learning is the process of deploying machines for understanding a system or an underlying process and making changes for its improvement. Here are the three types of machine learning methods you need to know about: Supervised Learning: It is based on the outcomes of a similar process in the past. Supervised learning helps in predicting an outcome based on historical patterns. Some of the algorithms for supervised learning include SVMs, Random Forest and Linear Regression etc. Unsupervised Learning: This learning method remains devoid of an existing outcome or pattern. Instead, it focuses on analyzing the connections and the relationships between data elements. An example for an unsupervised learning algorithm is Kmeans clustering. Reinforcement Learning: Reinforcement Learning(RL) is a type of machine learning technique that enables an agent to learn in an interactive environment by trial and error using feedback from its own actions and experiences. Some of the algorithms for RL include Q-Learning and Deep Q Network etc. Model Evaluation Once you are done with picking the right machine learning algorithm, next comes its evaluation. The stability of a model means it can continue to perform over time. The assessment will focus on evaluating (a) the overall fit of the model, (b) the significance of each predictor, and (c) the relationship between the target variable and each predictor. We also want to compare the lift of a newly constructed model over the existing model. You need to validate the algorithm to check whether it produces the desired results for your business. Techniques such as cross-validation or even ROC (Receiver operating characteristic) curve, work well for generalizing the model output for new data. If the model appears to be producing satisfying results, you are all good to go! Model Deployment Deploying machine learning models into production can be done in a wide variety of ways. The simplest form is the batch prediction. You take a dataset, run your model, and output a forecast on a daily or weekly basis. The most common type of prediction is a simple web service. The raw data is transferred via the so-called REST API in real time. This data can be sent as arbitrary JSON which allows complete freedom to provide whatever data is available. Model Monitoring Over time a model will lose its predictability due to many reasons: the business environment may change, the procedure may change, more variables may become available or some variables become obsolete. You will monitor the predictability over time and decide to re-build the model. Tips to Optimize Data Science Modeling To get the best out of the data science models, some of the methods to optimize data science modeling are: Data set Selection Training a good model is a balancing act between generalization and specialization. A model is unlikely to ever get every prediction right because data is noisy, complex, and ambiguous. A model must generalize to handle the variety within data, especially data that it hasn't been trained on. If a model generalizes too much, though, it might underfit the data. The model needs to specialize to learn the complexity of the data. Alternatively, if the model specializes too much, it might overfit the data. Overfitted models learn the intricate local details on the data that they are trained on. When they're presented with new data or out-of-sample data, these local intricacies might not be valid. The ideal is for the model to be a good representation of the data on the whole, and to accept that some data points are outliers that the model will never get right. Performance Optimization/Tuning Objective is to improve efficiency by making changes to the current state of the data model. Essentially, data model performs better after the data model goes through optimization. You might find that your report runs well in test and development environments, but when deployed to production for broader consumption, performance issues arise. From a report user's perspective, poor performance is characterized by report pages that take longer to load and visuals taking more time to update. This poor performance results in a negative user experience. Poor performance is a direct result of a bad data model, bad Data Analysis Expressions (DAX), or the mix of the two. The process of designing a data model for performance can be tedious, and it is often underestimated. However, if you address performance issues during development, with the help of right visualization tools you will get better reporting performance and a more positive user experience. Pull only the Data You Need Wherever you can, limit the data pulled to the only columns and rows you really need for reporting and ETL (Extract, Transform and Load) purposes. There is no need to overload your account with unused data, as it will slow down data processing and all dependent calculations. Hyper Parameter Tuning The main way of tuning data science models is to adjust the model hyperparameters. Hyperparameters are input parameters that are configured before the model starts the learning process. They're called hyperparameters because models also use parameters. However, those parameters are internal to the model and are adjusted by the model during the training process. Many data science libraries use default values for hyperparameters as a best guess. These values might create a reasonable model, but the optimal configuration depends on the data that is being modeled. The only way to work out the optimal configuration is through trial and error. Applications of Data Science One needs to apply the above discussed methods and techniques in the data science toolkit appropriately to the specific analytics problems and evaluate the data that's available to address them. Good data scientists must always be able to understand the nature of the problem at hand and analyze, and see whether is it a clustering task, classification or regression one? and come up with the best algorithmic approach that can yield the expected answers given the characteristics and nature of the data. Data science has already proven to solve some of the complex problems across the wide array of industries like education, healthcare , automobile, e-commerce, agriculture etc. and yet yield improved productivity, smart solutions, improved security and care, business intelligence: Smart Gate Security: Objective is to expedite entry transactions and easily verify repeat visitors at gated community entrances with the help of License Plate Recognition (LPR). This gate security system captures an image of the license plate for each of the guest using the visitor lane to enter. Using LPR, the image is crossreferenced with the database of approved vehicles that are allowed entrance into the community. The gate will automatically open If the vehicle has been to the community before and the license plate is recognized as verified and permanent. ATM Surveillance: Today CCTV cameras deployed in the ATM premises mostly act as a way to provide the footage so that one can analyse the videos when mishap/crime takes place in the premises and may be help with the spotting of culprit. AI with the help of Deep Learning and Computer Vision has changed the way people analytics is done. With these advancements, vehicle analytics is helping in detecting and raising real-time alerts for suspicious activities in the ATM premises such as crowding in ATM, Face Occlusions, Anomaly Detection and Camera Tampering etc. Sentiment Analysis: Sentiment Analysis is contextual mining of the text that identifies and extracts subjective information from the source material, and while monitoring online conversations, it helps a business to get the understanding of the social sentiment of their brand, product or service. The most common text classification is done in sentiment analysis, where texts are classified as positive or negative. Sometimes the problem at hand can be slightly harder, classifying whether a tweet is about an actual disaster happening or not. Not all tweets that contain words associated with disasters are actually about disasters. A tweet such as, "California forests on fire near San Francisco" is a tweet that should be taken into consideration, whereas "California this weekend was on fire, good times in San Francisco" can safely be ignored. Vision based Brand Analytics: Mostly the content created today is visual, either as images, video or both. On a daily basis consumers communicate with images and video. Vision based Brand Analytics is the need of the hour to unlock hidden values from images and videos. With the applications such as Sponsorship monitoring, Ad monitoring and Brand monitoring etc., Brand Analytics delivers key impactful insights in real-time including Sponsorship ROI, competitor analysis and brand visual insights. . Data Cleaning You are required to clean the data while you are collecting it. The sooner you get rid of the redundancies, the better! Some of the common sources of the data errors include duplicated entries gathered from across many databases and missing values in variables across databases etc. Techniques to eliminate these errors include filtering out the duplicates by referring to the common IDs and filling in the missing data entries with the mean value etc. Exploratory Data Analysis (EDA) Data collection is time-consuming, often iterative, and quite often under-estimated. Data can be messy, and need to be curated in order to start the data exploratory analysis (EDA). Learning the data is a critical part of the research. If you observe missing values, you will research what the right values should be to fill in the missing values. One can build an interactive dashboard and see how your data becomes a mirror to important insights. The picture would be clear and now you would know what is driving the variable features of your business. For example, if it is the pricing attribute, you would know when the price fluctuates and why. Feature Engineering When seeking to get hold of key patterns in business, feature engineering can be deployed. This step can’t be ignored as it forms the prerequisite for finalizing a suitable machine learning algorithm. In short, if the features are strong, the machine learning algorithm would produce awesome results. Modeling/Incorporating Machine Learning Algorithms This makes for one of the most important steps as the machine learning algorithm helps build a workable data model. There are many algorithms to choose from. In the words of data scientists, machine learning is the process of deploying machines for understanding a system or an underlying process and making changes for its improvement. Here are the three types of machine learning methods you need to know about: Supervised Learning: It is based on the outcomes of a similar process in the past. Supervised learning helps in predicting an outcome based on historical patterns. Some of the algorithms for supervised learning include SVMs, Random Forest and Linear Regression etc. Unsupervised Learning: This learning method remains devoid of an existing outcome or pattern. Instead, it focuses on analyzing the connections and the relationships between data elements. An example for an unsupervised learning algorithm is K-means clustering. Reinforcement Learning: Reinforcement Learning(RL) is a type of machine learning technique that enables an agent to learn in an interactive environment by trial and error using feedback from its own actions and experiences. Some of the algorithms for RL include Q-Learning and Deep Q Network etc. Model Evaluation Once you are done with picking the right machine learning algorithm, next comes its evaluation. The stability of a model means it can continue to perform over time. The assessment will focus on evaluating (a) the overall fit of the model, (b) the significance of each predictor, and (c) the relationship between the target variable and each predictor. We also want to compare the lift of a newly constructed model over the existing model. You need to validate the algorithm to check whether it produces the desired results for your business. Techniques such as cross-validation or even ROC (Receiver operating characteristic) curve, work well for generalizing the model output for new data. If the model appears to be producing satisfying results, you are all good to go! Model Deployment Deploying machine learning models into production can be done in a wide variety of ways. The simplest form is the batch prediction. You take a dataset, run your model, and output a forecast on a daily or weekly basis. The most common type of prediction is a simple web service. The raw data is transferred via the so-called REST API in real time. This data can be sent as arbitrary JSON which allows complete freedom to provide whatever data is available. Model Monitoring Over time a model will lose its predictability due to many reasons: the business environment may change, the procedure may change, more variables may become available or some variables become obsolete. You will monitor the predictability over time and decide to re-build the model. Tips to Optimize Data Science Modeling To get the best out of the data science models, some of the methods to optimize data science modeling are: Data set Selection Training a good model is a balancing act between generalization and specialization. A model is unlikely to ever get every prediction right because data is noisy, complex, and ambiguous. A model must generalize to handle the variety within data, especially data that it hasn't been trained on. If a model generalizes too much, though, it might underfit the data. The model needs to specialize to learn the complexity of the data. Alternatively, if the model specializes too much, it might overfit the data. Overfitted models learn the intricate local details on the data that they are trained on. When they're presented with new data or out-of-sample data, these local intricacies might not be valid. The ideal is for the model to be a good representation of the data on the whole, and to accept that some data points are outliers that the model will never get right. Performance Optimization/Tuning Objective is to improve efficiency by making changes to the current state of the data model. Essentially, data model performs better after the data model goes through optimization. You might find that your report runs well in test and development environments, but when deployed to production for broader consumption, performance issues arise. From a report user's perspective, poor performance is characterized by report pages that take longer to load and visuals taking more time to update. This poor performance results in a negative user experience. Poor performance is a direct result of a bad data model, bad Data Analysis Expressions (DAX), or the mix of the two. The process of designing a data model for performance can be tedious, and it is often underestimated. However, if you address performance issues during development, with the help of right visualization tools you will get better reporting performance and a more positive user experience. Pull only the Data You Need Wherever you can, limit the data pulled to the only columns and rows you really need for reporting and ETL (Extract, Transform and Load) purposes. There is no need to overload your account with unused data, as it will slow down data processing and all dependent calculations. Hyper Parameter Tuning The main way of tuning data science models is to adjust the model hyperparameters. Hyperparameters are input parameters that are configured before the model starts the learning process. They're called hyperparameters because models also use parameters. However, those parameters are internal to the model and are adjusted by the model during the training process. Many data science libraries use default values for hyperparameters as a best guess. These values might create a reasonable model, but the optimal configuration depends on the data that is being modeled. The only way to work out the optimal configuration is through trial and error. Applications of Data Science One needs to apply the above discussed methods and techniques in the data science toolkit appropriately to the specific analytics problems and evaluate the data that's available to address them. Good data scientists must always be able to understand the nature of the problem at hand and analyze, and see whether is it a clustering task, classification or regression one? and come up with the best algorithmic approach that can yield the expected answers given the characteristics and nature of the data. Data science has already proven to solve some of the complex problems across the wide array of industries like education, healthcare , automobile, e-commerce, agriculture etc. and yet yield improved productivity, smart solutions, improved security and care, business intelligence: Smart Gate Security: Objective is to expedite entry transactions and easily verify repeat visitors at gated community entrances with the help of License Plate Recognition (LPR). This gate security system captures an image of the license plate for each of the guest using the visitor lane to enter. Using LPR, the image is cross-referenced with the database of approved vehicles that are allowed entrance into the community. The gate will automatically open If the vehicle has been to the community before and the license plate is recognized as verified and permanent. ATM Surveillance: Today CCTV cameras deployed in the ATM premises mostly act as a way to provide the footage so that one can analyse the videos when mishap/crime takes place in the premises and may be help with the spotting of culprit. AI with the help of Deep Learning and Computer Vision has changed the way people analytics is done. With these advancements, vehicle analytics is helping in detecting and raising real-time alerts for suspicious activities in the ATM premises such as crowding in ATM, Face Occlusions, Anomaly Detection and Camera Tampering etc. Sentiment Analysis: Sentiment Analysis is contextual mining of the text that identifies and extracts subjective information from the source material, and while monitoring online conversations, it helps a business to get the understanding of the social sentiment of their brand, product or service. The most common text classification is done in sentiment analysis, where texts are classified as positive or negative. Sometimes the problem at hand can be slightly harder, classifying whether a tweet is about an actual disaster happening or not. Not all tweets that contain words associated with disasters are actually about disasters. A tweet such as, "California forests on fire near San Francisco" is a tweet that should be taken into consideration, whereas "California this weekend was on fire, good times in San Francisco" can safely be ignored. Vision based Brand Analytics: Mostly the content created today is visual, either as images, video or both. On a daily basis consumers communicate with images and video. Vision based Brand Analytics is the need of the hour to unlock hidden values from images and videos. With the applications such as Sponsorship monitoring, Ad monitoring and Brand monitoring etc., Brand Analytics delivers key impactful insights in real-time including Sponsorship ROI, competitor analysis and brand visual insights. Data Cleaning You are required to clean the data while you are collecting it. The sooner you get rid of the redundancies, the better! Some of the common sources of the data errors include duplicated entries gathered from across many databases and missing values in variables across databases etc. Techniques to eliminate these errors include filtering out the duplicates by referring to the common IDs and filling in the missing data entries with the mean value etc. Exploratory Data Analysis (EDA) Data collection is time-consuming, often iterative, and quite often under-estimated. Data can be messy, and need to be curated in order to start the data exploratory analysis (EDA). Learning the data is a critical part of the research. If you observe missing values, you will research what the right values should be to fill in the missing values. One can build an interactive dashboard and see how your data becomes a mirror to important insights. The picture would be clear and now you would know what is driving the variable features of your business. For example, if it is the pricing attribute, you would know when the price fluctuates and why. Feature Engineering When seeking to get hold of key patterns in business, feature engineering can be deployed. This step can’t be ignored as it forms the prerequisite for finalizing a suitable machine learning algorithm. In short, if the features are strong, the machine learning algorithm would produce awesome results. Modeling/Incorporating Machine Learning Algorithms This makes for one of the most important steps as the machine learning algorithm helps build a workable data model. There are many algorithms to choose from. In the words of data scientists, machine learning is the process of deploying machines for understanding a system or an underlying process and making changes for its improvement. Here are the three types of machine learning methods you need to know about: Supervised Learning: It is based on the outcomes of a similar process in the past. Supervised learning helps in predicting an outcome based on historical patterns. Some of the algorithms for supervised learning include SVMs, Random Forest and Linear Regression etc. Unsupervised Learning: This learning method remains devoid of an existing outcome or pattern. Instead, it focuses on analyzing the connections and the relationships between data elements. An example for an unsupervised learning algorithm is K-means clustering. Reinforcement Learning: Reinforcement Learning(RL) is a type of machine learning technique that enables an agent to learn in an interactive environment by trial and error using feedback from its own actions and experiences. Some of the algorithms for RL include Q-Learning and Deep Q Network etc. Model Evaluation Once you are done with picking the right machine learning algorithm, next comes its evaluation. The stability of a model means it can continue to perform over time. The assessment will focus on evaluating (a) the overall fit of the model, (b) the significance of each predictor, and (c) the relationship between the target variable and each predictor. We also want to compare the lift of a newly constructed model over the existing model. You need to validate the algorithm to check whether it produces the desired results for your business. Techniques such as cross-validation or even ROC (Receiver operating characteristic) curve, work well for generalizing the model output for new data. If the model appears to be producing satisfying results, you are all good to go! Model Deployment Deploying machine learning models into production can be done in a wide variety of ways. The simplest form is the batch prediction. You take a dataset, run your model, and output a forecast on a daily or weekly basis. The most common type of prediction is a simple web service. The raw data is transferred via the socalled REST API in real time. This data can be sent as arbitrary JSON which allows complete freedom to provide whatever data is available. Model Monitoring Over time a model will lose its predictability due to many reasons: the business environment may change, the procedure may change, more variables may become available or some variables become obsolete. You will monitor the predictability over time and decide to re-build the model. Tips to Optimize Data Science Modeling To get the best out of the data science models, some of the methods to optimize data science modeling are: Data set Selection Training a good model is a balancing act between generalization and specialization. A model is unlikely to ever get every prediction right because data is noisy, complex, and ambiguous. A model must generalize to handle the variety within data, especially data that it hasn't been trained on. If a model generalizes too much, though, it might underfit the data. The model needs to specialize to learn the complexity of the data. Alternatively, if the model specializes too much, it might overfit the data. Overfitted models learn the intricate local details on the data that they are trained on. When they're presented with new data or out-of-sample data, these local intricacies might not be valid. The ideal is for the model to be a good representation of the data on the whole, and to accept that some data points are outliers that the model will never get right. Performance Optimization/Tuning Objective is to improve efficiency by making changes to the current state of the data model. Essentially, data model performs better after the data model goes through optimization. You might find that your report runs well in test and development environments, but when deployed to production for broader consumption, performance issues arise. From a report user's perspective, poor performance is characterized by report pages that take longer to load and visuals taking more time to update. This poor performance results in a negative user experience. Poor performance is a direct result of a bad data model, bad Data Analysis Expressions (DAX), or the mix of the two. The process of designing a data model for performance can be tedious, and it is often underestimated. However, if you address performance issues during development, with the help of right visualization tools you will get better reporting performance and a more positive user experience. Pull only the Data You Need Wherever you can, limit the data pulled to the only columns and rows you really need for reporting and ETL (Extract, Transform and Load) purposes. There is no need to overload your account with unused data, as it will slow down data processing and all dependent calculations. Hyper Parameter Tuning The main way of tuning data science models is to adjust the model hyperparameters. Hyperparameters are input parameters that are configured before the model starts the learning process. They're called hyperparameters because models also use parameters. However, those parameters are internal to the model and are adjusted by the model during the training process. Many data science libraries use default values for hyperparameters as a best guess. These values might create a reasonable model, but the optimal configuration depends on the data that is being modeled. The only way to work out the optimal configuration is through trial and error. Applications of Data Science One needs to apply the above discussed methods and techniques in the data science toolkit appropriately to the specific analytics problems and evaluate the data that's available to address them. Good data scientists must always be able to understand the nature of the problem at hand and analyze, and see whether is it a clustering task, classification or regression one? and come up with the best algorithmic approach that can yield the expected answers given the characteristics and nature of the data. Data science has already proven to solve some of the complex problems across the wide array of industries like education, healthcare , automobile, e-commerce, agriculture etc. and yet yield improved productivity, smart solutions, improved security and care, business intelligence: Smart Gate Security: Objective is to expedite entry transactions and easily verify repeat visitors at gated community entrances with the help of License Plate Recognition (LPR). This gate security system captures an image of the license plate for each of the guest using the visitor lane to enter. Using LPR, the image is cross-referenced with the database of approved vehicles that are allowed entrance into the community. The gate will automatically open If the vehicle has been to the community before and the license plate is recognized as verified and permanent. ATM Surveillance: Today CCTV cameras deployed in the ATM premises mostly act as a way to provide the footage so that one can analyse the videos when mishap/crime takes place in the premises and may be help with the spotting of culprit. AI with the help of Deep Learning and Computer Vision has changed the way people analytics is done. With these advancements, vehicle analytics is helping in detecting and raising real-time alerts for suspicious activities in the ATM premises such as crowding in ATM, Face Occlusions, Anomaly Detection and Camera Tampering etc. Sentiment Analysis: Sentiment Analysis is contextual mining of the text that identifies and extracts subjective information from the source material, and while monitoring online conversations, it helps a business to get the understanding of the social sentiment of their brand, product or service. The most common text classification is done in sentiment analysis, where texts are classified as positive or negative. Sometimes the problem at hand can be slightly harder, classifying whether a tweet is about an actual disaster happening or not. Not all tweets that contain words associated with disasters are actually about disasters. A tweet such as, "California forests on fire near San Francisco" is a tweet that should be taken into consideration, whereas "California this weekend was on fire, good times in San Francisco" can safely be ignored. Vision based Brand Analytics: Mostly the content created today is visual, either as images, video or both. On a daily basis consumers communicate with images and video. Vision based Brand Analytics is the need of the hour to unlock hidden values from images and videos. With the applications such as Sponsorship monitoring, Ad monitoring and Brand monitoring etc., Brand Analytics delivers key impactful insights in real-time including Sponsorship ROI, competitor analysis and brand visual insights.
