Module I: Data Engineering Basics
Basic Concepts of Different Types of Data Structures; File Formats, and Sources of Data;
Different Types of Data Repositories; The ETL Process and Big Data Platforms; Process and
Tools for Gathering Data; Wrangling Data; Mining, Analysing Data and Visualising Data;
Understanding of Gathering, Wrangling, Mining, Analysing, and Visualising Data
Different Types of Data, File formats and Process
Raw facts, statistics, or information are referred to as data. Data may be gathered through
databases, files, sensors, web applications, or other data streams. Based on its structure and
properties, data can be divided into many types, such as structured, unstructured, semistructured, time series, geospatial and big data.
Data is represented and stored using file formats so that it may be read, written and processed
by software applications simply. For diverse sorts of data, many file formats are employed and
each one has advantages and use cases.
The process of converting unprocessed data into useful information is known as data
processing. Data cleaning, integration, analysis and visualisation are all aspects in the process
that produce crucial insights for decision-making.
In other words:
·
Data: A collection of information gathered through observations, measurements, research,
or analysis is called data.
·
File format: A file format is a common method of encoding data for storage in a computer
file. It describes the use of bits to encrypt data on a digital storage medium. There are both
proprietary and open-source file formats.
·
Data Process: Any organisation cannot benefit from data in its raw form. Data processing
is the process of taking raw data and turning it into information that can be used. An
organisation's team of data engineers typically performs it in a step-by-step manner. The
unprocessed data is gathered, sorted, processed, examined and stored before being presented
in a readable format.
Basic Concepts of Different Types of Data Structures
A data structure is an intelligently created format for organising, processing, accessing and
storing data. Data structures might be basic or complicated, but they are all designed to
organise data for a specific purpose. Data structures make it easy for users to access the data
they require and make proper use of it. Data structures frame the organisation of information in
a way that is easier for machines and people to understand.
In computer science and computer programming, a data structure can be selected or made to
store data such that it can be used in a variety of ways. In some cases, the basic operations of
the algorithm and the design of the data structure are tightly intertwined. Each data structure
contains information about the data's values, connections and, on rare occasions, functions that
can be used to change the data.
For instance, in an object-oriented programming language, the class description includes a
reference to the data structure and any accompanying methods. These functions are not
regarded as being a part of the data structure, despite the fact that they might be created to
work with the data structure in non-object oriented languages.
Why are data structures important?
Standard fundamental data types found in most computer programming languages, like integers
or floating-point numbers, are frequently insufficient to express the logical intent for data
processing and application. However, program s that receive, manipulate and output information
need to be aware of how data should be organised in order to make processing easier. By
logically combining the data pieces, data structures make it possible for the efficient use,
persistence and sharing of data. They provide a formal model that describes how the data
elements should be arranged.
The basis for increasingly complicated applications is data structures. Data elements are
grouped into logical units that represent abstract data kinds pertinent to the algorithm or
application in order to construct them. An example of an abstract data type is "customer name".
It consists of character strings for the "first name," "middle name," and "last name."
The optimum data structure should be chosen for each activity in addition to utilising data
structures. A poor choice of data structure may result in slow runtimes or unresponsive code.
The following five considerations should be made while selecting a data structure:
·
What kind of data will be kept on file?
·
What purpose will that data serve?
·
Where should newly created data persist or be stored?
·
What's the most effective approach to arrange the data?
·
What factors should be taken into account while managing memory and storage
reservations?
Classification of Data Structure:
Data structures are utilised in a wide range of applications in our daily lives. Different logical and
mathematical problems are addressed by using a range of various data structures. By using
data structure, a very large amount of data can be arranged and processed in a very short
period of time. Let's look at a few data structures that are used in different situations.
1.
Linear data structure:A linear data structure is one in which the data items are arranged
sequentially or linearly, with connections between each element and its neighbours to the right
and left of it.
There are several examples of linear data structures, including the array, stack, queue,
linked list and others.
Static data structure:A static data structure's memory footprint is fixed. The elements of a
static data structure are easier to access. One example of this data structure is an array.
2.
Dynamic data structure:The dynamic data structure's size is not fixed. Given the
complexity of the code in terms of memory (and space), it is possible for it to be randomly
updated while the program is running, which is considered efficient.
Examples of this type of data structure include queue and stack.
Non-linear data structure: Data pieces are not arranged linearly or sequentially in non-linear
data structures. An entire non-linear data structure cannot be dissected in a single operation.
Non-linear data structures include things like trees and graphs.
Need of Data structure :
The organisation of the data and the construction of the algorithm are related. The data display
needs to be easy to understand for both the developer and the user to implement the action
efficiently.
A straightforward way for controlling, retrieving, storing and organising data is provided by data
structures.
Here is a list of the needs for data structure.
·
It is simple to modify data structures.
·
It takes much less time.
·
Save memory for storage.
·
Data visualisation is simple.
·
Access to the sizable database is simple.
Arrays:
A set of components are retained close to one another in memory as part of a linear data
structure called an array. The idea is to store items together for easy access. It allows for the
quick processing of a large amount of data. The subscript 0 denotes the first element of the
array. A variety of operations can be performed on an array, such as searching, sorting,
inserting, traversing, reversing and deleting.
Characteristics of an Array:
·
An array has the following traits among others:
·
The index-based data structure used by arrays makes it simpler to rapidly identify each
element in an array.
·
The array can be effectively used if a user needs to store several values of the same data
type.
·
By storing information in a two-dimensional array, an array may also manage complicated
data structures.
·
Arrays are also used to implement other data structures like stacks, queues, heaps, hash
tables, etc.
·
In an array, searching is a pretty simple task.
Operations performed on array:
·
Initialisation: A value can be assigned to an array either when it is declared or at a later
time using an assignment statement.
·
Accessing elements: An array's items are accessed using their index, which ranges from 0
to the array size minus one and starts at 0.
·
Binary search or linear search techniques can be used to search an array for a specific
element.
·
Sorting array elements: Algorithms like bubble sort, insertion sort and rapid sort can be
used to arrange array elements in either ascending or descending order.
·
A single element can be added to an array at a precise point, but doing so requires
relocating other components in the array, which can take some time.
·
Elements can be removed from an array by rearranging the elements that follow them to fill
the empty space.
·
Element updating: An array's elements can be changed or updated by giving a new value
to a particular index.
·
Element traversal: An array's elements can be visited one at a time as they are traversed
in order.
·
Some of the most frequent operations on arrays are those listed above. Based on the
specifications of the problem and the programming language employed, the precise operations
and methods performed may change.
Applications of Array in Real Life:
·
It is common practice to store data for mathematical computations in an array.
·
In image processing, it is employed.
·
Additionally, record management employs it.
·
Book pages are another actual illustration of an array.
·
Additionally, it is utilised in order boxes.
Linked list:
The components of a linked list, a linear data structure, are not stored at successive positions in
memory. Pointers are used to connect the items in a linked list, as seen in the diagram below:
Types of linked lists:
·
Singly-linked list
·
Doubly linked list
·
Circular linked list
·
Doubly circular linked list
Characteristics of a Linked list:
A linked list contains the following qualities among others:
·
Links are stored in additional memory in a linked list.
·
The size of the elements does not need to be known when the linked list is being
initialised.
·
Stacks, queues, graphs, etc. are implemented using linked lists.
·
The Head is the node at the top of the linked list.
·
The final node's next pointer is always set to NULL.
·
Insertion and deletion are simple in a linked list.
·
Each node in a linked list contains a pointer or link that points to the address of the node
after it.
·
Linked lists are easily capable of changing their size at any time.
Operations performed on Linked list:
Each node in a linked list has a value and a reference to the node after it in a linear data
structure. Here are a few frequent operations on linked lists:
·
Initialisation: The first node in a linked list can be referenced by constructing a head node.
Each succeeding node has a value and a link to the one after it.
·
Inserting elements:The linked list's head, tail, or a precise location can all be used to insert
elements.
·
Deleting elements:Elements can be removed from a linked list by changing the preceding
node's reference to point to the subsequent node, which essentially removes the current node
from the list.
·
Finding elements: To find a specific element in a linked list, start at the head node and
work your way down the list by following the references to the following nodes.
·
Updating elements: A linked list's elements can be modified by changing the value of a
particular node.
·
Traversing elements: A linked list's items can be navigated by beginning at the head node
and moving down the references to the subsequent nodes until the end of the list is reached.
·
Reversing a linked list: A linked list can be reversed by changing each node's references
such that they now point to the prior node rather than the next node.
Some of the most frequent operations on linked lists are those mentioned above. Based on the
specifications of the problem and the programming language employed, the precise operations
and methods performed may change.
Real-Life Applications of a Linked list:
·
Round-Robin scheduling keeps track of the turn in multiplayer games using a linked list.
·
The picture viewer makes advantage of it. The previous and next buttons can be used to
access the linked previous and next images.
·
Songs on a playlist are connected to the songs before and after them.
Stack: A stack is a linear data structure that preserves the order of operations. Last in, first out,
or LIFO, determines the order. Only one end of the process can be used for data entry and
retrieval. Inserting and withdrawing data from a stack is also referred to as push and pop action.
Sorting, removing the centre piece from a stack and stack reversal using recursion are just a
few of the many operations that may be performed on a stack.
Features of a Stack: A stack has the following features, among others:
·
Numerous algorithms, like the Tower of Hanoi, tree traversal, recursion, etc., utilise the
stack.
·
Stack can be constructed using a linked list or an array.
·
It adheres to the principle of "Last In First Out," meaning that a piece put first will appear
last and vice versa.
·
The top of the stack is where the insertion and deletion are done, at one end of the stack.
Stack overflow occurs when the allotted amount of space for the stack is reached and further
pieces are added.
Operation performed on stack
The Last-In-First-Out (LIFO) concept is implemented via a stack, which is a linear data
structure. Here are a few frequent actions carried out on stacks:
·
it.
Push: A new element can be added to the top of the stack by pushing other elements onto
·
Pop: The top element on the stack can be removed by performing a pop operation, which
also removes the element that was most recently added onto the stack.
·
Peek: Using a peek operation, the top element can be examined without being taken out of
the stack.
·
IsEmpty: The existence of the stack can be checked using this function.
·
Size: A size operation can be used to figure out how many elements are in the stack.
These are a few of the most frequent actions carried out on stacks. Based on the specifications
of the problem and the programming language employed, the precise operations and methods
performed may change. Stacks are frequently employed in a variety of applications, including
the evaluation of expressions and the implementation of function call stacks in computer
programs.
Real-Life Applications of Stack:
·
The layer of dinner plates stacked one on top of the other is an actual stack. You can pull a
dish out of the stack and place it on top of the others. But this particular plate was the most
recent to be added to the collection. You must take out every plate above the plate at the
bottom of the stack in order to get to it.
·
Browsers keep track of previously visited websites using stack data structures.
·
Mobile call logs use stack data structures as well.
Queue:
A linear data structure called a stack maintains the order of operations. LIFO, or last in, first out,
establishes the hierarchy. Data entry and retrieval are limited to one end of the process. Push
and pop operation is another name for adding and removing data from a stack. The numerous
operations that can be carried out on a stack include sorting, removing the centre component
from a stack and stack reversal via recursion.
Characteristics of a Queue:
The following are some of the characteristics of the line:
·
A FIFO (First In First Out) structure describes the queue.
·
The elements added before the new element in the queue must all be removed in order to
remove the last member of the queue.
·
An ordered list of items with related data types is referred to as a queue.
Operation performed on queue:
First-In-First-Out (FIFO) is a linear data structure that is implemented by queues. Here are a few
frequent actions carried out in queues:
·
Enqueue: A new element can be added to the end of the queue by pushing an existing
element to the back.
·
Dequeue: By carrying out a dequeue action, the front element can be taken out of the
queue, thereby taking out the first element that was added.
·
Peek: Using a peek operation, the front element can be examined without being taken out
of the queue.
·
IsEmpty: The queue's status can be checked to see if it's empty.
·
Size: A size operation can be used to calculate how many elements are in the queue.
These are some of the actions taken on queues most frequently. Based on the specifications of
the problem and the programming language employed, the precise operations and methods
performed may change. Applications where queues are frequently utilised include handling
communication between processes, scheduling tasks and many others.
Applications of Queue in Real Life:
·
A single-lane, one-way road where the first car to arrive will be the first to leave is an
example of a queue in the real world.
·
A more practical illustration can be found in the line at the ticket booths.
·
Another illustration of a queue is a line of cashiers in a business.
·
Those riding an escalator
Tree
A tree is a type of non-linear, hierarchical data structure in which items are organised in a
manner that bears resemblance to the structure of a tree. The term "root node" is used to
designate the highest node in a tree structure. Data can assume several forms and are found
within every node. The network consists of a central node, along with peripheral nodes and sub-
nodes that are interconnected by edges. Tree data structures, being non-linear in nature, offer
enhanced efficiency and simplicity in data retrieval. The terminologies associated with a tree
structure encompass various concepts, such as nodes, roots, edges, tree height, tree degree
and more elements.
Characteristics of a Tree:
The tree has several distinct qualities, including the following:
·
An alternative name for a tree is a recursive data structure.
·
The distance from the root node to the leaf node that is the longest in a tree is known as
the height of the root.
·
One can also determine the depth from the top to any node in a tree. The depth of the root
node is 0.
Operation performed on tree:
A non-linear data structure called a tree is made up of nodes and edges. Following are some
typical tree operations:
·
Insertion: Additional nodes may be inserted into the tree to expand it vertically or to
establish a new branch.
·
Deletion: To delete a node from the tree, update the parent node's references to take out
the current node's reference.
·
Search: In a tree, you can search for elements by starting at the root node and moving up
the tree based on the value of the current node until you find the node you're looking for.
·
Traversal: A tree's elements can be navigated through in-order, pre-order and post-order
traversal, among other methods.
·
Height: The number of edges from the root node to the farthest leaf node can be used to
calculate the height of the tree.
·
Depth: Number of edges from the root node to the current node can be used to calculate a
node's depth.
·
Balancing: The tree can be balanced to ensure that its height is kept to a minimum and
that its nodes are distributed as evenly as feasible.
These are a few of the most typical actions taken with regard to trees. Based on the
specifications of the problem and the programming language employed, the precise operations
and methods performed may change. In applications like finding, sorting and storing hierarchical
data, trees are frequently utilised.
Real-Life Applications of Tree:
·
Tree data structures are useful for game development in real life.
·
Additionally, it supports database indexing.
·
An effective machine-learning technique frequently used in decision analysis is a decision
tree. Its flowchart-like layout makes the data easier to interpret.
·
A tree data structure is also used by Domain Name Server.
·
Any social networking site is the most typical application for a tree.
Graph
A graph, which is a type of non-linear data structure, consists of vertices (sometimes referred to
as nodes) and edges. A finite set of edges and a finite collection of vertices connect a pair of
nodes. The utilisation of graphs is employed in order to address complex and intricate
programming challenges. The language employed includes terminologies such as "Path,"
"Degree," "Adjacent vertices," "Connected Components," and various more.
Characteristics of Graph:
The graph has the following qualities, among others:
·
The eccentricity of a vertex is defined as the greatest distance between that vertex and all
other vertices.
·
The center of the graph is thought to be the vertex with the lowest degree of eccentricity.
·
The radius of a connected graph is the minimum value of eccentricity from all vertices.
Operation performed on Graph:
A non-linear data structure made up of nodes and edges is called a graph. Following are some
typical graph operations:
·
Add Vertex: To represent a new node in the network, additional vertices can be added.
·
Add Edge: To depict a connection between two nodes, edges can be inserted between
vertices.
·
Remove Vertex: Update the references of nearby vertices to delete the reference to the
current vertex in order to remove a vertex from the graph.
·
Remove Edge: To remove an edge, update the references of the nearby vertices to delete
the reference to the edge that is now there.
·
Depth-First Search (DFS): A depth-first search involves visiting the vertices of a graph in
order to navigate it.
·
Breadth-First Search (BFS): A graph can be traversed by performing a breadth-first search
(BFS), which involves exploring the vertices in that order.
·
Shortest Path: Algorithms like Dijkstra's algorithm or the A* algorithm can be used to find
the shortest path between two vertices.
·
Connected Components: Finding groups of vertices that are connected to one another but
not to any other vertices in the graph will reveal the connected components of the network.
·
Cycle detection: During a depth-first search, cycles in a graph can be found by looking for
back edges.
These are a few of the most typical graph operations. Based on the specifications of the
problem and the programming language employed, the precise operations and methods
performed may change. Graphs are frequently employed in applications including social
networks, computer networks and routing issues.
Real-Life Applications of Graph:
·
Google Maps is one of the most popular real-world instances of a graph, where cities are
found as vertices and the paths that connect them are found as edges.
·
Another real-world example of a graph is a social network, in which each member is a
node and the edges of the graph are all of their connections with other members.
·
In physics and chemistry, molecules are also studied using graphs.
File Formats and Sources of Data
File formats are commonly employed in order to store and secure data within files. Various
types of data, such as text, images, audio, video and other forms, can be saved using a diverse
range of file formats. Every file format possesses unique characteristics and is specifically
designed to cater to a certain usage scenario.
A delimiter, consisting of one or more letters, is employed to demarcate the boundaries of
distinct entities or values. This delimiter is utilised to segregate values within each line or row of
delimited text files, which are text-based data storage files. Various characters can be employed
to divide the values within a dataset, although the most frequently utilised delimiters include the
comma, tab, colon, vertical bar and space.
Comma-Separated Values (or CSVs): Comma-separated values (CSVs) and tab-separated
values (TSVs) are the predominant file formats commonly employed within this particular
category. Tab-separated values (TSVs) employ a tab character as the delimiter, as opposed to
a comma. TSVs serve as an alternative to CSV format when text data contains literal commas
that cannot be utilised as delimiters. Tab stops in running text are not common. A row, or
horizontal line, with a group of values separated by a delimiter serves as the text file's
representation of each record.
Each column contains data of a distinct type, with the top row acting as a column header. For
instance, one column can be of the date type, while another might be of the text or integer type.
Delimited files, which support field values of any length, are a common format for presenting
simple information schema. Nearly all apps now in use can handle them. One technique for
setting limits in a data stream is the use of delimiters.
Microsoft Excel Open XML developed the spreadsheet file type known as the Microsoft Excel
Open XML Spreadsheet, or XLSX. It is an XML-based file format developed by Microsoft. A
workbook, or XLSX file, may contain a number of worksheets. The cell is situated at the
intersection of the rows and columns in each spreadsheet. In each cell, information is kept.
Because XLSX uses the open file format, a large number of different apps can be used with it.
It is one of the more secure file formats because it can't save harmful code and can use and
save all of Excel's features. The Extensible Markup Language, also known as XML, is a markup
language having established rules for expressing data.
XML:The XML file format may be read by both humans and machines. It describes itself as a
language used to send data over the internet. Although there are some differences, XML and
HTML are similar in certain ways. Consider a predefined tag that is not used in XML as an
example. One of them is HTML. Data exchange across systems is facilitated by XML's platform
and programming language independence.
PDF:To deliver documents that are independent of application software, hardware and
operating systems and can be viewed on any device, Adobe developed the Portable Document
Format, or PDF. This format is frequently used for filling out information on forms and in legal
and financial documents.
JSON: JSON, often known as JavaScript Object Notation, is an open standard utilised for
transmitting structured data over the internet. The file format is a programming languageagnostic data format that may be interpreted by any programming language. JSON is widely
acknowledged as a highly effective approach for transmitting data of varying sizes and
structures, encompassing audio and video content. This recognition stems from its user-friendly
nature, compatibility with a diverse array of web browsers and its quick implementation process.
One possible explanation for this phenomenon is the prevalence of JSON format as the
preferred data format for many APIs and Web Services.
Sources of Data
Data sources might be internal or external to the company, primary, secondary, or third-party
data sources. To better understand what we mean by primary, secondary and third-party data
sources, let's look at a few examples. Information that you have acquired directly from a source
is referred to as "primary data". It could originate from internal resources like corporate data,
CRM, HR, or workflow platforms.
Additionally, the inclusion of data collected through surveys, interviews, conversations,
observations and focus groups is a viable approach. Secondary data refers to information that
has been obtained from pre-existing sources, like external databases, scholarly articles,
published works, training resources, online searches, or publicly accessible financial records.
It is also possible to add information from focus groups, talks, observations and polls that were
conducted by other parties. The term "third-party data" refers to information gathered from
aggregators who compile data from many sources into extensive datasets purely for the
purpose of selling it. We'll now look at a few of the various data sources you could employ.
Databases contain various types of data, including primary, secondary and third-party data.
Most firms manage their operations, workflows and clients using internal software. External
databases are available for subscription-based rental or purchase.
Many businesses have already made the switch to the cloud or are in the process of doing so
and it is swiftly emerging as a source for real-time data and on-demand insights. Businesses
can access publicly accessible data through the Internet. individuals for either personal or
business usage.
There is a significant collection of freely available data on the Internet. Publicly available
textbooks, official documents, papers and articles, as well as social media and interactive
websites like Facebook, Twitter, Google and YouTube, could all fall under this category.
Instagram is gaining popularity as a source of user information and opinions. Businesses use
these data sources to get quantitative and qualitative insights. Customers who are already
clients and those who are considering doing business with them.
Sensor data, which is frequently utilised as a data source, is produced by wearable technology,
smart homes, smart cities, smart phones, medical equipment and even household appliances.
In data exchange, a source of third-party data that entails the voluntary sharing of data between
data providers and data consumers, people, companies and governments can all be both data
suppliers and consumers.
It is possible to communicate data from business applications, sensor technology, social media
activity, location information and customer behaviour information. As part of a survey,
questionnaires are distributed to a particular group of individuals. Identifying, for instance, if
current customers are prepared to make a purchase of a new product version. Surveys can be
carried out on paper or online.
Along with population data, census data is a well-liked source for acquiring household data such
as wealth and income. To get qualitative data, such as people's opinions and experiences,
interviews are a valuable tool. Think about a conversation held to understand more about the
problems a customer support agent deals with on a regular basis.
Observations in person or over the phone could be used to conduct interviews. As part of
studies, participants are observed in a certain setting or while carrying out a specific job.
analysing the user experience while they visit an e-commerce site, for instance.
Information about how simple it is for them to find products and make purchases, gleaned from
surveys, interviews and observations. Studies' primary, secondary and third-party data may be
presented. Never before have data sources been as varied and active as they are today.
They also undergo continual change. You can explore problems and solutions in fresh, pertinent
ways by using secondary and third-party data sources to complement your primary data.
Different Types of Data Repositories
Various forms of data can be stored, managed andorganised in data repositories for simple
access and retrieval. Each kind of data repository fulfills a different function and is appropriate
for a particular use case. The following are a few typical types of data repositories:
Relational Database Management Systems (RDBMS):
Using specified schemas, RDBMS is a structured data repository that stores data in tables. For
data management and retrieval, it makes use of SQL (Structured Query Language). RDBMS is
frequently used for transactional applications, reporting and business analytics and is well suited
for managing structured data.
Relational DataBase Management Systems are known as RDBMS. We may create, delete and
update relational databases using this application. A relational database is a type of database
that organises data into rows and columns for tabular retrieval and storage. It is a tiny subset of
DBMS that was developed by E.F. Codd in the 1970s. The major DBMS are all built on
relational DBMS concepts, including SQL, My-SQL and ORACLE.
The foundation of a relational DBMS is the relationship between the values in each table. It can
easily simulate searches and handle massive volumes of data.
What exactly is a table?
Tables are database objects that are used to hold data in relational database management
systems (RDBMS). In essence, this table, which includes a number of columns and rows, is a
collection of linked data entries.
A table is the most prevalent and fundamental type of data storage in a relational database. The
program that follows shows an example of a customer table.
Table: Customers
ID
Name
Age
Address
Salary
1001
Amit
33
Gorakhpur
2000
1002
Tanvi
33
Delhi
1000
1003
Meha
35
Noida
1200
1004
Digvijay
38
Haldwani
2500
1005
Neha
42
Almora
1800
1006
Atharv
25
Greater Noida
800
1007
Priyanka
28
Delhi
2000
What exactly is a field?
Each table is broken up into smaller units called fields. In the Customers table, there are the
fields ID, Name, Age, Address and Salary.
A field is a table column used to keep track of unique information about each database entry.
What is record or a row?
A record, often known as a row of data, is the term used to describe each individual element in
a table. For instance, the Customers table contains seven records. Below is a single row of
information or a single entry from the Customers table.
1001
Amit
33
Gorakhpur
2000
What exactly is a column?
A column in a table is a vertical item that contains all the data associated with a certain field.
For instance, the Customers database's Address column, which displays location information,
might appear as follows.
Address
Gorakhpur
Delhi
Noida
Haldwani
Almora
Greater Noida
Delhi
What is the meaning of a NULL value?
A NULL value in a table denotes a value in a field that seems to be blank, suggesting that the
field is empty.
Understanding the distinction between a NULL value and a zero value or a field with spaces is
crucial. During the record generation process, a field with a NULL value was left empty.
Relational database management systems emulate the following characteristics to preserve
data integrity:
Entity Integrity:The database table cannot contain any duplicate records.
Referential Integrity:The only rows of such tables that can be deleted are those that other tables
don't need. Otherwise, there can be a data inconsistency.
User-defined Integrity:The access and confidentiality guidelines are determined by the users.
Domain integrity:The columns of the database tables are contained within certain structural
bounds based on default values, data types, or ranges.
Characteristics
·
Data must be saved in tabular form, that is, as rows and columns, in a database file.
·
A record/tuple is the term used to describe each database row. The accumulation of these
entries makes up the table's cardinality.
·
An attribute/field is the term used to describe each table column. The table's diversity
consists of a group of related columns.
·
The DB table cannot contain any duplicate records. The use of a candidate key eliminates
data duplication. A set of characteristics known as the Candidate Key is necessary for each
record to be uniquely identified.
·
Foreign keys are used to connect tables to one another.
·
Additionally, database tables accept NULL values, which implies that if any of the
elements' values are blank or missing, the value becomes a NULL value, which is different from
zero. (NOTE: The primary key cannot have a NULL value.)
Advantages
·
Simple to Manage:Each table can be changed independently of the others without
affecting them.
·
Security:Due to the several levels of security, it is more secure. Shared data may have
restricted access.
·
Flexible:Without requiring changes to numerous files, data may be updated in one place.
To handle more records, databases can easily be enlarged, improving scalability. Additionally, it
makes using SQL queries simpler.
·
Users:In a client-side design, RDBMS can hold a large number of users.
·
allows for the simple storage and retrieval of vast amounts of data.
·
Simple Data Handling:
Ø Faster data retrieval is possible thanks to the relational architecture.
Ø Data redundancy or duplication is prevented thanks to keys, indexes and normalisation
techniques.
Ø Data consistency is guaranteed by RDBMS's adherence to the ACID (Atomicity Consistency
Isolation Durability) rules for data transactions.
·
Fault Tolerance:In addition to enabling simultaneous access, database replication helps
the system recover in the case of a disaster like a power outage or an unexpected shutdown.
Disadvantages
·
High cost and extensive setup requirements are required for these systems in order for
them to function.
·
Scalability: As more data is added, more servers are required, along with more power and
memory.
·
Complexity: A lot of data makes it difficult to understand linkages and may hinder
performance.
·
Structured boundaries: The numerous boundaries that surround the fields or columns in a
relational database system may cause data loss.
NoSQL Databases:
Non-relational data repositories known as NoSQL databases offer scalability and flexible
schema design. They are appropriate for content management systems, big data applications
and real-time analytics because they can handle significant amounts of unstructured or semistructured data.
NoSQL databases, which stand for "non SQL" or "non relational," enable the storage and
retrieval of data. This data is modelled differently from the tabular relations used in relational
databases. Although these databases first debuted in the late 1960s, the term "NoSQL" wasn't
coined for them until the early 2000s.
Large-scale data analytics and real-time internet applications are increasingly using NoSQL
databases. The phrase "not only SQL" emphasises the possibility of SQL-like query languages
in NoSQL systems.
An advantage of a NoSQL database is its ease of design, horizontal scaling to server clusters
and tighter control over availability. NoSQL databases employ different default data structures
than relational databases, which enables NoSQL to carry out some operations more quickly.
The problem that a NoSQL database is meant to solve determines how applicable it is. There is
a misconception that NoSQL databases' data structures are more adaptable than relational
databases' tables.
Partition tolerance, performance and consistency are frequently traded off in NoSQL databases.
Wider adoption of NoSQL storage is hampered by the use of low-level query languages, a lack
of standardised interfaces and significant prior investments in relational databases.
Although true ACID transactions (atomicity, consistency, isolation and durability) are absent
from the majority of NoSQL databases, some, such as OrientDB, Symas LMDB, MarkLogic,
Aerospike, FairCom c-treeACE, Google Spanner (though it is technically a NewSQL database)
and Aerospike, have made them a key component of their designs.
Most NoSQL databases support eventual consistency, which enables the progressive
dissemination of database alterations across all nodes. The occurrence of stale readings, which
is attributed to the utilisation of obsolete data, may arise from data searches that do not
promptly provide recent findings. Furthermore, it is worth noting that certain NoSQL systems
may have instances of lost writes and other forms of data loss.
Certain NoSQL systems offer features such as write-ahead logging in order to mitigate the risk
of data loss. When engaging in distributed transaction processing across multiple databases,
ensuring data consistency becomes significantly more complex. This is a problem for relational
and NoSQL databases alike. Cross-database referential integrity requirements are still not
supported by relational databases as of today. Only a few systems can implement ACID
transactions and the X/Open XA standards for distributed transaction processing.
Advantages of NoSQL
There are several benefits to using NoSQL databases like MongoDB and Cassandra. The main
advantages are high scalability and availability.
High Scalability:NoSQL databases use sharding to provide horizontal scale. Data division and
distribution across several machines while preserving the data's order is known as sharding. In
order to manage the data, new machines must be added horizontally and existing machines
must have more resources added vertically. While horizontal scaling is simple to perform,
vertical scaling is more challenging. Examples of databases with horizontal scaling include
MongoDB, Cassandra and others. NoSQL can handle a lot of data since it is scalable. NoSQL
scales effectively to accommodate growing data volumes.
High Availability:The auto replication functionality present in NoSQL databases ensures that
data is able to revert to a previous consistent state in the event of a failure, hence enhancing its
accessibility.
NoSQL's Disadvantage:
Due to the fact that they are primarily designed for storage and have extremely limited
functionality, NoSQL databases have a highly constrained emphasis. Relational databases
perform better than NoSQL databases in the area of transaction management.
Open-Source:NoSQL is a cost-free database to use. There isn't yet a reliable NoSQL standard.
Thus, there is a considerable likelihood that two database systems will differ from one another.
Big data technologies are designed to make managing enormous volumes of data as easy as
possible. But it's not quite that easy. It is much more difficult to manage data in a NoSQL
database than it is in a relational database. Particularly NoSQL is notorious for being
challenging to put up and even more challenging to manage on a regular basis.
GUI Mode Tools to Access the Database are Not Flexible:The market does not offer many
versatile GUI mode solutions for database access.
Backup:One significant drawback of several NoSQL databases, like MongoDB, is backup. A
consistent technique for backing up data is not provided by MongoDB.
Large Document Size: Some database systems, including MongoDB and CouchDB, store data
in JSON format. This indicates that documents are quite large (due to BigData, network
bandwidth and speed) and that having specific key names actually makes documents larger.
Data Warehouses: In order to facilitate business intelligence (BI), reporting and data analysis,
huge volumes of historical and current data from multiple sources are kept in a centralised,
integrated repository known as a data warehouse. It provides a lone source of truth for data
analysis and reporting, acting as a foundation for decision-making processes within businesses.
Data warehouses are made to simplify complex queries, data aggregation and trend analysis,
giving users insightful information about the performance and trends of their businesses.
Key Characteristics of Data Warehouses:
·
Subject-Oriented: Data warehouses are arranged according to particular topics or areas of
interest, including sales, marketing, finance, or customer information. They concentrate on
giving users a thorough understanding of the data around these topics, facilitating analysts'
access to pertinent data.
·
Integration of Data: Data warehouses combine information from different sources,
including operational databases, external systems and data streams, into a single, coherent
perspective. Through this integration, the organisationas a whole is guaranteed accurate and
consistent data.
·
Time-Variant: Data warehouses offer the ability to analyse data changes over time and
store historical data. The time-variant function enables trend analysis, pattern recognition and
performance tracking for analysts.
·
Non-Volatile: Data is not edited or updated within a data warehouse because those
systems are intended to be read-only. Instead, data is routinely updated or refreshed from the
source systems to guarantee data consistency and integrity.
·
Optimised for Query Performance: Data warehouses are designed to handle analytical
processing and complicated queries. To reduce the time it takes for queries to respond, they
employ strategies including data denormalisation, segmentation and indexing.
Components of Data Warehouses:
·
ETL (Extract, Transform, Load): Data is extracted from diverse source systems,
transformed into a uniform format and loaded into the data warehouse using a process known
as ETL (Extract, Transform, Load), or ETL operations. ETL guarantees data consistency and
quality within the warehouse.
·
Data Storage: A relational database management system (RDBMS) is frequently used by
data warehouses to store data. RDBMSs like Oracle, Microsoft SQL Server and PostgreSQL
are often used.
·
Data modelling: Star schema, also referred to as snowflake schema, is a particular data
modelling technique used by data warehouses. These schemas organise the data into a core
fact table and dimension tables on either side, making it easy to retrieve the data for analysis.
·
Business intelligence (BI) Tools: BI technologies are used in conjunction with data
warehouses to make data processing, reporting andvisualisation easier. Tableau, Power BI and
QlikView are examples of well-known BI tools.
·
Metadata Management: In order to offer information about the data, such as data sources,
data definitions, transformations and data lineage, metadata management is crucial in data
warehouses.
Benefits of Data Warehouses:
·
Improved Decision-Making: Data warehouses offer a solitary, dependable source of data
for analysis, facilitating informed decision-making based on precise and consistent facts.
·
Advanced Data Analysis: Data warehouses enable advanced data analysis and provide
deeper insights into business patterns and performance. They also support complex queries
and data mining techniques.
·
Increased Business Agility: Data warehouses enable firms to react swiftly to shifting
business needs and make data-driven choices by centralizing and integrating data.
·
Enhanced Performance: When compared to accessing data directly from operational
databases, data warehouses are optimised for query performance, which speeds up data
retrieval and analysis.
·
Scalability: Data warehouses are appropriate for enterprises with substantial data
requirements because they can handle massive volumes of data.
Data Lakes:
A data lake is a centralised repository that has the capacity to store data of any scale, both
structured and unstructured. Better decisions can be made using dashboards and visualisations
and you can leverage big data processing, real-time analytics and machine learning without first
organising the data.
Companies that are successful in extracting economic benefits from their data will outperform
their competitors. Based on a survey conducted by Aberdeen, it was observed that
organisations who adopted data lakes exhibited a superior performance in terms of organic
revenue growth, surpassing comparable businesses by a margin of 9%.
The aforementioned leaders effectively leveraged novel data sources within the data lake,
encompassing log files, click-stream data, social media content and internet-connected devices,
to facilitate innovative forms of analytics, such as machine learning. The organisation
demonstrated an enhanced ability to promptly recognise and capitalise on prospects for
business expansion through the acquisition and retention of clientele, optimisation of operational
effectiveness, regular maintenance of equipment and the implementation of well-informed
decision-making processes.
A data mart is a focused data repository that is designed to cater to a certain subject matter or
sector within the context of a data warehouse. The utilisation of a data mart enables teams to
expedite data access and gain prompt insights, as it eliminates the need for manual data
collection from multiple sources or navigating a complex data warehouse.
Key characteristics of a data lake include:
Flexibility: Whether structured, semi-structured, or unstructured, data lakes embrace them all.
Organisations may store a variety of data formats, including text, photos, videos, log files,
sensor data, social media data and more because to this versatility.
Scalability: Massive amounts of data can be handled via data lakes. With few changes to the
underlying architecture, they can scale horizontally by adding more storage nodes to meet
expanding data requirements.
Cost-Effectiveness: Data lakes, which store data in its unprocessed form, might be more
affordable than typical data storage systems, especially when handling big datasets. Bypassing
upfront data modelling and schema construction costs.
Schema-on-Read: Data lakes employ a schema-on-read method as opposed to typical
databases, which use a schema-on-write approach (data is structured and verified before
storage). This indicates that the structure is applied when the data is read or queried and that
the data is not modified or altered during ingestion.
Data Exploration and Analysis: Ad hoc analysis and data exploration on the raw data are made
possible by data lakes, which are used by data scientists, analysts and other users. Users are
not restricted by pre-defined schemas when using data to gain value and insights.
Integration with Big Data Tools: Big data processing frameworks like Apache Hadoop and
Apache Spark frequently connect with data lakes. With the aid of these tools, users can
disseminate process and analyse the data kept in the lake.
Data Governance and Security Challenges: Data governance and security must be carefully
considered while managing data lakes. Data lakes can turn into data swamps with unchecked
data sprawl and significant security issues if suitable controls aren't in place.
Metadata Management: Understanding the content and context of the data in the lake requires
knowledge of the metadata. Data is cataloged and indexed using metadata management
technologies, making it simpler to find and effectively utilise the data.
Data Transformation and Preparation: Despite the focus on storing data as-is in data lakes,
some amount of data preparation and transformation may still be required for specific analytical
use cases. When necessary, the raw data is processed and transformed using data integration
tools into a more structured manner.
Data Mart
A data mart facilitates expedited data retrieval for a designated team or business unit inside an
organisation. The process of gathering and consolidating data from several platforms may be a
laborious task, prone to errors and ultimately costly when undertaken by a marketing team
seeking to enhance the efficacy of their campaigns over the holiday season.
Teams regularly interact and work together on spreadsheets when charged with locating data
from several sources. The so-called "spreadsheet nightmare" is a common consequence that
involves human mistake, uncertainty, challenging reconciliations and several sources of truth.
As a centralised area where pertinent data is gathered and organised before being used to
create reports, dashboards andvisualisations, data marts have grown in popularity.
The benefits of a data mart
A data mart focused on a certain group or area of business provides a number of benefits:
The centralisation of a data mart ensures the existence of a single source of truth, so enabling
choices to be made within a department or organisation based on consistent and reliable data.
The aforementioned advantage is significant as it allows stakeholders to prioritise decision-
making and action-taking rather than engaging in factual disputes. Moreover, the reliability of
the data and subsequent forecasts may be ensured.
Enhanced Efficiency in Data Retrieval: The enterprise data warehouse facilitates convenient
data access and integration for certain business teams and users, enabling them to retrieve the
necessary data and merge it with information from various additional sources. Once the users
have established links to their chosen data sources, they can access real-time data as required,
eliminating the requirement for regular data extraction assistance from the IT department.
Increased productivity is a notable advantage experienced by both business and IT teams.
The acceleration of insights facilitates expedited decision-making processes.A data mart
facilitates the analysis of data at the departmental level, while a data warehouse facilitates
decision-making at the company level. Analysts possess the ability to focus their attention on
specific challenges and opportunities within domains such as finance and human resources. By
efficiently extracting insights from data, analysts are able to expedite the decision-making
process and arrive at more accurate and prompt judgements.
Implementing an enterprise data warehouse to cater to the comprehensive needs of a complete
organisation can be a time-consuming and labor-intensive endeavour. Conversely, a data mart
is established to cater to the specific requirements of certain business units and requires access
to a reduced range of data types. Consequently, the process of implementation is expedited and
simplified.
Creating Agile and Scalable Data Management:Data marts are adaptable data management
systems that can take into account shifting business needs, such as the capacity to use
knowledge from earlier initiatives to help with current responsibilities. Teams can update and
modify their data mart based on fresh and developing analytics projects.
Transient Analysis:A thorough examination of online sales during a two-week campaign before
a team meeting is an example of a temporary data analytics work. Teams can put up a data
mart fast to finish such a project.
ETL Process and Big Data Platforms
ETL (Extract Transform Load): The utility of utilising data in analytics, data science and machine
learning program s to provide business insights escalates in tandem with the growth of data
volume, data sources and data types within organisations. The necessity to give priority to these
endeavours places mounting demands on data engineering teams, as it is a vital stage
preceding the transformation of raw, unrefined data into reliable, sanitised data.
The ETL process, also known as extract, transform and load, is a data engineering methodology
that involves the collection of data from diverse sources, subsequent transformation into a
dependable resource and then integration into systems that enable end users to access and
leverage the data for addressing business challenges in the future.
How Does ETL Work?
Extract
The initial stage of this process involves extracting data from a wide range of sources, such as
business systems, APIs, sensor data, marketing tools, transaction databases and other relevant
sources. It is evident that many data types presented here consist of structured outputs derived
from frequently employed systems, whereas others comprise semi-structured JSON server logs.
There exist multiple techniques for conducting the extraction process. There exist three distinct
methodologies for the extraction of data:
Partial Extraction: The simplest method to obtain the data is to receive a notification from the
source system whenever a record is updated.
Partial Extraction (with update notification): Even though not all systems can alert users to
updates, they can nonetheless lead users to the entries that have changed and provide an
excerpt of the revised data.
Full Extract: Certain systems lack the capability to discern the specific data that has undergone
alteration. In this particular scenario, the sole viable course of action is to employ a full
extraction method in order to eliminate the data from the system. To ascertain the alterations,
this approach necessitates the possession of a duplicate of the preceding excerpt in an identical
format.
Transform
The last phase is the transformation of the unprocessed data acquired from various sources into
a standardised format that is compatible with many applications. During this phase, the data
undergoes regular cleaning, mapping and transformation processes to conform to a
predetermined schema, hence fulfilling operational requirements. This approach employs a
diverse range of adjustments in order to ensure the integrity and quality of the data.
Rather of directly loading data into the final data source, it is common practice to first save the
data in a staging database. This procedure ensures a timely reversal in the case of an error.
During this phase, it is possible to generate audit reports to ensure adherence to regulatory
requirements and identify and rectify any inaccuracies in the data.
Load
The process of moving converted data from a staging location to a target database, which may
or may not have previously existed, is known as the load function. The requirements of the
application will determine how simple or complicated this solution is. Each of these procedures
can be finished using bespoke code or ETL tools.
Data Pipelines
The technology that permits ETL activities to take place is referred to as an ETL pipeline
(sometimes called a data pipeline). A group of tools and procedures known as data pipelines
are used to move data from one system to another, where it may be stored and maintained in a
different way. Additionally, pipelines make it possible for data to be automatically gathered from
several sources, transformed and consolidated into a single high-performance data storage.
Challenges with ETL
Even though ETL is essential, building and maintaining reliable data pipelines has become one
of the trickier parts of data engineering due to the exponential expansion in data sources and
types. It takes time and effort to create pipelines that guarantee the accuracy of the data. Data
pipelines are built using complex programming and limited reusability.
Although the underlying code is fairly similar, a pipeline created in one environment cannot be
used in another, making data engineers the bottleneck and forced to reinvent the wheel
repeatedly. In addition to pipeline construction, managing data quality in intricate pipeline
networks is difficult. The entire data collection is commonly depreciated by allowing bad data to
pass through a pipeline undetected.
To maintain quality and ensure reliable insights, data engineers must create a significant
amount of custom code to execute quality checks and validation at each stage of the pipeline.
Finally, when pipelines expand and become more complex, businesses are forced to manage
them with an increased operational burden, which makes it incredibly challenging to maintain
data reliability. It costs time and money to set up, scale, restart, patch and upgrade the
infrastructure for data processing.
Pipeline faults are hard to spot and more harder to fix due to a lack of visibility and tooling.
Despite these challenges, every business that aspires to be insights-driven needs a reliable ETL
process.
Teams throughout the organisation are compelled to base decisions on hunches and faulty
metrics and reports in the absence of ETL solutions that uphold a baseline of data reliability.
Data engineers want tools to democratize and streamline ETL, making the ETL process easier
and enabling data teams to build and use their own data pipelines to gain insights more quickly
and scale further.
Massive amounts of structured, semi-structured and unstructured data can be worked with
using big data processing methods in order to gain value from them.
Massive volumes of structured, semi-structured and unstructured data can be handled via big
data platforms, which are specialised software and hardware ecosystems. These platforms give
businesses the ability to store, handle andanalyse enormous amounts of data in order to get
insightful knowledge and make data-driven choices. Big data platforms provide fault-tolerance,
high performance and scalability to meet the particular difficulties presented by big data
applications. Several well-known big data systems are:
Hadoop: A group of technologies known as Hadoop are used to store and analyse enormous
volumes of data in a distributed fashion. A Hadoop-based data warehouse called Hive is used
for data analysis and trawling. Spark is a distributed data analytics framework for carrying out
complex data analyses in real-time.
Massive datasets can be stored and analysed across computer clusters using the open-source
Hadoop framework, which is based on Java. In the Hadoop distributed system, a node is a
single computer, whereas a cluster is a group of nodes. Hadoop scales from a single node to an
unlimited number of nodes, each with its own capacity for computation and storage. Hadoop is a
dependable, scalable and economical system for storing unformatted data.
Hadoop can be used to: Include new data formats, such as streaming audio, video, sentiment
data from social media and clickstream data, as well as structured, semi-structured and
unstructured data that isn't generally used in a data warehouse.
Real-time, self-service access should be available to all stakeholders. To optimize and cut costs
in your enterprise data warehouse, consolidate data across the company and shift "cold" data to
a Hadoop-based system. One of the four main parts of Hadoop is the Hadoop Distributed File
System, or HDFS, a massive data storage system that runs on inexpensive hardware
connected via a network.
HDFS provides huge data storage that is scalable and reliable by distributing files over several
nodes. Large files are split up among many computers so they can be accessed simultaneously.
As a result, computations can run concurrently on every node that stores data. By replicating file
blocks across numerous nodes, it also eliminates data loss, making it fault-tolerant.
Let's examine an illustration to learn how this functions.
Consider a list of all Americans' phone numbers. The ones for people with last names starting
with A may go on server 1, those for those with B on server 2 and so on. This phonebook's
fragments would be Hadoop-stored throughout the cluster. To reconstruct the entire phonebook,
your program would require the blocks from each server in the cluster. In order to ensure
availability in the event of a server loss, HDFS replicates these smaller pieces by default to two
more servers.
Besides increasing availability, this provides a variety of benefits. For improved scalability, it
enables the Hadoop cluster to divide work into smaller parts and run those processes across all
cluster nodes. Data proximity, or the practice of moving computing closer to the node where the
data is kept, is a last advantage. This is essential when working with large amounts of data
since it lowers network congestion and increases throughput.
The following are some of the other advantages of using HDFS:
·
HDFS is made to recognize and automatically fix hardware issues, enabling speedy
recovery.
·
You can access streaming data thanks to HDFS's high data transfer rates.
·
Huge data sets can be accommodated using HDFS, which can support several clusters of
hundreds of nodes or computers.
·
HDFS offers a high degree of mobility as it can run on a variety of hardware platforms and
is compatible with a number of underlying operating systems.
Hive: Large-scale datasets that are stored in Hadoop Distributed File System (HDFS) or other
comparable data storage systems, such as Apache HBase, can be effectively accessed,
modified and administered through the utilisation of Hive, an open-source data warehousing
software. Due to its foundation on Hadoop, Hive exhibits a notable lag in query execution,
rendering it less optimal for applications that want prompt outcomes. Hadoop is designed to
efficiently handle extensive sequential scans.
Hive is not designed for transaction processing, particularly in scenarios that entail a high
volume of write operations. Hive possesses characteristics that facilitate straightforward SQL
data retrieval and is more appropriate for data warehousing activities such as ETL, reporting
and data analysis.
Hive was specifically developed to facilitate interaction with massive amounts of data by
individuals who possess familiarity with SQL, utilising a SQL-like interface known as HiveQL.
This design aims to cater to non-program rs. Traditional relational databases are not suitable for
managing large datasets due to their design, which primarily caters to interactive queries on
small to medium-sized datasets. In contrast, Hive employs batch processing as a means to
achieve efficient operations over a large distributed database.
Hive is capable of converting HiveQL queries into MapReduce or Tez jobs, which are then run
by YARN, the distributed job scheduling component of Apache Hadoop. The system seeks
information from distributed storage systems such as Amazon Simple Storage Service (S3) or
Hadoop Distributed File System (HDFS). The metastore serves as a storage system for Hive,
housing the metadata of databases and tables. It is designed to provide straightforward data
abstraction and discovery, utilising either a database or file-based approach.
To provide smooth compatibility among Hive, Apache Pig and MapReduce, Hive incorporates
HCatalog, a layer for managing tables and storage that retrieves data from the Hive metastore.
Pig, MapReduce andHCatalog leverage the identical data structures as Hive through the
utilisation of the metastore, hence obviating the necessity to redefine metadata for every engine.
WebHCat, a RESTful API for HCatalog, enables the retrieval and utilisation of Hive metadata by
custom applications or third-party integrations.
Spark: Huge volumes of data may be extracted and processed using Spark, a general-purpose
data processing engine, for a number of applications, including Interactive Analytics, Streams
Processing, Machine Learning, Data Integration and ETL. It uses in-memory processing to
speed up computations significantly and only spills to disc when memory is low. A handful of the
programming languages supported by Spark include Java, Scala, Python, R and SQL.
Its autonomous clustering technology enables it to function both independently and on top of
other infrastructures like Hadoop. It is very versatile because it can retrieve data from many
different data sources, including as HDFS and Hive. The capacity to process streaming data
quickly and carry out intricate analyses in real-time is the core use case for Apache Spark.
An open-source distributed processing method for big data workloads is Apache Spark. It
makes use of speedy query execution and in-memory caching for quick analytic queries against
any size of data. It includes development APIs in Java, Scala, Python and R and supports code
reuse across a variety of workloads, including batch processing, interactive queries, real-time
analytics, machine learning and graph processing.
In addition to FINRA, Yelp, Zillow, DataXu, the Urban Institute and CrowdStrike, it is used by a
wide range of businesses. Apache Spark has become one of the most well-liked large data
distributed processing frameworks, with 365,000 meetup members in 2017.
The spark tools, which are used for effective and scalable data processing in big data analytics,
are the major software components of the spark framework. The Spark framework is being
made available under the Apache license. GraphX, MLlib, Spark Streaming, Spark SQL and
Spark Core are just a few of the five important data processing technologies that are included.
Graph data is handled and managed using the GraphX graph data analysis tool. On a
distributed dataset, machine learning is applied using the MLlib Spark tool. On the other side,
stream data processing is done using Spark Streaming. The most widely used tool for analysing
structured data is Spark SQL. The Spark Core tool is responsible for managing RDD (Resilient
Data Distribution).
Process and Tools for Gathering Data
The acquisition of data is a fundamental and indispensable stage in numerous disciplines,
including research, analytics, decision-making and problem-solving. This document provides a
comprehensive overview of the data collection procedure, including a description of each stage
and a list of frequently employed tools to facilitate data gathering.
Define objectives and research questions:
·
The objective of data gathering should be well articulated, together with the exact
questions that are intended to be answered.
·
Identify the necessary data required to effectively handle the objectives at hand.
Plan data collection:
·
The initial step involves making a determination regarding the data sources to be utilised,
which may include internal sources such as company databases or external sources such as
publicly available datasets.
·
The selection of appropriate data collection methods encompasses a range of options,
including surveys, interviews, observations, experiments and other relevant techniques.
·
Formulate a comprehensive schedule and financial plan for the execution of data collection
endeavours.
Identify data sources:
·
Internal sources of information within an organisation encompass various data repositories
such as databases, customer records and sales data.
·
Some examples of external sources that can be utilised for research purposes are
government databases, industry publications and open data repositories.
Select data collection methods:
·
Surveys: Utilisation of online platforms such as Google Forms or SurveyMonkey, as well
as traditional paper-based surveys.
·
Interviews can be conducted using several methods, including face-to-face interactions,
telephone conversations, or the utilisation of video conferencing systems such as Zoom or
Skype.
·
Observations: The collection of data by direct visual examination or the utilisation of
cameras and sensors for remote monitoring.
·
Experiments refer to controlled tests that are conducted to quantitatively determine the
effects of specified factors.
Design data collection instruments:
·
Surveys involve the creation of questionnaires that consist of structured or open-ended
questions.
·
Interviews: Construct interview guides or question scripts.
·
Observations: Establishing the parameters and conditions for observation.
·
The experimental setup and techniques will be outlined in this section.
Collect the data:
·
Conduct the data gathering procedure in accordance with the selected methodologies and
instruments.
Organise and store the data:
·
Design and implement a robust data storage system that effectively guarantees both
security and accessibility.
·
Utilise databases or cloud storage options such as SQL databases, Amazon S3 and
Google Cloud Storage.
Clean and preprocess the data:
·
One should do a thorough examination to identify any flaws, inconsistencies, or missing
values.
Data cleaning technologies such as Excel and OpenRefine, as well as computer languages like
Python and R, are commonly employed for the purpose of data cleaning and preprocessing.
Analyse the data:
·
Employ statistical applications such as R, SPSS and SAS, or data analysis platforms like
Excel and Tableau, to investigate the data and extract meaningful observations.
Interpret the results:
·
In light of the data analysis conducted, it is possible to draw conclusions and answer the
research problems at hand.
Tools for data gathering
A range of technologies are accessible to facilitate the data gathering process, contingent upon
the particular methods and purposes of data collection. The following are few widely used
instruments for the collection of data:
Online Surveys and Forms:
·
Google Forms is a cost-free and user-friendly software application that facilitates the
creation of online surveys and questionnaires.
·
SurveyMonkey is a robust survey platform that offers a wide range of question types and
analysis capabilities.
·
Typeform is a contemporary and interactive tool for constructing forms that effectively
promotes user involvement.
Interviews and Video Conferencing:
·
Zoom is a prevalent video conferencing platform that is extensively utilised for the purpose
of conducting remote interviews and facilitating discussions.
·
Skype is a communication platform that facilitates audio and video chats, making it wellsuited for conducting distant interviews and organising focus groups.
Observations and Data Collection:
·
Both Excel and Google Sheets are valuable tools for organising and documenting
observations and data in a tabular manner.
·
Mobile applications can be tailored to suit certain data collection tasks.
·
The utilisation of social media monitoring tools such as Hootsuite, Brandwatch, or Mention
enables the collection of data from various social media networks.
·
Web scraping solutions like as Scrapy, Beautiful Soup (Python libraries), or import.io are
utilised for the purpose of extracting data from websites.
Data Analytics and Visualisation:
·
Python and R are programming languages that are frequently utilised for the purposes of
data analysis and visualisation.
·
Tableau is a robust data visualisation platform that enables the creation of dynamic and
informative dashboards.
·
Microsoft Excel is a software application that is well-suited for performing fundamental data
analysis and visualisation activities.
GPS and Location-based Data Collection:
·
GPS-enabled devices or mobile applications are utilised for the purpose of acquiring data
and coordinates dependent on the user's position.
Sensor Data Collection:
·
The Arduino is a microcontroller platform that enables the construction of customised
sensors for the purpose of data collection.
·
The Raspberry Pi is a cost-effective and adaptable computing device that may be utilised
for a wide range of data collection endeavours.
Crowdsourcing and Data Collection Platforms:
·
Amazon Mechanical Turk (AMT) is a widely used crowdsourcing platform that facilitates
the recruitment and engagement of individuals for the purpose of doing data gathering tasks.
·
Ushahidi is a software platform designed to facilitate the collection and visualisation of data
on maps, with a particular focus on its application in disaster mapping.
Internal Data Collection and Management:
·
Structured Query Language (SQL) databases, such as MySQL and PostgreSQL, are
commonly employed for the purpose of handling structured data.
·
Data management tools, such as Airtable, are utilised for the purpose of efficiently
organising and analysing data.
Paper-based Data Collection:
·
The utilisation of conventional pen-and-paper techniques for conducting surveys and
questionnaires.
When making decisions on the selection of data gathering technologies, it is important to take
into account many variables. These elements include the level of complexity associated with the
project, the needs for data security and privacy, any budgetary limitations and the technical
proficiency of the data gathering team. It is imperative to consistently verify that the selected
tools adhere to pertinent data protection legislation and ethical issues.
Wrangling Data, Mining and Analysing Data
Data wrangling is the process of cleaning, organising and enhancing raw data to create the
required format for faster, better decision-making. Top organisations nowadays increasingly
engage in data wrangling. More effort must be spent choosing, organising and cleaning data
before conducting a more in-depth analysis due to the increasing variety and unstructured
nature of data. In addition, business users have less time to wait for prepared data from
technical staff because data now informs practically all business decisions.
This necessitates the implementation of a self-service framework and a shift away from ITcentric data preparation towards a more inclusive data wrangling paradigm. By utilising the selfservice paradigm and employing data wrangling tools, analysts are able to effectively handle
intricate data sets in a more efficient manner. This enables them to generate more precise
responses and ultimately arrive at superior findings. The utilisation of data wrangling technology
by an increasing number of enterprises has been observed as a consequence of its capacity to
facilitate data preparation for analysis.
Benefits of Data Wrangling
·
By transforming data into a format that is compatible with the final system, data wrangling
increases the usability of the data.
·
It allows users to organise and automate the data-flow process as well as construct data
flows fast and easily utilising a simple user interface.
·
combines a variety of data types and their sources (such as files, databases and web
services), among others.
·
Allow users to communicate data-flow approaches and analyse enormous amounts of data
with ease.
Steps in Data Wrangling
Similar to most data analytics procedures, data wrangling is an iterative process where the data
engineer must repeat these steps in order to reach the desired predictions. Data wrangling can
be divided into the following six steps:
1. Discovering: The first step in data wrangling is to analyse the data before imputing it.
The criteria that can be utilised to demarcate and partition the data must be identified in
this step and wrangling must be done methodically.
2. Structuring: The majority of the time, the raw data gathered in the form of user
information is unstructured. The data should be reorganised in a manner that is better
suited to the analytical approach being used. To make the data more usable, it should
be divided into categories based on the categorisation from the first stage. Feature
engineering is the act of selecting a single column that could be split into two or a single
row that could be split in order to better analysis.
3. Cleaning: In processed datasets, outliers are inevitable and can bias the study's findings.
The dataset should be cleaned for the best results. This process involves a thorough
data cleaning to guarantee accurate data analysis. Null values should be imputed and
formatting should be consistent in order to create processed data of higher quality.
4. Enriching: The fourth phase is when the data is upgraded after it has been processed.
Because of this, you must evaluate the data and choose whether to upscale,
downsample, or enhance it. Resampling data can be done in two ways: either by
downsampling the data or by upsampling the data to produce synthetic data.
5. Validating: After data has been processed, it is checked for consistency and quality
using a set of repetitive programming techniques called validation. To ascertain whether
the fields in the data set are accurate, you might need to validate the data or check to
see if the attributes are usually distributed.
6.
Publishing: Data wrangling's sole goal is to provide the cleaned, processed data for
additional usage. For the benefit of users and clients, the entire data wrangling procedure
should, if necessary, be meticulously documented.
Data Mining
The process of removing a valuable resource from the soil, such coal or diamonds, is known as
"mining" in general. Computer science names for data mining include knowledge extraction from
data, data/pattern analysis, data archaeology and data dredging. In essence, it is a method for
obtaining important facts from huge amounts of data or data warehouses. It's obvious that the
phrase is a little baffling on its own. Coal and diamonds are the ultimate outcomes of the
extraction process involved in mining activities. However, it should be noted that in the field of
data mining, the result obtained from the extraction procedure does not constitute raw data. The
outcomes of data mining, as commonly understood, encompass the patterns and insights that
are obtained upon completion of the extraction process. Data mining can be perceived as a
pivotal stage within the process of extracting or discovering knowledge.
The term "Knowledge Discovery in Databases" was initially introduced by Gregory PiatetskyShapiro in the year 1989. Conversely, the term "data mining" has gained increasing prevalence
within both media and corporate spheres. The terms "data mining" and "knowledge discovery"
are often employed synonymously.
In contemporary times, the practice of data mining has become ubiquitous, finding application in
various domains where substantial volumes of data are subjected to processing and storage.
Banks commonly employ the practice of 'data mining' to identify prospective consumers who
may exhibit interest in credit cards, personal loans, or insurance. Financial institutions analyse
this data in order to identify trends that can assist them in predicting the potential interest of their
customers in personal loans or other financial products. This is made possible by the availability
of transaction records and detailed profiles maintained by the banks on its clientele.
Data Analysis
It is extremely human to gather and store data for analysis. Statistics has its roots in systems
that have been used for thousands of years to track grain stocks, taxes and population. The
20th century saw an explosion in related fields including statistical process control, operations
research and cybernetics.
The field of data analysis goes by many names, including business intelligence (BI), analytics,
data science and decision science and its practitioners hold a variety of positions. Marketers,
product managers, business analysts anda number of other people also analyse data. We will
alternatively refer to the individual using SQL to understand data as a data analyst or data
scientist. We will use the term "BI tools" to describe the programs used to create reports and
dashboards.
The history of computers has enabled and is entwined with data analysis in the modern sense.
Data analysis combines classic statistical methods with the computing capabilities of computers.
Data discovery, data interpretation and data transmission are all components of data analysis.
Data analysis is frequently done to help people make better decisions—both manually and
increasingly automatically by technology.
Although good methodology is essential, analysis involves more than just arriving at the correct
number. It's all about curiosity, enquiring and discovering the "why" behind the statistics. It
involves finding and analysing clues regarding the behaviour of people and enterprises by
analysing trends and anomalies. In other cases, such as in a scientific environment or an online
experiment, analysis is done on a data collection that has been obtained to address a particular
question.
Data collected for analytics reasons, such as tracking user interactions on websites and mobile
applications, or data generated as a result of conducting business, such as sales of a
company's products, are also subject to analysis. This data has a wide range of potential
applications, from troubleshooting to planning changes to user interfaces (UI), but it frequently
comes in a format and volume that require processing before providing results.
It's difficult to imagine an industry that data analysis hasn't affected: manufacturing, retail,
finance, healthcare, education and even government have all undergone changes as a result of
it. Since Billy Beane's early years as Oakland Athletics general manager, made famous by
Michael Lewis' book Moneyball (Norton), data analysis has been used by sports teams.
Marketing, sales, logistics, product development, user experience design, customer service
centers, human resources and other fields use data analysis. Data engineering and data
science have become quite popular thanks to the convergence of concepts, applications and
computational capacity.
By definition, historical data is used for data analysis and it's crucial to keep in mind that
previous events don't always foretell future events. Organisations are dynamic, just because the
environment is dynamic—new products and processes are launched, rivals emerge and fall and
sociopolitical climates change. Data analysis is criticised for having a backwards perspective.
Although that description is accurate, we have seen businesses benefit greatly from looking at
past data. We can better understand the traits and actions of customers, suppliers and
processes by mining past data.
Using historical data, we can create accurate predictions and ranges of results that, while
occasionally being incorrect, are frequently correct. Data from the past might highlight holes,
flaws and openings. Organisationsare able to maximize efficiency, save money and lessen risk
and fraud. Additionally, it can aid businesses in identifying opportunities and serve as the
foundation for brand-new goods that please consumers.
Let's examine some of the crucial phases of any data analysis process.
Understanding The Issue And The Desired Outcome: The initial stage in data analysis is to
comprehend the issue at hand and the desired result to be attained. You must decide where
you are and where you want to go before you can start the analytical process.
Defining A Metric That Is Easy To Understand:This phase of the process includes selecting the
metrics to be used. For instance, how many units of product X were sold in a specific place. In a
quarter or during festival season, you decide the data you'll need, the data sources you'll need
to collect this data from and the suitable tools for the job once you've decided what you're going
to measure and how you're going to measure it.
Data Cleansing:After collecting the data, the next step is to address any data quality issues that
can jeopardize the accuracy of the study. This stage is essential since only clean data can
guarantee the accuracy of the study. You'll scan the data for outliers and missing or incomplete
values.
For instance, a customer demographics data collection with an age field value of 150 is an
anomaly. Additionally, you'll standardize the data that arrives from various sources. Data
analysis and mining You will extract the data, evaluate it from several perspectives andanalyse
it after it has been cleansed.
You might need to alter your data in a number of different ways to understand trends, build
correlations and uncover patterns and variations.
Interpreting the Result: After you've analysed your data and maybe undertaken additional
research (which could be an iterative cycle), it's time to interpret your results. Consider whether
your analysis can be defended against criticism as you interpret your results and whether there
are any limitations or circumstances under which it might not be true.
A presentation of your findings The ultimate goal of any analysis is to have an impact on
judgment. The ability to effectively and concisely explain and present your findings is just as
important as the data analysis process itself. You have a wide range of presentation options for
your data, including reports, dashboards, graphs, maps, charts and case studies.
Descriptive Analysis
The initial category of data analysis is descriptive analysis. The cornerstone of data analysis is
its foundation. In contemporary business environments, the utilisation of data has become a
pervasive and essential practice. Descriptive analysis employs dashboards to elucidate
historical data, so addressing the inquiry of "what occurred."
The monitoring of Key Performance Indicators (KPIs) is a prevalent utilisation of descriptive
analysis within the realm of business. Key performance indicators (KPIs) provide a means of
evaluating the performance of a firm by comparing it to a predetermined set of standards.
The following examples illustrate the application of descriptive analysis in the field of business:
·
KPI dashboards
·
Monthly revenue reports
·
Sales leads overview
Diagnostic Analysis
Following your response to "what happened," you should probe further and ask "why did it
happen?" Diagnostic testing is used in this situation.
To identify the underlying causes of the results, diagnostic analysis delves further into the
descriptive analytics data. Organisations employ this form of analytics due to its capacity to
establish more linkages among data points and discern patterns of behaviour.
The collection of thorough information is a crucial aspect of diagnostic analysis. It is probable
that you have already acquired some understanding of the situation with the emergence of new
challenges. No further effort will be required on your part as the necessary information is
already available andall of the concerns will be interrelated.
There are several business uses for diagnostic analysis, including:
·
A freight company looking into the reason behind sluggish shipments in a certain area.
·
A SaaS firm investigating whether marketing strategies increased the number of trials.
Predictive Analysis
The objective of predictive analysis is to provide insights into the likelihood of future events or
outcomes. Drawing upon historical data, this particular form of analytics provides
prognostications for prospective results in subsequent periods.
Similar to descriptive and diagnostic research, this particular form of analysis extends beyond
the aforementioned approaches. The acquired knowledge is utilised in the field of predictive
analysis to generate reasoned forecasts regarding future events. This research is grounded in
the utilisation of statistical modelling techniques, necessitating supplementary investments in
terms of human resources and technological infrastructure to facilitate accurate forecasting. It is
imperative to consider that forecasting is inherently speculative and that the attainment of
dependable and complete factual information is indispensable for ensuring the accuracy of
predictions.
Diagnostic and descriptive analysis are commonly employed in the field of business, however
predictive analysis often serves as an initial indicator of potential risks for many firms. Certain
organisations may face limitations in terms of their workforce capacity, which may hinder their
ability to fully deploy predictive analytics across all desired areas. Some individuals or
organisations may lack the capacity or willingness to allocate resources for the establishment of
interdisciplinary analytical teams or the provision of training for existing teams.
There are a multitude of business applications that can be utilised for predictive analysis,
encompassing:
·
Risk Assessment
·
Sales Forecasting
·
The utilisation of consumer segmentation as a means to ascertain the likelihood of lead
conversion.
·
The utilisation of predictive analytics within customer success teams.
Prescriptive Analysis
The most sought-after type of data analysis is the last one, yet few businesses are truly
equipped to perform it. The most recent development in data analysis is called prescriptive
analysis, which uses the knowledge from all earlier research to determine the optimum course
of action in a certain circumstance or set of options.
Prescriptive analysis leverages contemporary data management methodologies and
technologies. Organisations must ensure their preparedness to allocate the requisite time and
resources, as this is a significant commitment at the organisational level.
Artificial Intelligence (AI) serves as a prominent illustration of prescriptive analytics.In order to
constantly learn and form wise decisions, AI systems take in a huge amount of data. Welldesigned AI systems are able to communicate and even take action on these decisions. Without
the involvement of a human, daily business activities can be done and optimised thanks to
artificial intelligence.
Many prominent data-driven organisations, such as Apple, Facebook and Netflix, rely heavily on
the utilisation of prescriptive analytics and artificial intelligence (AI) techniques to augment their
decision-making processes. The transition to predictive and prescriptive analytics may present
difficulties for certain firms. With the advancement of technology and the training of more
specialists in data, more organisations will move into the data-driven space.
Visualising Data
One of the interactive techniques for coming up with fresh concepts and discoveries is data
visualisation. It is a dynamic tool that expands research possibilities and quickens the pace of
science. Due to the extensive use of the Internet and Web, a significant amount of data is
produced every day. It is necessary to understand sizable and complex data sets. When there is
access to a large amount of data, it must be processed using a variety of data processing
techniques and presented using a variety of methodologies.
Every business must utilise data visualisation to manage their data properly and use it to its
fullest potential in order to transform it into knowledge. It involves converting numerical data into
a visual representation.
The development, design and use of graphical representations of data to facilitate
understanding is known as data visualisation. It's frequently referred to as scientific or
information visualisation.
People have used images, graphs, charts and maps for millennia to understand facts and
information. The development of computers has allowed for the rapid handling and processing
of massive volumes of data. Data visualisation is now developing into a science and art hybrid
that will become more and more prominent over the coming years.
Although it might be challenging to visualize data, this format makes it more simpler to interpret
data than text, numbers, or large tables with several rows and columns. Selecting the
appropriate data visualisation technique may be aided by an understanding of the data and its
structure. All visualisation methods aim to approach the same issue in various ways.
Explanation and exploration are the two categories into which data visualisation may be
separated. Exploration data visualisation is useful when there is a lot of data but little knowledge
about it and the aims are ambiguous. When a lot of data is available but we don't know what it
is, explanatory data visualisation is utilised. These two groups support the visual presentation of
facts.
Importance of Visualisation
Due to the extensive use of the Internet and Web, a significant amount of data is produced
every day. It is necessary to understand sizable and complex data sets. Every organisation that
keeps records must handle data and come to conclusions. When there is access to a large
amount of data, it must be processed using a variety of data processing techniques and
presented using a variety of methodologies.
Because it lets them to efficiently manage data and maximize the use of that data to turn it into
knowledge, it is essential to the success of any organisation. It involves converting numerical
data into a visual representation. Data visualisation approaches employ a variety of computer
graphics effects. Data visualisation helps stakeholders make quick and informed decisions while
also enhancing learning, comprehension and reasoning. It also enables a better understanding
of pattern recognition, trend analysis and the extraction of pertinent data from images.
The precise, understandable and effective communication of complex data is greatly aided by
data visualisation. Businesses are able to make better-informed judgments since it genuinely
absorbs data in novel and helpful ways. It illustrates the connections and trends between
operational activities. Data visualisation is a new business language, to put it another way.
The precise, understandable and effective communication of complex data is greatly aided by
data visualisation. Businesses are able to make better-informed judgments since it genuinely
absorbs data in novel and helpful ways. It illustrates the connections and trends between
operational activities. Data visualisation is a new business language, to put it another way.
Traditional Data Visualisation Techniques
Data can be transformed into a visual form that cannot be manipulated directly by a human
using a variety of tools and techniques. Some of the most well-liked database-connected
multifunctional programs that can be used to visualize data and produce exceptional results
include Microsoft Word, Microsoft Excel, Microsoft Spreadsheet and Microsoft PowerPoint.
These programs can be useful to organisations that don't need highly specialised data
visualisation.
Data is represented using pie charts, line charts, bar charts, area charts, graphs, maps, heat
maps and other conventional data visualisation approaches.
Understanding of Gathering, Wrangling and Mining
Data wrangling is the process of organising, manipulating and cleansing raw data so that
analysts can use it to make decisions quickly. Data wrangling, often referred to as data cleaning
or data munging, enables businesses to work with more complex data in a shorter amount of
time, provide more accurate results and make better decisions. The specific steps vary from
project to project depending on your data and the goal you're trying to achieve. A growing
number of businesses are using data wrangling tools to prepare data for downstream analytics.
Wrangling is the process of "digesting" data (also known as "munging," thus the alternate name
"data wrangling techniques") and making it useful (or usable) for the system. It might be
considered a step in the process of getting ready for any other data-related operation.
Data wrangling is usually done in addition to mapping. Data Mapping is the part of data
wrangling that includes matching up source data fields to their corresponding target data fields.
Mapping is focused with tying together disparate components, whereas Wrangling is concerned
with data transformation.
Main Purpose of Data Mining
Businesses commonly integrate data mining with various techniques from diverse fields,
including statistics, machine learning, pattern recognition, database and data warehouse
systems, information retrieval, visualisation and more. This amalgamation facilitates a
comprehensive understanding of the data, enabling the identification of concealed patterns,
future trends andbehaviours.
Data mining is the process of categorizing and summarizing data using computers to examine it
from a variety of perspectives, dimensions and angles.
Any type of data, including those from data warehouses, relational databases, multimedia
databases, spatial databases, time-series databases and the World Wide Web, can be mined
using data mining techniques.
Data Mining as a whole process
The data mining process consists of these three fundamental steps:
·
Data cleansing, integration, selection and transformation are all components of preprocessing data.
·
Exact data mining occurs during data extraction.
·
A step in the evaluation and presentation of data is the analysis and presentation of the
findings.
Real-life examples of Data Mining
A method for analysing a customer's purchases in a supermarket is market basket analysis
(MBA). The objective is to use the concept to pinpoint the items that a customer purchases in
tandem. What are the chances that someone who purchases bread will also purchase butter?
This study assists in promoting business specials and discounts. The similar process is carried
out using data mining.
Protein folding is a technology that closely analyses biological cells and foretells how proteins
will interact and function. This study may help identify the reasons and future treatments for the
cancer, Parkinson's and Alzheimer's diseases brought on by protein misfolding.
Detecting fraud In today's world of mobile devices, data mining can be used to compare
suspicious phone behaviour by examining mobile device usage. This might make it easier to
identify copied phone calls. Comparing recent transactions to earlier purchases when using a
credit card can also assist identify fraudulent activity.
Several instances of successful data mining applications encompass corporate intelligence,
Web search, bioinformatics, health informatics, finance, digital libraries and digital governments.
Analysing and Visualising Data
Data visualisation approaches employ a variety of computer graphics effects. It supports
decision-making by stakeholders that is swift and efficient. It also enables a better
understanding of pattern recognition, trend analysis and the extraction of pertinent data from
images.
Although it might be challenging to visualize data, this format makes it more simpler to interpret
data than text, numbers, or large tables with several rows and columns. Selecting the
appropriate data visualisation technique may be aided by an understanding of the data and its
structure.
Data must be handled and presented in an intelligible way because the world is being inundated
with information on a regular basis. Making better judgments requires analysing data in a variety
of ways, which visualisation can help with. Data visualisation is the process of presenting data
and information graphically or visually. It is rapidly becoming recognised as an effective, widely
used and acceptable method for reading and analysing enormous, complex data collections. It
has evolved into a quick and easy method of spreading information around the world.
Module II: Relational Database
Relational Database Concepts; Using Relational Databases; MySQL and PostgreSQL;
Database Design Project
RDBMS Concepts
A sophisticated software system called a relational database management system (RDBMS)
makes it easier to create, organise and manipulate relational databases. An RDBMS is
fundamentally based on the ideas of the relational model, which was first put forth by Dr. E.F.
Codd in 1970 and is centered around the idea of storing data in tables with rows and columns.
Each column denotes a different record property and each row denotes a distinct record.
RDBMSs use primary keys to identify records specifically and foreign keys to create
connections between tables.
The normalisation procedure makes sure that data is efficiently formatted, minimizing
duplication and averting anomalies. RDBMSs use SQL (Structured Query Language) to
communicate with the database, giving users the ability to do a variety of tasks like data
querying, inserting, updating and deletion. RDBMSs provide data consistency and dependability
by supporting ACID qualities (Atomicity, Consistency, Isolation, Durability). Since RDBMSs are
built on the principles of tables, primary keys, normalisation, SQL and ACID features, they are
widely utilised and essential to contemporary data management techniques.
Relational Database Concepts
Connected data points are saved and made available within a relational database, which is a
specific type of database. The relational model serves as the fundamental framework for
organising data in the form of tables, offering a straightforward approach to data representation
within relational databases. In a relational database, every row within a table represents a
record that possesses a unique identifier referred to as the key.
Determining the relationships between data points is facilitated by the inherent simplicity of the
task, as the attributes of the data are encapsulated within the columns of the table and each
record generally encompasses a value for each respective property.
The three most widely used RDBMS are MS SQL Server, DB2 and Oracle. An illustration of a
record-based approach is the relational model. Because the database is organised into fixedformat records of various types, record-based models get their name. There are records of a
specific type in each table. A specific amount of fields or characteristics are specified for each
record type. The qualities of the various record kinds are represented by the table's columns.
The most popular data model is the relational data model, which provides the foundation for the
great majority of today's database systems. Dr. E.F. Codd, an IBM research scientist and
mathematician, created the relational model. Even though many contemporary DBMS do not
meet the Codd's definition of an RDBMS, they are nonetheless regarded as RDBMS. The
relational model was created with two key goals in mind: to further reduce data redundancy and
to enhance data integrity inside database systems.
The relational model, which draws inspiration from Dr. Codd's 1970 article entitled "A Relational
Model of Data for Large Shared Data Banks," has been widely influential in the field. This essay
addressed many concepts pertaining to relational database management systems. The
relational data architecture use the relation data structure only for the representation of entities
and their relationships.
The properties of a relation are represented by its columns, whilst the rows of the relation are
commonly known as its tuples. The attribute of each column is selected from a collection of
values within the domain. The domain of an attribute encompasses the set of values that it can
potentially take on.
The relational data model is relatively young in historical terms.Either network or hierarchical
models served as the foundation for the first database systems.In commercial data processing
applications, the relational data model has cemented its position as the default data model. Due
to its success in this area, it is now being used in computer assisted design systems and other
areas in addition to data processing.
Difference b/w DBMS and RDMS
Parameters
DBMS
RDBMS
Storage
Stores data in the form of a file
Stores data in the form of tables
Database Structure
Hierarchical arrangement of data
Stores data in the form of rows and
columns within tables
Number of Users
Allows one user at a time
Allows more than one user at a time
ACID
Does not use the ACID form of data
storage
Uses the ACID model
Type of Program
Manages the data in a computer
Maintains the relationships of tables in a
database
Hardware and Software
Needs
Not many hardware and software
requirements
Needs a good set of hardware and
software requirements
Integrity Constraints
Does not support integrity constraints
Supports integrity constraints
Normalisation
Cannot be normalised
Supports normalisation
Distributed Databases
No support for distributed databases
Allows distributed databases
Data Handling Capacity
Cannot handle large amounts of data
Able to handle high amounts of data
Data Access
Individual data access
Easy and straightforward data access
Data Relationship
No relationships defined for the data
Defines relationships using foreign keys
Data Security
Lack of data security
Good data security due to several log
files
A relational database example
Presented below are two tables that could be employed by a small enterprise for the purpose of
managing and executing product orders. The first table comprises individual records that pertain
to customer information. Each record includes details such as the client's name, address, phone
number, as well as shipping and billing information.
In the database, individual attributes are organised in separate columns, while each row is
assigned a unique identifier, commonly referred to as a key. The subsequent table is a client
order table, wherein each record encompasses the customer ID, the requested product, the
quantity, as well as size and colour preferences, among other relevant details. However, it does
not include the consumer's name or contact information.
The sole shared attribute between these two databases is the ID column, which serves as the
primary key. The relational database is capable of establishing a relationship between the two
tables due to the presence of a shared column. The database is capable of accessing the
customer order table to retrieve pertinent information regarding the product order.
Subsequently, it utilises the customer ID obtained from said table to search for the customer's
billing and shipping details in the customer info table. This process occurs when the company's
order processing application submits an order to the database. Subsequently, the appropriate
item can be extracted from the warehouse, ensuring timely delivery of the customer's purchase
and facilitating the receipt of payment by the business.
Basic RDBMS Features:
·
This feature offers the capability to retain data within tables.
·
The system allows concurrent access by several users and is managed by a single user.
·
Virtual tables are a viable means of storing pertinent data.
·
The primary key is employed in isolation to uniquely identify the rows.
·
Data is consistently organised in a tabular format, with rows and columns serving as the
primary means of storage.
·
Indexes are employed for the purpose of retrieving data.
·
The keys serve the purpose of facilitating the sharing of columns among tables.
People utilise the Relational Database Management System in the real world to gather
information, process it and offer services. For instance, in a ticket processing system,
information about us (such as our age and gender) and our travel (such as our source and
destination) is gathered before the ticket is given to us.
RDBMS Architecture
·
The Secondary Storage Devices (SSD), such as disks and tapes, store all data as well as
metadata and logs.Application programs are the ones that are employed by businesses to carry
out their daily operations. These applications offer the functionality needed for the company's
daily operations. They are created using high level languages (HLLs), such as Java, C and
others, which work with SQL to interface with databases.
·
The SQL commands are translated into a lower level language by the RDBMS compiler,
which is then processed and stored in the secondary storage device.
·
Setting up the database's structure using a command processor is the responsibility of the
database administrator (DBA). The DBA uses the Data Definition Language, or DDL, to add
columns, drop tables and perform other operations. Other commands are also used by the DBA
to set restrictions and access controls.
·
Application program rs use a compiler to compile the applications, producing executable
files (application programs that have been built), before storing the data on a secondary storage
medium.
·
The role of the data analyst is to manipulate the data in the database using the query
compiler and query optimizer (which use relational attributes for query execution).
·
The RDBMS Run Time System interacts with the transaction manager and buffer manager
in addition to running the compiled queries and application programs.
·
In order to speed up processes and better manage disk space, Buffer Manager temporarily
saves database data in main memory using the paging mechanism.
·
Transaction Manager deals with the idea that a task can either be completed entirely or not
at all (Atomicity attribute). Let's say Geeks wishes to send his sister some money. The system
crashes in the middle of him sending the money. It should never happen that he sends money
but his sister doesn't get it. The transaction manager handles this. The money would either be
transferred to Geeks' sister or refunded to Geeks.
·
A system called a log keeps track of all transactions so that they can be undone in the
event of a system failure (such as a disk failure or a power outage, for example).
·
After a failure, Recovery Manager takes over the system to restore it to a stable state. The
Recovery Manager considers the log files, undoes any incomplete transactions and then reflects
the whole transaction in the database.
Relational Database Structure
The installation step defines the database and the database structure. Whether the database is
an Oracle Database, IBM® DB2®, or Microsoft SQL Server affects the structure of the
database.
a database that may be used to edit data using the relational model of data and is viewed as a
collection of tables. Every database contains:
·
a collection of system catalog tables outlining the data's logical and physical organisation
·
a configuration file that includes the database's parameter values
·
a recovery log with active transactions and transactions that can be archived
The logical data structures, such as data tables, views and indexes, are differentiated from the
physical storage structures due to the utilisation of the relational paradigm. As a result of this
segregation, database administrators possess the ability to govern the physical storage of data
without exerting any influence over the logical access to this data. For example, altering the
name of a database file does not have any impact on the tables that are contained within it.
Database operations encompass well-defined actions that enable program s to modify the data
and structures within a database. These operations can be classified into two categories: logical
and physical. Physical operations are responsible for determining the means by which data is
accessed and subsequently executing the required activity. On the other hand, logical
operations allow an application to articulate the specific information it requires.
Relational databases conform to specific integrity criteria in order to ensure the consistent
accuracy and availability of data. In order to mitigate the inclusion of erroneous data into the
database, an integrity rule may be established, wherein the prohibition of duplicate rows inside a
table is stipulated.
·
Data dictionary tables: The data dictionary tables of the database contain the structure of a
relational database.
·
Integrity checker: The integrity checker, a database configuration tool, can be utilised to
assess the state of the base layer data dictionary. The tool assesses the conformity of the data
dictionary with the underlying schema of the actual database. In the event that defects are
detected, the program will generate error messages that encompass detailed advice for
rectifying the identified issues.
·
Storage divisions: A database storage partition designates the area of a disk where a
database object is kept. Table spaces and file groups are the names of database storage
divisions in DB2 and Oracle and SQL Server, respectively.
·
Business objects are objects with a number of attributes and values, as well as operations
and connections to other business objects. Business data is contained in business objects,
which also model business behaviour.
·
User-defined objects: Objects can be generated in two different ways: by being natively
defined in the database, or by being created in the database. The Database Configuration app
always creates user-defined objects.
·
Object configuration levels: Levels must be applied to objects in order to indicate their
scope. You must build different properties for various items based on the level you give them.
An attribute value must exist at the level where users have authorisation in order for users to
access an object. Sometimes, the level of the database record determines the level that you
give to an item.
·
Database relationships: Join statements are used to construct associations between tables
in order to obtain data.
·
Business object characteristics Data that is related to a business object is contained in its
attributes. A database view column or a database table column is represented by a persistent
attribute. Because the data that is associated with a nonpersistent property is not kept in the
database, it only exists in memory.
·
Types of attribute data Many attributes are present in each database record. Each attribute
has a corresponding data type.
·
Database views: Based on a query that is executed on one or more database tables, a
database view is a subset of a database. Database views can be used to store sophisticated,
frequently used queries because they are preserved in the database as named queries.
·
Indexes: You can use indexes to improve data retrieval performance. Pointers to
commonly accessible data locations are provided by indexes. On the columns in an object that
you commonly query, you can establish an index.
·
Primary keys: When you give an attribute a primary key, the attribute's associated object is
uniquely identified by the key. Which attributes are used to build the main key depends on the
value in the primary column.
The relational model
During the early stages of database development, many applications employed distinct
structures for data storage. In order to effectively utilise data while developing applications,
developers must possess a comprehensive understanding of the unique data structure,
enabling them to locate the essential data. The aforementioned data structures exhibited limited
efficacy, posed difficulties in maintenance and presented challenges in optimising for optimal
application performance. The existence of several disparate data structures was a significant
difficulty, prompting the development of the relational database paradigm.
The standard representation and querying of data provided by the relational data model can be
utilised by any application. From the beginning, it has been recognised by developers that the
primary advantage of the relational database architecture is its utilisation of tables. These tables
offer a versatile, logical and effective method for the storage and retrieval of structured data.
With the adoption of structured query language (SQL) by database writers and query writers,
the relational model began to reveal an additional advantage. For numerous years, SQL has
served as the predominant language utilised for executing database queries. SQL, derived from
relational algebra, provides a mathematically coherent language that facilitates the optimisation
of database query performance. On the other hand, alternative approaches necessitate the
individual definition of each question.
Advantages and Disadvantages
Advantages
1. Data accuracy: The practice of data collection is commendable; nonetheless, it is imperative
to ensure responsibility for the accuracy and reliability of the processed data. Unfortunately, the
presence of duplication and redundancy issues inside conventional databases can lead to
significant mistakes, while also consuming a considerable amount of important time. In a
relational database, it is possible to establish connections across several tables, thereby
ensuring data accuracy and eliminating redundancy.
For example, in the event of an order being made, the address of the respective client can be
extracted directly from the table, as it encompasses all pertinent information about the
consumer inside a singular record. In the context of a relational database, it is possible to
consistently access the latest information without redundancy, even if its location is altered. This
is in contrast to the scenario where data is updated in one table but neglected in other tables,
resulting in potential complications in the future.
2. Data integrity: The integrity of the information in databases is another significant problem that
frequently arises. According to established criteria, a relational database features data type and
validity checks that guarantee the input is entered correctly. In order to ensure that the data is
complete, it also issues warnings when data is missing. Another excellent fail-safe is the
relational component. Because tables depend on one another to run, keeping records is
encouraged to prevent isolation and imperfection. This ensures precision and consistency.
3. High security: Relational databases employ multiple tables instead of requiring data to be
stored in a single, huge file. This suggests that certain information can be restricted from
individuals who do not have a need for it and can only be accessed by those who have a
legitimate requirement. For example, a sales manager may require access to numerical data
pertaining to their subordinates, whereas the product design team may not necessarily want
such information.
The roster of employees may potentially be accessible to a personal aide, although the
remuneration details of those personnel would not be included. Individuals utilising a relational
database possess the capability to access data from tables that are relevant to their specific
needs, all the while ensuring the confidentiality and restricted access to sensitive information.
In other words, adopting data segregation can improve security because some tables might be
private while others are open to all employees. The relational database selects which tables an
employee has access to when they input their username and password, protecting critical data.
4. Easy to access: In relational databases, concurrent access to data by several users is
possible, even during periods of data modification. The simultaneous retrieval or updating of
information by multiple individuals in commercial contexts might be of utmost importance. Unlike
other models, the process of gathering data does not follow a hierarchical pattern. Individuals
who possess the necessary authorisation can readily navigate the database, executing queries
on various tables and merging data from many tables to locate and examine the specific
information they seek.
5. Normalisation: Anomalies are debatably the worst thing when it comes to data storage. These
might result from the way information is entered, for instance. Because of this, a normalisation
process is essential for a reliable and useful database. It is possible to ensure the robustness
and dependability of the RDBMS by supplying a set of guidelines, traits and goals for its
architecture and structure.
While it is technically feasible to normalise any directory, a relational database offers a more
efficient and effective approach to achieving normalisation. The inclusion of standards and
objectives for the interconnections among different tables introduces an additional level of
security. This can make data more usable, safeguard it andguarantee its reliance. For instance,
normalisation will change a table of orders from one that displays a row for each order to one
that displays a row per client, with each order represented by a distinct column. The client
information is just displayed once and is therefore easier to obtain rather than being repeated.
6. Future Modification: Databases take time to set up and manage, therefore they should be
able to withstand changes in the future. They are more than simply a passing notion. Another
crucial element to a database is the categorisation of tables in an RDBMS, which makes it
easier to update records and structures in the future.
7. Flexibility: Your database serves varied objectives because no two firms are same. In
contrast to a business that offers an online service or produces items, for example, one that
sells goods will use data in a specific manner. Maximum flexibility is possible with a relational
database since you may design the exact tables you need and link them together based on your
unique requirements.
8. Simplicity: Relational databases are not sophisticated, but they are reliable. The relational
structure is free from query processing and complex, hierarchical systems, whereas other forms
of databases could need years of training or the ability to write code. The table structure is wellknown to most individuals and straightforward SQL queries will extract everything you need. It is
more simpler to build and utilise because it feels more natural to users when using it.
Disadvantages
Relational databases are frequently used in a variety of businesses to store financial data, track
merchandise and maintain employee information. Information is kept in tables (commonly
referred to as relations) in a relational database, which helps structure and organise data.
Relational databases have some shortcomings despite their widespread use.
Cost: One disadvantage of relational databases is the financial burden associated with the initial
setup and ongoing maintenance of the database system. In the majority of instances, it is
necessary to get specialised software for the purpose of establishing a relational database. A
relational database can be constructed via a diverse range of technologies, even in the absence
of coding proficiency.
The procedure of program installation and inputting essential data does require a significant
amount of time. If a business is of considerable size and requires a more robust database, it
becomes imperative to employ a program r proficient in Structured Query Language (SQL) for
the construction of a relational database, as well as a database administrator to oversee its
management post-construction. In order toutilise any desired data, it is necessary to either
manually input the data using the keyboard or import it from alternative data sources such as
text files or Excel spreadsheets.
Regardless of the scale of your enterprise, it is imperative to implement measures to safeguard
your data from unauthorised access. This is particularly crucial if your database contains legally
protected or personal information, such as health records, social security numbers, or credit
card details, in order to comply with regulatory regulations.
Abundance of Information: One additional challenge associated with relational databases is to
the escalation of information complexity. Relational databases are specifically engineered to
facilitate the categorisation of data based on shared characteristics. The presence of intricate
images, statistics, patterns and multimedia assets poses a challenge in terms of their
categorisation, therefore necessitating the development of object-relational database
management systems, a novel type of database. These systems possess the ability to scale
and are designed to effectively manage increasingly complex applications.
Stuctured Limits: Field lengths in certain relational databases are subject to limitations. When
constructing a database, it is necessary to specify the capacity of a field to accommodate data.
Data loss may occur when there is a discrepancy in the length of names or search words.
Isolated Databases: Complex relational database systems possess the capacity to generate
"islands of information" wherein the seamless flow of data between substantial systems
becomes challenging. With large companies or institutions, relational databases frequently
developed in several divisions in a variety of ways. For instance, it's possible that the hospital's
billing department utilised one database while the hospital's personnel department used
another. It can be difficult and expensive to get these databases to "talk" to one another, yet
effective patient and employee care depends on all the databases being involved in a complex
hospital system.
MySQL
Numerous small and large organisations employ MySQL, a rapid and straightforward relational
database management system (RDBMS). MySQL AB is a Swedish firm that engages in the
creation, promotion and provision of support for the MySQL database management system.
There exist numerous compelling factors contributing to the increasing popularity of MySQL,
which include:
·
The distribution of MySQL is governed by an open-source licence. Consequently, there is
no financial burden associated with its utilisation.
·
MySQL is a highly powerful program in its own right. It effectively oversees a substantial
percentage of the advanced functionalities offered by high-end database solutions.
·
MySQL utilises a standardised iteration of the widely recognised SQL data language.
·
MySQL supports a wide range of operating systems and programming languages,
including but not limited to PHP, PERL, C, C++ and JAVA.
·
Despite the presence of large amounts of data, MySQL demonstrates efficient and rapid
performance.
·
PHP, a highly popular programming language for web development, has strong
compatibility with MySQL.
·
MySQL has the capability to accommodate tables with a substantial number of rows,
reaching up to 50 million or even more, particularly in the context of managing extensive
databases. If the operating system has the necessary capabilities, it is possible to enhance the
default file size limit for a table from 4GB to a theoretical upper limit of 8 million terabytes (TB).
·
MySQL has the capability to be tailored to specific requirements. Program rshave the
ability to modify the MySQL software to suit their own contexts, owing to the open-source GNU
General Public Licence (GPL).
MySQL – Installation
All MySQL downloads can be found at MySQL Downloads (http://www.mysql.com/downloads).
Choose the required MySQL Community Server version number and the platform you'll be
running it on.
MySQL installation on Linux/UNIX RPM is the suggested method for installing MySQL on a
Linux machine. The following RPMs are available for download from the website of MySQL AB:
·
MySQL - The MySQL database server regulates user access, manages databases and
tables and handles SQL queries.
·
The MySQL client software allows users to connect to and communicate with the server.
·
Libraries and header files from MySQL-devel are useful when putting together other
programs that use MySQL. Shared libraries for the MySQL client are known as MySQL-shared.
·
Tools for benchmarking and performance testing the MySQL database server include
MySQL-bench.
The MySQL RPMs listed here were all created on SuSE Linux Systems, but they should all run
well on other Linux distributions as well.
To continue with the installation, you must now follow these instructions:
·
Use the root user to log into the system.
·
Go to the directory where the RPMs are located.
·
By running the following command, the MySQL database server can be installed. Never
forget to substitute your RPM's file name for the one in italics.
[root@host]# rpm -i MySQL-5.0.9-0.i386.rpm
The aforementioned command handles MySQL server installation, user creation, configuration
creation and automatic MySQL server startup.
All MySQL-related binaries can be found under /usr/bin and /usr/sbin. The /var/lib/mysql
directory will be used to create all databases and tables.
To install the remaining RPMs in the same way, there is an optional but suggested step in the
code box after this one:
[root@host]# rpm -i MySQL-client-5.0.9-0.i386.rpm
[root@host]# rpm -i MySQL-devel-5.0.9-0.i386.rpm
[root@host]# rpm -i MySQL-shared-5.0.9-0.i386.rpm
[root@host]# rpm -i MySQL-bench-5.0.9-0.i386.rpm
Installing MySQL on Windows
The inclusion of an installer for MySQL has significantly simplified the default installation
process on all Windows versions, compared to prior iterations.
To initiate the software installation process, please proceed by downloading the installer
package. Once the download is complete, extract the contents of the package to a desired
location on your system. Finally, execute the setup.exe file to commence the software
installation. The default installer, setup.exe, will provide step-by-step instructions for a
straightforward installation process, automatically installing all components into the C:mysql
directory.
The first time, start the server from the command prompt to test things out. Go to the mysqld
server's location, which is likely C:mysqlbin and type:
mysqld.exe –console
You will see some startup and InnoDB-related notifications if everything went fine. If not, a
permissions problem might exist. Make sure that whichever user (likely MySQL) the database
processes run under can access the directory that contains your data.
There is no especially elegant GUI method to shutdown the server and MySQL won't
automatically add itself to the start menu. As a result, if you frequently launch the server by
double clicking the mysqld executable, keep in mind that you should manually halt the process
using mysqladmin, Task Manager, Task List, or other Windows-specific tools.
The process of confirming the installation of MySQL
After the successful installation of MySQL, initialisation of the base tables and commencement
of the server, it is advisable to conduct a series of brief tests to verify the proper working of the
system.
Use the mysqladmin utility to check the status of the server.
To check the server version, use the mysqladmin binary. On Linux, this binary would be located
at /usr/bin and on Windows, it would be located in C:mysqlbin.
[root@host]#mysqladmin –version
On Linux, it will result in the following outcome. Depending on your installation, it might change:
mysqladminVer 8.23 Distrib 5.0.9-0, for redhat-linux-gnu on i386
If you do not receive such a message, your installation may have a problem and you will need
assistance to fix it.
Use the MySQL Client to run basic SQL commands.
Using the mysql command and the MySQL client, you can establish a connection to your
MySQL server. You do not currently need to provide a password because it will be left blank by
default.
You only need to use the command [root@host]#mysql
It ought to result in a mysql> prompt. Now that you are connected to the MySQL server, you
may use the mysql> prompt to run any SQL statement.
mysql> SHOW DATABASES;
+----------+
| Database |
+----------+
| mysql |
| test |
+----------+
2 rows in set (0.13 sec)
Post-installation Steps
The root MySQL user's password is left blank when MySQL is installed. After the database and
client have been successfully installed, you must set the root password specified in the following
code block:
[root@host]#mysqladmin -u root password "new_password";
Now, you would need to execute the following command to establish a connection to your
MySQL server:
[root@host]#mysql -u root -p
Enter password:*******
In order to avoid having to repeatedly type the whole path every time you wish to use the
command-line client, UNIX users will also want to add your MySQL directory to your PATH.
It would be something like - for bash.
export PATH=$PATH:/usr/bin:/usr/sbin
MySQL – Administration
Runningand Shutting down MySQL Server
First, make sure your MySQL server is operational. The command listed below can be used to
verify it:
ps -ef | grepmysqld
If MySql is active, you will see a list of the mysqld process in your search results. Using the
following command, you can start the server if it is not already running:
root@host# cd /usr/bin
./safe_mysqld&
Now, if you wish to stop a MySQL server that is already running, you can do so by using the
command:
root@host# cd /usr/bin
./mysqladmin -u root -p shutdown
Enter password: ******
Setting Up a MySQL User Account
To create a new user, it is sufficient to append a new entry to the user table within the MySQL
database.
The program demonstrates the process of creating a new user with the privileges of SELECT,
INSERT and UPDATE, using the password "guest123".
root@host# mysql -u root -p
Enter password:*******
mysql> use mysql;
Database changed
mysql> INSERT INTO user(host, user, password,
select_priv, insert_priv, update_priv)
VALUES ('localhost', 'guest',
PASSWORD('guest123'), 'Y', 'Y', 'Y');
Query OK, 1 row affected (0.20 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 1 row affected (0.01 sec)
mysql> SELECT host, user, password FROM user WHERE user = 'guest';
+-----------+---------+------------------+
| host | user | password |
+-----------+---------+------------------+
| localhost | guest | 6f8c114b58f2ce9e |
+-----------+---------+------------------+
1 row in set (0.00 sec)
Remember to use MySQL's PASSWORD() function to encrypt the new password when adding
a new user. Mypass' encryption in the example above is 6f8c114b58f2ce9e, as you can see.
Take note of the words FLUSH PRIVILEGES. The server is instructed to reload the grant tables
by this. Otherwise, until the server is restarted, you won't be able to connect to MySQL using the
new user account.
By setting the values of the following columns in the user table to 'Y' when performing the
INSERT query or by updating them later using the UPDATE query, you can additionally provide
additional privileges for a new user.
·
Select_priv
·
Insert_priv
·
Update_priv
·
Delete_priv
·
Create_priv
·
Drop_priv
·
Reload_priv
·
Shutdown_priv
·
Process_priv
·
File_priv
·
Grant_priv
·
References_priv
·
Index_priv
·
Alter_priv
Using the GRANT SQL command is an additional method of adding user accounts. The user
zara will be added to the RDMSTEST database in the example below with the password
zara123.
root@host# mysql -u root -p password;
Enter password:*******
mysql> use mysql;
Database changed
mysql> GRANT SELECT,INSERT,UPDATE,DELETE,CREATE,DROP
-> ON RDMSTEST.*
-> TO 'zara'@'localhost'
-> IDENTIFIED BY 'zara123';
Additionally, a new record will be made in the user table of the MySQL database.
NOTE: Until you add a semicolon (;) at the end of a SQL command, MySQL will not end the
command.
The /etc/my.cnf File Configuration
You should generally avoid modifying this file. It will have the following entries by default:
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
[mysql.server]
user=mysql
basedir=/var/lib
[safe_mysqld]
err-log=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
Otherwise, you shouldn't update any entry in this table; instead, you can designate a different
path here for the error log.
Administrative MySQL Command
The list of key MySQL commands that you will frequently use when working with a MySQL
database is provided below:
·
The command "USE Databasename" is utilised to specify a particular database within the
MySQL work area.
·
The command "SHOW DATABASES" is used to display a list of databases that may be
accessed by the MySQL Database Management System (DBMS).
·
The "SHOW TABLES" command is used to display the tables within a selected database
once the "USE" command has been executed.
·
The command "SHOW COLUMNS FROM tablename" is used to display various details
about the attributes of a table, including their names, data types, key information, nullability,
default values and other relevant information.
·
The command "SHOW INDEX FROM tablename" provides a comprehensive overview of
all indexes associated with the specified table, including the PRIMARY KEY.
·
The command "SHOW TABLE STATUS LIKE tablename\G" provides a comprehensive
overview of the performance and statistical information related to the MySQL Database
Management System (DBMS).
MySQL – PHP Syntax
Combining MySQL with other programming languages like PERL, C, C++, JAVA and PHP
works quite well. PHP is the most widely used of these languages due to its potential for
creating web applications. In this topic, the use of MySQL in a PHP context is emphasised
significantly.
Numerous functions are available in PHP to access the MySQL database and modify the data
stored inside. The PHP functions must be called in the same manner as other PHP functions.
The general format of the PHP functions used with MySQL is as follows:
mysql_function(value,value,...);
A word that describes the function's purpose typically makes up the second part of the function
name. Two of the functions that we shall employ are as follows:
mysqli_connect($connect);
mysqli_query($connect,"SQL statement");
The example below demonstrates a general PHP syntax for calling any MySQL function.
<html>
<head>
<title>PHP with MySQL</title>
</head>
<body>
<?php
$retval = mysql_function(value, [value,...]);
if( !$retval )
{
die ( "Error: a related error message" );
}
// Otherwise MySQL or PHP Statements
?>
</body>
</html>
MySQL – Connection
Connection to MySQL Using MySQL Binary Using the mysql binary at the command prompt,
you can create the MySQL database.
Example To connect to the MySQL server using the command prompt, follow this easy
example:
[root@host]#mysql -u root -p
Enter password:******
You will then be presented with the mysql> command prompt, from which you may issue any
SQL statement.
The output of the preceding command is shown in the code block below:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2854760 to server version: 5.0.9
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
You can use any other user in place of root, like we did in the example above.
Any user will be able to execute all SQL operations that user is permitted to.
Using the exit command at the mysql> prompt, you can detach from the MySQL database at
any moment.
mysql> exit
Bye
MySQL Connection Using PHP Script
To establish a database connection, PHP offers the mysql_connect() function. When
successful, this function produces a MySQL link identification; otherwise, it returns FALSE. It
requires five parameters.
Syntax:
connection mysql_connect(server,user,passwd,new_link,client_flag);
MySQL commands:
A popular open-source RDBMS database, MySQL offers customers scalable and highperformance web-based and embedded database applications. It includes a variety of helpful
commands and statements for using embedded and web-based database applications. The
command is a strong directive in MySQL that is used to carry out a certain action or service.
Knowing and using these queries to communicate with the MySQL databases in real-time is
incredibly beneficial for every developer. The structured query language (SQL) that MySQL
commands are based on can support a number of operating systems, including Linux, UNIX,
macOS and Windows.
The following command will display a list of all commands in MySQL:
mysql>HELP;
Let's look at the most popular commands and statements needed to set up and maintain a
MySQL database.
MySQL command-line client commands
This command enables us to establish a connection to the MySQL Server using the login and
passwords listed below.
mysql -u [username] -p;
Use the following syntax to connect to a specific database:
mysql -u [username] -p [database];
Use this syntax to create a new password:
mysqladmin -u root password your_password;
The "exit" command can be used to terminate the MySQL command-line client.
Using the command listed below, we can clear the terminal window in Linux:
mysql> system clear;
It is important to acknowledge that there is currently no built-in command available in the
Windows operating system that specifically clears the console window of MySQL.
Working with Database
Users have the ability to efficiently retrieve and oversee the organised compilation of data
entries that are kept within a database. The data is structured into tables, rows, columns and
indexes, facilitating efficient retrieval of required information. The following syntax can be
utilised to construct a database. Furthermore, it ascertains whether the database name has
already been utilised.
CREATE DATABASE IF NOT EXISTS db_name;
Use the syntax listed below to replace the current database with a different one you are working
on:
mysql> use db_name;
Using the syntax shown below, we can permanently delete a particular database and the files
that are associated with it:
DROP DATABASE IF EXISTS db_name;
To retrieve a list of all databases present on the current server, the following syntax can be
employed:
mysql> SHOW DATABASES;
Working with tables
A database table is a grouping of linked data that is recorded in a row-and-column structure.
The syntax listed below can be used to construct a new table. Additionally, it determines
whether the table name is already in use.
CREATE TABLE IF NOT EXISTS tab_name (
column_list (s)
);
To add a record to the table, we can use the INSERT statement, which is provided below:
INSERT INTO table_name( field1, field2,...fieldN ) VALUES ( value1, value2,...valueN );
The syntax listed below can be used to permanently delete a specific table:
DROP TABLE IF EXISTS tab_name;
Use the following syntax to display all tables in the current database:
mysql> SHOW TABLES;
Use the following statements to add a new column to the table or to remove any existing
columns:
-- Add a new column
ALTER TABLE tab_name ADD [COLUMN] colm_name;
-- Delete column
ALTER TABLE tab_name DROP [COLUMN] colm_name;
Using the following statements, we may add a primary key to the table or remove it from the
table:
-- Add a primary key
ALTER TABLE tab_name ADD PRIMARY KEY (colm_names);
-- Delete primary key
ALTER TABLE tab_name DROP PRIMARY KEY;
Use the following syntax to add an index with a specified name to a column:
ALTER TABLE tabl_name ADD INDEX [name] (column(s));
The following sentence can be used to demonstrate the structure of a table:
DESCRIBE tab_name;
The following sentence can be used to display a table's column information:
DESCRIBE tab_namecolm_name;
Working with Indexes
An index is a type of data structure that enables quicker record retrieval from a database table.
Using the following statement, we can add an index to the table:
CREATE INDEX [index_name] ON [table_name] (column names);
To drop an index, say the following:
DROP INDEX index_name;
Working with View
By combining one or more tables, the View function in MySQL allows you to construct a virtual
table. Using SELECT commands, we can add or remove a VIEW as follows:
CREATE [OR REPLACE] VIEW IF NOT EXISTS view_name AS SELECT column_list
FROM tables WHERE conditions;
The following statement can be used to remove a view:
DROP VIEW [IF EXISTS] view (s).;
The following statement can be used to rename a view:
RENAME TABLE view_name TO new_view_name;
Use the sentence below if you want to see all the views that are present in the database:
SHOW FULL TABLES [{FROM | IN }db_name] WHERE tab_type = 'VIEW';
Working with Stored Procedure
An SQL statement collection with a specific name that accepts inputs and executes specific
tasks is known as a stored procedure. The same code can be written in a database repeatedly
to save time. These MySQL statements can be used to create a stored procedure:
CREATE PROCEDURE procedure_name[ (parameter_list) ]
BEGIN
Declaration_section
Executable_section
END;
Use the statement below if you want to delete an existing stored procedure from the database:
DROP PROCEDURE [IF EXISTS] procedure_name;
Use the statement below if you wish to display all procedures that are available in the database:
SHOW PROCEDURE STATUS
[LIKE 'pattern' | WHERE condition];
Working with Triggers
A database trigger is a piece of procedural code that is automatically run whenever specific
events on a given table or view occur. The statements listed below can be used to establish a
new trigger:
CREATE TRIGGER trigger_name
{ AFTER | BEFORE } {INSERT | UPDATE| DELETE }
ON tab_name FOR EACH ROW
BEGIN
--variable declarations
--trigger code
END;
Use the following sentence to remove an existing trigger:
DROP TRIGGER [IF EXISTS] trigger_name;
Use the statement below to display all triggers that are present in the database:
SHOW TRIGGERS
[{FROM | IN} db_name]
[LIKE 'pattern' | WHERE condition];
Working with Stored Functions
A stored function in MySQL refers to a collection of SQL statements that are designed to
perform a certain activity or task and yield a solitary value. This feature belongs to one of the
stored program categories in MySQL. It is vital to own the CREATE ROUTINE database
privilege prior to the construction of a stored function. The subsequent text presents the syntax
for composing a stored function in MySQL.
DELIMITER $$
CREATE FUNCTION function_name(parameter(s))
RETURNS datatype
[NOT] {DETERMINISTIC, NO SQL, or READS SQL DATA}
BEGIN
-- SQL statements
END $$
DELIMITER ;
Use the following sentence to remove an existing trigger:
DROP FUNCTION [IF EXISTS] fun_name;
Use the following sentence if you want to display all triggers that are present in the database:
SHOW FUNCTION STATUS
[LIKE 'pattern' | WHERE condition];
Popular queries based on tables
Here, we'll see a few of the crucial table queries that were run to locate the necessary data.
Use the following syntax to display all table data:
SELECT * FROM tab_name;
Use the following syntax to display the data from a set of chosen columns in a table:
SELECT colm1, colm2... FROM tab_name;
The following statement can be used to retrieve the table's filtered results:
SELECT column_list(S) FROM tab_name WHERE condition;
Our tables occasionally have duplicate rows. However, we only wish to display distinct (unique)
rows, in which case the following statements will be carried out:
SELECT DISTINCT (column) FROM tab_name;
There are situations when a table's column names are inappropriate for the output. In that
scenario, we may use the following statement to modify the result's column name:
SELECT
Column(s) AS alias_name(s),
Expression AS alias.....
FROM tab_name;
The JOIN operation must be used to retrieve the output from many tables. Following is the
syntax for combining two tables:
SELECT select_list
FROM table1
JOIN table2 ON condition;
With this assertion, we can determine how many rows there are:
SELECT COUNT(*) FROM tab_name;
The following statement can be used to order the result set:
SELECT column_list FROM tab_name
ORDER BY colm1 ASC [DESC], colm2 ASC [DESC];
Use the syntax listed below to arrange rows into the result set:
SELECT column_list FROM tab_name
GROUP BY colm1, colm2 ...;
Use the syntax shown below to change the table's rows:
UPDATE table_name
SET column1 = value1 ... WHERE condition;
The syntax listed below can be used to remove every record from the table:
DELETE FROM tab_name;
Searching data from the table
The LIKE clause can be used to search for data, as demonstrated below:
SELECT column_list FROM tab_name
WHERE column LIKE '%pattern%';
As demonstrated below, we can also find text using a regular expression and the RLIKE
operator.
SELECT column_list FROM tab_name
WHERE column RLIKE 'regular_expression';
Control Option to the Administrator
Here, we'll look at the features that give database administrators and users control options.
System administrators can provide MySQL user accounts access to the database by defining
roles and privileges in the grant statement.
GRANT privilege_name(s)
ON object
TO user_account_name;
The revocation statement can be utilised by system administrators to withdraw privileges and
roles from user accounts that have previously accessed the database.
REVOKE privilege_name(s)
ON object
FROM user_account_name;
PostgreSQL
An open-source object relational DBMS (ORDBMS) is PostgreSQL. This DBMS has academic
roots because it was created by the academic community. It was initially created as a Postgres
database (created at UC Berkeley in the early 1980s). Around 1996, primarily, it was given the
official name PostgreSQL to reflect the addition of an ANSI SQL compatible translator. One of
the most robust, feature-rich open-source databases is this one.
PostgreSQL is an example of an object relational database management system. This resource
offers a comprehensive overview of the fundamental principles of object-oriented programming,
encompassing key concepts such as inheritance, complicated data types andspecialised
methods designed specifically for manipulating these data kinds.
However, the most of it primarily pertains to relationships. In reality, the extensive objectoriented capabilities of PostgreSQL are frequently not fully utilised. The attributes of
PostgreSQL that render it an exemplary database management system (DBMS) are
enumerated as follows:
·
Full ANSI-SQL 92 compliance: It supports:
o Additionally, the majority of ANSI 99 compliance, broad transaction support,
o Support is provided for BEFORE and AFTER triggers, the creation of stored procedures,
constraints and referential integrity with cascade update/delete.
·
The creation of user-defined database functions can be done using a variety of high-level
languages and native interfaces.
·
Native SQL, PgSQL (Postgres' version of Oracle PL/SQL or MS SQL Server's/Sybase
Transact-SQL), Java, C, C++ and Perl are all options.
·
Table structure inheritance is perhaps one of the more underutilised capabilities.
·
One of the key features of programming is the capacity to create custom data types that
possess specific attributes, operators and functions. The database incorporates advanced data
types, like IP Address, Geometries (Points, Lines and Circles) and Arrays, as built-in features.
·
Aptitude for defining aggregate functions.
·
Collections and sequences idea.
·
The software provides compatibility with a diverse range of operating systems,
encompassing Mac, Linux, Windows and Unix.
·
This database can be considered a vital resource for implementing web-based
applications. This might be attributed to its efficient performance and comprehensive range of
features. PostgreSQL, a database system with modest speed, provides suitable support for web
programming languages such as PHP and Perl.
Furthermore, due to its compatibility with ODBC and JDBC drivers, MySQL can be conveniently
utilised in many programming languages such as ASP, ASP.Net and Java. MySQL is often cited
as a rapid database solution on the internet, regardless of its open source status. The query
processing time of the system is similar to that of MySQL. Nevertheless, when considering its
characteristics, PostgreSQL is undoubtedly a database that merits careful consideration.
PostgreSQL Architecture
The PostgreSQL fundamental architecture on the Unix operating system is depicted in the figure
below.
Figure: The Architecture of PostGresSQL
A straightforward process per-user client/server model is used by Postgres.
The following processes cooperate throughout a session on a PostgresSQL database:
·
The PostGres process itself, a supervisory daemon process (also known as postmaster),
·
The front-end user application process, such as the psqlprogram , is accompanied by one
or more back-end database server processes.
A singular postmaster process is responsible for managing a database collection, referred to as
an installation or site, on a solitary host machine. The client interface library is responsible for
receiving requests from client program s that require access to a database stored at a
designated installation. The initiation of a novel backend server process by the library involves
the transmission of user requests to the postmaster across the network. Subsequently, the
postmaster establishes a connection between the client process and the newly designated
server.
Henceforth, the client process and the backend server process engage in mutual interaction
without the intervention of the postmaster. Consequently, the postmaster process remains
consistently active and prepared to receive requests from new clients. It is important to note that
client and server activities may be initiated and concluded as required.
Can a client process connect to a backend server process more than once? A single client can
establish many connections to backend server processes using the libpq library. Please take
note that these client processes are single-threaded processes, not multi-threaded ones. Libpq
does not currently support multithreaded front-end and back-end connections.
Files that are accessible on a client machine, subject to appropriate permissions, may not be
accessible on the database server machine, or may require the use of a different filename for
access. The system architecture involves the utilisation of either a client-server model or a frontend/back-end configuration, wherein various operations are distributed across separate
workstations.
Please be aware that the Postgressuperuser or administrator's user ID is used to manage the
Postmaster and Postgres servers. Also, keep in mind that the Postgressuperuser does not
necessarily need to be a unique user and that it should never be the root user of a UNIX
system. To this Postgressuperuser belong all database-related files.
Figure below shows the establishing of connection in PostgresSQL.
Figure: Establishing connection in PostgresSQL
User interfaces
The following PostgreSQL interfaces are available for information access:
·
Programs for Postgres terminal monitoring, such as psql: It is a SQL command level
interface that enables interactive entry, editing and execution of SQL instructions.
·
Programming Interface: The LIBPQ subroutine library can be used to create C programs.
Because of this, SQL instructions can be sent from the host language, C and answers and
status messages can be returned to your program.
Setting up Postgres on your computer: As a user, you must complete the client installation
because Postgres is a client/server database management system (a client application interface
is the interactive monitor psql). /usr/local/pgsql is one of the often used directories where
PostGres may be installed on Unix-based computers. We will thus presume that Postgres has
been set up under the /usr/local/pgsql directory. You should replace this directory name with the
name of the directory where PostGres was installed if you did. All PostGres commands are set
up in the /usr/local/pgsql/bin directory. As a result, in Unix, you must add this directory to your
shell command path.
For instance, you must add the following to the Berkeley C shell or any of its variations, such as
csh or tcsh:
% set path = ( /usr/local/pgsql/bin path )
in the home directory's .login file.
You must include the following on the Bourne shell or its variations, such as sh, ksh, or bash:
% PATH=/usr/local/pgsql/bin
PATH % export PATH
to your home directory's profile file.
Various Interfaces Other user interfaces for PostgreS include pgAdmin 3 from
http://www.pgadmin.org for Windows, Linux, BSD/nix and an experimental port for Mac OS X.
An entire PostgreSQL administrative interface that was made available under the Artistic
License. Written in C++ and wxWindows, it resembles Microsoft's Enterprise Manager in certain
ways. It permits ad hoc inquiries and administration of practically all database items.
The original PostgreSQL GUI is available for most platforms as PGAccess at
http://www.pgaccess.org. It is a Tcl/Tk-created database browser in the MS Access style. It
supports graphically assisted queries as well as browsing, adding and updating tables, views,
functions, sequences, databases and users. Additionally being developed is a designer for
forms and reports.
On the Internet, there are numerous open source tools that are comparable. Let's now get into
more detail about the PostgreSQL interface that is most frequently used, psql.
Opening the Interactive Monitor (psql) You can handle a client application if:
·
With the correct user id and password, you are permitted to access the database because
the site administrator properly initiated the postmaster procedure.
Two distinct connection styles are supported as of Postgres v6.3. Which are:
·
TCP/IP network connections or
·
Restricted database access to local (same-machine) socket connections only
These options are important in the event that you have trouble connecting to a database. For
instance, if a Postgres command (like psql or createdb) returns the following error message:
% psql template1
Connection to database 'postgres' failed.
connectDB() failed: Is the postmaster running and accepting connections at 'UNIX
Socket' on port '5432'?
or
% psql -h localhost template1
Connection to database 'postgres' failed.
connectDB() failed: Is the postmaster running and accepting TCP/IP (with -i)
connections at 'localhost' on port '5432'?
It is either because the postmaster is not functioning or because you are trying to connect to the
incorrect server host. Similar to the previous error message, the next one indicates that the
postmaster was started as the incorrect user by the website administrator.
FATAL 1:Feb 17 23:19:55:process userid (2360) != database owner (268)
Accessing a Database
The next step is to begin viewing the database after you have a legitimate account. You can use
the following command to access the database using PostgreSQL's mydb database:
% psqlmydb
The following message could appear:
Welcome to the POSTGRESQL interactive sql monitor:
Please read the file COPYRIGHT for copyright terms of POSTGRESQL
type \? for help on slash commands
type \q to quit
type \g or terminate with semicolon to execute query
You are currently connected to the database: template1
mydb=>
The prompt indicates that your SQL queries are ready to be run on the terminal monitor. These
inquiries must be entered into a workspace that the terminal monitor keeps up. The psql
program also responds to escape codes that start with the backslash character, "\" which you
must have used when writing in C. You can access guidance on Postgre SQL command syntax,
for instance, by typing:
mydb=> \h
When the query is finished, type: to send the workspace's contents to the Postgres server.
mydb=> \g
This tells the server that it's okay to handle the query. The "\g" is not required if the query is
ended with a semicolon. The semicolon at the end of a query's line causes psql to process it
automatically.
Your queries can be kept in a file. You can type: to read your queries from a file like this:
mydb=> \i filename
To exit psql and return to UNIX, type
mydb=> \q
SQL queries may contain white space, such as spaces, tabs and new line characters. You can
leave comments as well. "--" is used to indicate single-line comments. "/*... */" indicates
multipleline comments and comments inside a line.
SQL Variation and Extensions
In 1986, the American National Standards Institute (ANSI) standardised SQL. It was
standardised in 1987 by the International Standards Organisation (ISO). The ANSI/ISO
standard was adopted by the US government's Federal Information Processing Standard (FIPS)
in 1989 and a new standard known as SQL89 or SQL1 was issued.
To serve the needs of commercial DBMS developers, the SQL89 standard was purposefully left
unfinished. But the ANSI committee strengthened the standard with the 1992-approved SQL92
standard (also known as SQL2). This standard addressed a number of shortcomings in SQL89
and introduced conceptual SQL features that, at the time, were more advanced than those
offered by the RDBMSs of the day. In actuality, the SQL92 standard was about six times longer
than its forerunner. Due to the size of SQL 92, the standards' authors established three levels of
compliance: Entry-level conformance (only minimal improvements over SQL 89), Intermediatelevel conformance (a set of significant advancements that are typically attainable) and Full
conformance (full compliance with all SQL 92 features).
In 1999, the ANSI/ISO organisation published the SQL99 standard (also known as SQL3).
Object-relational database principles, call level interfaces and integrity management are some
of the more sophisticated and previously unaddressed aspects of contemporary SQL systems
that are now covered by this standard. SQL99's Core SQL99 and Enhanced SQL99 degrees of
conformance replace SQL92's levels of compliance. Currently, PostgreSQL complies with the
majority of the SQL92 Entry-level standard as well as a large number of the Intermediate- and
Full-level capabilities. Additionally, many of the new capabilities in SQL99 are quite comparable
to the arrays, functions and inheritance pioneered by PostgreSQL in the object-relational
paradigm.
Additionally, PostgreSQL offers a number of extensions to SQL. Here are a few of these
extensions:
·
Numerous non-standard types are supported by PostgreSQL. These include abstract data
types including void, complex, domains, record, trigger andcstring. Like any array, it also
includes polymorphic types.
·
It is trigger-compatible. Additionally, it enables the development of routines that can be
saved and run on the server.
·
Numerous procedural programming languages, including PL/pqSQL, PL/Tcl, PL/Python
and others, are supported.
Transaction Management
A transaction is used to perform each SQL query. While processing such queries are all-ornothing kinds while they are performing adjustments, which leads to several beneficial features.
This guarantees the recoverability and integrity of queries. Take this case into consideration:
UPDATE students SET marks = marks +10;
Assume that out of the 2000 records, the aforementioned query has updated the first 200
records and is currently working on the 201st record. If a user resets the computer at this point
to end the query, the recovery mechanism will ensure that none of the student's entries are
changed when the database is restarted. To change the marks as requested, the query must be
executed once more. As a result, the PostGres has ensured that the query doesn't lead to any
issues with recovery or integrity.
This DBMS's feature is really helpful. Imagine you were running a query to raise staff salaries at
your organisation by Rs. 500 when the power went out while the update process was in
progress. Without transactions support, some but not all of the people's records in the query
may have been modified. It would be challenging to pinpoint the UPDATE's failure. Which
records were changed and which ones weren't? is what you want to know. You can't just run the
query again since some people who may have already earned their Rs. 500 raise might also
receive an additional Rs. With the transactions system in place, you don't need to worry about it
because any updates to the data will be undone when the DBMS restarts by first recovering
from the failure. As a result, you may just run the query again.
Multistatement Transactions
Every SQL query in PostGres executes by default in its own transaction. Take these two similar
questions as an illustration:
mydb=> INSERT INTO table1 VALUES (1);
INSERT 1000 1
OR
mydb=> BEGIN WORK;
BEGIN
mydb=> INSERT INTO table1 VALUES (1);
INSERT 1000 1
mydb=> COMMIT WORK;
COMMIT
An ordinary INSERT query is the first one. PostgreSQL automatically starts a transaction before
the INSERT command is executed. Following the INSERT, the transaction is committed. Any
query without an explicit transaction will automatically complete this phase. The INSERT,
however, requires explicit transaction statements in the second version. The transaction is
started with BEGIN WORK and it is committed with COMMIT WORK. The only difference
between the two searches is the inferred BEGIN WORK...COMMIT WORK instructions. Both
queries produce the identical database state. The ability to combine several queries into a
single transaction is where these transactions-related statements really shine. In this scenario,
either all of the queries run to completion or none of them do. For instance, either both of the
INSERTs in the following transaction will be successful, or neither will.
mydb=> BEGIN WORK;
BEGIN
mydb=> INSERT INTO table1 VALUES (1);
INSERT 1000 1
mydb=> INSERT INTO table1 VALUES (2);
INSERT 2000 1
mydb=> COMMIT WORK;
COMMIT
Both multi-version concurrency control and two-phase locking have been implemented in
PostgreSQL. All of the isolation levels specified in SQL standards are supported by the multiversion concurrency control. They are as follows:
·
Read uncommitted
·
Read committed
·
Repeatable Read and
·
Serialisable.
Storage and Indexing
The fact that PostgreSQL defines a number of system columns in all tables is an intriguing issue
here. Although the user is typically unaware of these system columns, explicit queries can
reveal their contents. These columns typically provide meta-data, or information about the
information found in a table's records.
As a result, each record would have attribute values for both the user-defined and systemdefined columns in a table. The system columns are listed in the following table.
System Columns Table
A primary key must be deliberately created by the database creator in order to distinguish
between two records with the same field values. PostgreSQL appends each entry with an object
identifier number, or OID, that is specific to that table in order to prevent this from happening.
Since no two records in the same table will ever have the same OID, no two records in a table
are also unique. This is guaranteed by the oid.
Operating system files are used internally by PostgreSQL to store data. Each table has a
separate file and each file contains a sequence of data items. On the database, an index can be
made. An index is kept as a separate file that the user can sort on one or more columns.
Indexes
Indexes enable quick retrieval of particular table rows. Finding a specific row in a large table
with an index only takes a fraction of a second, however non-indexed rows take longer to
analyse the same data. Index creation is not automated in PostgreSQL. Users can define
indexes for attributes or columns that are often utilised for information retrieval.
You could, for instance, make an index like this:
mydb=> CREATE INDEX stu_name ON Student (first_name);
Although it is possible to create many indexes, each index should be capable of illustrating its
effectiveness in retrieving data from the database. It is important to note that the utilisation of an
index introduces additional costs in terms of storage capacity and system performance, as any
modification made to a record may require corresponding updates to the associated index. An
index can be constructed by utilising many columns. The initial column in a multi-column index,
followed by the subsequent column, is utilised for the purpose of sorting.
Numerous implementations of indexes are supported by PostgreSQL. Which are:
·
B-Tree indexes are the standard type of index. For comparison and range questions, they
are helpful.
·
Hash Indexes: This index makes use of linear hashing. Comparatively speaking, B-tree
indexes are preferable over such indexes.
·
R-Tree indexes: For purposes of determining operations like overlap, etc., these indexes
are produced on built-in spatial data types such as box and circle.
·
GiST Indexes: Generalised search trees are used to build these indexes. These indexes
are helpful for full text indexing and are therefore helpful for information retrieval.
Query processing operations
After being sent to PostgreSQL, the query travels through the following processes (in
chronological order) to be answered:
·
The client application program establishes a connection with the PostgreSQL server. The
application program sends the query to the server, then waits for the server to respond with the
results after processing it.
·
The server's parser validates the query syntax after receiving it from the client and builds a
query tree.
·
The rewriting system chooses the rules from the system catalogues that might apply to the
query tree using the query tree that the parser generated as input. The change is then carried
out in accordance with the rules. Additionally, it converts any query that is directed at a view into
a query that uses the basic tables.
·
The planner/optimiser generates a query plan from the (rewritten) query tree as input for
the executor. All the potential routes that could lead to the same outcome are organised into a
list. The cheapest path is ultimately selected after estimating the cost of each path's execution.
A comprehensive query assessment plan that the executor can employ is developed from this
cheapest approach.
·
The executor generates the desired result by iteratively traversing the planner's query
evaluation plan tree.
Below is an explanation of the procedure in more detail:
Data packets generated by TCP/IP requests for database access on local or remote Unix
Domain sockets are used to transmit a query to the backend (which may be the query
processor). Following the loading of the query into a string, the parser receives the string and
passes it on to the lexical scanner scan.l, which tokenizes the query string's words. The query
type, such as Create or Select queries, is then determined by the parser using another
component, gram.y and the tokens. The appropriate query-specific structure has now been
loaded.
The statement is then classified as simple (e.g., CREATE USER, ANALYSE, etc.) or complex
(SELECT / INSERT / UPDATE / DELETE). Statement-specific functions handle simple utility
instructions; however, more in-depth processing is required for complicated statements.
Complex searches may specify columns that need to be modified, such as INSERT and
UPDATE, return data columns, or both. These columns' references are transformed into
TargetEntry entries, which can then be linked together to form the query's target list. In
Query.targetList, the target list is kept.
The query is now changed to reflect the intended VIEWS or the application of any rules that
might be applicable. Then, based on the query structure and the activities to be carried out in
order to execute the query, the optimizer constructs the query execution plan. The Executor
receives the Plan, executes it and returns the results to the Client.
Query Optimisation
The planner's or optimizer's job is to select the best course of action from the available options.
There are numerous ways to run the query tree of a given SQL query, all of which basically yield
the same set of results. The query optimizer is unable to compare each of these potential
execution plans to determine which is most likely to complete operations quickly. Thus, within
the boundaries of the predetermined time and space complexity, the optimizer must construct a
viable (non optimum) query plan. Here, PostgreSQL uses a genetic query optimizer.
Database Design Project: Part 1
The creation of a model centralised relational music store application is the goal of this project.
Information about consumers and artists and their works must be stored in this application. The
capability in this context is to update, remove and insert records for the various entities. The
database was created for the managers and salespeople at a music store. Orders are taken
over the phone or by email. The database clerk must be able to accommodate consumer
requests. Finding the ideal album and acquiring this album are among these wishes.
The project team opted to integrate extra functionalities later after completing the main
functionality.
The primary function is:
·
In the context of this discussion, the user's request pertains to the actions of adding,
deleting and updating. Information pertaining to customers
·
Revise the album, artists and songs by incorporating additions, deletions and updates to
the existing information.
·
Amend and modify a customer order. A client order comprises the customer's information
and the information pertaining to the product(s) being ordered.
This choice was decided because to the team's inexperience with Java and the four-week time
frame that was available. The team's minimal goal is to display these fundamental functions in a
Graphic User Interface (GUI) that is user-friendly. This factor is taken into account by the
database's architecture, which aims to ensure a built-on architecture. It must be simple to
implement new functions. These additional features include, for instance:
·
brief descriptions of the concepts of Customer, Product and Order.
·
checking the application for errors.
·
Possibility of setting up customer information for the music store manager. The
management can learn more about the consumers thanks to this.
·
The client can select their preferred musical selections. The Music Store will create a CD
with these tracks on it.
Architecture
Here, we use an Entity Relationship Diagram (ERD) and a Relational Schema Definition to
show the database architecture. The ERD displays the database's general organisation and
interactivity. The tables that will be built in the database are described by the Relational Schema
Definition.
ERD
The entities Customer, Product, Artist, Track andCustomerorder are described in the entity
relationship diagram (see figure below). Each entity has a primary key, which is highlighted in
the diagram below. A product (album) is made up of tracks (songs) and has the medium type
CD, DVD, Tape, or Records. Each entity's attributes are listed in a circle, with the primary keys
highlighted. The entities are connected in a systematic and straightforward way by the
relationships such as orders, lists, productions and media tracks.
Figure: ERD of the Music store Database
The following explanation and illustration of mapping cardinalities uses arrows:
·
Numerous items have the potential to have multiple artists.
·
A single product has the potential to encompass many tracks.
·
A single consumer order has the potential to encompass many products.
·
A single client has the potential to generate many customer orders.
As was mentioned in the introduction, this team decided on the simple structure to make
implementation easier. The ERD outlines the key guidelines the project is adhering to at every
stage of development.
Relational Table definition
The foundation for building the database's tables is this definition. It provides, in general, the
same data as the ERD but in a more focused manner.
Figure: Relational Table Definition
Database Design Project: Part 2
Functionality Design
The technical design schedule's depiction of the diagram primarily serves to give a "high level"
overview of the application structure. As a result, not all classes or attributes may be shown in
the diagram. For the sake of greater clarity, classes and other elements that don't directly
elucidate or show anything "out of the ordinary" have been left out.
Presentation of the program structure
The set of classes (each defined separately in the "class definition file") that make up the
program structure that we have chosen at this time. Additionally, there is a main routine in the
"core" program structure. There is a JFrame class there. Along with the distinct form objects and
menus, this (extended) JFrame object also stores the calls to the relevant (different) classes
and sub-windows/screens. Along with the objects required to use and maintain the database
and its connection.
Therefore, the'main' class of the JFrameWnd maintains a somewhat significant role in the
maintenance of the database. In the event that a different function or class doesn't operate as
intended, this should prevent locking issues. A Connection pointer or reference is supplied to
the constructor of an object whenever a class is called or formed so it can modify the database
for however long is necessary before leaving the Connection pointer to the garbage collection.
Implementation
The three following elements make up the implementation.
Music store database
The University of Amsterdam contributed the database. The course's assistants were database
administrators. JDBC (Java database connectivity) is supported by the database. SQL is the
query language that is used to create the tables. The following two examples of the code are
provided:
Each attribute is given a variable type during the creation of the tables using the Relational
schema definition. All additional primary keys in table TRACK were constructed as data types
SERIAL, with the exception of the attribute TRACKTITLE. This implies that an ID is
automatically produced for each database record. The goal is to maintain overall stability and
prevent all IDs from having to be created by the music store manager or clerks.
Music store application
Java is used to implement the application. JDBC is used to establish a connection to the
database and get the relevant data. The database connection code is provided below as an
example. SQL queries can also be called directly from Java programs via JDBC. This enables
users to create, edit, or view stored data in the database.
try{
Class.forName("com.mysql.jdbc.Driver");
myConnection=DriverManager.getConnection(
"jdbc:mysql://123.156.5.73/dbpc-t6","username", "password"
);
}
catch(Exception e){
System.out.println("Failed to get connection");
e.printStackTrace();
}
}
Music store graphic user interface (GUI)
Java is used to construct the GUI. The Java package Java.swing was therefore the most
popular.
User Manual of application:
The GUI is made up of a Menu that displays the following tabs: Menu, Customer, Product,
Order, Settings and Help. The cashier at the music store has access to view, modify, or add
entries for customers, orders and products. Additional functions include the menu, settings and
help categories. To close the application, use the "Menu" option. The connection status is
displayed in the settings menu. Assistance offers further support. An example of the menu items
Customer, Product and Order is shown below. New windows displaying the information will
open through the various sub items.
Menu item Customer
'View All Customer', 'Insert New Customer', 'Change Customer' and 'Delete Customer' are subitems under this item. View All Customer in Figure below displays a list of each customer. They
are arranged alphabetically to make searching easier. The consumers' characteristics are
mentioned in full. One can be chosen and the client information can then be deleted or altered.
The "Refresh button" is required to update the screen following a change. The clerk must select
the 'Add customer' sub-item to add a new customer, which launches a new window. The other
sub-items are subject to the same rules.
Menu item Product
The overall structure of the "Product" and the "Customer" are same. 'View All Products,' 'Insert
New Product,' 'Change Product Details,' and 'Delete Product' are included in this. 'View All
Products' is depicted in a screenshot in Figure be;ow.
Menu item Order
Figure following displays the 'Insert New Order' menu option. The cashier has a selection of
products and customer IDs. The details of the produced order can be seen below.
Module III: SQL for Data Science
Learn and Apply Foundational Knowledge of the SQL Language; Create a Database in the
Cloud; Use String Patterns and Ranges to Query Data; Sorting and Group Data in Result Sets
and Data Type; Analyse Data Using Python
SQL Basics
The language used to interact with databases is SQL. Structured Query Language is the name
of the acronym, which is pronounced either like "sequel" or by speaking each letter individually,
as in "ess cue el." The debates and contradictions around SQL will continue after this, but most
people will understand what you mean no matter how you express it. Whether SQL counts as a
programming language or not is a matter of significant controversy. In the same manner that C
or Python are general-purpose languages, this one isn't. Without a database, SQL is essentially
a text file with data in tables. Although SQL cannot create a website, it is effective at managing
data in databases. Practically speaking, the most important thing is that SQL may assist you in
completing the task of data analysis.
SQL is a comprehensive database language. The computer programming language SQL, also
referred to as Sequel or simply S-Q-L, is used to run nonprocedural queries on relational
databases. Database querying is the process of using SQL to retrieve data from a database.
A relational database is implemented using an RDBMS, or relational database management
system. An RDBMS performs all of the essential activities of DBMS software in addition to a
variety of other tasks that make the relational model easier to understand and use. RDBMS
users have access to a special data manipulation language that they can use to manipulate
data. Database structures are defined by means of a data defining language. Users of the
system can type commands or use a graphical user interface to enter commands at a terminal
with an RDBMS interface to save and retrieve data. The DBMS subsequently processes the
commands.
Basics SQL Language
Following Edgar Codd's creation of the relational model in the 1960s, IBM was the first to create
SQL databases. The relational model served as a theoretical justification for using relationships
to manage data. IBM contributed to the theory's advancement by developing the first databases,
but it also had business concerns, just like Oracle, Microsoft and every other corporation that
has since made databases available for purchase. There has always been conflict between
computer theory and business reality.
In 1987, the International Organisation for Standards (ISO) recognised SQL as a standard and
in 1986, the American National Standards Institute (ANSI) did the same. Although all significant
databases begin with these principles when implementing SQL, many of them contain
customisations and functions that make life simpler for their users. Without certain adjustments,
moving SQL between databases will become more challenging as a result of them.
To access, modify and retrieve data from database objects, SQL is needed. Databases may
contain one or more schemas that offer organisation and structure as well as including
additional items. Tables, views and functions are the items within a schema that are most
frequently utilised in data analysis. Data is stored in fields found in tables. An index is a unique
type of data structure that makes it possible to retrieve data more quickly. Tables may include
one or more indices.
A database administrator typically creates index definitions. Views can be referred to in the
same way as a table because they are essentially stored queries. Commonly used sets of
calculations or processes can be kept and quickly referred to in queries thanks to functions.
They are often produced by a DBA, or database administrator. An overview of database
organisation is shown in the figure below.
Figure: Overview of database organisation and objects in a database
DDL is used to define and manage the structure of database objects, such as tables, indexes
and constraints. It focuses on the creation, alteration and deletion of database schema
elements. Here are some key DDL commands:
·
CREATE: This command is used to create new database objects like tables, views,
indexes and schemas.
·
ALTER: ALTER is used to modify existing database objects. You can use it to add, modify,
or drop columns, constraints, or indexes.
·
DROP: DROP is used to delete database objects. You can use it to remove tables,
indexes, views, or other schema elements.
·
TRUNCATE: TRUNCATE is used to remove all the rows from a table but retain the table
structure. It's faster than DELETE for large datasets because it doesn't log individual row
deletions.
·
COMMENT: This command allows you to add comments or descriptions to database
objects for documentation purposes.
Data Manipulation Language (DML):
DML is used to interact with and manipulate the data stored within the database. It focuses on
querying, inserting, updating and deleting data.
·
SELECT: SELECT is used to retrieve data from one or more tables. It allows you to specify
the columns you want to retrieve, apply filters and join multiple tables.
·
INSERT: INSERT is used to add new rows of data into a table. You specify both the table
and the values to be inserted.
·
UPDATE: UPDATE is used to modify existing data in a table. You specify the table, the
columns to be updated and the new values.
·
DELETE: DELETE is used to remove rows from a table based on specified criteria.
·
MERGE: MERGE (or UPSERT) is used to perform an INSERT, UPDATE, or DELETE
operation based on a condition. It's often used for data synchronisation between tables. Its
syntax can be more complex than other DML commands.
These are the fundamental DDL and DML commands in SQL. DDL focuses on the database
structure, while DML is all about interacting with and manipulating the data within that structure.
Effective use of these commands is essential for managing and querying relational
databases.sation.
Advantages of SQL
·
Ease of Use: SQL is relatively easy to learn and use, especially for those familiar with
English-like syntax. Its declarative nature allows users to specify what they want to retrieve or
manipulate from the database without needing to specify how to do it, leaving the query
optimisation to the database management system (DBMS).
·
Universal Language: SQL is a standardised language and most relational database
management systems (RDBMS) support it. This means that once you learn SQL, you can work
with a wide range of database systems, including popular ones like MySQL, PostgreSQL,
Oracle, Microsoft SQL Server and SQLite.
·
Data Retrieval: SQL excels at querying databases to retrieve specific data. It provides a
rich set of commands for filtering, sorting and aggregating data, allowing users to extract
valuable insights from large datasets efficiently.
·
Data Manipulation: SQL supports powerful data manipulation operations, including
INSERT, UPDATE, DELETE and MERGE, which enable users to modify data within a database
with precision and control.
·
Data Integrity: SQL databases enforce data integrity rules through constraints like unique
keys, primary keys and foreign keys. These constraints help maintain the accuracy and
consistency of data, preventing data corruption and ensuring reliable data storage.
·
Scalability: SQL databases can handle large amounts of data and are highly scalable. As
data grows, you can optimize performance through indexing and partitioning techniques,
ensuring that your applications remain responsive.
·
Transaction Management: SQL supports transactions, which allow multiple database
operations to be grouped together as a single unit. This ensures that all or none of the
operations are completed, maintaining data consistency.
·
Security: SQL databases offer robust security features, including user authentication,
access control and data encryption, to protect sensitive data from unauthorised access and data
breaches.
·
Reporting and Analysis: SQL can be used to create complex queries that generate reports
and perform data analysis. Business Intelligence (BI) tools often rely on SQL for querying and
reporting purposes.
Data Modeling: SQL allows users to create sophisticated data models that represent complex
relationships between data entities. This is particularly useful for designing databases that
accurately reflect real-world scenarios..
Operation of SQL
The fundamental component of any programming language is its collection of operators.
Operators are symbolic representations that allow for the execution of specific logical and
mathematical operations on operands. The operator performs operations on the operands.
There exist three distinct categories of SQL operators.
On the information kept in the tables, we can apply a number of arithmetic operators.
Arithmetic Operators are:
1.
+ [Addition]: On the data items—which can be either single columns or many columns—
it is utilised to perform addition operations.
Implementation:
SELECT employee_id, employee_name, salary, salary + 100
addition;
AS "salary + 100" FROM
Output:
employee_id
employee_name
salary
salary+100
1
alex
25000
25100
2
rr
55000
55100
3
jpm
52000
52100
4
ggshmr
12312
12412
Each employee's wage has been increased by 100 in this case, which is an addition operation
on a single column.
2.
- [Subtraction]: The data items can either be single columns or multiple columns and the
subtraction operation is performed on the data items.
Implementation:
SELECT employee_id, employee_name, salary, salary - 100 AS "salary - 100" FROM
subtraction;
Output:
employee_id
12
employee_name
Finch
salary
15000
salary-100
14900
22
Peter
25000
24900
32
Warner
5600
5500
42
Watson
90000
89900
3.
/ [Division]: When you wish to identify the entities that are interacting with every other
entity of a group of entities of various types, division is frequently necessary.
When analysing queries with the keyword "all," the division operator is utilised.
There are several circumstances in which the division operator is utilised:
·
Which individual possesses accounts in all financial institutions within a specific urban
area?
·
Which students have successfully completed all the courses necessary for graduation?
All of these queries define a set that contains some items following the keyword "all," and the
final result only includes units that meet these criteria.
Note: SQL implementations do not provide division. Other procedures can be used to represent
it, though.(similar to cross join, except for, in)
SQL Implementation of Division
Given two relations(tables): R(x,y) , S(y).
R and S : tables
x and y : column of R
y : column of S
All different values of x from R that are related with all values of y from S are given by R(x,y) div
S(y).
Computation of Division : R(x,y) div S(y)
Steps:
By calculating R(x) x(cross join) S(y), say r1, you may find out all conceivable combinations of
S(y) and R(x).
·
Subtract actual R(x,y) from r1, say r2
·
x in r2 are those that are not associated with every value in S(y); therefore R(x)-r2(x) gives
us x
·
that are associated with all values in S
Queries
Implementation 1:
SELECT * FROM R
WHERE x not in ( SELECT x FROM (
(SELECT x , y FROM (select y from S ) as p cross join
(select distinct x from R) as sp)
EXCEPT
(SELECT x , y FROM R) ) AS r );
Implementation 2 : Using correlated subquery
SELECT * FROM R as sx
WHERE NOT EXISTS (
(SELECT p.y FROM S as p )
EXCEPT
(SELECT sp.y FROM R as sp WHERE sp.x = sx.x ) );
Relational algebra
Using steps which is mention above:
All possible combinations
r1 ← πx(R) x S
x values with “incomplete combinations”,
r2x ← πx(r1-R)
and
result ← πx(R)-r2x
R div S = πx(R)- πx((πx(R) x S) – R)
Examples
Supply Schema
Here sid means supplierID and pid means partsID.
Tables: suppliers(sid,pid) , parts(pid)
1. Find suppliers that supply all parts.
Ans 1 : Using implementation 1
SELECT * FROM suppliers
WHERE sid not in ( SELECTsid FROM ( (SELECT sid, pid FROM (select pid from parts) as p
cross join
(select distinct sid from supplies) as sp)
EXCEPT
(SELECT sid, pid FROM supplies)) AS r );
Ans 2: Using implementation 2
SELECT * FROM suppliers as s
WHERE NOT EXISTS (( SELECTp.pid FROM parts as p )
EXCEPT
(SELECT sp.pid FROM supplies sp WHERE sp.sid = s.sid ) );
4.
* [Multiplication]: It is employed for data item multiplication.
Implementation:
SELECT employee_id, employee_name, salary, salary * 100
addition;
AS "salary * 100" FROM
Output:
employee_id
employee_name
salary
salary * 100
1
Finch
25000
2500000
2
Peter
55000
5500000
3
Warner
52000
5200000
4
Watson
12312
1231200
Here, we have multiplied the salaries of each employee by 100, or one column of multiplication.
5.
% [Modulus]: When dividing one set of data by another, it is used to obtain the residual.
Implementation:
SELECT employee_id, employee_name, salary, salary % 25000 AS "salary % 25000" FROM
addition;
Output:
employee_id
1
employee_name
Finch
salary
25000
salary % 25000
0
2
Peter
55000
5000
3
Warner
52000
2000
4
Watson
12312
12312
Here we have done modulus of 100 to each Employee’s salary i.e, modulus operation on single
column.
Comparison Operators in SQL
A comparison operator, also known as a relational operator, is a symbol used in mathematics to
compare two values.
When comparing one expression to another, comparison operators are utilised. A comparison
can have three possible outcomes: TRUE, FALSE, or UNKNOWN (UNKNOWN is the result
when an operator contains one or two NULL expressions).
The 'agents' table should be queried to retrieve data for all columns with the following
requirement:
1.
‘commission’ is equal to .15,
SQL statement that can be used:
SQL Code:
SELECT * FROM agents WHERE commission = 0.15;
Output:
SQL Greater than ( > ) operator
To determine whether one phrase (or number) exceeds another, use the larger than operator.
Example:
The 'agents' table should be queried to retrieve data for all columns with the following
requirement:
1. 'commission' is greater than .14,
SQL statement that can be used:
SQL Code:
SELECT * FROM agents WHERE commission>0.14;
Output:
SQL Less than ( < ) operator
To determine whether one expression (or number) is less than another, use the less than
operator.
Example:
The 'agents' table should be queried to retrieve data for all columns with the following
requirement:
1. 'commission' is less than .12,
You can use the SQL statement listed below:
SQL Code:
SELECT * FROM agents WHERE commission <0.12;
SQL Greater than or equal to ( >= ) operator
To determine whether an expression (or number) is greater than or equal to another one, use
the greater than equal to operator.
Example:
The 'agents' table should be queried to retrieve data for all columns with the following
requirement:
1. 'commission' is greater than or equal to .14,
You can use the SQL statement listed below:
SQL Code:
SELECT * FROM agents WHERE commission >= 0.14;
SQL Less than or equal to ( <= ) operator
To determine whether an expression (or number) is less than or equal to another one, use the
less than equal to operator.
Example:
The 'agents' table should be queried to retrieve data for all columns with the following
requirement:
1. commission is less than or equal to .12, the following SQL statement can be used :
SQL Code:
SELECT * FROM agents WHERE commission <= 0.12;
Output:
SQL Not equal to ( <> ) operator
When comparing the equality of two numbers or expressions, the not equal to operator is
employed.
Example:
The 'agents' table should be queried to retrieve data for all columns with the following
requirement:
1. commission is not equal to .15, the following SQL statement can be used :
SQL Code:
SELECT * FROM agents WHERE commission <>0.15;
Output
Creating Database in the Cloud
A cloud database, which is a curated and structured collection of data, is hosted on a cloud
computing platform that can be categorised as public, private, or hybrid. The design and
operation of a cloud database closely resemble that of an on-premises database, which
operates on servers located within a company's own data centre. The divergence between the
two lies mostly in the deployment and maintenance of the database.
For example, regardless of whether it is deployed on-premises or in a cloud environment, the
database maintains a consistent interface for end users and applications. Cloud databases,
similar to their on-premises counterparts, have the capability to store many types of data,
including structured, unstructured, or semistructured data, depending on the individual database
software being employed.
However, employing a cloud database alters the roles that the data management and IT teams
play. The database platform is installed and managed by cloud service providers in managed
services settings. This lessens the everyday management work that database administrators
(DBAs) and IT operations staff have typically performed. After that, a DBA can handle additional
duties including streamlining databases for specific applications and monitoring the utilisation
and expense of cloud database systems.
Database deployments are certainly moving towards the cloud, just like other IT systems. A
year faster than expected, Gartner anticipated that cloud databases will generate 50% of all
global revenues for database management systems (DBMS) in 2022 in a research on cloud
databases that was released in December 2021. Additionally, according to a study of 753 cloud
customers performed in late 2021 by the provider of IT management tools Flexera, 55% of
respondents claimed their companies used cloud-based data warehouses, while 49% used
cloud-based relational database services and 38% used NoSQL database services.
How cloud databases work
Databases are utilised by organisations to collect, arrange and furnish data to executives and
employees for the purpose of operational and analytical endeavours. Cloud databases often
provide similar functionalities for data processing, management and access as their onpremises counterparts. The migration of on-premises databases, which are now utilised,
alongside the applications they facilitate, is a common practice to transition them to cloud-based
environments.
The cost of a system is determined by the extent to which system resources are utilised. These
resources can be allocated as needed to meet processing demands, in contrast to the
conventional practice of purchasing software licences. As a viable option, individuals have the
opportunity to prearrange reservations for database instances, typically spanning a minimum
duration of one year, in order to avail themselves of advantageous pricing for regular workloads
characterised by foreseeable capacity requirements.
Companies using public cloud databases can choose between the two deployment strategies
listed below:
Self-managed database: Database that is self-managed In this infrastructure as a service (IaaS)
setting, a virtual machine running on a system run by a cloud provider hosts the database.
Servers, operating systems and storage hardware are all managed and supported by the
provider. The user organisation, however, is in charge of the deployment, management and
upkeep of the database. As a result, for the DBA, who maintains complete management control
over the database, it's comparable to an on-premises setup.
Managed database service:Database as a Service (DBaaS) environments are comprehensively
administered by the vendor, who may be a cloud platform provider or another database vendor
that operates its cloud Database Management System (DBMS) on the infrastructure of the
platform provider. The DBaaS idea entails the management of both system infrastructure and
database platform on behalf of the customer.
The Database Administrator (DBA) assumes the responsibility of monitoring the database and
engages in collaborative efforts with the Database-as-a-Service (DBaaS) provider for certain
administrative tasks. Meanwhile, the vendor is entrusted with the provisioning, backups, scaling,
patching, upgrades and other essential duties related to database management. Cloud data
warehouse systems also offer accessible choices for comparable data warehouse as a service
(DWaaS).
Furthermore, several cloud service providers, like as Oracle and Amazon Web Services (AWS),
offer their DBaaS technologies for deployment within on-premises data centres, forming either a
private cloud or a hybrid cloud infrastructure that integrates both public and private clouds.
Instead of delivering the systems to a customer's data centre to run there, as is the case in a
typical DBaaS scenario, the provider sends the systems to the customer's data centre and
controls the databases remotely from there.
Types of cloud databases
There are many different cloud databases that may be used, matching the various database
technologies that can be installed on-site. Every reputable database provider currently provides
their software on the cloud. This comprises both already-existing on-premises databases with
cloud compatibility and cloud-native databases created expressly for usage in cloud
environments.
The main database types that cloud users can utilise are as follows:
Relational Databases: Since the 1990s, SQL-based relational software has dominated the
database industry and continues to be the most frequently adopted technology. It is especially
well suited for transaction processing and other applications using structured data.
NoSQL Databases:NoSQL systems are preferable for unstructured data since they do not have
the inflexible relational database models. Document databases, graph databases, wide-column
stores and key-value databases are the four main NoSQL product groups.
Multimodel Databases:They execute a wider range of applications because they support several
data models. As a result of add-ons, such as the inclusion of a graph module to a relational
DBMS, many relational and NoSQL databases today qualify as multimodel databases.
Distributed SQL Databases:These technologies, which were once referred to as NewSQL,
disperse relational databases among numerous compute nodes to build transactional systems
that can scale to NoSQL-like levels.
Cloud data warehouses:They were first created to offer data warehousing capabilities for BI and
reporting applications, but they today frequently assist data lake creation, machine learning and
other advanced analytics activities.
Create a MySQL database hosted in Google Cloud
The definitions of a few terms used in connection with AppSheet are included in the list below.
·
Database Instance: The structure that looks after a group of database files is known as a
database instance. Many databases may be stored in one instance. You will create a public IP
address in the instruction that follows so that you can connect to the database instance.
·
Databases are collections of data. Typically, a database contains many tables. Each
database will show up as a separate data source in AppSheet.
·
Schema: Schema and database are interchangeable terms in this context. It is a data set
made up of various tables. The word "schema" is used in MySQL Workbench. In both AppSheet
and Google Cloud, the word "database" is utilised.
·
Table: The real data are kept in a table. A table typically has numerous columns, each of
which contains a particular sort of data (such as Name, Email, Phone, etc.). A different record is
present in each row. Each field in a form that is being viewed in AppSheet corresponds to a
column in a table and saving the form adds a new row to the database.
Create a Database Instance
1.
Log in to Google Cloud and select SQL from the left menu.
1.
Click Create Instance.
2.
Select a database engine. MySQL is used in this lesson.
Complete the instance data.
1.
Select a unique instance ID. Your database will be identified in your Google Cloud
account in this way.
2.
Pick a root password.=.
3.
Select the area. US-West and EC-West are the locations of the AppSheet servers. Based
on where the majority of your users are, choose between these two regions for the greatest
performance.
4.
To view more setting choices, click Show options.
1.
In the context of connectivity, it is important to verify that the option for Public IP is
selected. By selecting this option, Google Cloud will generate an IP address that may be utilised
to establish a connection between your AppSheet account and the database instance.
2.
Select the option labelled "Add network."
1.
Type in the IP addresses that are provided under Manage IP addresses and firewall
information. As a result, your database can be accessed directly by the AppSheet server. The
IP addresses must be in CIDR notation for Google Cloud. Add /32 to the end of each IP address
to represent it in CIDR notation. 20.189.130.98, for instance, becomes 20.189.130.98/32.
2.
Decide on the machine and storage options. Choose a computer type that has sufficient
memory to accommodate your largest table.
3.
Adjust other parameters as necessary. For the other options, the default values are
generally accurate.
4.
Press the Create Database Instance button. It will take a little while to finish this.
Create a user account for the database instance
1.
To access the information page after the database instance has been established, click
on the instance id.
3. For the account that will be used to login to AppSheet, create a user name and password.
Note down the login information. You can modify the password after creating the account, but
you cannot get the password back.
Enter a name for your database name and click on Create
Create a Database - Method 2: Using database visualisation tool
A typical method of interacting with the database is using a data visualisation tool. MySQL
Workbench is used in this lesson to access and edit the database.
1.
You must add the IP address of your workstation to the firewall allow list in order to use
MySQL Workbench to connect to the database. By typing "What's my IP address" into your
browser, you can get this information.
2.
Follow the same procedures to authorise your IP address in Google Cloud as you did for
AppSheet IP addresses.
3.
Create a new MySQL connection by launching MySQL Workbench.
·
The hostname corresponds to the public IP address assigned to your database.
·
The username and password correspond to the user account that was previously
established.
4.
First, initiate the process by selecting the option "Test the Connection." Once this step is
completed, proceed by clicking the "Ok" button to confirm the action.
5.
Create a new schema.
Connect AppSheet to the Database
1.
Go to the My Account > Sources tab.
2.
Click + New Data Source.
3.
Give the data source a name, then select Cloud Database or On-premises Database.
4.
Information about the new database connection dialogue box appears.
5.
Set up the database connection details.
·
Type: Select MySQL.
·
Server: Enter the MySQL database server hostname or IP address.
·
Database: Enter the database name.
·
Username: Enter the username of the database user.
·
Password:Enter the password for the database user.
6.
To verify the connection details, click Test.
7.
In order to store the data source in your AppSheet account following a successful test,
click Authorise Access.
Connect apps to the database
Your AppSheet account is now linked to the database and schema. The steps for adding tables
from a database from within an AppSheet app are the same as those for adding tables from a
spreadsheet. See Include a table.
The database can show up as database-1 in your AppSheet account. In AppSheet, you can
modify the name of your database as follows:
1.
Navigate to My account > Sources.
2.
You can update a database by clicking on it.
3.
In the box that appears, you can change the name. Click Cloud Database after entering a
new name, such as AppSheet_Demo.
1.
The identical connection details should be entered, then click Authorise Access. The
database will now show up in AppSheet with the changed name.
Using String Patterns and Ranges to Query Data
A flexible and effective technique to query and filter data based on certain conditions is to use
string patterns and ranges in SQL. While comparison operators provide filtering based on
numerical ranges, the LIKE operator permits pattern matching in character columns utilising
wildcard characters. A regular expression can match patterns even more effectively. Data
analysts and developers can effectively retrieve the information they require from databases by
using these strategies to derive useful insights from the data. Database indexing and query
optimisation must be taken into account when employing these approaches to enable effective
data retrieval, particularly in large-scale data scenarios.
The LIKE operator and comparison operators are used to filter and get specific data from a
database table when text patterns and ranges are used to query data in SQL. These methods
enable data analysts and developers to carry out intricate data searches and selections by
offering flexible and strong querying of text data and numerical ranges.
One of two wildcards can be used in conjunction with the LIKE operator:
·
The percent sign
·
The underscore _
The % symbol can represent a string of zero, one, or many characters. The underscore denotes
a single digit or letter. It is possible to combine the symbols.
Syntax
The basic syntax of % and _ is as follows:
SELECT FROM table_name
WHERE column LIKE 'XXXX%'
or
SELECT FROM table_name
WHERE column LIKE '%XXXX%'
or
SELECT FROM table_name
WHERE column LIKE 'XXXX_'
or
SELECT FROM table_name
WHERE column LIKE '_XXXX'
or
SELECT FROM table_name
WHERE column LIKE '_XXXX_'
It is possible to combine N criteria by using the AND or OR operators. Any text or numeric value
could be used in this situation as XXXX.
Example
Here are a few instances of parts that use distinct LIKE clauses along with the "%" and "_"
operators:
The following illustration would display each entry from the CUSTOMERS table where SALARY
starts with 200:
SQL> SELECT * FROM CUSTOMERS
WHERE SALARY LIKE '200%';
This would result in the following:
Previous Activity
Jump to...
Sorting in Result Sets and By Data Type
The SQL ORDER BY clause allows for the sorting of data in either ascending or descending
order based on one or more columns. Certain databases possess the capability to
autonomously arrange the outcomes of a query in an ascending manner.
The data in a database table can also be sorted in a preferred order using the ORDER BY
clause in addition to this. In this scenario, sorting a table's records in a conventional manner
(such as alphabetical or lexical order) may not be possible; instead, the records may be sorted
according to any external criterion. For instance, the contents of orders placed can be sorted
depending on the dates such orders are made in an ORDERS table that contains a list of orders
made by various clients of an organisation. This is based on first come, first serve, so it is not
required to be ordered alphabetically.
Syntax
The ORDER BY clause's basic syntax, which is used to arrange the results in either ascending
or descending order, is as follows:
SELECT column-list
FROM table_name
[WHERE condition]
[ORDER BY column1, column2, ..columnN] [ASC | DESC];
The ORDER BY clause might include more than one column. Whatever column you want to use
for sorting must be present in the column list.
Sorting Results in Ascending Order
The records in a database table can be sorted using the Order By Clause in SQL either
automatically or by using the "ASC" keyword in the clause condition. To further grasp this, let's
look at an example.
Example
Consider the following records in the CUSTOMERS table:
The example that follows would sort the outcome by NAME and SALARY in ascending order.
SELECT * FROM CUSTOMERS ORDER BY NAME;
Output
Sorting Results in Descending Order
However, you must specify the "DESC" keyword in the clause condition to organise the records
in a database table in descending order. To further grasp this, let's look at an example.
Example
An example of a code block that will sort the output by NAME in decreasing order is shown
below.
Select * From Customers Order By Name Desc;
Output
Sorting Results in a Preferred Order
Using the CASE statement within the ORDER BY clause, one can also arrange the records in a
table in the order of their choice. All of the values are listed in the clause along with the order in
which they should be sorted; if no number is provided for the values, they are sorted in
ascending order by default.
Example
The SELECT query used would be as follows in order to retrieve the rows in their own desired
order.
SELECT * FROM CUSTOMERS
ORDER BY (CASE ADDRESS
WHEN 'DELHI'
THEN 1
WHEN 'BHOPAL' THEN 2
WHEN 'KOTA'
THEN 3
WHEN 'AHMEDABAD' THEN 4
WHEN 'MP'
THEN 5
ELSE 100 END) ASC, ADDRESS DESC;
Output
Group Data in Result Sets and By Data Type
The result set is grouped using the SQL GROUP BY clause based on common values found in
the result set. Following the extraction of all records from tables, the result set is grouped. A
group of rows can be created based on the values of a particular column or expression by using
the GROUP BY clause in conjunction with a SELECT query. Every group's summary
information can be obtained using the SQL AGGREGATE function and each group can then
apply these results. The GROUP BY clause's primary function is to summarize database data,
hence reporting queries frequently utilise this command. These searches are known as grouped
queries since just one result is returned after the GROUP BY clause has been used.
The WHERE condition is used to limit the data from the tables and the GROUP BY clause is
used to group the returned result set according to the supplied column name. You can use both
the WHERE condition and the GROUP BY clause. The HAVING clause can be used to filter the
set of results that have been grouped. After the GROUP BY clause, this is used.
Syntax of the SQL GROUP BY Clause
1 SELECT * from Table_Name
2 GROUP BY Column_Name1,
3 [Column_Name2,….]
4 [HAVING CONDITION];
·
[SELECT * from Table_Name] – To retrieve records from a database, use the standard
SELECT command.
·
GROUP BY Column_Name1 – The result set is grouped using the column name supplied
in this clause. Here, you can enter the names of one or more columns.
·
[Column_Name2,….] – This is not required. One column name must be included in the
GROUP BY clause in order to group the result set. If you enter multiple column names, the
result set will initially group by the first column's value before moving on to the following
column(s).
·
[HAVING CONDITION] – This clause was used to "CONDITION"-filter the returned data
set. Using the HAVING clause, any condition can be specified.
Using GROUP BY with Aggregate Functions:
When used in conjunction with the GROUP BY clause, the AGGREGATE functions are quite
powerful. Most frequently, GROUP BY clauses are combined with AGGREGATE functions in
SQL GROUP BY queries.
Here, we'll look at a variety of GROUP BY Clause examples using a variety of AGGREGATE
Function combinations:
SQL GROUP BY with SUM() function
SQL GROUP BY with COUNT() function
SQL GROUP BY with COUNT() and SUM()
SQL GROUP BY on more than one column
SQL GROUP BY with WHERE clause
SQL GROUP BY with HAVING clause
SQL GROUP BY with SUM() function:
If the user wants to see the total amount of salary paid to each department, they can use the
DEPARTMENT_ID and SALARY columns in the Employees database. In order to group the
result set using DEPARTMENT_ID and obtain the salary sum for each department, we will
utilise the SUM Aggregate function.
SELECT DEPARTMENT_ID,
SUM(SALARY) as Total_Salary
FROM Employees
GROUP BY DEPARTMENT_ID;
The outcome of running the SQL GROUP BY query mentioned above is shown below.
SQL GROUP BY with COUNT() function:
We have a field in the Employees table named DEPARTMENT_ID because, let's say, the user
wants to know how many employees are in each department. Therefore, we can use
DEPARTMENT_ID to group the result set and we'll utilise the COUNT Aggregate function to get
the number of employees by department.
SELECT DEPARTMENT_ID,
COUNT(*) as Number_Of_Emloyees
FROM Employees
GROUP BY DEPARTMENT_ID;
The outcome of running the SQL GROUP BY query mentioned above is shown below.
SQL GROUP BY with COUNT() and SUM() function:
In the Employees database, we have columns called DEPARTMENT_ID and SALARY. Let's
say the user wants to know the number of employees in each department and the total salary
paid to each department. Therefore, we may utilise the DEPARTMENT_ID to group the result
set and we'll also use the COUNT() and SUM() aggregate procedures. The functions COUNT()
and SUM() are used to calculate the total number of employees by each department.
SELECT DEPARTMENT_ID,
COUNT(*) as Number_Of_Emloyees,
SUM(SALARY)
FROM Employees
GROUP BY DEPARTMENT_ID;
Following is the result upon executing the above SQL GROUP BY query,
SQL GROUP BY on more than one column:
Sort data by Department ID, Job ID and Salary SUM with Department
The Employees database contains columns for DEPARTMENT_ID, pay andJob_ID. These
columns may be utilised to determine the count of employees in each department and the
aggregate pay provided to each department based on Job ID. Hence, by use the
DEPARTMENT_ID and subsequently the JOB_ID, it is possible to initially group the result set in
this particular scenario. The SUM() Aggregate function will be utilised to calculate the total
salary for each Job_id within every department.
SELECT DEPARTMENT_ID, JOB_ID, SUM(SALARY) FROM Employees GROUP BY
DEPARTMENT_ID, JOB_ID;
Following is the result upon executing the above SQL GROUP BY query,
SQL GROUP BY with WHERE clause:
Within the Employees table, there exist columns denoted as DEPARTMENT_ID and SALARY.
Suppose the user seeks to ascertain the aggregate sum of remunerations disbursed to
employees affiliated with DEPARTMENT_ID = 50. To retrieve only the relevant data from the
employee table pertaining to DEPARTMENT_ID = 50, we will deploy the WHERE condition on
DEPARTMENT_ID = 50. The result set can be organised by include the DEPARTMENT_ID in
addition to the WHERE clause.
SELECT DEPARTMENT_ID, SUM(SALARY) FROM Employees WHERE DEPARTMENT_ID =
50 GROUP BY DEPARTMENT_ID;
The outcome of running the aforementioned SQL GROUP BY query is shown below.
SQL GROUP BY with HAVING clause:
Sort employees by Department ID and Count with Department and HAVING count >1
Let's say the customer wants to know how many employees are present in each department
where there are at least two employees. In order to group the result set using
DEPARTMENT_ID and determine the number of employees by department, we will utilise the
COUNT() aggregate function.
In this case, the HAVING clause will be used to obtain a greater staff count>1.
SELECT DEPARTMENT_ID, COUNT(*) as Number_Of_Emloyees FROM Employees GROUP
BY DEPARTMENT_ID HAVING COUNT(*) > 1;
The outcome of running the aforementioned SQL GROUP BY query is shown below.
·
The result set is divided into more manageable groups using the SQL GROUP BY clause.
·
Following the retrieval of rows from a table, the process of grouping may be performed.
·
The same values that are present in the result set are used to group the data in the
GROUP BY clause.
·
The SQL SELECT query is used in conjunction with the GROUP BY Clause.
·
Although it cannot be used on grouped results, the WHERE clause is used to get rows
based on a specific condition.
·
The GROUP BY clause's result set is filtered using the HAVING clause. After the GROUP
BY clause, this is used.
Analyse Data Using Python
Data analysis is a method for gathering, arranging and changing data so that predictions about
the future and data-driven decisions can be made. Finding potential answers to business
problems is also helpful. Data analysis is broken down into six steps. As follows:
·
Ask or Specify Data Requirements
·
Prepare or Collect Data
·
Clean and Process
·
Analyse
·
Share
·
Act or Report
Python possesses an exceptional assemblage of data-centric Python packages, rendering it an
exemplary programming language for executing data analysis tasks. One such program,
Pandas, significantly streamlines the procedure of importing and analysing data.Here we used
Pandas to analyse data on a Country Data.csv file from UN public Data Sets of the well-known
website "statweb.stanford.edu" for this work.
Analysing Numerical Data with NumPy
The Python package known as NumPy offers a high-performance multidimensional array object
and accompanying capabilities for manipulating and analysing these arrays. The
aforementioned Python package serves as a fundamental component for scientific computing.
Arrays in NumPy
A NumPy Array is a collection of things, often numbers, organised in a table format. Each item
in the array is indexed by a tuple of positive integers. The concept of rank in Numpy pertains to
the amount of dimensions that an array possesses. The configuration of the array is determined
by a tuple of numerical values that specify the dimensions of the array along each axis.
Creating NumPy Array
There exist various methodologies and distinct hierarchies that can be employed in the
construction of NumPy arrays. Additionally, this can be constructed utilising many data types,
such lists, tuples and so on. The determination of the type of the final array is based on the type
of the items present in the sequences. There exist a multitude of routines inside the NumPy
library that facilitate the construction of arrays by employing initial placeholder data. These
mechanisms decrease the necessity for costly array expansion operations.
Create Array using numpy.zeros(shape, dtype = None, order = ‘C’)
import numpy as np
b = np.zeros(2, dtype = int)
print("Matrix b : \n", b)
a = np.zeros([2, 2], dtype = int)
print("\nMatrix a : \n", a)
c = np.zeros([3, 3])
print("\nMatrix c : \n", c)
Output
Matrix b :
[0 0]
Matrix a :
[[0 0]
[0 0]]
Matrix c :
[[0. 0. 0.]
[0. 0. 0.]
[0. 0. 0.]]
Operations on Numpy Arrays
Arithmetic Operations
Addition:
import numpy as np
# Defining both the matrices
a = np.array([10, 27, 131, 20])
b = np.array([12, 15, 210, 230])
# Performing addition using arithmetic operator
add_ans = a+b
print(add_ans)
# Performing addition using numpy function
add_ans = np.add(a, b)
print(add_ans)
# The same functions and operations can be used for
# multiple matrices
c = np.array([1, 2, 3, 4])
add_ans = a+b+c
print(add_ans)
add_ans = np.add(a, b, c)
print(add_ans)
Output:
[ 2242341250]
[ 2242341 250]
[ 2344344254]
[ 2242341250]
Analysing Data Using Pandas
Python Pandas is used to manipulate relational or labelled data and offers a variety of data
structures for doing so. The NumPy library serves as the foundation for this library. Typically,
this module is imported as:
import pandas as pd
Pd is used as a shorthand for the Pandas in this sentence. Although it is helpful to write less
code each time a method or property is called, utilising the alias to import the library is not
required. In general, Pandas offers two data structures for data manipulation, namely:
·
Series
·
Dataframe
Series:
There are multiple approaches and hierarchical structures that can be utilised in the
development of NumPy arrays. Moreover, this can be implemented by employing various data
types, such as lists, tuples and other similar structures. The classification of the resulting array
is contingent upon the nature of the elements contained inside the sequences.
The NumPy package offers a wide range of functions that enable the creation of arrays using
initial placeholder data. These methods reduce the need for expensive array expansion
operations.
The dataset can be accessed from several storage systems, including SQL databases, CSV
files, Excel files and others. Additionally, data structures like lists and dictionaries can also be
utilised to load the information, employing the Series() function.
Python Pandas Creating Series
import pandas as pd
import numpy as np
# Creating empty series
ser = pd.Series()
print(ser)
data = np.array(['g', 'e', 'e', 'k', 's'])
ser = pd.Series(data)
print(ser)
Output:
Dataframe:
The term "Pandas DataFrame" refers to a tabular data structure that is two-dimensional, has the
ability to change in size and can contain different types of data. This structure is characterised
by having named axes for both rows and columns.
The organisation of data in a data frame involves the arrangement of information in a tabular
format consisting of rows and columns. This particular data structure is characterised by its twodimensional nature. The three primary components of a Pandas DataFrame are the data, rows
and columns.
It can be produced using the Dataframe() function from many file types and data structures, just
like a series.
Python Pandas Creating Dataframe
import pandas as pd
# Calling DataFrame constructor
df = pd.DataFrame()
print(df)
# list of strings
lst = ['Welcome', 'To', 'The', 'World',
'of', 'python']
# Calling DataFrame constructor on list
df = pd.DataFrame(lst)
df
Output
Empty DataFrame
Columns:
[]
Index:
[]
0
0
Welcome
1
To
2
The
3
World
4
of
5
Python
Creating Dataframe from CSV
Using the read_csv() function, a dataframe can be produced from the CSV files.
import pandas as pd
# Reading the CSV file
df = pd.read_csv("Iris.csv")
# Printing top 5 rows
df.head()
Output
Filtering DataFrame
To subset rows or columns of a dataframe according to labels in the given index, use the
dataframe.filter() function in Pandas. Keep in mind that a dataframe's contents are not filtered
by this method. The labels of the index are subjected to the filter.
Python Pandas Filter Dataframe
import pandas as pd
# Reading the CSV file
df = pd.read_csv("Iris.csv")
# applying filter function
df.filter(["Species", "SepalLengthCm", "SepalLengthCm"]).head()
Output
Pandas GroupBy
Groupby is a fairly straightforward idea. We can arrange categories into groups and then give
each group a function. We utilise the Groupby notion because actual data science projects
require you to work with big volumes of data and repeatedly test different things. The term
"groupby" typically describes a procedure including one or more of the following steps:
·
Splitting: By applying certain requirements to datasets, we can divide data into groups.
·
Applying:This method entails applying a function to every group separately.
·
Combining:After applying groupby and results into a data structure, we merge many
datasets in this step.
The method involved in the Groupby notion can be better understood by looking at the following
illustration.
1. Sort the Team column's distinctive values into groups.
3. Put the other information in the buckets.
Python Pandas GroupBy
import pandas as pd
# Define a dictionary of personnel information
data1 = {'Name': ['Jai', 'Anuj', 'Jai', 'Princi',
'Gaurav', 'Anuj', 'Princi', 'Abhi'],
'Age': [27, 24, 22, 32,
33, 36, 27, 32],
'Address': ['Nagpur', 'Kanpur', 'Allahabad', 'Kannuaj',
'Jaunpur', 'Kanpur', 'Allahabad', 'Aligarh'],
'Qualification': ['Msc', 'MA', 'MCA', 'Phd',
'B.Tech', 'B.com', 'Msc', 'MA']}
# Dictionary to DataFrame conversion
df = pd.DataFrame(data1)
print("Original Dataframe")
display(df)
# applying the groupby() fucntion to the data to create Name value groups.
gk = df.groupby('Name')
# Let's print the first entries for each group that was created.
print("After Creating Groups")
gk.first()
Output:
Applying function to group:
After grouping the data, we apply a function to each group. The operations we use to
accomplish this are:
·
Aggregation: We construct a summary statistic (or statistics) about each group through this
approach. Calculate group sums or means, for instance.
·
Transformation:We carry out various group-specific calculations during this procedure and
we then return a like-indexed. Filling NAs within groups, for instance, with values taken from
each group
·
Filtration: In this procedure, some groups are eliminated based on a group-wise
computation that determines if a statement is true or false. For instance, Using the group sum or
mean to filter data
Pandas Aggregation
In the process of aggregation, a summary statistic for each group is calculated. For each group,
the aggregated function returns a single aggregated value. Several aggregating processes can
be carried out on the grouped data after data has been divided into groups using the groupby
function.
Python Pandas Aggregation
import pandas as pd
import numpy as np
# Define a dictionary of personnel information
data1 = {'Name': ['Jai', 'Anuj', 'Jai', 'Princi',
'Gaurav', 'Anuj', 'Princi', 'Abhi'],
'Age': [27, 24, 22, 32,
33, 36, 27, 32],
'Address': ['Nagpur', 'Kanpur', 'Allahabad', 'Kannuaj',
'Jaunpur', 'Kanpur', 'Allahabad', 'Aligarh'],
'Qualification': ['Msc', 'MA', 'MCA', 'Phd',
'B.Tech', 'B.com', 'Msc', 'MA']}
# Dictionary to DataFrame conversion
df = pd.DataFrame(data1)
# utilising the aggregate technique to achieve aggregation
grp1 = df.groupby('Name')
grp1.aggregate(np.sum)
Output
Visualisation with Matplotlib
Matplotlib, a Python library, is widely regarded as an exceptional tool for data visualisation due
to its user-friendly interface. The library is built upon the utilisation of NumPy arrays and is
designed to seamlessly integrate with the extensive SciPy ecosystem. It encompasses a
diverse range of visualisations, such as line plots, bar charts, scatter plots, histograms and other
additional graphical representations.
Pyplot
A Matplotlib package called Pyplot offers a MATLAB-like user interface. Pyplot offers functions
that work with the figure, such as creating a figure, labelling the plot and adding a plot area to a
figure.
# Python program to show pyplot module
import matplotlib.pyplot as plt
plt.plot([1, 2, 3, 4], [1, 4, 9, 16])
plt.axis([0, 6, 0, 20])
plt.show()
Output
Bar chart
A bar plot, sometimes referred to as a bar chart, is a visual representation that utilises
rectangular bars whose lengths and heights are directly proportional to the corresponding
numbers they represent, in order to illustrate a certain category of data. It is possible to
represent data using either horizontal or vertical bar graphs. The bar chart illustrates the
comparisons among the different categories. The production of the desired outcome can be
achieved through the utilisation of the bar() technique.
Python Matplotlib Bar Chart
Here we will use the iris dataset only
import matplotlib.pyplot as plt
import pandas as pd
df = pd.read_csv("Iris.csv")
# This will plot a simple bar chart
plt.bar(df['Species'], df['SepalLengthCm'])
# Title to the plot
plt.title("Iris Dataset")
# Adding the legends
plt.legend(["bar"])
plt.show()
Output
Exploratory Data Analysis
Data analysis utilising some visual techniques is known as exploratory data analysis (EDA).
Using this method, we may obtain comprehensive details on the statistical summary of the data.
Additionally, we will be able to handle duplicate values and outliers as well as identify any trends
or patterns in the collection.
Note: Iris Dataset will be used.
Information Gathering for the Dataset
To determine the shape of the dataset, we will use the shape parameter.
Shape of Dataframe
df.shape
Output:
(150, 6)
We can observe that the dataframe has 150 rows and 6 columns.
Let's move on to the columns and their respective data types. We'll employ the info() function for
this.
Details about the Dataset
df.info()
Output:
As we can see, just one column has categorical information, while the others are all numeric
columns with non-Null entries.
Relation between variables
We'll examine the relationship between petal length and width as well as the relationship
between sepal length and sepal width.
Comparing Sepal Length and Sepal Width
# importing packages
import seaborn as sns
import matplotlib.pyplot as plt
sns.scatterplot(x='SepalLengthCm', y='SepalWidthCm',
hue='Species', data=df, )
# Placing Legend outside the Figure
plt.legend(bbox_to_anchor=(1, 1), loc=2)
plt.show()
Output
The pattern depicted above shows that the:
·
Setosa species' sepals are broader but shorter in length.
·
The Versicolor Species is in the middle of the other two species in terms of sepal length
and width.
·
Sepal widths are lower but sepal lengths are longer in the species Virginica.
Python code for pairplot
# importing packages
import seaborn as sns
import matplotlib.pyplot as plt
sns.pairplot(df.drop(['Id'], axis = 1),
hue='Species', height=2)
Output
Module IV: Big Data Architectures
What is Big Data?; Impact of Big Data; Parallel Processing, Scaling, and Data Parallelism;
Tools of Big Data; Beyond the Hype; Big Data Use Cases; Viewpoints about Big Data; What is
Hadoop; An Introduction to Map Reduce; The Hadoop Ecosystem/Common Components:
Introducing HDFS, Hive, HBase, and Spark, Working with HDFS; Working with Hbase; Why Use
Apache Spark?; Functional Programming Basics; Parallel Programming Using Resilient
Distributed Datasets; Scale-out/ Data Parallelism in Apache Spark; Data Frames and SparkSQL
Big Data Introduction and Tools
The large volume of organised and unstructured data being generated at a rate that was
previously unheard of from a variety of sources, including social media, sensors, mobile
devices, websites and more, is referred to as "big data."
Why is Big Data Important?
Big Data's capacity to offer insightful information and business intelligence has made it a crucial
asset for organisations in a variety of industries. Businesses can get a competitive edge by
analysing big data to make data-driven decisions, spot trends, find patterns and identify trends.
Several important uses for big data include:
·
Business intelligence: Businesses utilise Big Data analytics to learn more about consumer
behaviour, market trends and operational effectiveness. This information helps them make
better decisions and plan their strategies.
·
Personalisation: Based on unique user preferences and behaviours, big data enables
personalised marketing and product recommendations.
·
Healthcare: Big Data analytics in the field of healthcare can enhance patient outcomes,
optimise therapeutic approaches and spot patterns that can be used to avoid disease.
·
Internet of Things (IoT): The enormous amount of data that IoT devices create may be
analysed to boost productivity and optimise processes across a range of industries, including
manufacturing, transportation and utilities.
·
Big Data analytics may be used to analyse social media to assist organisations better
understand customer sentiment, brand perception and engagement.
Introduction to Big Data
An increasing amount of data is produced continuously. Fresh data are generated by all of
these and more and these data must be kept for some reason. Mobile phones, social
networking and imaging technologies utilised in medical diagnosis are a few examples. Devices
and sensors automatically produce diagnostic data, which must be recorded and processed
right away.
Maintaining this tremendous input of data is difficult enough, but it is far more difficult to analyse
large amounts of it to uncover important trends and extract useful information, especially when
the data doesn't adhere to traditional notions of data structure. The challenges posed by the
data flood present an opportunity to change business, government, research and daily life.
A number of industries have led the way in developing their capacity to gather and use data.
•
By applying rules developed from processing billions of transactions, credit card firms can
accurately identify fraudulent purchases made by their consumers by keeping track of every
purchase they make.
•
Mobile phone providers examine users' calling habits to find out, for instance, whether a
caller's go-to contacts are on a competing network. The mobile phone provider can proactively
offer the customer a perk to stay in her contract if the competing network is running an alluring
promotion that could lead her to leave.
•
The primary product for businesses like LinkedIn and Facebook is data itself. These
businesses' valuations heavily depend on the data they collect and host, which increases in
intrinsic value as the data volume increases.
Big Data is the term used to describe the enormous volume of data that is too complex and vast
to be effectively processed by conventional data processing and storage techniques.
Big Data has completely changed how businesses manage and analyse massive volumes of
data. Innovative technologies and approaches are used to address the problems it creates,
giving businesses, academic research and a variety of industries useful insights and
opportunities. Big Data will continue to be an important area for data analysis and decisionmaking in the future as the volume of data keeps increasing exponentially.
The three Vs—Volume, Velocity and Variety—define the term "Big Data".
1.
Volume: Compared to conventional databases and data management solutions, big data
involves a vast volume of data. Terabytes, petabytes and even exabytes of data may be
present.
2.
Velocity: Big Data is produced, gathered and processed at a previously unheard-of rate.
Numerous sources, including social media, sensors, machinery and online transactions,
continuously provide data.
3.
Variety: Structured, semi-structured and unstructured data are only a few of the different
formats and types of big data. Text, pictures, videos, audio, log files, posts on social media and
more are all included.
Big data is any significant amount of organised, semi-structured and unstructured data that can
be mined for information and in which individual records are no longer significant in favour of
aggregates only. Big data is data that is difficult to process with traditional techniques.
History of Big Data Analytics
When organisations first began storing and analysing enormous volumes of data on computers,
in the early days of computing, big data analytics has a long history. However, it wasn't until the
late 1990s and early 2000s that big data analytics started to really take off as businesses turned
more and more to computers to help them make sense of the enormous volumes of data that
were being produced by their operations.
Today, big data analytics is a vital tool for companies of all sizes operating in a range of
industries. By using the power of big data, businesses are now able to discover things about
their consumers, markets and surrounding environment that were previously unimaginable.
As the field of big data analytics continues to advance, we might expect to see even more
amazing and ground-breaking applications for this technology in the years to come.
Applications for big data analytics commonly use data from internal and external sources, such
as weather data or customer demographic information provided by external information service
providers. Streaming analytics applications are also becoming more common in big data
contexts as clients attempt to perform real-time analytics on data streamed into Hadoop
systems through stream processing engines like Spark, Flink and Storm.
Early big data systems were typically implemented locally, especially in large organisations that
collected, organised andanalysed massive volumes of data. The process of setting up and
operating Hadoop clusters on the cloud has been made simpler by cloud platform providers like
Amazon Web Services (AWS), Google and Microsoft. The same is true for vendors of Hadoop,
such as Cloudera, who offer the implementation of the big data framework across the AWS,
Google and Microsoft Azure clouds. Users may now rapidly create cloud clusters, use them for
as long as necessary and then take them offline thanks to usage-based pricing that does not
require ongoing software licences.
Big data has significantly benefitted supply chain analytics. Big data and quantitative methods
are combined with supply chain analytics to enhance decision-making. In particular, big supply
chain analytics expands the data sets for analysis beyond the standard internal data available
on ERP and SCM systems. Additionally, big supply chain analytics uses highly effective
statistical methods on both recent and historical data sources.
Challenges of Big Data
In a broader context, here are 10 big data issues that businesses should be aware of and some
solutions for them.
1. Managing large volumes of data
By definition, big data refers to enormous amounts of data spread across numerous platforms
and systems. According to Szybillo, merging the incredibly large data sets extracted from CRM
and ERP systems, as well as other data sources, into a coherent and understandable big data
architecture is the first challenge for enterprises.
He claimed that by making small adjustments, it is easier to pinpoint insights once you have a
sense of the data being gathered. To accomplish this, design an architecture that enables
progressive modifications. Making significant changes could lead to the emergence of new
challenges.
Moving a big data initiative from boardroom discussions to business insights involves a number
of challenges for businesses.
Enterprises confront a number of difficulties in guiding a big data endeavour from boardroom
talks to business insights.
2. Identifying and resolving data quality problems
When data quality problems infiltrate big data platforms, analytics algorithms and artificial
intelligence systems built on big data may produce mediocre results. As data management and
analytics teams strive to gather more and varied types of data, these issues may worsen and
become more difficult to audit. When Bunddler, an online marketplace for hiring web shopping
assistants who help individuals make orders and arrange deliveries, reached 500,000 clients, it
saw these issues firsthand. The utilisation of big data to create highly personalised experiences,
identify opportunities for upselling and follow emerging trends was a crucial growth driver for the
business. Controlling data quality effectively was a major problem.
Pavel Kovalenko, CEO of Bunddler, said that "you must constantly monitor and correct any data
quality issues." He claims that mistakes and duplicate entries are common, especially when
data is compiled from many sources. Kovalenko's team developed an intelligent data identifier
that can find duplicates with minuscule data changes and spot any potential faults to verify the
accuracy of the data they collect. As a result, business insights from data analysis are now more
precise.
3. Handling challenging data preparation and integration tasks
The challenge of collecting and storing vast volumes of data of all kinds is handled by big data
platforms, as is the quick retrieval of data needed for analytics. Rosaria Silipo, a Ph.D. and lead
data scientist at open source analytics platform provider Knime, asserts that data collection may
still be challenging.
The integrity of an enterprise's collected data repositories depends on how frequently they are
updated. This necessitates ongoing access to a variety of data sources in addition to certain big
data integration tools.
Some companies, without taking into account how the heterogeneous data will be integrated,
use a data lake as a catch-all repository for vast amounts of big data collected from numerous
sources. For instance, different business domains offer data that is helpful for combined
analysis, but this data often has inconsistent underlying semantics that must be addressed.
According to Silipo, initiatives involving ad hoc integration may require a significant amount of
rework. For the highest return on investment on big data initiatives, it is frequently preferable to
adopt a strategic approach to data integration.
4. Cost-effectively and efficiently scaling huge data systems
Businesses run the danger of losing a lot of money if they don't have a plan for what to do with
their enormous volumes of data. According to ZL Tech's head of corporate solutions, George
Kobakhidze, organisations need to recognise that big data analytics begins with data entry.
Curating firm data repositories also requires continuous retention procedures to cycle out
obsolete information because data from before the COVID-19 epidemic is typically no longer
reliable in today's market.
Therefore, before deploying big data solutions, data management teams should map out the
many types, formats and purposes of data. As Travis Rehl, vice president of product at cloud
management platform provider CloudCheckr, notes, this is easier said than done.
"Often, you start with one data model and expand out, but you quickly realise the model doesn't
fit your new data points and you suddenly have technical debt to resolve," he explained.
Data reuse can be done effectively and affordably with the help of a generic data lake and the
right data structure. In a data lake, Parquet files, for instance, usually offer a better
performance-to-cost ratio than CSV dumps.
5. Assessing and choosing big data technologies
A range of big data technologies are available to data management teams and the features of
the various solutions frequently overlap.
Teams should begin by assessing the present and future demands for data from batch and
streaming sources, such as mainframes, cloud apps and third-party data services, recommends
Lenley Hensarling, chief strategy officer at Aerospike. He suggests taking into consideration
enterprise-grade streaming choices like Apache Kafka, Apache Pulsar, AWS Kinesis and
Google Pub/Sub because they all enable seamless data transfer between cloud, on-premises
and hybrid cloud systems.
The next step is for teams to assess the sophisticated data preparation skills needed to feed
emerging analytics technologies like AI and machine learning. It is also vital to make plans for
the site of data processing. Teams need to consider how to run analytics and artificial
intelligence (AI) models on edge servers while also making it simple to update the models when
latency is a problem. The price of installing and maintaining the hardware and software that run
on-premises, in the cloud, or at the edge must be contrasted with these capabilities.
6. Developing commercial insights
Data teams may be tempted to prioritise big data technologies over results. Silipo has shown
that in many instances, the importance placed on how to use the data is substantially less.
Big data applications in companies must take into account scenarios like creating KPI-based
reporting, identifying useful forecasts and making various types of recommendations in order to
yield meaningful business insights.
A varied mix of business analytics experts, statisticians and data scientists with skills in machine
learning will be required to take part in these activities. She thinks that by bringing together that
team and the big data engineering team, the ROI of developing a big data environment can be
increased.
7. Finding and keeping big data experts in your workforce
According to Mike O'Malley, senior vice president of strategy at SenecaGlobal, a company that
specialises in software development and IT outsourcing, "Finding and keeping workers with big
data skills is one of the biggest challenges in big data software development."
Big data is a trend that won't go away anytime soon. Cloud architects and data scientists will be
among the most in-demand professions in 2021, claims S&P Global. Working with companies
that provide software development services and have already built up talent pools is one way to
fill them.
Pablo Listingart, the founder and owner of ComIT, a nonprofit organisation that offers free IT
training, suggests working with HR to identify and address any gaps in present big data skills.
According to him, many big data initiatives fail because of inaccurate assumptions and
estimations that are used throughout the entire project. The right team will be able to identify
hazards, evaluate their seriousness and address a variety of big data issues.
The development of a culture that draws in and keeps great talent is equally important.
According to Vojtech Kurka, CTO of customer data platform supplier Meiro, he first believed that
by inserting a few SQL and Python scripts in the right places, he could solve any data issue.
Over time, he realised that he could make a big difference by hiring the right people and
creating a secure workplace environment that keeps workers happy and motivated.
8. Preventing out-of-control spending
According to David Mariani, founder and CTO of data integration firm AtScale, another common
big data fear is the "cloud bill heart attack." Many firms make the error of estimating the cost of
their new big data infrastructure using the KPIs for current data usage.
One issue is that organisations underestimate the sheer amount of computer resources that
would be required as access to larger data sets is increased. Big data platforms may now
disclose richer, more detailed data more easily thanks to the cloud, which also makes it possible
for pricing to rise as a result of cloud systems' ability to expand flexibly in response to customer
demand.
An on-demand pricing technique can result in price hikes as well. Using set resource pricing is a
good strategy, however it won't solve the issue completely. Even though the metre stops at a
certain point, poorly designed program s may continue to consume resources that impact other
users and workloads. Building fine-grained query controls is another advised practice.
When talking about large data deployments, Rehl of CloudCheckr advises data management
teams to bring up the cost issue with business and data engineering teams. Software
developers should be in charge of efficiently supplying the data and DevOps should be in
charge of monitoring and managing the proper archiving rules and growth rates. It is the
responsibility of the firm to be clear about what it needs.
9. Governing ecosystems for massive data
Data governance issues becoming harder to resolve when big data applications proliferate
across more systems. Modern cloud architectures allow businesses to capture and store all of
the data they acquire in its raw form, which exacerbates this difficulty. Numerous program s may
unintentionally contain protected information areas.
"In my experience, without a data governance strategy and controls, much of the benefit of
broader, deeper data access can be lost," Mariani continued.
A good practice is to approach data as a product with built-in governance requirements from the
beginning. Giving self-service access that does not require monitoring of each new use case will
be simpler if more time is spent up front identifying and managing significant data governance
concerns.
10. Ensuring that the use cases and context of the data are clear
Additionally, businesses often place too much emphasis on technology without understanding
the context of the data and its practical uses.
Adam Wilson, CEO of Trifacta, a provider of data wrangling tools, observed that "often a tonne
of effort is put into thinking about big data storage architectures, security frameworks and
ingestion, but very little thought is put into onboarding users and use cases."
Teams must think about who and how the data will be improved. Individuals closest to the
business challenges must collaborate with those closest to the technology in order to control
risk and ensure proper alignment. To do this, one must think about democratising data
engineering. Additionally, it helps to develop a few straightforward end-to-end use cases to
engage stakeholders, accomplish quick wins and examine limits.
E-Content
Parallel Processing, Scaling and Data Parallelism
Big data analytics depend heavily on parallel processing since it enables quick and effective
analysis of massive amounts of data. Processing large datasets using big data analytics
frequently requires more resources than a single machine or conventional processing
techniques can provide. Data analysis becomes faster and more scalable thanks to parallel
processing, which allows for the spread of data and computations across numerous processors
or nodes. Here is how big data analytics makes use of parallel processing:
A key component of big data analytics is parallel processing. Big data analytics systems can
effectively handle and analyse enormous volumes of data by utilising distributed computing and
parallelism. This enables businesses to get insightful information, make data-driven decisions
and find patterns and trends buried inside gigantic datasets. Despite the difficulties, parallel
processing is still a vital tool for meeting the needs of contemporary big data analytics.
1. Distributed Data Storage: Data is generally dispersed across several nodes or servers in a
distributed storage system, such as Apache Hadoop Distributed File System (HDFS), when big
data analytics is being used. Data may now be retrieved and processed concurrently from
several nodes thanks to parallel processing, which ensures data is used effectively and lessens
data transfer bottlenecks.
2. MapReduce Paradigm: A crucial part of large data analytics, the MapReduce programming
style significantly relies on parallel processing. The process consists of two stages: the Map
phase, in which data is processed concurrently across several nodes to produce intermediate
key-value pairs and the Reduce phase, in which intermediate results are compiled to create the
final output. This method is appropriate for large-scale analytics projects since it supports
parallel and distributed data processing.
3. Parallel Data Transformation: Data transformations and calculations that are difficult are
frequently used in big data analytics. Faster data preparation and analysis are made possible by
parallel processing, which enables data transformations, filtering, aggregation and other
activities to be carried out concurrently across numerous nodes.
4. Parallel Machine Learning:Algorithms for machine learning can be computationally
demanding, particularly those utilised in large data analytics. In order to speed up model training
and increase scalability, parallel processing enables the distribution of training and assessment
tasks across numerous nodes.
5. Real-Time Stream Processing: Real-time data streams must be continuously analysed in big
data analytics. Real-time data handling requires parallel processing because it allows for the
simultaneous processing of incoming data across numerous nodes, ensuring low-latency
analysis and decision-making.
6. Batch Processing: Large datasets must be processed in batches for many big data analytics
activities. Data processing is sped up and execution time is decreased because to parallel
processing's effective distribution of batch jobs across numerous nodes.
7. Scalability: Parallel processing is naturally scalable, enabling businesses to cope with
growing data quantities and processing requirements. Nodes can be added to the cluster as
needed to keep up with the cluster's computational demands as data accumulates.
8. Fault Tolerance: Big data analytics systems frequently run on computer clusters and errors
are frequent. By replicating data and restarting unsuccessful jobs on healthy nodes, faulttolerant parallel processing frameworks like Apache Spark and Hadoop ensure robustness.
Challenges in Parallel Processing for Big Data Analytics:
·
Data Skew: When data is distributed unevenly, some nodes process more data than
others, leading to data skew. This may result in an unbalanced load and slow down operations
as a whole.
·
Network Overhead: In distributed systems, node-to-node communication can cause
network overhead, which can have an effect on performance, particularly when data needs to
be sent back and forth between nodes.
·
Task Synchronisation: To prevent conflicts and guarantee accuracy, it is important to
carefully synchronise the execution of tasks across several nodes.
·
Complexity of Debugging: Due to the increased complexity of distributed systems,
debugging and optimising parallel processing program s can be difficult.
In big data analytics, scaling refers to a system's capacity to manage increased data quantities,
rising computational needs and bigger workloads without compromising performance or
efficiency. Scaling is necessary to make sure that data analytics systems can handle the evergrowing data amount and complexity as data continues to expand exponentially. In big data
analytics, vertical scaling and horizontal scaling are the two main methods of scaling.
1. Vertical Scaling (Scaling Up):
To handle more workloads and data volumes, a single system can be vertically scaled by
upgrading its hardware. Usually, this is accomplished by giving the server more processing
power, memory, or storage space. For instance, a single server's throughput and ability to
analyse bigger datasets can be increased by adding additional RAM or CPU cores. When data
volumes are moderate and it is feasible to upgrade existing hardware to handle the processing
demands, vertical scaling is appropriate.
Advantages of Vertical Scaling:
·
Because only one machine is involved, management is made easier.
·
For modest to medium-sized data volumes, it may be cost-effective.
·
Easier to install because the present system needs to only be slightly altered.
Limitations of Vertical Scaling:
·
limited scalability since hardware has upper limits.
·
Given that there is only one machine, there is a higher danger of single points of failure.
·
Processing excessively huge amounts of data could become expensive.
2. Horizontal Scaling (Scaling Out):
To manage greater workloads and data volumes, a distributed computing system can be
horizontally scaled by adding more machines or nodes. In this method, processing operations
are carried out concurrently while the data is disseminated across a number of computers. The
main technique used in big data analytics to handle enormous datasets and live data streams is
horizontal scaling.
Advantages of Horizontal Scaling:
·
Highly scalable, allowing for the installation of new nodes to handle expanding data
volumes.
·
increased fault tolerance as the risk of single points of failure is decreased by the use of
many machines.
·
Cost-effective because it employs common hardware for processing massive amounts of
data.
Limitations of Horizontal Scaling:
·
A rise in complexity brought on by data segmentation and distributed computing.
·
For efficient load balancing and data synchronisation, a resilient distributed system is
needed.
·
overhead incurred during the distribution of data and tasks across nodes due to network
communication.
Scaling Techniques in Big Data Analytics:
1.
Sharding: Data is divided into smaller portions, or shards and distributed across a number
of nodes using a technique called sharding. Scalability and parallel processing are made
possible by the fact that each node is only in charge of processing a portion of the data.
2.
Load Balancing: In a distributed system, load balancing makes sure that the burden is
distributed equally among all nodes. It maximises the use of computational resources and aids
in the prevention of data skew.
3.
Replication: To increase fault tolerance and lower the chance of data loss in the event of
node failures, replication involves replicating data across many nodes.
4.
Distributed File Systems: Massive datasets can be stored and managed over several
nodes using distributed file systems, such as Hadoop Distributed File System (HDFS) and
Amazon S3, which offer scalability and fault tolerance.
5.
Data Partitioning: By using particular criteria, such as data ranges, hash keys, or
geographic areas, data partitioning entails breaking the data up into smaller portions. This
makes it possible to get and handle data more effectively.
6.
Cluster Management Systems: Systems for managing a distributed cluster's resources
and task coordination among nodes include Apache YARN and Kubernetes.
Data Parallelism
A major idea in big data analytics is data parallelism, which focuses on partitioning large
datasets into smaller ones and processing them concurrently across several nodes or
processors. This method makes it possible to handle data in an effective and scalable manner,
which speeds up the examination of large volumes of data. Data parallelism is a key method
used to run parallel calculations on massive data in distributed computing frameworks like
Apache Hadoop and Apache Spark. Data parallelism in big data analytics functions as follows:
1. Data Partitioning: The first stage in data parallelism is to divide the input data into smaller
sections, or partitions. Each partition, which is distributed among several nodes in the
computing cluster, comprises a portion of the data.
2. Parallel Processing: After the data has been disseminated and partitioned, each node
processes its allocated data partition independently. The parallel application of the same data
processing logic to each partition enables the concurrent execution of data processing
operations.
3. Results Aggregation: The final output is created by aggregating the results of each node's
computation after processing. To acquire a thorough analysis of the entire dataset, the
aggregation process aggregates the results from several nodes.
Advantages of Data Parallelism in Big Data Analytics:
•
Scalability: As more nodes may be added to the computer cluster to address expanding
data volumes and computational needs, data parallelism makes it simple to scale.
•
Efficiency: Data parallelism ensures efficient use of computing resources and cuts
processing time by partitioning the data into smaller chunks and processing them in parallel.
•
Fault Tolerance: Frameworks for distributed computing like Apache Hadoop and Spark
have built-in fault tolerance mechanisms. The system can automatically retry failed jobs on
healthy nodes if a node fails during computation.
•
High Throughput: Data parallelism offers high throughput data processing, making it
appropriate for batch processing of huge datasets and real-time data streams.
Use Cases of Data Parallelism in Big Data Analytics:
•
Big Data Processing: For processing and analysing large datasets, such as logs, sensor
data and social media interactions, data parallelism is frequently employed in big data analytics.
•
Machine learning: To efficiently execute complex algorithms, machine learning uses data
parallelism to train models on numerous huge datasets simultaneously.
•
Real-time stream processing is made possible by data parallelism, which enables lowlatency analysis and decision-making.
•
Data parallelism is used to process and analyse huge text corpora for sentiment analysis,
topic modelling and language translation. Text analysis and natural language processing.
Challenges of Data Parallelism:
•
Data Skew: When there is an imbalance in the distribution of data among nodes, it can
result in data skew, which slows down processing overall since some nodes must process more
data than others.
•
Communication Overhead: If the data transfer is frequent and significant, communication
overhead may represent a bottleneck while collecting results from different nodes.
•
Synchronisation: Data parallelism needs effective synchronisation techniques to make
sure that all nodes finish their work prior to aggregation, preventing discrepancies in the result.
•
Design of Algorithms: Not all algorithms are easily parallelizable. The intrinsic
dependencies of some algorithms make parallel processing difficult.
Tools of Big Data
Large and complicated datasets that can't be simply handled by conventional data processing
systems must be processed and analysed as part of big data. The Big Data ecosystem is made
up of a variety of tools and technologies that have been developed to handle, process
andanalyse such massive quantities. Here are a few of the crucial devices employed in the big
data field:
1. Apache Hadoop: One of the key pieces of the Big Data ecosystem is Apache Hadoop. For
managing massive datasets, it offers a distributed storage and processing platform. Hadoop is
made up of MapReduce for distributed data processing and Hadoop Distributed File System
(HDFS) for storage.
Large-scale datasets can be distributedly processed and fault-tolerantly analysed using the
open-source Apache Hadoop distributive computing platform. The Apache Software Foundation
now maintains it after Doug Cutting and Mike Cafarella originally developed it. One of the key
components of the Big Data ecosystem, Hadoop has emerged as a vital tool for effectively
managing enormous amounts of data.
Important Elements of Apache Hadoop
•
HDFS, the Hadoop Distributed File System: A distributed file system called HDFS offers
scalable and dependable storage for massive amounts of data. For fault tolerance, it separates
data into blocks and maintains several copies of each block on various cluster nodes.
•
MapReduce: MapReduce is a processing engine and programming model utilised within
the Hadoop framework for the purpose of distributed data processing. The data processing
operations are categorised into two distinct stages: the Map phase, where input is concurrently
processed across several nodes and the Reduce stage, where the output generated from the
Map phase is integrated.
Apache Hadoop's main attributes are:
•
Distributed Processing: Data processing operations can be split across several cluster
nodes with Hadoop, allowing for simultaneous processing of data and better performance.
•
Scalability: Hadoop can accommodate petabytes and even exabytes of data thanks to its
distributed architecture, which enables it to extend horizontally by adding more nodes to the
cluster.
•
Fault Tolerance: Hadoop provides fault tolerance by replicating data blocks across a
number of cluster nodes. Data integrity and availability are guaranteed since data can be
retrieved from replicas in the event that a node fails.
•
Cost-Effectiveness: Hadoop uses commodity gear, which is less expensive in terms of
processing huge datasets than specialised technology.
•
Flexibility: Hadoop is suited for a variety of data sources and formats because it can
manage a variety of data kinds, including structured, semi-structured and unstructured data.
•
Ecosystem: Hadoop includes a large ecosystem of libraries and tools that expand its
capability and let users accomplish things like data analytics, machine learning, real-time
processing and more.
2. Apache Spark:An open-source, distributed computing system called Apache Spark allows for
in-memory data processing. It is intended for real-time data streaming, high-speed data
processing, machine learning and graph processing.
Apache Spark is a distributed computing platform that is open-source and made for processing
and analysing enormous amounts of data. It was first created in 2009 by the UC Berkeley
AMPLab and then contributed to by the Apache Software Foundation. The foundation of Spark
is in-memory data processing, which makes it possible to handle data quickly and effectively. It
offers high-level APIs in a variety of programming languages, making it usable by developers of
all backgrounds.
Key Features of Apache Spark:
•
Processing Data in Memory: Spark processes data in memory instead of reading and
writing it to disc, which greatly accelerates computing.
•
Distributed computing: Spark scales data processing and analytics by distributing data
over several cluster nodes and processing it in parallel.
•
Data Abstraction: Spark offers high-level abstractions, such as DataFrames, Datasets
and RDDs (Resilient Distributed Datasets), for working with both organised and unstructured
data.
•
Multiple Language Support: Spark is available to developers with a variety of language
preferences thanks to its support for multiple languages, including Scala, Java, Python and R.
•
Processing in Batches and Real-Time: Spark offers real-time processing for processing
data streams using Spark Streaming and Structured Streaming, as well as batch processing,
where data is processed in fixed-size chunks.
•
Machine Learning and Graph Processing: The machine learning (MLlib) and graph
processing (GraphX) libraries included in Spark enable users to carry out complex analytics jobs
on the same platform.
•
Built-in Libraries: Spark has built-in libraries for SQL, machine learning, graph processing
and data streaming, which eliminates the need for third-party dependencies.
3. Apache Hive: Built on top of Hadoop, Apache Hive is a data warehousing and SQL-like
querying tool. It enables users to interact with and analyse data stored in Hadoop by writing
SQL-like queries.
Built on top of Apache Hadoop, Apache Hive is an open-source data warehousing and SQL-like
querying tool. The Facebook Data Infrastructure team created it and later the Apache Software
Foundation received its contribution. Users can more easily interact with and analyse massive
datasets stored in the Hadoop Distributed File System (HDFS) or other comparable storage
systems thanks to Hive, which offers a high-level abstraction over Hadoop.
Key Features of Apache Hive:
•
Hive employs Hive Query Language (HQL), which is comparable to SQL and is therefore
familiar to users with SQL knowledge. In order to execute data analysis and transformation on
Hadoop data, it enables users to create SQL-like queries.
•
Hive uses a "schema on read" approach, in contrast to typical databases that enforce
schema at write time. Users can choose the data schema at the moment a query is executed,
giving them flexibility in how they handle varied and changing data.
•
Hive uses tables to create data abstractions for both structured and semi-structured data.
To analyse data without moving it, users can establish external tables over the data already
saved in HDFS or other storage systems.
•
User-defined functions (UDFs), user-defined aggregation functions (UDAFs) and userdefined table-generating functions (UDTFs) are all supported by Hive. Users can create unique
functions to process data in accordance with their own needs thanks to this extensibility.
•
Planning and optimisation of queries: Hive generates query plans and executes them in a
distributed fashion to optimise queries. To boost query performance, it makes advantage of
methods like predicate pushdown and map-side joins.
•
Hive's smooth integration with other Hadoop ecosystem parts including HDFS, Apache
HBase, Apache Spark and Apache Tez enables users to take advantage of these tools' features
in their data processing pipelines.
Hive Architecture: Hive consists of the following major components:
•
Hive Metastore: The Hive Metastore is a central database where information about Hive
tables, databases, partitions and other related data is stored. It aids in keeping the statistics and
schema of the data kept in Hadoop.
•
Hive Driver: The Hive Driver is in charge of taking user-submitted HQL queries and
parsing them. It creates a query execution plan and coordinates the query's execution across
several Hive components.
•
Hive Execution Engine: Execution engines supported by the Hive framework include
MapReduce, Apache Tez and Apache Spark. The driver's query plan is processed by the
execution engine, which then runs it on the underlying Hadoop infrastructure.
4. Apache HBase:A NoSQL database called Apache HBase offers random read and write
access to vast amounts of structured data. It is designed to be used for real-time data
processing and is developed on top of HDFS.
Built on top of the Hadoop Distributed File System (HDFS), Apache HBase is an open-source,
scalable, distributed NoSQL database. It is made to store and manage massive amounts of
structured data in a distributed, fault-tolerant manner. HBase is a component of the Apache
Hadoop ecosystem and offers random read and write access to data, making it appropriate for
low-latency and real-time data processing.
Key Features of Apache HBase:
•
Column-Family Data Model: Data is organised into tables using the column-family data
paradigm, which HBase employs. Each table may contain a number of different column families.
Multiple columns can be included in each column family, enabling flexible and dynamic data
architecture.
•
Distributed and Scalable: Because HBase is created to work in a distributed environment,
it has the ability to grow horizontally by adding more nodes to the cluster. Petabytes of data can
be handled effectively.
•
Fault Tolerance: By duplicating data across numerous cluster nodes, HBase offers fault
tolerance. Data availability and integrity are guaranteed since data can be retrieved from
replicas in the event that a node fails.
•
Storage choices: HBase enables both in-memory and on-disk choices for data. It can
keep less often accessible data on disc to save money while caching frequently accessed data
in memory for low-latency access.
•
High Performance: HBase is excellent for real-time data processing and applications
demanding low-latency access to data due to its high read and write throughput.
•
Schema Flexibility: HBase does not impose a hard structure, allowing users to flexibly
add or modify columns without interfering with data access.
•
Built-in Versioning: HBase maintains multiple copies of the data, enabling users to
retrieve older copies and run time-based queries.
HBase Architecture:
HMaster: The cluster's main node in charge of coordination and management is called the
HMaster. It manages administrative duties like failover, splitting and region assignment.
RegionServers: Worker nodes known as RegionServers maintain and store data regions. Each
RegionServer in the cluster is capable of managing numerous data regions and data is
dispersed among the many RegionServers.
ZooKeeper: For distributed coordination and consensus in HBase, ZooKeeper is employed. It
facilitates coordination between HMaster and RegionServers and aids in managing the
scattered nature of HBase.
5. Apache Pig: A high-level data flow language and execution framework called Apache Pig is
used to handle and analyse big datasets. It makes Hadoop's sophisticated data transformations
easier to program .
Apache Pig is a high-level scripting language and open-source platform created to make it
easier and more efficient to process huge datasets on Apache Hadoop. Yahoo! created it and
the Apache Software Foundation later received contributions. Pig offers a data flow language
called Pig Latin that simplifies the difficulties of designing MapReduce tasks in Java and makes
working with Hadoop easier for data analysts and engineers.
Key characteristics of Apache Pig
Data Flow Language: Pig Latin is a high-level data flow language that enables users to define
data processing processes in a way that is more logical and succinct than when constructing
Java MapReduce jobs.
Integration with Hadoop Ecosystem: Pig abstracts the underlying difficulties of building
MapReduce program s, allowing users to concentrate on data transformation and analysis
rather than the specifics of implementation.
Schema Flexibility: Because Pig does not impose a rigid schema on data, it is appropriate for
working with semi-structured and unstructured data.
Opportunities for Optimisation: Pig automatically optimises data processing operations,
optimising the execution plan to boost efficiency and use less resources.
Extensibility: Pig enables the inclusion of unique functions for specialised data processing tasks
by allowing users to write User Defined Functions (UDFs) in Java, Python, or other languages.
Integration with Hadoop Ecosystem:Pig's smooth integration with other elements of the Hadoop
ecosystem, such as HDFS and Apache HBase, enables users to take advantage of these tools'
features in their data processing pipelines.
6. Apache Kafka:Real-time data streaming and message queuing are made possible by the
distributed streaming platform Apache Kafka. It is frequently employed for stream processing
and data intake.
An open-source, distributed streaming technology called Apache Kafka was created by LinkedIn
and eventually added to the Apache Software Foundation. For the purpose of developing data
pipelines and applications, it is made to handle real-time data streaming and event processing.
Large volumes of data streams can be distributedly ingested, stored and processed using
Kafka's high-throughput, fault-tolerant and scalable technology.
Key Concepts of Apache Kafka:
Topics: Kafka divides streams of data into topics, which serve as channels or classifications for
the data. Consumers read data from certain topics where producers have written data.
Partitions: There may be one or many partitions for each topic. Data distribution across several
nodes in a Kafka cluster is made possible through partitions, enabling for scalability and
simultaneous data processing.
Brokers: The distinct servers or nodes that make up a Kafka cluster are called brokers. They
take care of data replication and dissemination as well as data storage and management for the
assigned partitions.
Producers: Applications or systems that write data to Kafka topics are known as producers.
They make data records available for customer consumption by publishing them to Kafka
brokers.
Consumers: Applications or systems that read data from Kafka topics are called consumers.
They process data records in real-time or batch mode and subscribe to one or more subjects.
Consumer Groups: Consumer groups are organisations of consumers. A topic's data is
individually read by each consumer group and the messages inside a topic are dispersed
among the group's members.
Connectors: A framework for creating and executing connectors, known as Kafka Connect,
makes it easier to integrate Kafka with other systems or data sources.
Kafka Architecture: Kafka can effectively manage massive data streams because of its
distributed and fault-tolerant architecture. The principal elements of Kafka's architecture are as
follows:
•
Producers: Data records are published by producers to Kafka brokers. They have a
choice in what subjects they publish data about.
•
Brokers: Data for one or more topics is stored and managed by Kafka brokers. They take
care of data dissemination, partitioning and replication.
•
ZooKeeper: ZooKeeper is used in Kafka for administration and coordination activities like
managing broker metadata, choosing a leader and identifying broker failures.
•
Consumers: Kafka topics' data records are read by consumers. They may be a part of a
consumer group that would divide up the processing of data among several customers.
Beyond the Hype
In isolation, big data lacks inherent value. The potential utility of it becomes evident just when
employed to exert influence on decision-making processes. Organisations necessitate efficient
approaches to convert substantial quantities of swiftly evolving and heterogeneous data into
meaningful findings, thereby facilitating evidence-based decision-making. The figure presented
below depicts the five distinct stages that comprise the whole process of acquiring insights from
vast amounts of data, as outlined by Labrinidis and Jagadish (2012).
The two fundamental subprocesses of data management and analytics are constituted by the
following five steps. Data management encompasses a set of methodologies and
supplementary resources that are employed to collect, store and preprocess data in order to
facilitate subsequent analysis. On the contrary, analytics pertains to the methodologies
employed in the examination and extraction of valuable insights from extensive datasets.
Consequently, big data analytics can be regarded as an integral component of the broader
procedure of obtaining valuable insights from vast volumes of data.
Figure: Processes for extracting insights from big data
Across industries, data analytics has become a disruptive force that offers firms useful insights
and competitive benefits. There is now much enthusiasm and anticipation surrounding the
possibilities of data analytics due to the emergence of big data, advanced analytics and artificial
intelligence (AI). While there is no doubting the huge influence data analytics may have, it is
important to see past the marketing gimmick and comprehend the real-world difficulties,
restrictions and moral dilemmas related to its application.
Going beyond the hype in data analytics is using data analytics in a practical and pragmatic way
to benefit businesses and organisations. Although the term "data analytics" has gained
popularity and is said to offer a variety of advantages, it is important to look past the first
excitement and comprehend the difficulties and complications associated with its application.
Organisations can then use data analytics initiatives to make educated decisions and gain
valuable insights. The following are some essential components of going past the hype in data
analytics:
1. Define Clear Objectives: Setting clear and precise objectives is crucial before beginning any
data analytics project. Aligning analytics efforts with strategic objectives involves determining
the company issues or opportunities that data analytics can address. By avoiding the temptation
of pursuing analytics for analytics' sake, one can save money and assure a targeted approach.
2. Quality Data Matters: The calibre of the data utilised for analysis has a significant impact on
the success of data analytics. Organisations must invest in data governance, data cleansing
and data quality assurance because GIGO is a prevalent hazard. For the purpose of producing
precise and useful insights, a robust base of trustworthy data is essential.
3. Realistic Expectations: Data analytics can provide insightful information, but it is not a
panacea for all issues. Organisations should recognise the boundaries of data analytics and
have realistic expectations for what it can accomplish. Making educated decisions may
occasionally require the addition of extra criteria or domain knowledge because data alone may
not always have all the answers.
4. Address Data Privacy and Ethics: Data privacy and ethical standards must be prioritised by
organisations because data analytics entails handling sensitive and private information. Building
trust with customers and stakeholders is largely dependent on putting strong data security
measures into place, receiving informed consent for data usage and protecting user privacy.
5. Collaboration and Skill Sets: Data scientists, domain specialists, business analysts and IT
experts must work together because data analytics is a multidisciplinary area. These teams
must effectively communicate in order to make sure that analytics initiatives are in line with
business requirements and can produce results that can be put into practice.
6. Agile Approach: Since data analytics is an iterative process, initiatives utilising the agile
methodology might be beneficial. Flexibility and improved adaptability to shifting business
requirements are made possible by breaking down complicated problems into smaller,
achievable tasks and iterating regularly based on feedback and insights.
7. Measure Success: Organisations should evaluate the results of their data analytics program s
to get past the hype. Monitoring key performance indicators (KPIs) and assessing how analytics
affect business results give useful feedback for ongoing development and support expenditures.
8. Balancing Automation and Human Expertise: Although automation and AI are essential to
data analytics, human skill is still critical. Human intuition and judgement are needed in order to
comprehend the subtleties of the data, formulate the appropriate inquiries and interpret the
findings.
9. Scalability and Infrastructure: Scalability becomes an issue as data volumes increase.
Businesses need to make sure their data analytics infrastructure can handle growing data loads
and analysis needs. Solutions built on the cloud may be flexible and scalable.
10. Continuous Learning: Because the field of data analytics is continually expanding,
businesses need to promote a continual learning environment. Analytics efforts are made to
remain relevant and successful by encouraging data professionals to stay up to date with the
newest technologies, methods and best practices.
Big Data Use Cases and Viewpoints about Big Data
Big Data Analytics Use Cases in Various Industries
1. Banking and Finance (Fraud Detection, Risk and Insurance and Asset Management)
Future-oriented banks and financial institutions are making use of big data in a variety of ways,
from expanding into new countries and seizing market possibilities to reducing fraud and
managing investment risk. Big data analytics can be a potent tool for these organisations to use
in order to achieve a competitive edge.
View the following illustration to see several examples of how big data analytics are being used
in the banking and finance industries:
Big data analytics is predicted to grow at a CAGR of 22.97% from 2021 to 2026, according to
recent studies. Government rules and the volume of data being produced are driving up
demand for big data analytics in the industry.
2. Accounting
Since data is the lifeblood of accounting, adopting big data analytics would undoubtedly
increase the value of the accounting company. The accounting industry engages in a wide
range of tasks, including numerous audits, ledger checking and maintenance, transaction
management, taxation, financial planning, etc.
Big data analytics may assist auditors who work with many types of data, whether structured or
unstructured, by
·
Outliers identification
·
Exclude exceptions
·
Focus on data blocks of greatest risk areas
·
Visualize data
·
Connect financial and non-financial data
·
Compare predicted outcomes for improving forecasting etc
Big data analytics will lessen accounting redundancies and increase regulatory efficiency.
3. Aviation
According to studies, the market for aviation analytics will reach $3 billion by 2025 and grow at a
rate of 11.5% CAGR.
The following are the main factors boosting the aviation market:
·
Increasing demand for optimised business operations
·
COVID-19 outbreak affecting the normal aviation operations
·
Mergers, acquisitions and joint ventures
Recent developments and modifications in the aviation industry's Original Equipment
Manufacturer (OEM) and user segments Cloud-based real-time data gathering and analytics,
which call for a variety of data models, are one of the most lucrative big data analytics prospects
in the aviation sector.
Similar to other industries, big data analytics has enormous promise in the airline sector as well,
enhancing everything from routine tasks like maintenance, resource allocation, flight safety and
flight services to corporate objectives like loyalty program s and route optimisation.
The following illustration depicts the numerous aviation-related data generation points (flights
only), which can be a legitimate use case for big data analytics:
4. Agriculture
According to UN predictions, the world's population will reach 9.8 billion by 2050 and agriculture
will need to change to meet the needs of such a big population. But in addition to making most
farmlands unusable for farming, climate change has also affected rainfall patterns and dried up
certain water supplies.
This implies that farmers must enhance all other farming-related operations in addition to
growing crop yield.
Stakeholders in agriculture and agribusiness can benefit from big data analytics in the following
ways:
Approaches for precision farming that are based on cutting-edge technology like big data, IoT,
analytics, etc.
·
Offer advance warnings and climate change predictions
·
Ethical and wise use of pesticides
·
Farm equipment optimisation
·
Supply chain optimisation and streamlining
5. Automotive
Big data analytics has a broad range of applications in the automobile business, which is a
conglomeration of various independent industries, whether it be for research and development
or marketing strategy. The vehicle industry generates enormous amounts of data every single
day as a vital component of the infrastructure supporting numerous vital public and private
ecosystems!
It is one of the most important use cases for big data analytics as a result.
Typical applications include:
•
A thorough cost study of alternative designs and concepts can be used to improve the
design and production process.
•
Limitations on vehicle use and maintenance.
•
Tracking and keeping an eye on the manufacturing procedures to guarantee flawless
production.
•
Predicting market trends for the automotive industry's innovations, manufacturing and
sales.
•
Analysis of the logistics and supply chain.
•
Streamlining production to keep ahead of the competitors in the market.
•
Excellent analytics to design vehicles that are very user-friendly and perform well.
View the following picture to get a general understanding of the use cases for big analytics in
the automobile industry's value chain:
6. Biomedical Research and Healthcare (Cancer, Genomic medicine, COVID-19 Management)
According to recent statistics, the healthcare big data analytics industry would reach a value of
67.82 billion USD by 2025. The healthcare sector is a massive one that produces mountains of
data that are vital for research, government, insurance companies, hospitals and individuals.
Big data analytics can assist medical researchers in developing more successful and focused
treatment plans as well as in locating medicinal supplies around the world with the right analysis
of enormous data blocks.
Big data analytics can help the healthcare sector in a variety of ways, including improving
treatment facilities, developing better medications and forecasting pandemic or epidemic
outbreaks to tame their intensity.
To understand better, look at the following image:
The management of COVID-19 is also greatly aided by big data analytics, which forecasts
outbreaks, red zones and provides vital information to frontline staff.
And finally, big data analytics becomes an effective tool for biomedical research when we
consider:
·
Data sourcing, processing and reporting
·
Predicting trends and offering hidden patterns from historic data blocks
·
For the development of personalised medicine, genome research and personal genetic
data processing.
Big data analytics has several applications in the biomedical research and healthcare sectors
and these applications can be the subject of in-depth discussion on their own.
There are many big data analytics applications in biomedical informatics:
7. Business and Management
Unstructured data management is cited as a serious issue by 95% of firms, while big data and
AI are being invested in by 97.2% of corporate organisations for a variety of business goals,
including streamlining processes, implementing digitisation and introducing automation.
However, commercial organisations experience a variety of data pain issues, including:
·
Data silos
·
Unstructured data
·
Fragmented data
·
Database incompatibility
·
Unstructured data storage and management
·
Data loss due to cyber crimes
With its vast capabilities set, big data analytics might therefore be a saving grace for the
management and simplification of corporate processes.
8. Cloud Computing
At least one big data workload is currently being performed on the cloud by 45% of enterprises
worldwide and public cloud services will be the source of 90% of analytics and data innovation.
Security is just one of several difficulties that cloud computing faces. In fact, security is rising to
the top of commercial organisations' global concerns.
The stringent network, data and server requirements of big data analytics also influence
commercial organisations all over the world to outsource the headache and operational
overloads to third parties. Many new opportunities are being created as a result, supporting big
data analytics and assisting organisations in overcoming architectural challenges.
9. Cybersecurity
Big data security analytics is a new trend in cybersecurity that assists corporate organisations in
enhancing security through the following methods:
·
Identify outliers and anomalies in security data to detect hostile or suspicious actions.
·
Automation of work processes for responding to threats, such as stopping blatant malware
attacks.
53% of businesses who are already utilising big data security analytics claim to have reaped
significant rewards from doing so.
10. Telecom
Advanced technology and data analytics are about to unleash a massive tsunami of digital
change and revolution on the telecom industry. Big data analytics emerges as a reliable tool to
address several difficulties as the number of smartphone users rises and technologies like 5G
are ready to enter underdeveloped nations.
The following graphic displays some applications:
Big data applications in the telecom sector include:
Optimizing Network Capacity:Analysis of network utilisation for controlling network constraints,
rerouting bandwidth and interpreting infrastructure expenditures with data-driven insights from
various fields.
Telecom Customer Churn:Because there are so many options on the market, business owners
constantly run the danger of losing clients to rivals.
The brands can address the problem with great clarity thanks to the insights gained from data
on customer happiness, market research and service quality.
New Product Offerings:Instead of relying on generic brand products, telecom businesses can
develop innovative product offerings with the help of predictive analytics and in-depth market
research that are distinct, address customer pain points and address usability issues.
The Hadoop Ecosystem and Common components
One of the most important advancements in big data is the hadoop ecosystem. It offers
resources for data storage and analysis as well as a foundation on which other businesses can
build their apps.
As a result, you may either use the Hadoop ecosystem to develop your applications or just as a
tool in your infrastructure for storing and processing data. Due to its adaptability, it is a strong
choice for enterprise-level businesses that need to analyse massive amounts of data quickly.
To learn more about the Hadoop ecosystem, keep reading.
What Is Hadoop?
A cluster of computers can be used to store and handle massive amounts of data thanks to the
Hadoop software architecture.
Hadoop consists of two separate components:
1. HDFS (Hadoop Distributed File System)
To make it simpler for you to access big volumes of data via requests from your clients or apps,
it enables you to store massive amounts of data across numerous servers in a distributed way.
2. MapReduce
It lets your computer to quickly process these requests, regardless of how big they are.
What Is the Hadoop Ecosystem?
You can create applications on top of Apache Hadoop with the aid of the Hadoop Ecosystem,
which is a set of tools, libraries and frameworks.
Hadoop is ideally suited for big data issues because it offers huge parallelism with low latency
and high throughput.
In order to make it simpler to integrate Hadoop with other frameworks like Spark or Kafka for
real-time processing or machine learning activities, the Hadoop ecosystem definition expands
this capabilities with additional tools.
Hadoop Ecosystem Architecture With Diagram
Here's a Hadoop ecosystem diagram –
The four key parts of the Hadoop ecosystem architecture are data processing, data access,
data management and data storage.
1. Data Storage: Where all of your raw data is stored is the first step in describing the Hadoop
ecosystem. It might be stored locally or in the cloud.
2. Data Processing:During the second phase of the Hadoop ecosystem for Big Data, the data
undergoes analysis and transformation to derive meaningful insights that may be further
employed for subsequent research.
3. Data Access: In this third stage of the Hadoop ecosystem, you may query your data sets
using tools like Hive or Pig and take actions like filtering out particular rows, sorting them by
specific columns or values within them (like location or birthdate), etc.
4. Data Management: The final step in the Hadoop ecosystem design entails carefully
preserving all the work we've done on data sets in earlier phases so we may refer to it again if
necessary.
Introducing HDFS
The partitioning of a dataset among multiple physical machines becomes imperative when the
size of the dataset surpasses the storage capacity of a single system. Distributed filesystems
refer to a type of filesystem architecture that manages storage resources over a network of
interconnected computers. Distributed filesystems are more sophisticated than traditional disc
filesystems since they are network based, which brings with it all the challenges of network
programming. One of the hardest issues, for instance, is getting the filesystem to accept node
failure without losing data.
HDFS, which stands for Hadoop Distributed Filesystem, is the name of the distributed filesystem
that comes with Hadoop. (Occasionally, "DFS" may be mentioned casually or in older
documentation or settings; it refers to the same thing.) Nevertheless, it is worth noting that
Hadoop possesses a versatile filesystem abstraction, hence enabling us to explore its
integration with diverse storage systems, including but not limited to the local filesystem and
Amazon S3. The Hadoop Distributed File System (HDFS) serves as the primary filesystem for
Hadoop.
The Design of HDFS
Hadoop Distributed File System (HDFS) is a file system designed specifically for the storage of
large files, utilising clusters of cost-effective hardware. It is optimised for handling streaming
data access patterns. Let us delve more into the examination of this claim.
Very large files:In this sense, "very large" refers to files that are terabytes, gigabytes, or
hundreds of megabytes in size. Currently active Hadoop clusters may hold petabytes of data.
Streaming data access:The fundamental principle of HDFS is predicated on the notion that a
write-once, read-many-times paradigm is the optimal approach for data processing. In general,
a dataset is initially generated or obtained from a specific origin and afterwards, various
investigations are conducted on this dataset over a period of time. The duration required to read
the complete dataset holds greater importance compared to the latency involved in reading the
initial record, as subsequent analyses typically necessitate a substantial portion, if not the
entirety, of the dataset.
Commodity hardware: Hadoop doesn't need high-end, dependable hardware. It is intended to
run on commodity hardware clusters, which, at least for large clusters, have a high likelihood of
node failure across the cluster (generally accessible hardware that is available from numerous
vendors)3. In the event of such a failure, HDFS is built to continue operating without causing the
user to notice any interruptions.
Examining the applications for which HDFS is less effective is also worthwhile. While this could
change in the future, the following are some areas where HDFS is currently not a suitable fit:
Low-Latency Data Access:HDFS will not function properly with applications that need tens of
milliseconds or less to access data. Keep in mind that HDFS is designed to provide large
amounts of data quickly, possibly at the sacrifice of latency. Right now, HBase is a preferable
option for low-latency access.
Lots of Small Files:The number of files in a filesystem is constrained by the memory available
on the namenode because it stores filesystem metadata in memory. Every file, directory and
block typically occupy 150 bytes. As a result, you would require at least 300 MB of memory if,
for instance, you had a million files, each taking up one block. Millions of files can be stored, but
billions are beyond the hardware's existing capacity.
Multiple Writers, Arbitrary File Modifications:A single writer may write to a file in HDFS. Writes
are always performed in an append-only manner at the end of the file. Both multiple writers and
edits at arbitrary file offsets are not supported. (These may receive support in the future, but
they are probably not going to be very effective.)
HDFS Concepts
The term "block size" refers to the minimum amount of data that can be accessed or stored on a
disc. In addition, it should be noted that file systems designed for a singular disc handle data in
units known as blocks, which are typically sized as multiples of the disk's block size. Disc blocks
are commonly characterised by a size of 512 bytes, but filesystem blocks typically have a size in
the range of a few kilobytes.
When engaging in the process of reading or writing a file, regardless of its duration, the user of
the filesystem is generally uninformed about this activity. Nevertheless, there are tools available
for the purpose of managing filesystems at the block level, such as df and fsck.
The presence of a block is also observed in HDFS, but with a default size of 128 MB. Similar to
a file system designed for a single disc, the Hadoop Distributed File System (HDFS) partitions
files into individual blocks of a predetermined size, which are then stored independently. In
comparison to a single-disk filesystem, a file in HDFS that is less than a single block does not
occupy a whole block's worth of underlying storage. For example, when a file with a size of 1
MB is stored using a block size of 128 MB, only 1 MB of disc space is utilised, rather than 128
MB. In the context of this topic, the term "block" denotes an HDFS block when employed without
any additional specification.
There are many advantages to a distributed filesystem having a block abstraction. The first
advantage is the most obvious: a file can exceed the capacity of any individual network disc.
The blocks from a file can use any of the discs in the cluster because there is no need that they
be stored on the same disc. In fact, storing a single file on an HDFS cluster with blocks that
filled every disc in the cluster would be doable, if exceptional.
Second, simplifying the storage subsystem by using blocks rather than files as the unit of
abstraction. All systems should strive for simplicity, but distributed systems with their wide range
of failure scenarios require it even more. Blocks are dealt with by the storage subsystem, which
simplifies storage management (since blocks have a fixed size, it is simple to figure out how
many can be stored on a given disc) and removes concerns about metadata (since blocks are
just chunks of data to be stored, file metadata such as permissions information does not need to
be stored with the blocks, so another system can handle metadata separately).
Blocks also work well with replication to provide availability and fault tolerance. Each block is
copied to a small number of physically distinct machines (usually three) in order to protect
against corrupted blocks, disc failure and machine failure. In the event that a block is no longer
available, a client-transparent copy may be read from another place. In order to restore the
replication factor to its regular level, a block that has become unavailable due to corruption or
machine failure might be copied from its alternate locations to other active computers. The
blocks in a well-liked file might also have a high replication factor set by some applications to
distribute the read load across the cluster.
The fsck command in HDFS recognises blocks, just like its disc filesystem counterpart. For
instance, jogging:
% hdfsfsck / -files –blocks
will list each file's individual blocks in the filesystem.
Namenodes and Datanodes
An HDFS cluster consists of two types of nodes: a namenode, which serves as the master anda
number ofdatanodes, which function as the workers. The control of the filesystem namespace is
vested in the namenode. The system maintains a record of the hierarchical structure of the
filesystem and the associated metadata for all files and folders. The namespace image and the
edit log are two files that save data and are permanently stored on the local disc. The
namenode possesses knowledge of the datanodes that store all the blocks associated with a
particular file. However, it does not continuously store the positions of these blocks, as their
determination is carried out by the datanodes during system initialisation.
Through the utilisation of the name node and data nodes, a client is able to gain access to the
filesystem by means of interaction. The user code does not require knowledge of the namenode
and datanodes for its operation, as the client provides a filesystem interface that is similar to the
Portable Operating System Interface (POSIX).
The primary components responsible for the functioning of the filesystem are the datanodes.
When directed by clients or the namenode, the storage nodes perform the tasks of storing and
retrieving blocks. Additionally, they periodically communicate with the namenode by providing
lists of the blocks they have stored.
The presence of the namenode is essential for the utilisation of the filesystem. Indeed, in the
event of the destruction of the machine housing the namenode, it would result in the loss of all
files stored on the filesystem. This is due to the absence of a mechanism to reconstruct the files
from the blocks stored on the datanodes. Ensuring the resilience of the namenode is of utmost
importance due to this factor and Hadoop provides two methodologies to accomplish this.
One approach involves creating a duplicate copy of the files that constitute the enduring state of
the filesystem metadata. Hadoop has the capability to save the permanent state of the
namenode in many filesystems. The aforementioned texts possess atomicity and synchronicity.
Write operations frequently entail the utilisation of both a local disc and a remote NFS mount.
The secondary namenode, despite its nomenclature, does not fulfil the role of a namenode and
can potentially serve other purposes. The major purpose of this function is to periodically merge
the edit log and namespace image, hence preventing the edit log from experiencing excessive
growth. Due to the requirement of comparable CPU power and memory resources for the
completion of the merge process, it is common practice for the secondary namenode to operate
on a distinct physical computer from the namenode.
In the event of a failure of the namenode, a backup of the merged name space image is
retained. The occurrence of data loss is quite probable when facing a complete primary failure
due to the secondary namenode's delay in maintaining an up-to-date state compared to the
primary namenode. In this particular scenario, the conventional course of action is executing the
secondary as the fresh primary and duplicating the namenode's metadata files, which are based
on NFS, onto it.
Block Caching
A datanode typically reads blocks from the disc, however in cases when files are often visited,
the blocks could be intentionally stored in the datanode's memory, in an off-heap block cache. A
block is typically only cached in one datanode's memory, though this can be changed on a perfile basis. By scheduling jobs to run on the datanode where a block is cached, job schedulers
(for MapReduce, Spark and other frameworks) can benefit from cached blocks and improve
read speed. For instance, a small lookup table that is utilised in a join is an excellent candidate
for caching.
By adding a cache directive to a cache pool, users or program s can advise the namenode
which files to cache (and for how long). An administrative grouping for controlling resource
utilisation and cache permissions is called a cache pool.
HDFS Federation
Because the namenode maintains a reference to each file and block in the filesystem in
memory, scaling becomes constrained on very large clusters with lots of files. By adding
namenodes, which each control a portion of the filesystem namespace, a cluster can scale
thanks to HDFS federation, which was first made available in the 2.x release series. One
namenode, for instance, might be responsible for all of the files rooted under /user and another
name node, for files under /share.
Each namenode in federation is responsible for managing a namespace volume, which contains
the namespace's metadata and a block pool, which has all of the blocks for the namespace's
files. Because namespace volumes are independent of one another, namenodes cannot
communicate with one another and the availability of the namespaces controlled by other
namenodes is not impacted by the failure of one namenode. However, because block pool
storage is not partitioned, datanodes can store blocks from different block pools by registering
with each namenode in the cluster.
Clients employ client-side mount tables to map file paths to namenodesin order to access a
federated HDFS cluster. The viewfs:// URIs and ViewFileSystem are used to control this in
configuration.
HDFS High Availability
The secondary namenode is used to construct checkpoints and replicate namenode metadata
across several filesystems, providing data loss prevention but not high availability of the
filesystem. Still a single point of failure (SPOF), the namenode. All clients, including MapReduce
jobs, would be unable to read, write, or list files if it did fail since the namenode is the only
location where the file-to-block mapping and metadata are stored. A fresh namenode would
need to be brought online before the entire Hadoop system could resume operation.
In this case, the administrator establishes a new primary namenode with one of the filesystem
metadata replicas, configures datanodes and clients to use this new namenode and recovers
from the failed namenode. The new namenode cannot respond to queries until it has completed
the following steps: loading its namespace image into memory; replaying its edit log; and
receiving enough block reports from the datanodes to exit safe mode. The time it takes for a
namenode to start from scratch on large clusters with plenty of files and blocks can be 30
minutes or longer.
Regular maintenance is also a concern because of the lengthy recovery time. In fact, the case
for planned downtime is arguably more significant in practice because unexpected failure of the
namenode is so uncommon.
By including support for HDFS high availability (HA), Hadoop 2 fixed this issue. A pair of
namenodes are set up as an active-standby pair in this implementation. The standby namenode
assumes control in the event that the main namenode fails, allowing client requests to be
fulfilled without much disruption. In order for this to be possible, a few architectural adjustments
are required:
·
The edit log must be shared among the namenodes using highly accessible shared
storage. In order to synchronise its state with the current namenode, a standby namenode first
reads the shared edit log all the way to the end before continuing to read fresh entries as they
are added by the active namenode.
·
Because the block mappings are maintained in a namenode's memory rather than on disc,
datanodes must deliver block reports to both namenodes.
·
Clients must be set up to manage namenode failover using a user-transparent technique.
·
The standby, which routinely checkpoints the namespace of the current namenode,
assumes the function of the secondary namenode.
The highly available shared storage has two options: an NFS filer or a quorum journal manager
(QJM). The QJM is the suggested option for the majority of HDFS deployments because it is a
specialised HDFS implementation created with the express objective of delivering a highly
accessible edit log. Each edit in the QJM must be written to the majority of the journal nodes
because it operates as a group of journal nodes.
The system can survive the loss of one journal node as there are often three. It's important to
note that the QJM implementation does not employ ZooKeeper, despite the fact that this setup
is comparable to how ZooKeeper operates.
The standby namenode has the most recent state accessible in memory, including the most
recent edit log entries and an updated block mapping, so if the active namenode fails, it can
take over extremely rapidly (in a few tens of seconds). Because the algorithm needs to be
cautious when determining that the active namenode has failed, the actual observed failover
time will be longer in practice (around a minute).
The administrator can still start the standby from scratch if, in the odd event that the standby is
unavailable when the active fails. Given that the process is a normal operational routine built
into Hadoop, it is neither better nor worse than the non-HA situation from the perspective of
operations.
Failover and fencing
A new component in the system known as the failover controller is in charge of managing the
switch from the current namenode to the standby. There are different failover controllers,
however ZooKeeper is used by default to guarantee that only one namenode is operational. A
lightweight failover controller process is run by each namenode and is responsible for
monitoring each namenode for failures (using a basic heartbeat mechanism) and initiating a
failover should a namenode fail.
In certain circumstances, an administrator has the ability to initiate a failover manually, such as
during a planned maintenance event. The failover controller facilitates a seamless transition for
both namenodes to switch roles, which is commonly known as a graceful failover.
Nevertheless, in the case of an ungraceful failover, it is challenging to ascertain with certainty if
the malfunctioning namenode has ceased its operations. In certain scenarios, the continued
operation of the previously active namenode, despite its belief that it is still the active
namenode, can be disrupted by a slow network or a network partition, resulting in a failover
transition. Fencing is a strategic approach employed by the High Availability (HA)
implementation in order to effectively neutralise the potential negative impact and corruption that
may be caused by the previously active namenode.
Implementing an SSH fencing command to terminate the namenode's process is a prudent
course of action. This is due to the constraint imposed by the QJM, which allows only one
namenode to write to the edit log at any given time. However, it is important to note that despite
this limitation, there is a possibility for the previously active namenode to transmit outdated read
requests to clients. Due to the limitation of having only one active namenode for writing
purposes, it becomes necessary to employ more robust fencing strategies when utilising an
NFS filer for the shared edit log. This is precisely why the use of Quorum Journal Manager
(QJM) is recommended in such scenarios.
The repertoire of fencing strategies include the utilisation of a remote management command to
obstruct the network port of the name node, as well as the revocation of access to the shared
storage directory, typically accomplished by the application of a vendor-specific NFS command.
STONITH, an acronym for "Shoot The Other Node In The Head," refers to a final recourse in
which a dedicated power distribution unit is employed to forcibly terminate the operation of the
host computer, therefore isolating the previously active namenode.
The client library manages client failover in a transparent manner. Client-side setting is used as
the simplest implementation to manage failover. The client library tries each namenode address
until the operation is successful using the HDFS URI, which employs a logical hostname that is
mapped to two namenode addresses (in the configuration file).
File Permissions in HDFS
Similar to the POSIX paradigm, HDFS's file and directory permissions model. The letters "r,"
"w," and "x" stand for the three different sorts of permissions: read, write and execute. To read
files or list a directory's contents, the read permission is necessary. To create or delete files or
directories in a directory, the directory must have the write permission. A directory needs the
execute permission in order to access its children, but a file doesn't need it because HDFS
doesn't allow file execution (unlike POSIX).
Each file and directory has an owner, a group and a mode. The mode encompasses the
permissions allocated to the user who owns the file, the permissions granted to members of the
group and the permissions assigned to users who do not fall into either category of owner or
group member.
By default, Hadoop works without security enabled, resulting in the absence of client identity
verification. Users have the ability to assume the identity of any client by establishing an
account under that specific name on the remote system, as clients are situated at a distance. To
mitigate the risk of inadvertent modification or removal of critical sections of the filesystem, it is
advisable to enable permissions, which are often enabled by default (as indicated by the
dfs.permissions.enabled attribute).
When permissions checking is turned on, the owner permissions are only examined if the
client's username corresponds to the owner and the group permissions are only examined if the
client is a group member; otherwise, all other permissions are examined.
Hadoop Filesystems
HDFS is merely one implementation of Hadoop's abstract concept of filesystems. the
org.apache.hadoop.fs abstract class for Java.There are numerous actual implementations of
FileSystem, which in Hadoop provides the client interface to a filesystem. The table below lists
the primary ones that come with Hadoop.
Table: Hadoop filesystems
Hadoop offers a variety of filesystem interfaces and it often selects the appropriate filesystem
instance to talk with using the URI scheme. For instance, all Hadoop filesystems can be used
using the filesystem shell that we first encountered in the preceding section. Type: to display a
list of the files in the local filesystem's root directory.
% hadoop fs -ls file:///
Even while it is possible (and occasionally very convenient) to run MapReduce program s that
access any of these filesystems, you should pick a distributed filesystem with data locality
optimisation when processing huge amounts of data, particularly HDFS.
Interfaces: Since Hadoop is written in Java, the Java API is used to mediate most filesystem
interactions. For instance, the filesystem shell is a Java application that performs filesystem
operations using the Java FileSystem class. In this section, a quick discussion of the alternative
filesystem interfaces is provided. Since the other Hadoop filesystems often contain tools to
access the underlying filesystem (FTP clients for FTP, S3 tools for S3, etc.), these interfaces
are most frequently used with HDFS, although many of them can be used with any Hadoop
filesystem.
HTTP:Hadoop makes accessing HDFS difficult for non-Java program s by making its filesystem
interface available via a Java API. The WebHDFS protocol makes it simpler for other languages
to communicate with HDFS by exposing an HTTP REST API. Keep in mind that the native Java
client is faster than the HTTP interface, so if at all feasible, avoid using it for very big data
transfers.
There are two ways to access HDFS over HTTP: directly, in which case the HDFS daemons
directly serve HTTP requests to clients; and indirectly, in which case a proxy (or proxies)
accesses HDFS on the client's behalf using the standard DistributedFileSystem API. The two
approaches are shown in the following figure. The WebHDFS protocol is used by both.
In the initial configuration, the namenode and datanodes' integrated web servers act as
endpoints for WebHDFS. WebHDFS is enabled by default as the configuration parameter
dfs.webhdfs.enabled is set to true. The process of reading and writing files involves the first
transmission of operations to the namenode. Subsequently, the namenode sends an HTTP
redirect to the client, specifying the datanode responsible for streaming the file data either from
or to the client. The management of file metadata activities is undertaken by the namenode.
One or more independent proxy servers are required for the second method of accessing HDFS
over HTTP. (Since the proxies are stateless, they can be used in front of a regular load
balancer.) The client never directly reaches the namenode or datanode since all traffic to the
cluster is routed through the proxy. This enables the implementation of tougher firewall and
bandwidth-limitation regulations. When transferring data across Hadoop clusters housed in
various data centres or connecting to a Hadoop cluster that is hosted in the cloud through an
external network, proxies are frequently used.
Clients can access WebHDFS and the HttpFS proxy using webhdfs (or swebhdfs) URIs
because both provide the same HTTP (and HTTPS) interface. Using the httpfs.sh script, the
HttpFS proxy is started separately from the namenode and datanode daemons and by default
listens on a different port (1400).
Hive, HBase and Spark
Apache Hive
The capability to perform extensive analyses on a large scale is facilitated by the
implementation of Apache Hive, a distributed and fault-tolerant data warehousing solution. A
data warehouse provides a centralised storage facility for data that can be readily evaluated to
facilitate decision-making based on data. Hive users have the capability to utilise SQL for the
purposes of reading, writing and effectively managing vast amounts of data, reaching the scale
of petabytes.
Hive is built upon Apache Hadoop, an open-source platform renowned for its ability to efficiently
store and analyse vast amounts of data. The Hive framework is closely integrated with Hadoop
and designed to efficiently process large volumes of data, often reaching the scale of petabytes.
Hive is notable due to its provision of a SQL-like interface that enables users to query vast
databases via Apache Tez or MapReduce.
Apache Hive is a data warehouse and ETL program that offers users a SQL-like interface to
interact with the Hadoop distributed file system (HDFS). This integration allows for seamless
communication between the user and Hadoop. The system is built on top of the Hadoop
framework. The software program provides a platform for doing data analysis and inquiry. The
utilisation of SQL syntax facilitates the reading, writing and management of extensive datasets
that are stored in distributed storage systems. The system was not designed with the intention
of accommodating the requirements for online transaction processing (OLTP). Data
warehousing activities, such as data encapsulation, ad-hoc queries and the processing of large
datasets, are frequently utilised for this purpose. The purpose of using many input formats is to
enhance performance, fault tolerance, scalability, extensibility and loose coupling.
The development of Hive was first undertaken by Facebook, Amazon, Netflix and Netflix. This
platform provides standard SQL functionalities for analytical purposes. In order to do SQL
operations and run SQL queries on distributed data, conventional SQL queries are formulated
using the MapReduce Java API. Hive provides enhanced mobility due to its utilisation of SQLbased query languages, such as NoSQL, which are commonly employed by a significant portion
of data warehousing software.
The Apache Hive project, which is centred around the Hadoop ecosystem, pertains to a
software project for data warehousing. The system provides a query and analysis interface that
bears resemblance to SQL, specifically designed for handling large datasets stored in the HDFS
distributed file system for Hadoop or other similar storage systems.
The process of defining data searches, transformations and analytics in Hive can be
accomplished by utilising a language known as HiveQL, which bears similarities to SQL. In
order to handle the data, MapReduce tasks are generated that incorporate HiveQL statements,
which are subsequently performed on the Hadoop cluster.
Hive possesses several features, such as support for splitting, indexing and user-defined
functions (UDFs), that contribute to its efficacy as a tool for big data analysis. Furthermore, the
system provides a range of optimisation techniques, such as predicate pushdown, column
reduction and query parallelisation, in order to improve the performance of queries.
Hive is a versatile tool that may be employed for a range of data processing tasks, including but
not limited to data warehousing, ETL (extract, transform, load) pipelines and ad-hoc data
analysis. The utilisation of this technology is prevalent within the big data industry, namely
among enterprises that have adopted the Hadoop environment as their primary framework for
data processing.
Components of Hive:
HCatalog: It is a Hive component that serves as both a table and a layer for managing stores in
Hadoop. It enables users to simply read from and write to the grid when combined with various
data processing tools like Pig and MapReduce.
WebHCat: It offers a service that users can use to execute jobs via an HTTP interface for
Hadoop MapReduce (or YARN), Pig, Hive, or function Hive metadata operations.
Modes of Hive:
Local Mode: It is utilised when Hadoop is created in pseudo mode, when there is just one data
node, when the data size is smaller relative to the local machine as a whole and when
processing smaller datasets on the local system will be faster.
Map Reduce Mode: While Hadoop is constructed with several data nodes and data is
distributed across different nodes, it will work on enormous datasets and run queries in parallel
and it is utilised to gain improved performance while processing massive datasets.
Characteristics of Hive:
·
Data is loaded before databases and tables are created.
·
Only structured data that is stored under tables can be managed and accessed using the
data warehouse called Hive.
·
When dealing with structured data, MapReduce lacks optimisation and usability features
like UDFs while the Hive framework has both.
·
Hadoop programming directly interacts with the files. Hive can therefore use directory
structures to split the data and enhance query performance.
·
Hive supports a number of file formats, including TEXTFILE, SEQUENCEFILE, ORC,
RCFILE and others.
·
Hive employs the Derby database for storing metadata for a single user and MYSQL for
storing shared or multiple user metadata.
Features of Hive:
·
In order to speed up queries, it offers indexes, including bitmap indexes. Index type with
bitmap index and compaction as of 0.10.
·
The RDBMS's metadata storage speeds up semantic tests during query execution.
·
User-defined functions (UDFs) that are pre-built for manipulating dates, texts and other
data-mining tools. Hive is strengthened to expand the UDF set to handle the use-cases for
which preset functions are not sufficient.
·
The Hadoop Ecosystem uses algorithms like DEFLATE, BWT and snappy to operate on
compressed data.
·
It processes the data into the Hadoop File Distributed File System (HDFS) and stores the
schemas in a database.
·
It was designed with Online Analytical Processing (OLAP) in mind.
·
Hive Query Language (HVL or HiveQL), a general term for a variety of querying
languages, is delivered.
Advantages:
·
Scalability:Apache Hive is a scalable option for big data processing because it is made to
handle massive volumes of data.
·
Familiar SQL-like interface:Hive employs a language called HiveQL, which is similar to
SQL and is simple to understand and use for SQL users.
·
Integration with Hadoop ecosystem:Hive works well with the rest of the Hadoop
ecosystem, allowing users to leverage tools like Pig, MapReduce and Spark to process data.
·
Supports partitioning and bucketing:Hive enables bucketing and partitioning, which reduce
the amount of data read during queries and hence enhance query performance.
·
User-defined functions:Users of Hive can create custom functions that can be incorporated
into HiveQL searches.
Disadvantages:
·
Limited real-time processing: Since Hive is intended for batch processing, it might not be
the ideal solution for handling real-time data.
·
Slow performance: Because Hive is based on Hadoop, which is designed for batch
processing rather than interactive querying, it may run slower than typical relational databases.
·
Difficult to use for beginners: Although Hive utilises a language similar to SQL, it still
requires users to be familiar with Hadoop and distributed computing.
·
Lack of support for transactions: Transactions are not supported by Hive, which can make
it challenging to maintain data consistency.
·
Limited flexibility: Because Hive was created expressly to work with Hadoop, its
applicability in different systems may be constrained. This makes Hive less adaptable than
alternative data warehousing technologies.
HBase
Built on top of the Hadoop Distributed File System (HDFS), Apache HBase is a distributed,
scalable and consistent NoSQL database that is open-source. It offers random read and write
access to huge data and is made to manage massive amounts of structured data. HBase is
based on Google's Bigtable and is a component of the Apache Hadoop ecosystem.
Key Concepts and Architecture:
Column-Family Data Model: Column families, which are logical groups of columns, are how
HBase organises data. Each column family may include any number of columns and each row
may have several column families. This adaptable data model enables changes to the schema
design without affecting the data already in existence.
Distributed and Scalable:HBase is made to be distributed among several Hadoop cluster nodes.
For high availability and fault tolerance, it duplicates and splits data automatically among nodes.
It is suitable for managing enormous volumes of data because to its scalability.
Consistency and High Availability: For read and write operations, HBase offers excellent
consistency, maintaining data integrity throughout the distributed cluster. Additionally, it provides
data replication for disaster recovery and high availability.
Region Server and Region: Data is divided into regions by HBase and a region server oversees
each region. A region in a table represents a number of rows. A region divides into two new
regions when its data volume exceeds a configurable threshold, dispersing the data among
several area servers.
HBase Master and ZooKeeper: The management of metadata and the organisation of region
assignment are tasks that fall under the purview of the HBase master. HBase node coordination
and master election assistance are provided by ZooKeeper.
HBase Coprocessors: HBase enables the running of coprocessors—custom software—directly
on the region server. This makes it possible for program rs to increase the capability of HBase
and carry out intricate calculations close to the data, which enhances performance.
Basic Operations and Data Model:
Put Operation: A put operation is used to add data to HBase. Name of the table, row key,
column family, column qualifier and cell value are all specified. HBase duplicates the data based
on the replication factor and writes it to the appropriate region.
Get Operation:A get operation is used to obtain data from HBase. The table name, row key,
column family and column qualifier are all specified. HBase retrieves the information by reading
the data from the designated area.
Scan Operation: The HBase data can be retrieved in numerous rows using the scan method.
Based on predetermined criteria, it can be used to scan the entire table or just a subset of the
rows.
Delete Operation: To delete data from HBase, use the delete operation. It is capable of erasing
particular cells, entire rows, or even entire column families.
Use Cases of Apache HBase:
Real-time Data Processing: For processing real-time data, such as offering real-time web apps,
storing sensor data and developing Internet of Things (IoT) applications, HBase is frequently
utilised. It is suited for these use cases due to its capacity to manage massive data volumes
with low latency.
Social Media Analytics: Businesses can analyse and react to social media interactions in realtime thanks to HBase's suitability for storing and analysing social media data. This is essential
for customer involvement and social media marketing strategy.
Time Series Data Storage: HBase is ideally suited for the storage of time series data, which
includes log files, sensor readings and data from the stock market. It is a popular option for
these applications due to its effective storing and retrieval capabilities.
Machine Learning: Machine learning applications that need quick access to huge datasets
during training and inference can use HBase as a storage backend. For machine learning jobs
that require a lot of data, its capacity to process enormous amounts of data in real-time is
useful.
Advantages of Apache HBase:
·
Scalability and Distributed Architecture: HBase can manage petabytes of data over a
cluster of machines because to its distributed architecture. To guarantee high availability and
fault tolerance, it automatically distributes and replicates data.
·
High Performance: HBase is suitable for real-time data processing and analytics since it
offers low-latency read and write operations. Due to its effective data storage and retrieval
algorithms, it performs effectively even when dealing with big datasets.
·
Strong consistency for read and write operations is guaranteed by HBase, assuring data
integrity throughout the distributed system.
·
Being an open-source project, HBase benefits from a thriving community of developers
and contributors who work tirelessly to make the program better and offer support.
Challenges of Apache HBase:
·
Administration Complexity: Managing and setting up a distributed HBase cluster can be
difficult, calling for knowledgeable administrators to guarantee optimum performance and
resource allocation.
·
Limitations of the Data Model: Not all types of data can be handled by the column-family
data model since some use cases may call for intricate joins and relations, which are better
supported by conventional relational databases.
·
Limited Query Capabilities: Although HBase offers quick access to individual rows,
complicated ad-hoc queries involving numerous tables and joins might not run as smoothly.
Spark
Apache Spark is a distributed computing platform that is open-source and made for processing
and analysing enormous amounts of data. It is a component of the Apache Software Foundation
and is based on the idea of in-memory data processing, making data processing activities quick
and effective. Because Spark offers high-level APIs in numerous programming languages, it is
usable by developers of all levels of experience.
Key Features of Apache Spark:
·
Processing Data in Memory: Spark processes data in memory instead of reading and
writing it to disc, which greatly accelerates computing.
·
Distributed computing: Spark scales data processing and analytics by distributing data
over several cluster nodes and processing it in parallel.
·
Data Abstraction: Spark offers high-level abstractions, such as DataFrames, Datasets and
RDDs (Resilient Distributed Datasets), for working with both organised and unstructured data.
·
Spark is available to developers with a variety of language preferences thanks to its
support for multiple languages, including Scala, Java, Python and R.
·
Processing in Batches and Real-Time: Spark offers real-time processing for processing
data streams using Spark Streaming and Structured Streaming, as well as batch processing,
where data is processed in fixed-size chunks.
·
Machine Learning and Graph Processing: The machine learning (MLlib) and graph
processing (GraphX) libraries included in Spark enable users to carry out complex analytics jobs
on the same platform.
·
Reduced Requirement for External Dependencies: Spark includes built-in libraries for SQL,
machine learning, graph processing and data streaming.
Spark Components:
·
Task scheduling, memory management and fault recovery are the responsibilities of Spark
Core, which is the main Spark component. It offers the fundamental features needed for
distributed data processing.
·
Using DataFrames and Datasets, Spark SQL enables querying and manipulation of
structured data in a manner similar to SQL. It incorporates Hive and more data sources.
·
Spark Streaming: Spark Streaming divides data streams into micro-batches that may be
processed using Spark's batch processing capabilities, enabling real-time data processing.
·
Machine learning methods and tools are provided by MLlib (Machine Learning Library) for
creating and assessing machine learning models.
·
Spark's graph processing package, GraphX, enables users to run graph calculations and
analyse massive graphs.
·
Structured Streaming: A Spark Streaming extension, structured streaming enables users to
process structured data in real-time.
Use Cases of Apache Spark:
·
Spark is a popular tool for big data processing activities like ETL (Extract, Transform,
Load) operations, data preparation and data cleaning.
·
Data Analysis and Exploration: Spark is well suited for data analysis, exploratory data
analysis (EDA) and statistical calculations because to its strong data processing capabilities.
·
Machine learning: The MLlib package in Spark enables users to create and train machine
learning models on huge datasets, making it suited for machine learning applications.
·
Real-Time Analytics: Users can do real-time analytics on data streams using Spark
Streaming and Structured Streaming, enabling in-the-moment decision-making and monitoring.
·
Graph analytics: GraphX is useful for social network analysis and recommendation
systems since it enables users to carry out intricate graph computations and analyse massive
graphs.
Advantages of Apache Spark:
·
Speed and Performance: Spark's distributed computing architecture and in-memory
processing offer high-speed data processing, drastically cutting down processing time.
·
Spark is simple to work with and learn thanks to its high-level APIs and interaction with a
variety of languages.
·
Unified Data Processing Platform: Spark functions as a unified platform for multiple data
processing activities due to its integration of batch and real-time processing, SQL querying,
machine learning and graph processing.
·
Scalability: Spark's distributed computing paradigm enables it to scale with data and
computational resources in an effective manner.
·
Problems with Apache Spark
·
Memory management: To avoid out-of-memory issues and guarantee optimal
performance, in-memory processing calls for precise memory management.
·
Beginners may encounter a learning curve due to Spark's distributed design and multiple
components, despite the fact that it is user-friendly.
·
Cost of Cluster Setup: For small-scale applications in particular, setting up and maintaining
a Spark cluster can be expensive.
Working with HDFS
HDFS functions on a cluster of commodity hardware, wherein it consists of a single primary
NameNode and several supplementary datanodes. The nodes within the data centre are
spatially co-located. Subsequently, the data is partitioned into discrete units for the purpose of
storage and subsequently disseminated among multiple DataNodes. The replication of blocks
among nodes is a common practice aimed at reducing the probability of data loss. The function
of this system is to act as a contingency measure in the event of data loss.
Now, let us proceed to analyseNameNodes. The NameNode is the cluster node that possesses
knowledge about the data's contents, the block it is associated with, the size of the block and its
proper position. NameNodes are utilised for the purpose of overseeing file access,
encompassing permissions for writing, reading, appending, deleting and replicating data across
distinct data nodes.
In instances of increased data influx, the cluster can be dynamically reconfigured in accordance
with the server's capacity in real-time. Nodes have the capability to be added or withdrawn as
required.
Now, let us go to the topic of DataNodes. To ascertain the appropriate initiation and completion
time of an activity, DataNodes engage in continuous contact with NameNodes. Due to the
persistent flow of information, the NameNode possesses comprehensive knowledge regarding
the state of each DataNode.
The namemode possesses the capability to autonomously reassign a job to an alternative
operational node inside the identical datablockin the event that a DataNode is detected to be
underperforming. Just as DataNodes can collaborate in traditional file operations, they are also
capable of intercommunication. The NameNode plays a crucial role in maintaining the system
since it possesses knowledge about the DataNodes and their operational status.
The NameNode retrieves data blocks that are duplicated across many datanodes.
In order toutilise the Hadoop Distributed File System (HDFS), it is important to undertake the
installation and configuration of a Hadoop cluster. The available setup options include a single
node configuration, which is recommended for novice users and a cluster configuration
designed for larger, geographically distributed clusters. In order to effectively administrate and
manage your system, it is important to acquire proficiency in the usage of HDFS commands,
which are exemplified in the following list.
How does HDFS work?
The utilisation of NameNodes and DataNodes is a fundamental aspect of the Hadoop
Distributed File System (HDFS), as previously mentioned. The High-Performance Distributed
File System (HDFS) facilitates efficient data movement among compute nodes. Upon receiving
data, the Hadoop Distributed File System (HDFS) has the capability to partition the information
into smaller units known as blocks, which are subsequently distributed over several nodes
within the cluster.
The data is partitioned into blocks and distributed among the DataNodes for storage.
Replication of these blocks between nodes is employed to enable efficient parallel processing.
Various approaches can be employed to access, manipulate and observe data. The HDFS DFS
options "-get" and "-put" provide the retrieval and relocation of data as required.
Furthermore, the Hadoop Distributed File System (HDFS) is designed with a high level of
vigilance and efficiency in error detection. Data replication is a technique employed by the file
system to ensure redundancy and fault tolerance. It involves the creation of several copies of
each piece of data, which are then distributed across different nodes. This distribution is
designed in such a way that at least one duplicate of the data is stored on a separate rack from
the other copies.
This implies that in the event of a DataNode ceasing to transmit signals to the NameNode, the
cluster is able to sustain its operations in the absence of the DataNode. Subsequently, in the
event that the data node is retrieved, it can be allocated to a distinct cluster. Furthermore, due to
the replication of datablocks across several DataNodes, the act of deleting one will not lead to
any kind of file corruption.
HDFS components
It's crucial to understand that Hadoop is made up of three key parts. Hadoop YARN, Hadoop
HDFS and Hadoop MapReduce. Let's examine what these elements add to Hadoop:
·
Hadoop HDFS - The Hadoop storage component is called Hadoop Distributed File System
(HDFS).
·
Hadoop MapReduce - The Hadoop processing core is called MapReduce. It is possible to
create apps that process enormous volumes of data using this software platform.
·
Hadoop YARN - A part of Hadoop used for resource management is called Hadoop YARN.
It executes batch, stream, interactive and graph processing operations on data that is stored in
HDFS.
How to create an HDFS file system
Looking to construct an HDFS file system? The methods listed below will show you how to
create, amend and, if necessary, remove the system.
Listing your HDFS
You should be listed in HDFS as /user/yourUserName. Enter the following to view the contents
of your HDFS home directory:
hdfsdfs -ls
You won't be able to see much at this point because you're just getting started. Enter the
following to display a non-empty directory's contents:
hdfsdfs -ls /user
The names of all the other Hadoop users' home directories will then be visible to you.
Creating a directory in HDFS
Now that the test directory has been created, let's call it testHDFS. It'll show up in your HDFS.
Just type the following:
hdfsdfs -mkdirtestHDFS
Use the command you entered to list your HDFS to confirm that the directory is still present. The
testHDFS directory ought to appear on the list.
Use the HDFS full pathname to your HDFS to recheck the information. Enter:
hdfsdfs -ls /user/yourUserName
Before moving on to the following step, make sure this is functioning once more.
Copy a file
Create the file you want to copy first, then copy it from your local file system to HDFS. To begin,
type:
echo "HDFS test file" >>testFile
This will produce a brand-new file with the name testFile and the characters HDFS test file. To
confirm this, type:
ls
Enter the following to confirm that the file was indeed created:
cat testFile
The file must then be copied to HDFS. In order to transfer files from Linux to HDFS, you must
employ:
hdfsdfs -copyFromLocaltestFile
You should be aware that the command "-cp" is used to copy files within HDFS, so you must
use the option "-copyFromLocal".
All that's left to do is make sure the file was transferred properly. Enter the following to do this:
code>hdfsdfs -ls
hdfsdfs -cat testFile
Moving and copying files
The testfile was copied and placed in the base home directory. It can now be transferred into
the testHDFS directory you have established. Use these things:
hdfsdfs -mv testFiletestHDFS
hdfsdfs -ls
hdfsdfs -ls testHDFS/
First, your testFile was transferred from your HDFS home directory to the test directory you set
up. The third part of this command indicates that it has now been transferred to the test HDFS
directory after the second part of this command reveals that it is no longer in the HDFS home
directory.
Enter: to copy a file.
hdfsdfs -cp testHDFS/testFiletestHDFS/testFile2
hdfsdfs -ls testHDFS/
Checking disk usage
When utilising HDFS, it's helpful to check the disc space. You can achieve this by entering the
command below:
hdfsdfs -du
You will then be able to check how much HDFS space you are consuming as a result. By
entering: You can also see how much space is available in the cluster's HDFS by entering:
hdfsdfs -df
Removing a file/directory
You might need to remove a file or directory from the HDFS at some point. The command can
be used to accomplish this:
hdfsdfs -rm testHDFS/testFile
hdfsdfs -ls testHDFS/
You'll notice that the testHDFS directory and testFile2 you generated are still present. You can
delete the directory by typing:
hdfsdfs -rmdirtesthdfs
Then an error warning will appear, but don't be alarmed. Something along the lines of "rmdir:
testhdfs: Directory is not empty" will appear. Before it may be erased, the directory must be
empty. To get around this and delete a directory together with all of the files it holds, use the
"rm" command. Enter:
hdfsdfs -rm -r testHDFS
hdfsdfs -ls
How to install HDFS
You must keep in mind the difference between a single node and a multinode when installing
Hadoop. A single-node or multi-node cluster can be used, depending on your needs.
One DataNode is active in a single node cluster. On a single machine, it will house the
NameNode, DataNode, resource manager and node manager.
This is sufficient in some businesses. A single-node cluster, for instance, can be used in the
medical sector when conducting research that require data to be gathered, sorted and
processed sequentially. In contrast to data spread across several hundreds of machines, this
can readily handle data on a smaller scale. Install a single-node cluster by doing the following
steps:
·
The Java 8 Package can be downloaded. Save this file in the directory of your home.
·
Get the Java Tar file out.
·
A Hadoop 2.7.3 package can be downloaded.
·
Take the Hadoop tar file out.
·
In the bash file (.bashrc), add the Hadoop and Java paths.
·
The Hadoop Configuration files should be edited.
·
Edit the property in core-site.xml by opening it.
·
Edit the property and hdfs-site.xml.
·
Edit the attribute and the mapred-site.xml file.
·
Edit the property and yarn-site.xml.
·
Add the Java path to hadoop-env.sh after editing.
·
Format the NameNode by going to the Hadoop home directory.
·
Open the hadoop-2.7.3/sbin directory and launch each daemon individually.
·
Verify that each Hadoop service is active.
How to access HDFS files
Given that we are dealing with data, it should come as no surprise that security is strict when it
comes to HDFS. You can only view the information on your file system because HDFS is
technically virtual storage that spans the cluster; you cannot access the real particular data.
You can download the "jar" file from HDFS to your local file system to view HDFS files. The web
user interface of the HDFS is also accessible. Enter "localhost:50070" into the search box of
your browser after it has opened. You may then navigate to the utilities tab on the right side of
the screen and view the web user interface for HDFS. Then select "browse file system," which
displays a comprehensive list of all the files stored on your HDFS.
HDFS DFS examples
Here are a few samples of the most typical Hadoop commands.
Example
A directory must be deleted using the instructions listed below (note that this can only be done if
all of the files are empty):
$Hadoop fs -rmdir /directory-name
Or
$hdfsdfs -rmdir /directory-name
Working with HBase
We utilise Apache HBase because we need sporadic, continuous read/compose access to big
data. With Apache HBase, it is possible to have extremely large tables over collections of item
equipment. It is an example of a non-social database following Google's Bigtable.
Fundamentally, HBase competes with Hadoop and HDFS in the same manner that Bigtable
misbehaves on Google File System.
Working with HBase
Assume that a table's records are stored on the pages of memory. On the odd occasion that
they are not formally displayed in the memory, these pages are transmitted to the primary
memory. If one line owns a page and we need a specific section from each line, such as
compensation or rate of interest, for some analysis, each page containing the segment must
acquire the memory; as a result, this page in and page out will result in a great deal of I/O,
which may cause a delay in handling time.
Each segment will be stored in pages in section-based databases. There will be less I/O if we
need a specific feature because only the pages that include the preset segment should have
been delivered to the main memory and read. We don't need to load all of the pages comprising
lines and records into the memory and read through them all.
Therefore, a detail-oriented database, which is useful for research where we can get a few
sections and complete some numerical activities, serves the best for inquiries when we need to
get certain segments and not entire record(s) or set(s).
Application
·
Apache HBase can be used for writing-intensive applications.
·
Additionally, we use HBase since we must offer quick random access to the data that is
already available.
·
A few businesses, like Facebook, Twitter, Yahoo and Adobe, among others, also use
HBase internally.
Advantages
·
Both information pressure and productivity have benefited from it.
·
This encourages prompt data recovery.
·
Design and organisation are separated. It might very well be scaled down and as a result,
expanding it is not difficult at all.
·
For elite on total questions (COUNT, Total, AVG, MIN and MAX), this is helpful.
·
This is useful for allocating since it provides highlights of a program d slicing tool to
express more important sections to small children.
Why Should We Use HBase?
·
It can handle incredibly enormous scales of information and has a completely dispersed
engineering.
·
It composes activities and functions for reading that is utterly random.
·
It offers great security and straightforward information management.
·
A remarkably high compose throughput is provided.
·
Scaling is swift and constant to fulfil additional prerequisites.
·
Both organised and partially organised information kinds can be used with it.
·
When you don't have to bother with the full RDBMS capabilities, it's excellent.
·
It has a perfectly straight and measured adaptability highlight.
·
The content is thoroughly researched and composed with accuracy.
·
It is possible to arrange and automate table sharding effectively.
·
Programmable failover support is provided for various servers.
·
HBase Tables can support MapReduce operations.
·
Customer access is in line with Java APIs.
Why Do We Need HBase?
A dynamic NoSQL database called HBase is being developed today as Big Data becomes
increasingly dominant. Its foundations in Java programming are incredibly simple, allowing for
significant growth. There are numerous business circumstances where we are just working with
insufficient information, which requires us to search among billions of information fields that
coordinate precise criteria. It is extremely weak and powerful, capable of handling many types of
information and useful in a variety of commercial settings.
The table's segment arrangement makes it easy to find the right information among the vast
amount of information fields. With the right configuration and automation, it is not difficult to
partition the data into tables. It is ideally suited for the methodical preparation of information.
Explanatory preparation has high knowledge requirements, which enables inquiries to be more
complex than necessary on a single server. This is when the distributed stockpiling enters the
picture.
HBase is the best option for such applications because there is a demand for handling
numerous reads and writes, which is simply impossible using an RDBMS database. The
read/write speed of this innovation can be increased to even millions per second, providing it a
remarkable competitive advantage. It is frequently used by Facebook for applications that
provide continuous information, while Pinterest uses it for various tasks that may execute up to
5 million tasks per second.
Why Use Apache Spark
In the big data and analytics space, Apache Spark is a potent and adaptable open-source data
processing platform that has received extensive popularity. It is favoured for many use cases
because to its many benefits over conventional data processing methods. The following is a
thorough justification for using Apache Spark:
Speed: When compared to conventional disk-based processing systems like Hadoop
MapReduce, Spark's in-memory processing features greatly accelerate data processing
workloads. It reduces the number of times that data must be read from disc by caching it in
memory, enabling faster iterative algorithms and interactive querying.
Spark offers high-level APIs in several different languages, including Scala, Java, Python and R.
Because of this, even engineers and data scientists who are not knowledgeable about
distributed systems can use it. The API architecture makes difficult jobs simpler and enables
shorter code.
Spark provides a unifying framework for a range of data processing activities, including batch
processing, interactive queries, machine learning, graph processing and stream processing. As
a result, there is no longer a need to coordinate and handle numerous tools for various
activities.
Processing in Memory: Spark has the capacity to hold data in memory, which makes it ideal for
interactive data exploration, real-time analytics and iterative machine learning algorithms. This
feature lowers the latency of data access and accelerates computations.
Spark developers are in high demand since Apache Spark is advantageous to both you and
your company. Due to the high demand for Spark developers, businesses may provide alluring
bonuses and flexible work hours merely to secure their services. The average pay for a Data
Engineer with Apache Spark abilities, according to PayScale, is $100,362. People who are
interested in a profession in big data can learn Apache Spark. You may fill the skills gap for
employment involving data in a number of ways, but the ideal option is to enrol in formal training
that will provide you practical work experience and allow you to learn through hands-on projects.
Advanced Analytics: The machine learning (MLlib) and graph processing (GraphX) libraries
provided by Spark let you carry out challenging analytics jobs including classification,
regression, clustering and recommendation systems.
Fault Tolerance: Spark offers fault tolerance similarly to Hadoop by partitioning data and
duplicating it throughout the cluster. Spark can restore the lost data using the replicated copies
if a node fails.
Scalability: Spark's architecture enables horizontal scaling by expanding the cluster's nodes. In
order to ensure effective resource utilisation, it automatically distributes data and calculations
across nodes.
Spark's interactive querying feature is especially helpful for on-the-fly data exploration and
analysis. Data analysts' jobs are made simpler by the ability to conduct SQL-like queries on
structured data using technologies like Spark SQL.
Spark can read from and write to a variety of data sources, including HDFS, HBase, Cassandra,
JSON, Parquet, Avro and more. This adaptability makes data processing and integration for
many formats simpler.
Real-time processing of streams of data made possible by Spark Streaming makes it
appropriate for monitoring, fraud detection and sentiment analysis applications.
Community and Ecosystem: Spark has a sizable and vibrant open-source community, which
means that a wide range of resources, frameworks and tools are readily available to expand its
functionality and address diverse data processing issues.
Hadoop compatibility: Spark can operate on Hadoop YARN clusters, making it simple to interact
with pre-existing Hadoop ecosystems and efficiently utilise resources.
Adoption in Industry: Spark has been embraced by numerous businesses and institutions,
including tech behemoths and start-ups, for their data processing requirements. This
widespread use demonstrates its dependability and efficiency in practical situations.
Apache Spark is a tempting option for data processing and analysis across a wide range of
disciplines and use cases because to its speed, simplicity, unified framework, in-memory
processing, advanced analytics capabilities, fault tolerance, scalability and interoperability with
multiple data sources.
Functional Programming Basics
All links in the functional programming paradigm are created only through mathematical
operations. Declarative programming is employed. It emphasises "what to solve" more so than
"how to solve," which is the main focus of an imperative approach. It uses phrases rather than
statements. While an expression is tested to provide a value, a statement is executed to assign
variables. These functions have a few distinct qualities, which are discussed below.
The foundation of functional programming is lambda calculus:
The lambda calculus was developed by Alonzo Church to analyse calculations involving
functions. It is referred to as the smallest programming language in existence. The definition of
what can be calculated is given. Everything is calculable if it can be done with lambda calculus.
It has processing power on par with a Turing machine. It provides a conceptual framework for
describing and rating functions. Practically all functional programming languages in use today
have it as their base.
Alonzo Church taught Alan Turing how to build the Turing computer, which served as the
precursor to imperative programming.
Functional programming is supported by Haskell, JavaScript, Python, Scala, Erlang, Lisp, ML,
Clojure, OCaml, Common Lisp, Racket and other programming languages.
Concepts of functional programming:
Pure functions:Two main characteristics apply to these functions. First of all, regardless of
everything else, they always generate the same output for the same arguments.
Second, they don't affect any arguments, local or global variables, or input or output streams,
therefore they have no side effects.
Immutability is a later quality. The value that the pure function returns is the lone outcome.
They're predetermined.
Functional programming makes it simple to debug program s since there are no side effects or
hidden I/O in pure functions. Writing concurrent and parallel applications is also made simpler
by pure functions. When code is written in this manner, a smart compiler is able to do a variety
of things, including parallelize instructions, hold off on evaluating results until they are needed
andmemorise results because they are constant so long as the input is constant.
Recursion:Functional programming languages do not possess constructs such as "for" or
"while" loops. Recursion is employed in functional programming languages for the purpose of
implementing iteration. Recursive functions exhibit a behaviour in which they repeatedly invoke
themselves until they reach the most basic instance.
Referential transparency:Once defined, variables in functional program s maintain their value
throughout the whole program . Assignment statements do not exist in functional program s.
Instead, we create new variables whenever we need to store certain data. Because every
variable can be changed with its true value at any time during execution, this completely
eliminates the possibility of any negative repercussions. Any variable's state is constant at all
times.
Functions are First-Class and can be Higher-Order:First-class variables are seen as equivalent
to first-class functions. First-class variables has the capability to be returned from functions,
utilised as parameters in functions, or saved within data structures. Higher order functions are
characterised by their ability to both return functions and receive arguments from other
functions.
Variables are Immutable:In functional programming, once a variable has been initialised, it
cannot be changed. This really aids in maintaining state during a program's execution because
we can add new variables but not modify current ones. We can be completely confident that
once a variable is created and its value is set, it will never change.
Parallel Programming Using Resilient Distributed Datasets
The fundamental logical data units in Spark are known as RDDs (Resilient Distributed
Datasets). The objects in question constitute a dispersed assemblage that is stored within the
memory or on the discs of multiple machines within a cluster. A single RDD can be divided into
many logical partitions, allowing for the processing and storage of these partitions across
multiple cluster servers.
Resilient Distributed Datasets (RDDs) possess an inherent characteristic of being read-only,
hence maintaining immutability. RDDs are immutable, meaning they cannot be edited directly.
However, they can be converted or processed using coarse-grained operations to generate new
RDDs.
Users greatly benefit from the fact that an RDD in Spark can be cached and utilised once more
for subsequent changes. RDDs are said to evaluate lazily, delaying the evaluation until it is truly
necessary. This reduces time spent and increases effectiveness.
Features of an RDD in Spark
Here are some features of RDD in Spark:
·
Resilience:In the event of RDD failures, the restoration of lost data is facilitated with the
automatic tracking of data lineage information. An alternative term for it is fault tolerance.
·
Distributed:The data in a Resilient Distributed Dataset (RDD) is distributed across multiple
nodes. The data is distributed among multiple cluster nodes.
·
Lazy evaluation: Even if data is defined in an RDD, it is not loaded. Transformations are
effectively computed when an operation, such as count or collect, is invoked, or when the result
is saved to a file system.
·
Immutability:The data included within a Resilient Distributed Dataset (RDD) is immutable,
meaning that it cannot be modified after it has been stored. However, via the implementation of
alterations to the existing Resilient Distributed Datasets (RDDs), it is possible to generate novel
RDDs.
·
In-memory computation:A Resilient Distributed Dataset (RDD) is a data structure that
allows for the storage of data in random access memory (RAM) rather than on disc, hence
facilitating faster data retrieval.
·
Partitioning:Existing RDDs have the capability to be partitioned into distinct and
customizable logical units. The process of achieving this outcome involves implementing
modifications to the existing partitions.
The Resilient Distributed Dataset (RDD) concept lies at the heart of Spark. A read-only data
collection called an RDD serves as the main functional component of a Spark cluster and can
be distributed among a number of Spark cluster servers. The creation, transformation and
export of RDDs are so central to how Spark works that the entire Spark API may be seen as a
collection of these activities.
Every algorithm used by Spark is essentially a set of transformations applied to data that is
represented as an RDD. An RDD can be cached in memory on the Spark cluster compute
nodes, allowing it to be reused by numerous iterative activities. This is the main performance
driver of Spark.
Each node in the Spark cluster receives the input data that makes up an RDD in the form of
chunks, which are then distributed among all the nodes. Subsequently, every node engages in
parallel computation on its respective set of chunks. An RDD can be generated from several
data sources, such as files from HDFS or other file systems, directly from Scala arrays, or by
various transformations that can be implemented on an existing RDD. An RDD is a concrete
implementation in the Scala programming language.
RDDs are notable for their ability to be rebuilt utilising the idea of lineage in the event that an
RDD partition is lost, making them fault resilient. Spark keeps track of an RDD's ancestry by
documenting the transformations that were applied to produce it. In the event that a portion of
an RDD is lost, Spark will make use of this linage record to rapidly and effectively recompute the
RDD using the same procedures that were used to construct the original version.
By using this method of lineage recomputation, in-memory storage across a compute cluster
can be abstracted without the requirement for expensive data replication strategies. However,
users can manually flag a particular RDD to be check-pointed if the lineage chain grows long
enough. In order to prevent the need for lengthy lineage chains to be recomputed, checkpointed RDDs are written to disc or HDFS. Final result data is still handled by HDFS or another
file system and is not stored or archived by RDDs.
Using the Spark Core API, an RDD can be subjected to a variety of simultaneous operations.
The two categories of these operations are Transformations and Actions. Actions cause
computations to generate a return value from the data or write the data to external storage,
whereas transformations contain functions that will construct new RDDs from current data
sources. Due to the laziness of the Transformation operations, no calculation will take place
when these are called. To make the computation take place, an Action must be invoked.
There are a number of benefits to this lazy execution strategy for Spark's storage and
performance optimisation. Before a user's application is physically run on the cluster, Spark will
review the entire chain of Transformations. This makes it possible to build an overall picture of
the data lineage so that Spark can assess the entire transformation chain. By being aware of
the entire chain, Spark is able to atomically optimise before runtime by only using the data
necessary to produce the intended output.
The addition of Spark DataFrames has significantly enlarged the RDD paradigm. Similar to a
conventional relational database, a SparkDataFrameorganises the distributed collection of data
into labelled columns. Because DataFrames are more abstract than RDDs and have a schema,
Spark may take use of their organised nature to do more automatic optimisation at run time.
The DataFrames API contains additional domain-specific capabilities not provided by the RDD
API in addition to performance benefits, such as joins and aggregations that are well-known to
many data scientists.
Data Flow and Programming with Spark
Spark programming is quite similar to programming in other data flow languages. The only class
that must be written by end users to serve as a high-level program driver is one. This is in
contrast to MapReduce, where three classes—the driver, mapper and reducer—are needed to
make a typical program . The user-created driver program in Spark specifies how the input data
will be changed to produce the desired output. Using the offered APIs, users can develop their
own applications in Java, Scala, Python, or R.
In contrast, it should be noted that Spark is primarily designed to support the Scala
programming language, resulting in the early integration of new API capabilities in Scala. Due to
its functional programming paradigm, users who are more used with object-oriented
programming may encounter a learning curve when using Scala. Non-native speakers of a
certain language are obliged to wait until the latest Spark features are made available to them.
The program is submitted by the head node in the Spark cluster and thereafter distributed
across the cluster to be executed on the respective portions of the Resilient Distributed Dataset
(RDD) stored on each worker node.
A Spark program's data flow can be thought of as follows: In the beginning, the driver produces
one or more RDDs from a Hadoop or other data source. The RDD will then undergo changes
that result in the creation of new RDDs. The altered RDDs will then be subjected to a series of
actions to produce the program's final output.
The simplest approach to visualise how Spark works may be to think about a common sample
application. In this example, we'll look at how to write a straightforward log processing program
that can read a log file as input and count the instances of a certain keyword. This application's
overall goal is to take a log file that is kept on an HDFS, remove only the lines that are
necessary, load the file into memory and count the number of lines that are stored there.
The following steps are necessary to build this application in Spark. First, an RDD would be
made from the log file located on HDFS. The filter transformation would construct the second
RDD by filtering the original file and only selecting the lines that began with the word "ERROR."
The "ERROR" sub-string would be removed from the beginning and a third RDD would be
produced using the map transformation to split the string on white space.
The built-in count function would then be used to determine the total number of elements. For
such a straightforward application, it may appear inefficient to establish three RDDs, but
because the computation is only started when the count Action is called, Spark's execution
planning will make sure that only the necessary data is handled.
To get the most out of the Spark system, users must still exercise caution when developing their
algorithms. Even when running on a cluster, Spark may still need some configuration to reach
the best performance.
Scale-out / Data Parallelism in Apache Spark
You must arrange the necessary resources for the operation in order to get the best
performance out of the Apache Spark cluster. One of the performance elements you should
take into account when planning is the processing of data in parallel. You must split the data
and provide the task an adequate number of executors or cores in order to process data in
parallel. The amount of executors and cores in Spark are the primary variables taken into
account while determining parallelism.
The abstraction offered by Spark for distributed data is called RDD (Resilient Distributed
Dataset). A data file is divided into parts by RDD and distributed among nodes. Consequently,
each node in the cluster has a data piece in its memory. One of the primary abstractions that
Spark automatically manages internally is the distributed management of data in a cluster
context. Because of this abstraction, the developer may concentrate on the logic of data
processing rather than thinking about how to manage data in a distributed environment.
The diagram above demonstrates that when a spark program is launched, the data file that
needs to be processed is on disc (or another source). This file is taken by the Spark driver, who
splits it into pieces and distributes them among all nodes. The cluster in the aforementioned
image has 4 nodes assigned to it, dividing the data file partitions into 4 nodes.
Partition size is the next factor to consider. Each node has the option of having numerous file
partitions and this is typically the default setting. The default partition size is 64MB if the data file
is loaded from HDFS, which is the distributed disc storage system used by Hadoop. Assuming
the data file is 300MB in size, it will be split into 5 partitions. That also means that one node
must handle two data partitions while the other three nodes process only one.
Each node receives one or more executors when a Spark job is launched. The developer
decides this number at launch time. These are the executors that start tasks to process the data
in each partition, with one job handling each partition. Let's assume that the developer created a
Spark application that performs a single MAP transformation before saving the file to disc. This
is the key:
valtextFile = sc.textFile("hdfs://...")
val words = textFile.flatMap(line =>line.split(" "))
words.saveAsTextFile("hdfs://...")
Since each partition is handled by a separate job, saving data to disc requires a total of 5 MAP
tasks and 5 Save tasks. On a 4 node cluster, a total of 10 jobs will be run in order to process a
300MB file. Currently, when a map transformation is performed, four tasks will run concurrently
and then one job will run independently. Similar to the save to disc operation, one job will be
conducted independently once four tasks are completed in tandem. Let's say that a single Map
task takes two minutes to complete. In this case, four jobs will complete in parallel in two
minutes and a further two minutes will be required to complete the lone Fifth task. The Map
change takes 4 minutes to complete in total. Now, if it takes one minute to save a single
partition to disc, it will take two minutes to save five partitions to disc altogether. This task
requires a total of 6 minutes to complete.
What we can do to hasten the scenario described above. We have 4 nodes, however we are
aware that a map transformation requires 5 actions to be completed. If the default partition size
is 64 MB, then the data is divided up into 5 partitions. In 2 minutes, 4 tasks will run concurrently
and in another 2 minutes, the final task will run. The other three nodes will be idle while the fifth
task is running on one of the nodes. The cluster's resources will be underutilisedas a result of
this. Now that the partition size has been increased from the normal 64 MB to 75 MB, the data is
split into 4 equal partitions. Assuming that each partition is now processed in 2.5 minutes, we
can speed up the data processing time by increasing the partition size since it takes 4 jobs to
execute 4 partitions. The same thing will happen if you choose to save something to disc, in
which case it will take, say, 1.5 minutes to save all 4 disc partitions. Simply increasing the
partition size has the overall effect of making the job complete in 4 minutes as opposed to 6
minutes.
The amount of cores allotted to each executor is another factor to boost parallelism. One core
was expected to be assigned to each executor in the aforementioned scenario, as illustrated in
the diagram below. Keep in mind that driver program s on each node start executor JVM
processes to do out duties.
Let's say we have a 600MB file now. With 8 data partitions if the partition size is set to 75 MB,
each node will have two data partitions. We require two sets of four tasks for map
transformation and two sets of four tasks for the save to disc operation if we keep the number of
cores at one as in the previous section. Due to the parallel execution of two sets of both
transformation and action, the job will complete in this instance in 8 minutes. If we raise the
number of cores for each executor to 2, each executor will process its two partitions by running
two tasks concurrently. The processing time for a 600MB file will decrease to 4 minutes by
increasing the number of cores for each executor.
We must take into account the following factors in Spark in order to increase parallelism.
·
Size of the data file
·
Number of executors to launch on each node
·
Number of cores assigned to each executor on each node
·
RAM size for each executor
The amount of RAM allotted to each executor is a different aspect to consider. The cores that
operate on partitions share this RAM to complete operations. Therefore, if each executor with
two cores is given 2MB, then each task running on each core can consume 1MB RAM.
Another thing to keep in mind is that the amount of cores allotted to each executor determines
how much parallelism is possible. In other words, if an executor needs to handle two tasks and
two cores are allocated, then the executor will process both jobs simultaneously. Tasks will be
executed one after another if just one core is assigned. Therefore, Apache Spark's parallelism is
based on the number of cores and partitions.
The settings for the Spark application start are as follows.
spark-submit
--classorg.apache.spark.examples.SparkPi
--masteryarn-cluster
--num-executors10
--driver-memory4g
--executor-memory2g
--executor-cores4
lib/spark-examples*.jar
·
Spark-submit command launches the spark application
·
master defines that his application will run on the YARN cluster
·
num-executors defines how many executors are required to launched in cluster
·
driver-memory defines how much RAM to assign to the driver program
·
executor-memory defines how much RAM to assign to each executor
·
executor-cores defines how many cores should be assigned to each executor
Data Frames and SparkSQL
The SparkSession, made available in Spark 2.0, offers a consistent entry point for using the
Structured APIs to program Spark. To utilise a SparkSession, simply import the class and
create an instance in your program's code.
Use the sql() function on the SparkSession instance, spark, to execute any SQL query, such as
spark.sql("SELECT * FROM myTableName"). Executing all spark.sql queries in this way
produces a DataFrame that can be used for additional Spark operations.
Basic Query Examples
This section will focus on the analysis of the Airline On-Time Performance and Causes of Flight
Delays data set. This data set contains information pertaining to US flights, such as the date of
the flight, the duration of the delay, the distance travelled, as well as the origin and destination
airports. There is a CSV file with over one million records that is now accessible. The data will
be imported into a DataFrame by utilising a schema. Subsequently, the DataFrame will be
registered as a temporary view, with further elaboration on temporary views to be provided later.
Finally, SQL will be employed to execute queries on the DataFrame.
Code snippets with query examples are supplied and the book's GitHub repository has Python
and Scala notebooks with all of the code discussed here. These examples will give you a taste
of how to utilise the spark.sql programmatic interface to integrate SQL into your Spark
applications. You can query structured data in your Spark applications using this interface,
which is comparable to the DataFrame API in its declarative flavour.
As seen in the following example, you would typically manually construct a SparkSession
instance in a standalone Spark application. The SparkSession is generated for you and
available via the appropriately named variable spark in a Spark shell (or DataBricks notebook),
however.
Reading the data set into a temporary view will be the first step.
// In Scala
import org.apache.spark.sql.SparkSession
val spark = SparkSession
.builder
.appName("SparkSQLExampleApp")
.getOrCreate()
// Path to data set
val csvFile="/databricks-datasets/learning-spark-v2/flights/departuredelays.csv"
// Read and create a temporary view
// Infer schema (note that for larger files you may want to specify the schema)
valdf = spark.read.format("csv")
.option("inferSchema", "true")
.option("header", "true")
.load(csvFile)
// Create a temporary view
df.createOrReplaceTempView("us_delay_flights_tbl")
# In Python
from pyspark.sql import SparkSession
# Create a SparkSession
spark = (SparkSession
.builder
.appName("SparkSQLExampleApp")
.getOrCreate())
# Path to data set
csv_file = "/databricks-datasets/learning-spark-v2/flights/departuredelays.csv"
# Read and create a temporary view
# Infer schema (note that for larger files you
# may want to specify the schema)
df = (spark.read.format("csv")
.option("inferSchema", "true")
.option("header", "true")
.load(csv_file))
df.createOrReplaceTempView("us_delay_flights_tbl")
If you want to specify a schema, you can use a DDL-formatted
string. For example:
// In Scala
val schema = "date STRING, delay INT, distance INT,
origin STRING, destination STRING"
# In Python
schema = "`date` STRING, `delay` INT, `distance` INT,
`origin` STRING, `destination` STRING"
We can now use Spark SQL to execute SQL queries because we have a temporary view. The
queries you may use against a SQL table in, let's say, a MySQL or PostgreSQL database are
identical to those you might use. The purpose of this example is to illustrate how Spark SQL
provides an ANSI:2003-compliant SQL interface and how SQL and DataFrames can work
together.
Five columns make up the US flight delays data set:
·
An example of a string in the date column is 02190925. This maps to 02-19 09:25 am
when converted.
·
Between the scheduled and the actual departure times, the delay is indicated in minutes in
the delay column. The numbers are negative for early departures.
·
The distance in miles between the origin airport and the destination airport is displayed in
the distance column.
·
The IATA airport code for the origin is listed in the origin column.
·
The IATA airport code for the destination is listed in the destination column.
Let's test some hypothetical queries against this data set with that in mind.
We'll start by locating all flights with a distance of more than 1,000 miles:
spark.sql("""SELECT distance, origin, destination
FROM us_delay_flights_tbl WHERE distance > 1000
ORDER BY distance DESC""").show(10)
The data indicate that Honolulu (HNL) and New York (JFK) were the two cities with the longest
flights. Next, we'll list all flights with at least a two-hour delay between Chicago (ORD) and San
Francisco (SFO):
spark.sql("""SELECT date, delay, origin, destination
FROM us_delay_flights_tbl
WHERE delay > 120 AND ORIGIN = 'SFO' AND DESTINATION = 'ORD'
ORDER by delay DESC""").show(10)
There appeared to be numerous planes between these two destinations that experienced
substantial delays on various dates.
Let's try a more challenging SQL query that makes use of the CASE clause. In the following
example, independent of origin or destination, we want to label all US flights with a description
of the delays they encountered, such as Very Long Delays (> 6 hours), Long Delays (2-6
hours), etc. These legible labels will be added to a new column named Flight_Delays.
spark.sql("""SELECT delay, origin, destination,
CASE
WHEN delay > 360 THEN 'Very Long Delays'
WHEN delay > 120 AND delay < 360 THEN 'Long Delays'
WHEN delay > 60 AND delay < 120 THEN 'Short Delays'
WHEN delay > 0 and delay < 60 THEN 'Tolerable Delays'
WHEN delay = 0 THEN 'No Delays'
ELSE 'Early'
END AS Flight_Delays
FROM us_delay_flights_tbl
ORDER BY origin, delay DESC""").show(10)
With the spark.sql interface, you can conduct common data analysis operations, much like with
the DataFrame and Dataset APIs. The computations travel the same path in the Spark SQL
engine, producing the same outcomes.
An equivalent Data Frame API query can be used to express each of the three previous SQL
queries. For instance, the Python Data Frame API allows for the following expression of the first
query:
# In Python
from pyspark.sql.functions import col, desc
(df.select("distance", "origin", "destination")
.where(col("distance") > 1000)
.orderBy(desc("distance"))).show(10)
# Or
(df.select("distance", "origin", "destination")
.where("distance > 1000")
.orderBy("distance", ascending=False).show(10))
This yields the exact same outcomes as the SQL query:
These illustrations demonstrate how writing a standard SQL query to a relational database table
is equivalent to using the Spark SQL interface to query data. Despite being in SQL, the queries
have readability and semantics that are comparable to DataFrame.
Spark handles all the difficulties of building and managing views and tables, both in memory and
on disc, to enable you to query structured data as demonstrated in the earlier examples. That
brings us to our next subject, which is the creation and management of tables and views.
SQL Tables and Views
Databases are tables. Each table in Spark has relevant metadata attached to it, which includes
details about the table and its data, such as the schema, description, table name, data base
name, column names, partitions and the physical location of the data itself. A central metastore
houses all of this data.
Instead of having a unique metastore for Spark tables, Spark by default persists all table
metadata in the Apache Hive metastore, which can be found at /user/hive/warehouse. The
Spark configuration variable spark.sql.warehouse.dir, which can be set to a local or external
distributed storage, allows you to override the default location.
Managed Versus UnmanagedTables
Spark enables the creation of controlled and unmanaged tables. Spark controls both the
metadata and the data in the file store for a man-aged table. A local filesystem, HDFS, or an
object store like Amazon S3 or Azure Blob might be used for this. Spark simply manages the
metadata for an unmanaged table; you manage the data yourself in an external data source like
Cassandra.
A SQL command like DROP TABLE table_name deletes both the metadata and the data from a
managed table because Spark controls everything. The same command will only destroy the
metadata for an unmanaged table, not the actual data. In the part after this, we'll look at a few
illustrations of managed and unmanaged table creation.
Creating SQL Databases and Tables
A database has tables. Spark creates tables in the default database by default. You can issue a
SQL command from your Spark application or notebook to define your own database name.
Let's create both a managed and an unmanaged table using the US flight delays data collection.
We'll inform Spark we want to utilise a database we'll create called learn_spark_db first.
// In Scala/Python
spark.sql("CREATE DATABASE learn_spark_db")
spark.sql("USE learn_spark_db")
From this point forward, any table-creation commands we issue in our application will result in
the creation of tables in this database with the name learn_spark_db.
Creating a managed table
With the database learn_spark_db, you can run the following SQL statement to create a
managed table:
// In Scala/Python
spark.sql("CREATE TABLE managed_us_delay_flights_tbl (date STRING, delay INT,
distance INT, origin STRING, destination STRING)")
Using the DataFrame API, you can accomplish the same task as follows:
# In Python
# Path to our US flight delays CSV file
csv_file = "/databricks-datasets/learning-spark-v2/flights/departuredelays.csv"
# Schema as defined in the preceding example
schema="date STRING, delay INT, distance INT, origin STRING, destination STRING"
flights_df = spark.read.csv(csv_file, schema=schema)
flights_df.write.saveAsTable("managed_us_delay_flights_tbl")
In the learn_spark_db database, both of these queries will construct the managed table
us_delay_flights_tbl.
Creating an unmanaged table
As an alternative, you can make unmanaged tables from your own data sources, such as
Parquet, CSV, or JSON files kept in a file store that is available to your Spark application.
In SQL, use the following syntax to create an unmanaged table from a data source such a CSV
file:
spark.sql("""CREATE TABLE us_delay_flights_tbl(date STRING, delay INT,
distance INT, origin STRING, destination STRING)
USING csv OPTIONS (PATH
'/databricks-datasets/learning-spark-v2/flights/departuredelays.csv')""")
And within the DataFrame API use:
(flights_df
.write
.option("path", "/tmp/data/us_flights_delay")
.saveAsTable("us_delay_flights_tbl"))
Creating Views
Spark can build views on top of already-existing tables in addition to building new tables. Views
are ephemeral objects that vanish once your Spark application is terminated. They can be
global (visible to all SparkSessions on a specific cluster) or session-scoped (visible only to a
particular SparkSession).
In a database, creating views uses a syntax that is similar to creating tables. Once a view has
been created, it can be queried similarly to a table. A view is different from a table in that views
don't really contain the data; unlike tables, views vanish when your Spark application is through
running.
Using SQL, a view can be produced from an existing table. The following queries will generate
global temporary and temporary views that only contain that portion of the table, for instance, if
you want to work on the subset of the US flight delays data set with origin airports of New York
(JFK) and San Francisco (SFO):
-- In SQL
CREATE OR REPLACE GLOBAL TEMP VIEW us_origin_airport_SFO_global_tmp_view AS
SELECT date, delay, origin, destination from us_delay_flights_tblWHERE
origin = 'SFO';
CREATE OR REPLACE TEMP VIEW us_origin_airport_JFK_tmp_view AS
SELECT date, delay, origin, destination from us_delay_flights_tblWHERE
origin = 'JFK'
Using the DataFrame API, you may achieve the same results as seen below:
# In Python
df_sfo = spark.sql("SELECT date, delay, origin, destination FROM
us_delay_flights_tbl WHERE origin = 'SFO'")
df_jfk = spark.sql("SELECT date, delay, origin, destination FROM
us_delay_flights_tbl WHERE origin = 'JFK'")
# Create a temporary and global temporary view
df_sfo.createOrReplaceGlobalTempView("us_origin_airport_SFO_global_tmp_view")
df_jfk.createOrReplaceTempView("us_origin_airport_JFK_tmp_view")
Once these views are generated, you can query them in the same way that you would query a
table. Because Spark creates global temporary views in a global temporary database called
global_temp, keep in mind that you must use the prefix global_temp. when accessing a global
temporary view. For instance:
SELECT * FROM global_temp.us_origin_airport_SFO_global_tmp_view
The standard temporary view, in contrast, can be accessed without the global_temp prefix:
-- In SQL
SELECT * FROM us_origin_airport_JFK_tmp_view
// In Scala/Python
spark.read.table("us_origin_airport_JFK_tmp_view")
// Or
spark.sql("SELECT * FROM us_origin_airport_JFK_tmp_view")
Similarly to dropping a table, you can also drop a view:
-- In SQL
DROP VIEW IF EXISTS us_origin_airport_SFO_global_tmp_view;
DROP VIEW IF EXISTS us_origin_airport_JFK_tmp_view
// In Scala/Python
spark.catalog.dropGlobalTempView("us_origin_airport_SFO_global_tmp_view")
spark.catalog.dropTempView("us_origin_airport_JFK_tmp_view")
Data Sources for DataFrames and SQL Tables
Spark SQL offers an interface to a multitude of data sources, as seen in the aforementioned
Figure. Additionally, it offers a collection of standard techniques for using the Data Sources API
to read and write data to and from these data sources.
This section will go over some of the built-in data sources, file formats that are supported,
methods for loading and writing data and particular choices related to these data sources. But
first, let's take a deeper look at two high-level Data Source API constructs: DataFrameReader
and DataFrameWriter, which specify how you interface with various data sources.
DataFrameReader
The fundamental building block for reading data into a DataFrame from a data source is
DataFrameReader. It has a specified format and a suggested usage pattern:
DataFrameReader.format(args).option("key", "value").schema(args).load()
This way of connecting methods is typical of Spark and is simple to read.
Please take note that a SparkSession instance is the sole way to access a DataFrameReader.
In other words, you are unable to create a DataFrameReader instance. Use: to access it and
retrieve an instance handle.
SparkSession.read
// or
SparkSession.readStream
ReadStream returns an instance to read from a streaming source, whereas read returns a
handle to a DataFrameReader to read into a DataFrame from a static data source.
Different values can be passed as arguments to different public DataFrameReader methods.
These are listed in the table below, along with a selection of the supporting arguments.
Table: DataFrameReader methods, arguments and options
The documentation for Python, Scala, R and Java includes suggestions and instructions, but we
won't exhaustively list all the various combinations of arguments and choices. But it's important
to provide a few illustrations:
// In Scala
// Use Parquet
val file = """/databricks-datasets/learning-spark-v2/flights/summarydata/parquet/2010-summary.parquet"""
valdf = spark.read.format("parquet").load(file)
// Use Parquet; you can omit format("parquet") if you wish as it's the default
val df2 = spark.read.load(file)
// Use CSV
val df3 = spark.read.format("csv")
.option("inferSchema", "true")
.option("header", "true")
.option("mode", "PERMISSIVE")
.load("/databricks-datasets/learning-spark-v2/flights/summary-data/csv/*")
// Use JSON
val df4 = spark.read.format("json")
.load("/databricks-datasets/learning-spark-v2/flights/summary-data/json/*")
DataFrameWriter
In contrast to its cousin, DataFrameWriter stores or writes data to a specific built-in data source.
In contrast to DataFrameReader, you access its instance from the DataFrame you want to save,
not from a SparkSession. It has the following suggested usage patterns:
DataFrameWriter.format(args)
.option(args)
.bucketBy(args)
.partitionBy(args)
.save(path)
DataFrameWriter.format(args).option(args).sortBy(args).saveAsTable(table)
To get an instance handle, use:
DataFrame.write
// or
DataFrame.writeStream
Each of the DataFrameWriter methods' arguments accepts a distinct value. With a selection of
the supporting reasons, we list these in the table below.
Table:DataFrameWriter methods, arguments and options
Here is a succinct example to show how to use methods and arguments:
// In Scala
// Use JSON
val location = ...
df.write.format("json").mode("overwrite").save(location)
Module V: Apache Spark for Data Engineering and Machine Learning
Spark Structured Streaming; Graph Frames on Apache Spark; ETL Workloads; Spark ML
Fundamentals; Spark ML Regression and Classification; Spark ML Clustering
Apache Spark and It Techniques
An open-source platform for distributed computing called Apache Spark was created for large
data processing and analytics. On a single platform, it offers a unified data processing engine
that enables batch processing, real-time streaming, machine learning and graph processing.
Because of Spark's great performance and scalability, big datasets may be processed over
distributed clusters. Some of Apache Spark's primary methods and features are listed below:
·
Distributed Computing: Spark divides data processing responsibilities among numerous
cluster nodes, enabling parallel processing and greatly enhancing large data workload
performance.
·
Resilient Distributed Datasets (RDDs):RDDs are the core data structure of Spark. They
can be cached in memory to speed up data processing and are immutable and fault-tolerant.
·
DataFrames and Datasets:Higher-level abstractions built on top of RDDs are introduced by
Spark in the form of DataFrames and Datasets. A structured and schema-aware API for data
processing, akin to using tables in relational databases, is provided by DataFrames and
Datasets.
·
Transformations and Actions:To process and analyse data effectively, Spark enables a
variety of transformations (such as map, filter andgroupBy) and actions (such as count, collect
and save).
·
Lazy Evaluation:Spark uses lazy evaluation, which means that changes to data are stored
as a series of logical processes rather than being immediately executed. When an action is
invoked, the actual execution is started, allowing Spark to refine and improve the execution
plan.
·
Spark SQL:Spark SQL makes it simple to combine Spark with conventional relational data
processing by enabling users to run SQL queries directly on DataFrames.
·
Spark Streaming: A real-time data processing module called Spark Streaming enables
data processing and analysis in almost real-time. It offers assistance with ingesting and
processing data streams from Kafka, Flume, or Kinesis sources.
·
Machine Learning Library (MLlib):In order to assist distributed machine learning operations
on huge data, Spark MLlib provides a collection of machine learning algorithms and utilities.
·
GraphX: A graph processing library for distributed graph computation is called Spark
GraphX. It offers a collection of tools and algorithms for quickly processing and evaluating graph
data.
·
Spark GraphFrames:A Spark DataFrame API extension created specifically for handling
graph data is called GraphFrames. It offers a more advanced API for manipulating and
analysing graphs.
·
Spark Streaming + Kafka Integration:To consume and handle distributed streaming data in
real-time, Spark can be coupled with Apache Kafka.
·
MLflow: Databricks, the company behind Spark, developed MLflow as an open-source
platform for managing the entire machine learning lifecycle, including experimentation,
reproducibility and deployment.
Apache Spark is a well-liked option for big data processing and analytics across a variety of
sectors and use cases thanks to its adaptability, speed and support for different data processing
approaches.
Spark Structured Streaming
The continuous processing of unending streams of data is known as stream processing. Stream
processing systems changed from single-node processing engines to multiple-node, distributed
processing engines with the introduction of large data. The record-at-a-time processing
approach has traditionally been used to develop distributed stream processing, as seen in the
image below.
Figure: Traditional record-at-a-time processing model
According to the above Figure, the processing pipeline is made up of a directed graph of nodes.
Each node in the graph gets one record at a time continuously, processes it and then sends the
resulting record(s) to the following node. This processing paradigm is capable of generating
output in milliseconds after processing an input record within very low latencies, or very short
latency. This model can either recover from a failure extremely quickly with a lot of extra failover
resources, or it can use very little more resources but recover slowly. However, it is not very
effective at recovering from node failures and straggler nodes (i.e., nodes that are slower than
others).
The Advent of Micro-Batch Stream Processing
When Spark Streaming (also known as DStreams) was introduced by Apache Spark, the
established method was put to the test. It developed the concept of micro-batch stream
processing, in which the streaming computing is described as a continuous sequence of small
batch processing operations akin to map/reduce on discrete portions of the stream data (thus,
"micro-batches").
Figure: Structured Streaming uses a micro-batch processing model
The data obtained from the input stream is partitioned into micro-batches of 1-second duration,
as exemplified by the implementation in Spark Streaming. In the context of distributed
processing in a Spark cluster, small deterministic tasks are employed to handle each batch,
generating output in micro-batches. We gain two advantages over the conventional continuousoperator paradigm by breaking the streaming computation down into these tiny tasks:
·
Spark's agile task scheduling enables efficient recovery from failures and straggler
executors by rescheduling one or more copies of the jobs on alternative executors.
·
No matter how many times the job is repeated, the output data will always be the same
thanks to the deterministic nature of the tasks. Due to this key feature, Spark Streaming is able
to guarantee end-to-end precisely-once processing, which means that every input record was
processed exactly once in the output results.
This effective fault tolerance does come at the expense of latency; the micro-batch approach
typically achieves latencies of a few seconds (as low as half a second in some circumstances),
not millisecond-level latencies. However, we have found that the advantages of micro-batch
processing far outweigh the disadvantage of second-scale latencies for the vast majority of
stream processing use cases. This is because at least one of the following features can be seen
in the majority of streaming pipelines:
·
Less than a few second latencies are not required by the pipeline. It is not advantageous to
generate output with subsecond latencies, for instance, if hourly jobs are the only ones who
would view the streaming data.
·
Other pipeline segments are experiencing more significant delays. For example, if the
posts from a sensor to Apache Kafka are performed in batches to enhance throughput, it is not
possible for downstream processing systems to reduce the end-to-end latency below the delays
caused by batching. No amount of optimisation in these processing systems will overcome this
limitation.
Additionally, Spark's batch RDD API served as the foundation for the DStream API. DStreams
possess the functional semantics and fault-tolerance mechanism that are shared with RDDs.
The demonstration of Spark Streaming has showcased the feasibility of offering consistent
application programming interfaces (APIs) and semantics for all types of workloads, including
batch processing, interactive processing and streaming processing, all inside a single,
integrated processing engine. The advent of this fundamental paradigm shift in stream
processing has facilitated the development of Spark Stream, which has emerged as one of the
most widely adopted open source stream processing engines.
Spark Streaming (DStreams) Lessons Learned
The DStream API had drawbacks in addition to its advantages. The following significant areas
for improvement were found:
There isn't a single API for batch and stream processing.
When converting batch jobs to streaming jobs, developers have to actively modify their code to
use new classes even though DStreams and RDDs had consistent APIs (i.e., the same
operations and meanings).
Inability to distinguish between logical and practical plans
The DStream actions are carried out by Spark Streaming in the exact order that the developer
intended. Developers must hand-optimize their code to achieve the optimal performance
because there is no room for automatic optimisations because they effectively provide the exact
physical plan.
Lack of native support for event-time windows
Only the processing time (also known as the time each record is received by Spark Streaming)
is used by DStreams to create window actions. However, many use cases require windowed
averages to be calculated using event time—as opposed to the time the records were received
or processed—instead of the time they were generated. Building similar pipelines using Spark
Streaming proved challenging for developers due to the lack of native support for event-time
windows.
The Philosophy of Structured Streaming
Based on these insights from DStreams, Structured Streaming was created from the ground up
with the fundamental tenet that creating stream processing pipelines for developers should be
as simple as creating batch pipelines. The guiding concepts of structured streaming can be
summed up as follows:
A single, unified programming model and interface for batch and stream processing
Both batch and streaming workloads can use the straightforward API interface provided by this
unified paradigm. You can use well-known SQL or batch-like DataFrame queries on your stream
just like you would on a batch and you can let the engine deal with the subtle complications of
fault tolerance, optimisations and delayed data. We will look at some of the queries you might
write in the sections that follow.
A broader definition of stream processing
The distinction between batch processing and real-time processing has become much more
hazy as a result of the complexity of big data processing systems. Any application that
periodically processes data (e.g., every few hours) or continuously processes data (like
traditional streaming applications) should be expressible using Structured Streaming. This was
the goal with Structured Streaming: to extend its applicability beyond traditional stream
processing to a larger class of applications.
The Programming Model of Structured Streaming
Developers are accustomed to the term "table" while creating batch programs. By considering a
stream as an unbounded, continually appended table, as seen in Figure below, Structured
Streaming extends this idea to streaming applications.
Figure: The Structured Streaming programming model: data stream as an unboun‐ ded table
A new row is added to the unbounded input table for every new record that is received in the
data stream. Although Structured Streaming won't actually keep all the information, the result it
generates up until time T will be the same as if the entire input had been stored in a static,
bounded table and processed by a batch job.
The developer then defines a query on this hypothetical input table, as if it were a static table, to
determine the result table that will be written to an output sink, as illustrated in the figure below.
This batch-like query will automatically be transformed into a streaming execution plan using
Structured Streaming. Structured Streaming determines what state has to be kept in order to
update the outcome each time a new record arrives. This process is known as
incrementalisation. Finally, trigger policies are specified by developers to regulate when to
update the outcomes. Structured Streaming looks for new data—a new row in the input
database, for example—each time a trigger fires and incrementally updates the outcome.
Figure: The Structured Streaming processing model
The output mode is the model's final component. The developer will wish to write the updates to
an external system each time the result table is updated, such as a database (such as MySQL,
Cassandra) or a files system (such as HDFS, Amazon S3). Typically, we wish to write output in
steps. Three output modalities are offered by Structured Streaming for this purpose:
Append mode: The only rows written to the external storage will be those that have been added
to the result table since the most recent trigger. Only queries where current entries in the return
table cannot change (such as a map on an input stream) are suitable to this.
Update mode :Only rows in the result table that have changed since the last trigger will have
their values altered in the external storage. This mode is appropriate for output sinks that permit
in-place updates, such a MySQL table.
Complete mode: It will write the complete updated result table to external storage.
The logical operations on the data are easier to conceive and are also simpler to represent in
code when the data streams are thought of as tables. The ability to do computations on
streaming data is facilitated by the DataFrame API, as it allows for the manipulation of data in a
tabular format. This is made possible by Spark's DataFrame, which serves as a programmatic
representation of a table.
To construct an input DataFrame from a streaming data source, one can follow the procedure of
creating an input table. Subsequently, operations can be applied to the DataFrame in a similar
manner as those performed on a batch source.
The Fundamentals of a Structured Streaming Query
We'll go over a few high-level ideas in this section that you should be familiar with in order to
create structured streaming queries. Before discussing how to keep an active query under
control and determine its life cycle, we will first go through the essential procedures to define
and launch a streaming query.
Five Steps to Define a Streaming Query
The data processing logic for Structured Streaming is expressed using the same DataFrame
API as batch queries. To define a Structured Streaming query, you must be aware of a few
important distinctions. In this section, we'll design a straightforward query that reads streams of
text data over a socket and counts the words as we go through the process of defining a
streaming query.
Step 1: Define input sources
The initial step is to define a DataFrame from a streaming source, much like with batch queries.
Spark is required when accessing batch data sources, though.Streaming sources require spark,
whereas read can build a DataFrameReader.create a DataStreamReader using readStream.
You can use DataStreamReader similarly to DataFrameReader because it contains many of the
same functions. Here is an illustration of how to construct a DataFrame from a text data stream
that will be gotten over a socket connection:
# In Python
spark = SparkSession...
lines = (spark
.readStream.format("socket")
.option("host", "localhost")
.option("port", 9999)
.load())
// In Scala
val spark = SparkSession...
val lines = spark
.readStream.format("socket")
.option("host", "localhost")
.option("port", 9999)
.load()
The lines DataFrame is created by this code as an unbounded table of newly-lined text data that
is read from localhost:9999. It should be noted that, similar to batch sources using spark.read,
this does not begin reading streaming data right away; rather, it waits until the streaming query
is explicitly begun before setting up the relevant parameters.
The reading of data streams from Apache Kafka and all the file-based formats supported by
DataFrameReader (Parquet, ORC, JSON, etc.) is also natively supported by Apache Spark in
addition to sockets. Additionally, various input sources, both batch and streaming, can be
defined via a streaming query. These sources can then be integrated using DataFrame
operations like unions and joins.
Step 2: Transform data
The following code illustrates how to divide the lines into individual words and then count them
using the standard DataFrame operations:
# In Python
from pyspark.sql.functions import *
words = lines.select(split(col("value"), "\\s").alias("word"))
counts = words.groupBy("word").count()
// In Scala
import org.apache.spark.sql.functions._
val words = lines.select(split(col("value"), "\\s").as("word"))
val counts = words.groupBy("word").count()
counts is a streaming DataFrame, or a DataFrame on unbounded, streaming data, that displays
the running word counts that will be calculated once the streaming query has been launched
and the streaming input data has begun to be constantly processed.
It should be noted that if lines were a batch DataFrame, these methods to convert it would
function in the same manner. In general, the majority of DataFrame operations that are
applicable to a batch DataFrame are also applicable to a streaming DataFrame. You must be
aware of the two main categories of data transformations in order to comprehend which
operations are supported by structured streaming:
Stateless transformations: The information from prior rows is not needed for operations like
select(), filter(), map(), etc. to process the subsequent row; instead, each row can be treated
independently. These activities are stateless because there was never a prior "state" in them.
Batch and streaming DataFrames can both use stateless operations.
Stateful transformations: While preserving state is necessary for aggregation operations like
count() to mix data from several rows. More specifically, stateful transformations are any
DataFrame operations that involve grouping, joining, or aggregating. While most of these
operations are enabled by Structured Streaming, some combinations of them are not supported
since computing them incrementally would be computationally challenging or impossible.
Step 3: Define output sink and output mode
After data transformation, we may specify with DataFrame how to output the transformed
DataFrame.writeStream (used for batch data instead than DataFrame.write). This generates a
DataStreamWriter with additional methods to express the following, comparable to a
DataFrameWriter:
·
Information on where and how to write the output
·
Processing specifics (data processing and failure recovery)
Let's begin with the specifics of writing the output (we'll concentrate on the processing details in
the following phase). The following sample, for instance, demonstrates how to output the final
counts to the console:
# In Python
writer = counts.writeStream.format("console").outputMode("complete")
// In Scala
val writer = counts.writeStream.format("console").outputMode("complete")
In this context, the output mode has been specified as "complete" and the output streaming sink
has been marked as "console". The output mode of a streaming query determines the specific
section of the updated output that will be written out after processing new input data. The user
has the option to display either the cumulative counts of all observed words (full mode) or only
the counts of words that were changed in the most recent chunk of input data. This example
demonstrates the process of processing a new chunk of input data and updating the word
counts.
Append mode
In this default setting, only the recently appended rows in the result table or Data Frame, such
as the counts table, will be displayed in the output sink. Based on semantic theory, this option
guarantees that every row produced as output by the query will remain unaltered and unaffected
by any subsequent modifications or updates. Therefore, add mode is only enabled for searches,
specifically stateless queries, that do not modify data that has been previously outputted. In
contrast, our word count query does not support the add mode as it possesses the capability to
update counts that have been computed before.
Complete mode
At the conclusion of each trigger in this mode, the entire result table or DataFrame will be
output. This is supported by queries when it is anticipated that the output table will be far smaller
than the input data and may thus realistically be kept in memory. For instance, because the
counts data is probably going to be much lower than the input data, our word count query
supports complete mode.
Update mode
Only the rows of the result table or DataFrame that have changed since the last trigger are
reported in this mode at the conclusion of each trigger. This is different from append mode in
that the query may modify the output rows and output them once more afterwards. Update
mode is supported by most queries.
Structured Streaming natively enables streaming writes to files and Apache Kafka in addition to
publishing the output to the console. Additionally, you may use the foreachBatch() and foreach()
API methods to write to any location. In reality, you may write streaming outputs utilising batch
data sources by using foreachBatch() (albeit you will lose exactly-once guarantees).
Step 4: Specify processing details
The last step before running the query is to specify the specifics of the data processing. Keeping
with our word count example, we will discuss the procedure in the following manner:
# In Python
checkpointDir = "..."
writer2 = (writer
.trigger(processingTime="1 second")
.option("checkpointLocation", checkpointDir))
// In Scala
import org.apache.spark.sql.streaming._
valcheckpointDir = "..."
val writer2 = writer
.trigger(Trigger.ProcessingTime("1 second"))
.option("checkpointLocation", checkpointDir)
Using the DataStreamWriter that we created using DataFrame.writeStream, we have here
defined two different sorts of information:
Triggering details: The timing of when to start finding and processing freshly accessible
streaming data is indicated by this. There are four alternatives.
·
Default:The streaming query runs data in micro-batches, with the next micro-batch being
triggered as soon as the preceding micro-batch has finished, by default when the trigger is not
explicitly provided.
·
Processing time with triggerinterval: The query will start triggering micro-batches at the
predetermined interval if you explicitly supply the ProcessingTime trigger with an interval.
·
Once: The streaming query will run in this mode in exactly one micro-batch, processing all
the fresh data in a single batch before stopping. This is helpful if you wish to manage the
triggering and processing from a third-party scheduler that will restart the query according to a
personalised schedule (for instance, to reduce costs by just running a query once per day).
·
Continuous: As with Spark 3.0, this is an experimental mode where the streaming query
would continually process data rather than doing so in micro-batches. Although this mode can
only be employed for a tiny fraction of DataFrame operations, it can provide substantially
reduced latency (as low as milliseconds) than the micro-batch throttle modes.
Checkpoint location: In any HDFS-compatible filesystem, this is the directory where a streaming
query saves its progress information, or the data that has been successfully processed. This
metadata is utilised to pick up where the failed query left off in the event of failure. Setting this
option is therefore required for exact-once guarantees on failure recovery.
Step 5: Start the query
The final step is to start the query after everything has been defined, which you can do by using
one of the following:
# In Python
streamingQuery = writer2.start()
// In Scala
valstreamingQuery = writer2.start()
An active query is represented by the returned object of type streamingQuery, which can be
used to control the query.
As a nonblocking method, start() will finish as soon as the query has begun running in the
background. Using streamingQuery.awaitTermination() will cause the main thread to pause until
the streaming query has finished. AwaitTermination() will fail with the same exception if the
query fails in the background.
You can manually halt the query using streamingQuery.stop() or wait up to a timeout duration
using awaitTermination(timeoutMillis).
Putting it all together
Here is the whole code for reading text data streams over a socket, counting the words and
writing the counts to the console, to give a brief summary:
# In Python
from pyspark.sql.functions import *
spark = SparkSession...
lines = (spark
.readStream.format("socket")
.option("host", "localhost")
.option("port", 9999)
.load())
words = lines.select(split(col("value"), "\\s").alias("word"))
counts = words.groupBy("word").count()
checkpointDir = "..."
streamingQuery = (counts
.writeStream
.format("console")
.outputMode("complete")
.trigger(processingTime="1 second")
.option("checkpointLocation", checkpointDir)
.start())
streamingQuery.awaitTermination()
// In Scala
import org.apache.spark.sql.functions._
import org.apache.spark.sql.streaming._
val spark = SparkSession...
val lines = spark
.readStream.format("socket")
.option("host", "localhost")
.option("port", 9999)
.load()
val words = lines.select(split(col("value"), "\\s").as("word"))
val counts = words.groupBy("word").count()
valcheckpointDir = "..."
valstreamingQuery = counts.writeStream
.format("console")
.outputMode("complete")
.trigger(Trigger.ProcessingTime("1 second"))
.option("checkpointLocation", checkpointDir)
.start()
streamingQuery.awaitTermination()
A background thread continuously gets fresh data from the streaming source after the query has
started, processes it and then writes it to the streaming sinks. Let's now quickly examine the
inner workings of how this is carried out.
Under the Hood of an Active Streaming Query
The following series of events occurs in the engine once the query is launched, as shown in the
below Figure. A logical plan, which is an abstract representation of the calculation Spark SQL
performs to plan a query, is created from the DataFrame operations:
1.
This logical plan is examined and optimised by Spark SQL to make sure it can be
executed effectively and gradually on streaming data.
2.
Spark SQL starts a background process that runs the following loop repeatedly:
a.
The thread examines the streaming sources for the presence of fresh data based on the
set trigger interval.
b.
If available, a micro-batch is used to execute the new data. A Spark execution plan that
reads fresh data from the source, computes the updated result gradually and publishes the
output to the sink in accordance with the specified output mode is generated from the optimised
logical plan.
c.
The selected checkpoint location saves the precise range of data processed for each
micro-batch (e.g., the set of files or the range of Apache Kafka offsets) and any associated state
so that the query may deterministically reproduce the precise range if necessary.
3.
The inquiry may be terminated for one of the following reasons before this loop ends:
a.
The query has failed (either due to a cluster failure or a processing fault).
b.
Using streamingQuery.stop(), the query is specifically terminated.
c.
The query will automatically terminate after processing a single micro-batch that contains
all the available data if the trigger is set to Once.
Figure: Incremental execution of streaming queries
It's important to keep in mind that Spark SQL is used to execute the data in Structured
Streaming. As a result, the stream processing throughput is maximised by using the full capacity
of Spark SQL's hyperoptimised execution engine, offering significant performance benefits.
References:
Graph Frames on Apache Spark
A module for Apache Spark called GraphFrames offers DataFrame-based Graphs. It offers
high-level Python, Java and Scala APIs. It seeks to offer GraphX functionality as well as
expanded capability that makes use of Spark DataFrames. Finding motifs, DataFrame-based
serialisation and extremely powerful graph queries are all included in this expanded feature.
GraphFrames is now accessible as a Spark package, which you must load when you launch
your Spark application. However, in the future, it's possible that GraphFrames will be integrated
into the core of Spark. Except for GraphFrames' significantly improved user experience, there
should not be much of a performance difference between the two. Although employing
GraphFrames has a tiny overhead, it generally attempts to call down to GraphX where
appropriate and for the majority of users, the improvements in user experience far outweigh this
slight expense.
Not a database, Spark. Spark is a distributed computation engine, however it doesn't conduct
transactions or long-term data storage. A graph computation can be built on top of Spark, but it
differs fundamentally from a database. GraphFrames performs well for analytics and can grow
to considerably bigger workloads than many graph databases, however it does not enable
transactional processing and serving.
You'll need to point to the appropriate package in order toget setup. Run the following command
line commands to accomplish this:
./bin/spark-shell --packages graphframes:graphframes:0.5.0-spark2.2-s_2.11
// in Scala
valbikeStations = spark.read.option("header","true")
.csv("/data/bike-data/201508_station_data.csv")
valtripData = spark.read.option("header","true")
.csv("/data/bike-data/201508_trip_data.csv")
# in Python
bikeStations = spark.read.option("header","true")\
.csv("/data/bike-data/201508_station_data.csv")
tripData = spark.read.option("header","true")\
.csv("/data/bike-data/201508_trip_data.csv"
Building a Graph
The construction of the graph is a prerequisite. In order to achieve this objective, it is necessary
to define the edges and vertices, which are represented as DataFrames including columns that
have been assigned specific names. In this particular case, a directed graph is being
constructed.
The graph provides information regarding the source and destination. This will establish a
directional connection between the initial place of departure and the final destination inside the
framework of the bicycle excursion data. The GraphFrames library utilises naming rules for
columns in order to establish the graph's definition.
The identifier in the vertices database is referred to as "id" and is of the string data type. In the
edges table, the source vertex ID is represented as "src" and the destination vertex ID is
represented as "dst" for each edge.
// in Scala
valstationVertices = bikeStations.withColumnRenamed("name", "id").distinct()
valtripEdges = tripData
.withColumnRenamed("Start Station", "src")
.withColumnRenamed("End Station", "dst")
# in Python
stationVertices = bikeStations.withColumnRenamed("name", "id").distinct()
tripEdges = tripData\
.withColumnRenamed("Start Station", "src")\
.withColumnRenamed("End Station", "dst")
We can now combine the vertex and edge DataFrames we have so far to create a GraphFrame
object that describes our graph. Because we'll be using this data frequently in future queries,
we'll also make use of caching:
// in Scala
import org.graphframes.GraphFrame
valstationGraph = GraphFrame(stationVertices, tripEdges)
stationGraph.cache()
# in Python
from graphframes import GraphFrame
stationGraph = GraphFrame(stationVertices, tripEdges)
stationGraph.cache()
In order to get the expected results, we can now query our original DataFrame and examine the
graph's fundamental statistics as follows:
// in Scala
println(s"Total Number of Stations: ${stationGraph.vertices.count()}")
println(s"Total Number of Trips in Graph: ${stationGraph.edges.count()}")
println(s"Total Number of Trips in Original Data: ${tripData.count()}")
# in Python
print "Total Number of Stations: " + str(stationGraph.vertices.count())
print "Total Number of Trips in Graph: " + str(stationGraph.edges.count())
print "Total Number of Trips in Original Data: " + str(tripData.count())
This returns the following results:
Total Number of Stations: 70
Total Number of Trips in Graph: 354152
Total Number of Trips in Original Data: 354152
Querying the Graph
The simplest approach to interact with the graph is to simply query it, carrying out operations
like trip counting and destination filtering. As with DataFrames, GraphFrames provide
straightforward access to both vertices and edges. In addition to the IDs, sources and
destinations, our graph also included all the other columns in the data, so if necessary, we can
also query those.
// in Scala
import org.apache.spark.sql.functions.desc
stationGraph.edges.groupBy("src", "dst").count().orderBy(desc("count")).show(10)
# in Python
from pyspark.sql.functions import desc
stationGraph.edges.groupBy("src", "dst").count().orderBy(desc("count")).show(10)
We can also use any legitimate DataFrame expression to filter. In this case, We want to focus
on one particular station and the number of trips that enter and exit that station:
// in Scala
stationGraph.edges
.where("src = 'Townsend at 7th' OR dst = 'Townsend at 7th'")
.groupBy("src", "dst").count()
.orderBy(desc("count"))
.show(10)
# in Python
stationGraph.edges\
.where("src = 'Townsend at 7th' OR dst = 'Townsend at 7th'")\
.groupBy("src", "dst").count()\
.orderBy(desc("count"))\
.show(10)
Subgraphs
Subgraphs are only smaller versions of the main graph. The last section demonstrated how to
query a set of edges and vertices. This ability of the query to produce subparagraphs is useful:
// in Scala
val townAnd7thEdges = stationGraph.edges
.where("src = 'Townsend at 7th' OR dst = 'Townsend at 7th'")
val subgraph = GraphFrame(stationGraph.vertices, townAnd7thEdges)
# in Python
townAnd7thEdges = stationGraph.edges\
.where("src = 'Townsend at 7th' OR dst = 'Townsend at 7th'")
subgraph = GraphFrame(stationGraph.vertices, townAnd7thEdges)
Motif Finding
A graph can represent structural patterns using motifs. When we provide a motif, we aren't
actually inquiringfor data, but rather patterns in the data. Similar to Neo4J's Cypher language,
we define our query in GraphFrames using a domain-specific language. With the help of this
language, we may name and express combinations of vertices and edges. For instance, we
would express (a)- [ab]->(b) if we wanted to specify that a certain vertex a relates to another
vertex b by an edge ab. The names enclosed in parentheses or brackets do not represent
values; rather, they indicate what the columns for matching vertices and edges in the final
DataFrame should be called. If we don't want to query the outcome values, we can skip the
names (for example, (a)-[]->()).
Let's do a search on the bike data. Find all the rides that connect three stations in a "triangle"
layout, to put it simply. We do this by utilising the following motif and the find method to search
for that pattern in our GraphFrame. The initial station (a) denotes the starting point, while the
edge [ab] denotes the location of our subsequent station (b). Then, from station (c) to station
(a), we repeat this for stations (b) to (c):
// in Scala
val motifs = stationGraph.find("(a)-[ab]->(b); (b)-[bc]->(c); (c)-[ca]->(a)")
# in Python motifs = stationGraph.find("(a)-[ab]->(b); (b)-[bc]->(c); (c)-[ca]->(a)")
This query returns a DataFrame with nested fields for vertices a, b and c as well as the
corresponding edges. Now, we can query this just like a DataFrame. What is the quickest route
a specific bike has taken to get from station a to station b to station c and back to station a, for
instance? Our timestamps will be converted into Spark timestamps using the following logic and
after that, we'll compare them to ensure that the bike is the same one that is riding from station
to station and that the start times for each journey are accurate.
// in Scala
import org.apache.spark.sql.functions.expr
motifs.selectExpr("*",
"to_timestamp(ab.`Start Date`, 'MM/dd/yyyyHH:mm') as abStart",
"to_timestamp(bc.`Start Date`, 'MM/dd/yyyyHH:mm') as bcStart",
"to_timestamp(ca.`Start Date`, 'MM/dd/yyyyHH:mm') as caStart")
.where("ca.`Bike #` = bc.`Bike #`").where("ab.`Bike #` = bc.`Bike #`")
.where("a.id != b.id").where("b.id != c.id")
.where("abStart<bcStart").where("bcStart<caStart")
.orderBy(expr("cast(caStart as long) - cast(abStart as long)"))
.selectExpr("a.id", "b.id", "c.id", "ab.`Start Date`", "ca.`End Date`")
.limit(1).show(false)
# in Python
from pyspark.sql.functions import expr
motifs.selectExpr("*",
"to_timestamp(ab.`Start Date`, 'MM/dd/yyyyHH:mm') as abStart",
"to_timestamp(bc.`Start Date`, 'MM/dd/yyyyHH:mm') as bcStart",
"to_timestamp(ca.`Start Date`, 'MM/dd/yyyyHH:mm') as caStart")\
.where("ca.`Bike #` = bc.`Bike #`").where("ab.`Bike #` = bc.`Bike #`")\
.where("a.id != b.id").where("b.id != c.id")\
.where("abStart<bcStart").where("bcStart<caStart")\
.orderBy(expr("cast(caStart as long) - cast(abStart as long)"))\
.selectExpr("a.id", "b.id", "c.id", "ab.`Start Date`", "ca.`End Date`")
.limit(1).show(1, False)
We can observe that the shortest travel takes around 20 minutes. Pretty quick for three different
people riding the same bike, we presume!
It is important to acknowledge that, in this particular instance, the triangles obtained from our
motif query have to undergo a filtration process. To ensure the use of separate vertices, it is
advisable to employ this filtering method. In most cases, the query's different vertex IDs are not
obligated to correspond to unique vertices. One of the most powerful features of GraphFrames
is its ability to combine motif recognition with DataFarme queries on the created tables in order
to further refine, organise, or summarise the identified patterns.
Graph Algorithms
Just a logical representation of data is what a graph is. Numerous techniques from graph theory
are available for evaluating data in this format andGraphFrames enables us to use many of
these algorithms right away. This list will probably keep growing as GraphFrames is developed
and new algorithms are added to it.
PageRank
One of the most prolific graph algorithms is PageRank. Larry Page, cofounder of Google,
created PageRank as a study project on web page rankings. Unfortunately, a thorough
explanation of PageRank's operation is outside the purview of this topic. The broad explanation
is as follows:
In order to assess the significance of a website, PageRank algorithm evaluates both the number
and quality of hyperlinks directed towards a certain webpage. The fundamental proposition
posits that websites of greater significance are likely to receive a higher number of backlinks
from websites of lesser importance.
The generalizability of PageRank outside the online domain is quite good. We can use our own
data to apply this technique and identify significant bike stations (particularly, those that get a lot
of bike traffic). In this illustration, significant bike stations will have high PageRank values:
// in Scala
import org.apache.spark.sql.functions.desc
val ranks = stationGraph.pageRank.resetProbability(0.15).maxIter(10).run()
ranks.vertices.orderBy(desc("pagerank")).select("id", "pagerank").show(10)
# in Python
from pyspark.sql.functions import desc
ranks = stationGraph.pageRank(resetProbability=0.15, maxIter=10)
ranks.vertices.orderBy(desc("pagerank")).select("id", "pagerank").show(10)
The majority ofGraphFrames' algorithms can be accessed using methods that accept
arguments, as resetProbability in the PageRank example above. Most algorithms either provide
a fresh GraphFrame or a single DataFrame as their output. In the vertices, edges, or data frame
of the GraphFrame, the algorithm's output is kept as one or more columns. The algorithm for
PageRank returns a GraphFrame, from which we may extract the estimated PageRank values
for each vertex.
In-Degree and Out-Degree Metrics
We have a directed graph here. The reason for this is that the bike journeys are directional,
beginning in one place and ending in another. Counting the travels into or out of a particular
station is a frequent chore. As shown in the accompanying Figure, we will use a metric known
as in-degree and out-degree, respectively, to count journeys into and out of stations.
Because some users may have much more incoming connections (i.e., followers) than outgoing
connections (i.e., individuals they follow), this is especially relevant in the context of social
networking. You can search the social network for interesting users who might be more
influential than others by using the query below. There is an easy approach to query our graph
for this data using GraphFrames:
// in Scala
valinDeg = stationGraph.inDegrees
inDeg.orderBy(desc("inDegree")).show(5, false)
# in Python
inDeg = stationGraph.inDegrees
inDeg.orderBy(desc("inDegree")).show(5, False)
In response to a search for the stations, which were sorted by highest in-degree:
The same method can be used to query the out degrees:
// in Scala
valoutDeg = stationGraph.outDegrees
outDeg.orderBy(desc("outDegree")).show(5, false)
# in Python
outDeg = stationGraph.outDegrees
outDeg.orderBy(desc("outDegree")).show(5, False)
It's interesting to look at this measure, which is the ratio of these two numbers. A higher ratio
value will show us where many trips end (but few start), whereas a lower value will show us
where many journeys start (but few end):
// in Scala
valdegreeRatio = inDeg.join(outDeg, Seq("id"))
.selectExpr("id", "double(inDegree)/double(outDegree) as degreeRatio")
degreeRatio.orderBy(desc("degreeRatio")).show(10, false)
degreeRatio.orderBy("degreeRatio").show(10, false)
# in Python
degreeRatio = inDeg.join(outDeg, "id")\
.selectExpr("id", "double(inDegree)/double(outDegree) as degreeRatio")
degreeRatio.orderBy(desc("degreeRatio")).show(10, False)
degreeRatio.orderBy("degreeRatio").show(10, False)
These queries yield the following information:
Breadth-First Search
Using the edges in the graph, breadth-first search will look for ways to connect two sets of
nodes in our graph. The approach also functions for sets of nodes supplied using a SQL
expression, however in our situation, we might wish to do this to get the shortest routes to
various stations. With the maxPathLength parameter, we may set the maximum number of
edges to follow. Additionally, we can specify an edgeFilter to exclude edges that don't comply
with a criteria, such as travel during non-business hours.
To save this from taking too long, we'll pick two stations that are quite close to one another.
However, when you have sparse graphs with far-flung links, you can perform intriguing graph
traversals. Feel free to experiment with the stations to see if you can get far-off stations to link,
especially those in other cities:
// in Scala
stationGraph.bfs.fromExpr("id = 'Townsend at 7th'")
.toExpr("id = 'Spear at Folsom'").maxPathLength(2).run().show(10)
# in Python
stationGraph.bfs(fromExpr="id = 'Townsend at 7th'",
toExpr="id = 'Spear at Folsom'", maxPathLength=2).show(10)
Connected Components
As shown in the below Figure, a connected component defines a (undirected) subgraph that is
connected to other connected components but not to the larger graph.
ince they presume an undirected graph, the linked components algorithm does not immediately
apply to our current issue. However, we may still use the technique because it just makes the
assumption that our edges have no directionality. In fact, if we examine the bike sharing map,
we anticipate seeing two separate connected parts.
Figure: A map of Bay Area bike share locations
Setting a checkpoint directory that will save the job's status at each iteration is necessary in
order to run this algorithm. If the job crashes, you may pick up where you left off thanks to this.
Expect delays because this is likely one of the more expensive processes in GraphFrames right
now.
To run this method on your local machine, you will probably need to sample the data, like we do
in the following code example (sampling can assist you get a result without causing the Spark
program to crash due to garbage collection issues).
// in Scala
spark.sparkContext.setCheckpointDir("/tmp/checkpoints")
# in Python
spark.sparkContext.setCheckpointDir("/tmp/checkpoints")
// in Scala
valminGraph = GraphFrame(stationVertices, tripEdges.sample(false, 0.1))
val cc = minGraph.connectedComponents.run()
# in Python
minGraph = GraphFrame(stationVertices, tripEdges.sample(False, 0.1))
cc = minGraph.connectedComponents()
This query yields two related elements, though perhaps not the ones we might anticipate. We
might need more computing power to conduct a more thorough investigation because our
sample might not have all the necessary facts or information:
// in Scala
cc.where("component != 0").show()
# in Python
cc.where("component != 0").show()
Strongly Connected Components
Strongly linked components, which considers directionality, is another related approach that
relates to directed graphs and is included in GraphFrames. A subgraph that contains pathways
connecting every pair of vertices is known as a strongly connected component.
// in Scala
valscc = minGraph.stronglyConnectedComponents.maxIter(3).run()
# in Python
scc = minGraph.stronglyConnectedComponents(maxIter=3)
scc.groupBy("component").count().show()
A high-level API for working with graph data is provided by the library GraphFrames for Apache
Spark. On distributed Spark clusters, it is intended to handle large-scale graph processing and
analytics well. By expanding on Spark's DataFrame API, GraphFrames makes it simple to
combine graph data with structured and semi-structured data.
GraphFrames' salient attributes include:
·
Graph Representation: The vertices and edges of a graph are represented as rows in a
distributed DataFrame object by GraphFrames. Utilising Spark's DataFrame transformations
and actions for graph processing is possible with this structure.
·
Graph Operations: A wide range of graph operations and algorithms are available with
GraphFrames, including conducting graph queries, computing linked components, shortest
paths and PageRank, among others.
·
Seamless Integration: Since GraphFrames are built on top of DataFrames, it is simple to
incorporate them into pipelines for Spark-based data processing. Using Spark's unified API, this
integration enables users to study graph data alongside other structured data.
·
·
Graph Querying: using the help of GraphFrames' support for Graph Pattern Matching,
users can write graph queries using a syntax similar to SQL in order to extract subgraphs or
look for patterns in the graph data.
·
Property Graph Model: The Property Graph Model, which uses data frames with userdefined characteristics to represent both vertices and edges, is what GraphFrames conform to.
Flexible and extendable graph data representation is possible with this architecture.
·
Performance: GraphFrames are designed to take full advantage of Spark's distributed
processing capabilities, making it possible to analyse graphs effectively on massive data sets.
·
Graph Visualisation: Although GraphFrames primarily concentrate on graph processing,
they can be used in conjunction with other tools and libraries for graph visualisation, which
makes it simpler to comprehend and examine the structure and properties of the graph.
Numerous graph-related use cases, such as social network analysis, recommendation systems,
fraud detection, transportation and logistics optimisation and others, make extensive use of
GraphFrames.
You must include the GraphFrames package in your Spark environment in order to use
GraphFrames in your Spark application. On the official GraphFrames GitHub repository, located
at https://github.com/graphframes/graphframes, you can find the most recent version and
installation guidelines.
As soon as the library is installed, you may use the GraphFrames API to interact with graph
data by importing the required classes and functions into your Spark application. More
information on how to use the library efficiently for various graph processing tasks can be found
in the official documentation and examples on theGraphFrames GitHub repository.
ETL Workloads
The ETL landscape for data engineering, data science and machine learning has been changed
by business demands from various industries. Extraction, transformation and loading, or ETL, is
a time-consuming procedure. ETL pipelines must churn and load data continuously in order to
swiftly produce useable data.
The framework to up the ETL game is provided by Apache Spark. Organisations may automate
data pipelines to make data-driven choices more quickly. They are a crucial component of a
successful ETL process because they enable efficient and precise data aggregation from many
sources.
The ability of Spark to accommodate numerous data sources and programming languages was
well known. Spark ETL provides clean data for all types of data, including relational and semistructured data (like JSON).
Spark data pipelines are built to process massive volumes of data.
Extracting data from diverse sources, transforming it into a format that is consistent and useable
and then loading it into a target database or data warehouse for analysis and reporting is known
as ETL (Extract, Transform, Load). ETL is a technique used in data integration and data
warehousing. These three key processes are carried out as part of ETL tasks, which are crucial
for maintaining and getting data ready for business intelligence, analytics and decision-making.
The ETL process is broken down into the following steps:
Extract:
·
Data is sourced from several dissimilar systems, databases, applications, or external
sources during the extraction step. Relational databases, flat files, APIs, online services, cloud
storage and other resources are examples of these sources.
·
The objective is to obtain the necessary data from the source systems and transfer it to a
staging area or short-term storage for additional processing.
Transform:
·
To make the extracted data correct, consistent and acceptable for analysis, it is subjected
to a number of data cleansing, enrichment and modification processes during the transformation
step.
·
Data cleansing, data normalisation, data aggregation, data filtering, data enrichment
(adding computed fields or combining data from several sources) and other processes are
examples of data transformations.
·
Before loading the data into the intended location, transformations are carried out to
guarantee its integrity and quality.
Load:
·
The transformed and processed data is loaded into the intended location—a data
warehouse, data mart, or database—during the load phase for reporting and analysis.
·
The target destination often supports efficient data querying and reporting and has a
schema that is optimised for analytics.
·
The loading procedure could entail full refresh (replacing all data), incremental loading
(updating only new or updated data), or a hybrid of the two.
ETL workloads are typical across a range of data-driven applications and sectors. They are
necessary for combining, cleansing and putting data from various sources into a dependable
and uniform format for analysis. The ETL process is a vital step in the data integration and data
warehousing pipeline because it guarantees that the data utilised for reporting and analytics is
consistent, correct andup-to-date.
Organisations frequently employ ETL tools and platforms, which offer visual interfaces and
automated capabilities to design and carry out complicated ETL workflows, in order to carry out
ETL tasks effectively. With the use of these technologies, the ETL process can be streamlined,
manual work can be decreased and timely high-quality data delivery can be ensured to enable
business intelligence and data-driven insights.
The act of obtaining data from a source system is known as extraction. Using something like
JDBC, you could fetch files on demand or receive files according to a timetable. Typically, the
outcome is kept in a space known as the staging area.
Cleaning up and reorganising the data to meet your needs is transformation. For example, you
could change your tables' third normal form to a star structure, which is better suited for
conventional data warehousing.
The new data must then be loaded into the target system in a way that maintains
synchronisation. There are numerous particular situations and considerations. You'll need to
employ a number of the toolbox's tools in order to do all of this work effectively. To manage all
these tasks, specialised ETL tools are frequently utilised. You may have used software
produced by organisations like Informatica, IBM and Oracle, to name a few. There are also a
ton of open source options accessible. As you may have guessed by this point, you can pretty
much perform anything that you can do with these graphical tools from the prompt as well.
Cleaning and transforming data
Given all the publicity around data science, one may assume that the majority of the work done
in the subject involves creating innovative algorithms and using them while reciting spells, much
like a data Gandalf or Harry Potter.
Not really, regrettably. The majority of effort in the field of data analysis still involves
comprehending the data and getting it ready for use. No matter who will utilise information in the
end—humans or algorithms—it needs to be organised into tidy tables.
You'll encounter a colossal number of challenges along the road. Dealing with inaccurate
metadata, null values and poor data are a few examples. The format, structure, or content must
then typically be altered in some way.
To learn how to remove errors, let's create a DataFrame that contains them. We employ the
createDataFrame method to accomplish that. The first list includes the data and the second one
either defines the schema or is simply a list of column names, as in this instance. The latter
situation will result in the real schema being implied:
df = spark.createDataFrame(
[
('Store 1',1,448),
('Store 1',2,None),
('Store 1',3,499),
('Store 1',44,432),
(None,None,None),
('Store 2',1,355),
('Store 2',1,355),
('Store 2',None,345),
('Store 2',3,387),
('Store 2',4,312)
],
['Store','WeekInMonth','Revenue']
)
You'll see that there are a few mistakes in this. Since Python uses None values for Null, we
have a handful of those. The fourth row of the first store then has a poor value.
Finding nulls
Let's start by trying to identify the various problems. To begin, let's look for null values. For
instance, there is a cute little function called isNull. If you use it as a filter, all the rows with null
values will be returned. Try it now:
display(df.filter(df.Revenue.isNull()))
This is good, because in this example, we can use all three columns to search for data by doing
this. It's less enjoyable if you have 100 columns, though. The number of nulls for each column
might then be a preferable place to look first. There are several ways to accomplish this;
however, let's use SQL functions:
from pyspark.sql.functions import count, when, isnull
display(df.select(
[count(when(isnull(c), c)).alias(c) for c in df.columns]
))
We import the functions we require first. To find the none values we're seeking for, we need
isnull. When is a conditional check and then count aids in maintaining the total. Following that,
we count the nulls in each of the columns using a loop. We employ the alias command to
preserve the column names for the aggregates:
from pyspark.sql.functions import col
cols = [c for c in df.columns if df.filter(col(c).isNull()).count() > 0 ]
This is a different approach to the same issue. The result will list all columns with at least one
null value. The list will be in the cols variable, allowing you to use df.select(cols) or do logic
operations.
You should probably look at the actual rows with the missing data if there aren't a ton of nulls.
You could construct a query and manually apply filters for each column in the cols list, but
reduction in Python can handle it for you instead.
from functools import reduce
display(df.filter(reduce(lambda a1, a2: a1 | a2, (col(c).isNull() for c in
cols))))
We import yet another function this time. Each value in a list has a function that is run by
Reduce. Create a function that adds to numbers, for instance; apply reduce to a list of 1, 2, 3
and 4 and it will produce 10.
In this instance, the call's outcome will be either ((Day IS NULL) OR ((Revenue IS NULL)).
Then, filter receives this and executes the statement, returning the rows that match our
argument. Of course, you can do this directly across all columns:
display(df
.filter(
reduce(lambda a1, a2: a1 | a2, (col(c).isNull() for c in
df.columns))
)
)
Getting rid of nulls
Numerous programmatic possibilities are available to you. However, none of them are excellent.
They can't be since you're either discarding data or estimating values in some other manner.
Basically, you have to choose your poison based on the information you have. Consider a
couple of the instruments in your toolbox.
Simply eliminating the rows containing null values is the simplest course of action. Although
erasing data is never a pleasant experience, it is frequently a quick and effective fix. If you have
100 million rows, for example, losing a few hundred rows is probably not a big deal. Most of the
time, the outcome will be near enough.
Dropna is the command for this extreme row elimination. Let's examine what occurs when the
command is executed immediately. Any row that has at least one field that is null will be
removed as a result:
df2 = df.dropna()
display(df2)
Be aware that this command, like numerous others, will not function in place. Instead, a
DataFrame is returned. If you want to maintain both versions, you must assign the result back to
the original DataFrame or a new one. Nothing will happen if you simply run dropna without
giving it a task.
As you'll see, df2 has lost a few rows. The command mostly performed as expected. This is a
fairly simple method for getting rid of potentially problematic rows. This might be a viable
solution if a large dataset only contains a few rows. Using dropna, you may also clean off only
rows that are entirely empty. This occasionally occurs when line breaks are added by exporting
tools at a specific number of rows. Simply set the first argument to "all" to clear them.
Otherwise, "any" is the default:
df2 = df.dropna('all')
display(df2)
You can tell dropna to only look at a subset of the rows if you don't want to eliminate rows just in
case a certain column contains null values. This command will eliminate any rows with null
values in the two specified columns, but it will leave nulls in the Revenue column:
df2 = df.dropna(how='any', subset=['Store','WeekInMonth'])
display(df2)
Another parameter is available and it occasionally has a place. If you have a large number of
columns, you can set the minimum number of valid column values needed to maintain each
column. It is known as thresh. If you do, the first argument won't matter whether you type "any"
or "all" because it will be disregarded:
display(df.dropna(thresh = 2))
display(df.dropna(thresh = 3))
display(df.dropna(thresh = 4))
The first command only removed the row with all null values. We have at least two filled-in fields
in the remaining rows, which is the cause. All of our nulls vanish when the requirement is
increased to three. The following one is fascinating. Due to the fact that our table has only three
columns, it will actually eliminate every row. Take note of this minor feature.
Spark ML Fundamentals
These days, machine learning is receiving a lot of attention, but what precisely is it? Machine
learning, in general, is a method for identifying patterns in data by employing statistics, linear
algebra and numerical optimisation. Machine learning can be used to solve issues like
forecasting power use, figuring out if your movie has a cat and grouping objects with similar
properties.
A part of Apache Spark called Spark MLlib (Machine Learning Library) offers a collection of
machine learning tools and methods for processing massive amounts of data and distributed
machine learning. Because Spark MLlib is built to smoothly integrate with Apache Spark's
distributed computing capabilities, it is well suited for handling massive data and challenging
machine learning workloads. Here are some Spark MLlib fundamentals:
·
High-Level API: DataFrame-based API, a high-level API provided by Spark MLlib and built
upon Spark DataFrames, is available. Working with structured data and machine learning
techniques is made simple by this API.
·
Distributed Computing: By utilising Spark's distributed computing capabilities, Spark
MLlibis able to handle huge datasets quickly over a cluster of computers.
·
Algorithms and Pipelines: Machine learning algorithms like classification, regression,
clustering, collaborative filtering, dimensionality reduction and others are available through
MLlib. These techniques make it simple to incorporate feature engineering, model training and
data preparation into machine learning workflows.
·
Feature Transformers: Tokenisation, one-hot encoding, vectorisation and other feature
engineering operations are among the feature transformers available in MLlib for preprocessing
data and engineering features.
·
Model Selection and Evaluation: Cross-validation and hyperparameter tuning are only two
of the model evaluation and selection techniques that MLlib offers to assist users in selecting
the top models for their datasets.
·
Data Types: Both labeled and unlabeled data formats are supported by MLlib. While
unlabeled data is utilised for unsupervised learning tasks like clustering, labeled data is used for
supervised learning tasks like classification and regression.
·
Integration with Spark Ecosystem: Users may complete end-to-end data processing and
machine learning operations in a single environment because to Spark MLlib's easy integration
with other Spark components like Spark SQL and Spark Streaming.
·
Performance and Scalability: MLlib is appropriate for handling huge datasets and
distributed machine learning tasks since it is built to scale effectively across a cluster of
computers.
It's vital to keep in mind that Apache Spark offers two machine learning libraries: MLlib (a
DataFrame-based API) and Spark ML (a Pipeline-based API). While both libraries provide
comparable capabilities, Spark ML (debuted in Spark 2.0) is being developed more actively and
is the API of choice for new projects.
Overall, Spark MLlib is a fantastic choice for large data and distributed computing environments
since it is a strong and adaptable library that makes distributed machine learning operations
simpler. In order to create scalable and effective machine learning pipelines to process and
analyse large datasets, it enables data scientists and developers to take advantage of Spark's
strengths.
Unsupervised, supervised and reinforcement learning are a few of the several types of machine
learning. Let's quickly go through the distinctions between supervised and unsupervised
machine learning before moving on.
Supervised Learning
The primary goal of supervised machine learning is to make predictions about the output
label(s) based on a fresh input that lacks labels. The dataset comprises a sequence of input
records, wherein each record is accompanied by corresponding labels. This leads us to the two
distinct areas within the realm of supervised machine learning, namely classification and
regression. The output labels in question have the potential to be either discrete or continuous
in nature.
The objective of a classification problem is to categorise the given inputs into a discrete set of
classes or labels. In the context of binary classification, it is necessary to make a selection
between two distinct labels, such as "dog" or "not dog," as depicted in the accompanying
diagram.
Multiclass classification, often referred to as multinomial classification, allows for the prediction
of three or more discrete labels, such as the breed of a dog.
The value to predict in regression issues is a continuous integer, not a label. Accordingly, as
illustrated in the Figure below, you might forecast values that your model hasn't encountered
during training. For instance, you might create a model to forecast daily ice cream sales based
on the weather. Even if none of the input/output pairings that your model was trained on
contained the value $77.67, it might nonetheless correctly predict that value.
Unsupervised Learning
The labeled data needed for supervised machine learning can be extremely expensive or even
impossible to get. Unsupervised machine learning is useful in this situation. Unsupervised
machine learning enables you to comprehend the structure of your data better rather than
making a label prediction.
Take the original unclustered data on the left in the Figure below as an illustration. For each of
these data points (x1, y1), there is no known true label, but by using unsupervised machine
learning to our data, we can uncover the clusters that arise organically, as seen on the right.
Unsupervised machine learning has the potential to serve as a valuable tool for outlier discovery
or as a first stage in the process of supervised machine learning. For example, it can be
employed to decrease the dimensionality of the dataset, so aiding in reducing storage demands
or optimising subsequent activities. K-means, Latent Dirichlet Allocation (LDA) and Gaussian
mixture models are examples of unsupervised machine learning techniques utilised within the
MLlib framework.
Spark offers a comprehensive framework that facilitates data ingestion, feature engineering,
model training and deployment within an environment conducive to unified analytics. In the
absence of Spark, program rs would need to utilise a diverse range of disparate technologies in
order to successfully execute this set of processes, potentially encountering challenges related
to scalability.
Spark.mllib and spark.ml are two machine learning packages that are accessible for use. The
initial machine learning API, spark.mllib, was developed using the RDD API, which has been in
maintenance mode since the release of Spark 2.0. In contrast, the latest API, spark.ml, is built
upon Data Frames. The aggregate term used to refer to both packages of the Apache Spark
machine learning library is "MLlib".
Data scientists now have the ability to prepare their data and build models in a single
environment using spark.ml, eliminating the need to downscale their data to accommodate a
single computer. The Spark.ml library is designed to prioritise O(n) scale-out, meaning that the
size of the model expands in a linear fashion as the number of data points grows. This enables
Spark.ml to effectively manage vast quantities of data. We will discuss the trade-offs involved in
choosing between a distributed framework such as spark.ml and a single-node framework like
scikit-learn (sklearn). A considerable number of the application programming interfaces (APIs)
present in the Spark framework exhibit similarities to those found in scikit-learn, hence
facilitating a sense of familiarity among users.
Designing Machine Learning Pipelines
We will discuss how to build and fine-tune ML pipelines in this part. Many ML frameworks use
the idea of pipelines to arrange a sequence of actions that will be applied to your data. To
arrange your machine learning workflow, MLlib's Pipeline API offers a high-level API built on top
of DataFrames.
We will use the Inside Airbnb housing data set for San Francisco throughout this topic. It
comprises data on San Francisco Airbnb rentals, including the number of bedrooms, location,
review ratings, etc. Our aim is to develop a model to forecast the nightly rental rates for listings
in that city. Price is a continuous variable, hence this is a regression issue. We will walk you
through the process a data scientist would use to solve this problem, which includes feature
engineering, model creation, hyperparameter tuning and model performance evaluation. Like
most real-world data sets, this one is quite messy and can be challenging to model, so if you are
exploring on your own, don't feel guilty if your early models aren't perfect.
Let's establish a few terms related to MLlib before getting into the specifics:
Transformer
The function accepts a DataFrame as an input parameter and generates a new DataFrame with
one or more supplementary columns. Transformers utilise rule-based transformations on data,
serving two purposes: preparing the data for model training and generating predictions using a
learned MLlib model. These transformations do not include the learning of any parameters from
the data. The approach employed by the subject is referred to as "transform()".
Estimator
With the.fit() method, it "fits" (or learns) parameters from your DataFrame and then returns a
Model, a transformer.
Pipeline
assembles a number of estimators and transformers into a single model. Although pipelines are
estimators in and of themselves, pipeline.fit() output.A Pipe lineModel, a transformer.
The code samples and examples in this topic will help you grasp how these ideas fit together,
even though they may currently appear somewhat ethereal. However, we must import our data
and execute some data preparation before we can build our ML models and employ
transformers, estimators and pipelines.
Algorithms that use machine learning produce predictions or judgments based on training data,
frequently aiming to maximize a mathematical objective that describes how the system should
act, distinct learning problems exist, each with a distinct goal, such as classification, regression,
or clustering. We'll use categorisation as a straightforward illustration. This process entails
determining which of several categories an item (like an email) belongs to based on labeled
samples of other objects (like emails that are known to be spam or not).
For each item, a collection of features must be defined for each learning method in order to be
supplied into the learning function. For an email, for instance, some features might be the server
it originates from, the quantity of times the term free is used, or the text's colour. The most
difficult aspect of employing machine learning is frequently selecting the appropriate
characteristics. A product recommendation assignment, for instance, can benefit greatly from
the addition of a single new feature, such as the recognition that the book you should offer to a
user may also depend on the movies she has seen.
The majority of algorithms are only defined for numerical features (more precisely, a vector of
integers reflecting the value for each characteristic), hence feature extraction and modification
to create these feature vectors is frequently a crucial step. There are other approaches for
categorizing text, such as counting the frequency of each word, for example, in our spam versus
non-spam example.
The majority of machine learning algorithms aim to optimise a clearly specified mathematical
function by leveraging data that has been transformed into feature vectors. One possible
approach in classification is to determine a plane within the feature vector space that effectively
separates spam from non-spam samples. The selection of this plane is based on a defined
criterion of effectiveness, such as the proportion of properly categorised points.
The procedure will ultimately yield a model that symbolises the decision made in the learning
process, such as the chosen plane. The current model has the capability to make predictions for
new data points. For instance, it can determine the classification of a new email's feature vector
by identifying which side of the plane it falls on, so indicating whether it is categorised as spam
or not. The provided information presents an exemplar learning pathway.
Finally, because the majority of learning algorithms involve a number of factors that might
influence the outcomes, pipelines used in the real world will train different iterations of a model
and evaluate each one. In order to check whether the model overfits the training set, it is
common practice to segregate the input data into "training" and "test" sets and train solely on
the former. MLlib offers a variety of model assessment strategies.
Example: Spam Classification
We provide a quick demonstration of MLlib using Examples 11-1 through 11-3, which is a
relatively straightforward program for creating a spam classifier. The two MLlib algorithms used
in this application are Logistic RegressionWithSGD, which employs stochastic gradient descent
(SGD) to implement the logistic regression process andHashingTF, which creates term
frequency feature vectors from text data. The two files spam.txt and normal.txt, which each
include instances of spam and non-spam emails, one per line, are what we'll start with. Then,
using TF to create feature vectors from the text in each file, we train a logis­tic regression model
to distinguish between the two types of communications. The book's Git repository contains the
code and data files.
Data Types
MLlib has a few unique data types that are housed in the pyspark.mllib (Python) or
org.apache.spark.mllib (Java/Scala) packages, respectively. Principally, these are:
Vector
a vector in mathematics. Both dense vectors, in which every entry is stored and sparse vectors,
in which only the nonzero entries are stored to conserve space, are supported by MLlib. The
various vector kinds will be covered shortly. The class mllib.linalg.Vectors allows for the
construction of vectors.
LabeledPoint
a tagged data point used by supervised learning algorithms like regression and classification.
includes a label (a floating-point value) and a feature vector. accessible through the
mllib.regression package.
Rating
a user's evaluation of a product, used in the mllib.recommendation package to recommend
products.
Various Model classes
Each model, which is the end result of a training procedure, includes a predict() method that can
be used to apply the model to a single new data point or to an RDD of new data points.
The majority of algorithms operate directly on labeled points, ratings, or RDDs of vectors. You
can create these objects in any way you like, but often you will create an RDD by transforming
external data—for example, by importing a text file or running a Spark SQL command—and
then using a map() to convert your data objects into MLlib types.
Spark ML Regression and Classification
A set of regression analysis techniques are offered by the Spark ML Regression machine
learning library. With the use of one or more independent variables and a statistical technique
called regression analysis, you may predict the value of a dependent variable. A collection of
algorithms for linear regression, decision tree regression, random forest regression and
gradient-boosted tree regression are offered by Spark ML Regression.
Labelled training data is commonly employed in supervised learning to facilitate the prediction of
a variable based on object qualities. Classification and regression are two widely utilised forms
of supervised learning methods in this context. There exists a variation in the type of variable
that is being predicted among them. In the context of classification, the variable under
consideration is discrete, meaning that it possesses a finite number of distinct classes or values.
For example, when categorising emails, the classes may be defined as either spam or
nonspam, or even based on the language employed in the content. Regression is a statistical
technique used to make predictions about a continuous variable, such as an individual's height,
based on other variables such as age and weight.
The LabeledPoint class, located in the mllib.regression package of MLlib, serves the purpose of
facilitating both classification and regression tasks. A Label edPoint consists of a label, which is
a Double value but can be converted to discrete integers for classification and a features vector.
Simple linear algorithms, decision trees and forests are just a few of the classification and
regression techniques found in MLlib.
One of the most used approaches for regression is linear regression, which predicts the output
variable as a linear combination of the features. Additionally, MLlib supports Lasso and ridge
regression, also known as L1 and L2 regularised regression.
The mllib.regression function provides access to the linear regression
techniques.LassoWithSGD, RidgeRegressionWithSGD and Line arRegressionWithSGD
classes. These adhere to a naming convention that is used consistently across MLlib: issues
involving several algorithms have a "With" portion in the class name to indicate the algorithm
that was used. Stochastic Gradient Descent is used here.
Each of these classes has different tuning parameters:
numIterations
Number of iterations to run (default: 100).
stepSize
Step size for gradient descent (default: 1.0).
Intercept
Whether to add an intercept or bias feature to the data—that is, another feature whose value is
always 1 (default: false).
regParam
Lasso and ridge regularisation parameter (default: 1.0).
Each language has a slightly different syntax for calling the algorithms. Create a
LinearRegressionWithSGD object in Java or Scala, use setter methods to set the parameters
and then use run() to train a model. Instead, use the class method LinearRegressionWithSGD in
Python.train(), which accepts key/value input.
Example: . Linear regression in Python
from pyspark.mllib.regression import LabeledPoint
from pyspark.mllib.regression import LinearRegressionWithSGD
points = # (create RDD of LabeledPoint)
model = LinearRegressionWithSGD.train(points, iterations=200, intercept=True)
print "weights: %s, intercept: %s" % (model.weights, model.intercept)
Example. Linear regression in Scala
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.mllib.regression.LinearRegressionWithSGD
val points: RDD[LabeledPoint] = // ...
vallr = new LinearRegressionWithSGD().setNumIterations(200).setIntercept(true)
val model = lr.run(points)
println("weights: %s, intercept: %s".format(model.weights, model.intercept))
Example: Linear regression in Java
import org.apache.spark.mllib.regression.LabeledPoint;
import org.apache.spark.mllib.regression.LinearRegressionWithSGD;
import org.apache.spark.mllib.regression.LinearRegressionModel;
JavaRDD<LabeledPoint> points = // ...
LinearRegressionWithSGDlr =
new LinearRegressionWithSGD().setNumIterations(200).setIntercept(true);
LinearRegressionModel model = lr.run(points.rdd());
System.out.printf("weights: %s, intercept: %s\n",
model.weights(), model.intercept());
Keep in mind that we must call the.rdd() method on our JavaRDD in order to convert it to the
Scala RDD class. Due to the fact thatMLlib functions are intended to be callable from both Java
and Scala, this is a pattern that is used frequently throughout MLlib.
Once trained, a pre dict() function is included in the LinearRegressionModel that is returned in
all languages and may be used to predict a value on a single vector. Both the
RidgeRegressionWithSGD and LassoWithSGD classes exhibit similar behaviour and produce
comparable model classes. In fact, the whole of MLlib follows this pattern of an algorithm with
parameters updated by setters, returning a Model object with a predict() method.
Logistic regression
A linear separation plane between cases that are positive and those that are negative is found
using the binary classification method of logistic regression. A LogisticRegressionModel that can
predict new points is returned by MLlib when given LabeledPoints with label values of 0 or 1.
The linear regression algorithm, which was reviewed in the previous part and the logistic
regression technique have very similar APIs. One distinction is that it can be solved using either
of the two techniques, SGD or LBFGS.Although 4 LBFGS is typically the best option, it is not
included in some earlier versions of MLlib (MLlib versions prior to Spark 1.2). You may find
these algorithms in the mllib.classificationfile.LinearRegressionWithSGD's interface is mirrored
by the classes LogisticRegressionWithLBFGS and WithSGD. They use the same inputs that
linear regression does.
These methods' LogisticRegressionModel calculates a score between 0 and 1 for each point
that the logistic function returns. Then, depending on a user-set threshold, it returns either 0 or
1; by default, if the score is at least 0.5, it returns 1. Through setThreshold(), you can modify this
threshold. The raw scores will be returned by predict() if you completely disable it using
clearThreshold(). We advise leaving the threshold at 0.5 for datasets with an equal number of
positive and negative examples. You can either lower the threshold to reduce the number of
false negatives in imbalanced datasets or raise the threshold to reduce the number of false
positives (i.e., increase precision but decrease recall).
Support Vector Machines
Another binary classification method using linear separation planes that anticipates labels of 0
or 1 is Support Vector Machines, or SVMs. They are accessible via the SVMWithSGD class and
have parameters that are similar to those in linear and logistic regression. The provided
SVMModel makes predictions using a threshold similar to the LogisticRegressionModel.
Naive Bayes
The Naive Bayes multiclass classification technique utilises a linear function of the features to
determine the degree of belongingness of each data point to different classes, resulting in the
assignment of a corresponding score. In addition to its various applications, text categorisation
sometimes employs TF-IDF features. An implementation of MLlib The Multinomial Naive Bayes
algorithm is designed to handle input features that consist of nonnegative frequencies, such as
word frequencies.
The mllib.classification function in MLlib enables the utilisation of Naive Bayes algorithm.The
topic of discussion pertains to the Naive Bayes classification algorithm. The software alone
accepts the smoothing parameter lambda (referred to as lambda_ in Python). The method can
be invoked on a Resilient Distributed Dataset (RDD) consisting of LabeledPoints, where the
labels correspond to C classes and are represented by integers ranging from 0 to C-1.
The NaiveBayesModel provides access to two trained model parameters: theta, which is a
matrix of class probabilities for each feature (with dimensions C×D for C classes and D features)
and pi, which is a C-dimensional vector representing class priors. Additionally, the model allows
for the prediction of the most suitable class for a given data point using the predict() function.
Decision trees and random forests
A versatile model, decision trees can be applied to both classification and regression. They
stand in for a tree of nodes, each of which contains a leaf node that contains a prediction (such
as, "Is the person likely to buy a product?") and makes a binary decision based on a feature of
the data (such as, "Is a person's age greater than 20?"). Decision trees are appealing because
they allow both categorical and continuous data and make it simple to analyse the models. A
sample tree is shown in the figure below.
The mllib.tree function in MLlib can be used to train trees.Through the static methods
trainClassifier() and trainRegressor() of the DecisionTree class. The Java and Scala APIs also
use static methods as opposed to a Deci sionTree object with setters, which is different from
some of the other algorithms. The following factors are considered by the training methods:
data
RDD of LabeledPoint.
numClasses (classification only)
Number of classes to use.
impurity
Node impurity measure; can be gini or entropy for classification and must be
variance for regression.
maxDepth
Maximum depth of tree (default: 5).
maxBins
Number of bins to split data into when building each node (suggested value: 32).
categoricalFeaturesInfo
A map specifying which features are categorical and how many categories they
each have. For example, if feature 1 is a binary feature with labels 0 and 1 and
feature 2 is a three-valued feature with values 0, 1 and 2, you would pass {1: 2,
2: 3}. Use an empty map if no features are categorical.
The employed algorithm is fully explained in the online MLlib manual. The algorithm's cost
increases linearly as the number of training samples, the number of features andmaxBins
increase. In order to train a model more quickly for huge datasets, you might want to reduce
maxBins; however, doing so will compromise model quality. A DecisionTreeModel is the result
of the train() methods. With the help of predict() or toDebugString(), you can use it to output the
tree or predict values for a brand-new feature vector or an RDD of vectors. Because this object
can be serialised, you can use Java Serialisation to save it and load it into another program.
Finally, to create ensembles of trees, sometimes referred to as random forests, Spark 1.2
introduces MLlib's experimental RandomForest class in Java and Scala. It is accessible via
RandomForest.trainRegressor and trainClassifier. Along with the previously mentioned pertree
options, RandomForest also accepts the following ones:
numTrees
How many trees to build. Increasing numTrees decreases the likelihood of over‐
fitting on training data.
featureSubsetStrategy
Number of features to consider for splits at each node; can be auto (let the
library select it), all, sqrt, log2, or onethird; larger values are more expensive.
seed
Random-number seed to use.
Random forests yield a WeightedEnsembleModel that can predict() an RDD or Vector and
contains many trees (in the weakHypotheses field, weighted by weakHypothesisWeights). To
output all the trees, it also has a toDebugString function.
Spark ML Clustering
The unsupervised learning problem of clustering entails placing things in groups with a high
degree of similarity. Clustering can be used to make sense of unlabeled data, unlike the
supervised tasks we've encountered earlier where the data is labeled. It is frequently used in
anomaly detection (to identify points that are far from any cluster) and data exploration (to
discover how a new dataset appears).
K-means
The K-means algorithm, a widely recognised clustering method, is incorporated in MLlib.
Additionally, MLlib has a variant known as K-means||, which provides enhanced initialisation in
parallel scenarios. The initialisation procedure for K-means++, commonly utilised in single-node
settings, is indistinguishable from that of Kmeans||.
The selection of the optimal number of clusters, denoted as K, is a pivotal component in the Kmeans clustering algorithm. It is considered optimal to engage in the exploration of various
values of K until the average intracluster distance exhibits a significant decline, as the precise
number of clusters is often unknown beforehand. However, the approach is limited to
processing only one K at a time. In the MLlib library, the K-means algorithm necessitates the
inclusion of the following parameters, in addition to the value of K:
initialisationMode
The method for initializing cluster centers, which can be either "k-means||" or "random," typically
produces better results but is a little more expensive.
maxIterations
Maximum number of iterations to run (default: 100).
Runs
number of concurrent algorithm runs to be performed. Because K-means runs can stall in local
minima, MLlib's K-means support allows running from many starting places concurrently and
selecting the best result.
K-means is activated similarly to other algorithms by creating a mllib.clustering.KMeans object
in Python or Java, or by invoking KMeans.train in Scala or Java. An RDD of vectors are
required. If you use predict() on a new vector, K-means will return a KMeansModel that allows
you to access the clusterCenters (as an array of vectors) and return the cluster for that vector.
Even though a point is far from all clusters, predict() will always return the center that is closest
to it.
Collaborative Filtering and Recommendation
Collaborative filtering is a method for recommender systems that uses user feedback and
interactions to suggest new products. The appeal of collaborative filtering is that it just needs to
take into account a list of user/product interactions, whether they are "explicit" interactions (such
as ratings on a shopping website) or "implicit" interactions (such as when a user browses a
product page but does not rate the product). Collaborative filtering algorithms can generate
fresh recommendations by learning which items are similar to one another (since the same
users engage with them) and which users are similar to one another.
While the MLlib API refers to "users" and "products," collaborative filtering can be used for a
variety of purposes, including suggesting persons to follow on social media, tags to add to
articles, or music to add to radio stations.
Alternating Least Squares
The implementation of Alternating Least Squares (ALS), a widely recognised collaborative
filtering approach that exhibits good scalability on clusters, may be found in MLlib.6. The
location of the file is within the mllib.recommendationmodule.Amyotrophic lateral sclerosis (ALS)
group.
To ensure that the dot product between a user's vector and a product's vector closely
approximates the score, the Alternating Least Squares (ALS) algorithm computes a feature
vector for each user and product. The aforementioned parameters are utilised.
Rank: Useful feature vector size; higher ranks can result in more accurate predictions, but they
cost more to compute (default: 10).
Iterations: Number of iterations to run (default: 10).
Lambda: Regularisation parameter (default: 0.01).
Alpha: A constant used to calculate implicit ALS confidence (default: 1.0).
numUserBlocks, numProductBlocks: Number of blocks to divide user and product data into to
control parallelism; by default, MLlib will choose this number automatically, but you can give -1
instead.
In order to utilisemllib.recommendation, you must give ALS an RDD.Every rating object has
three identifiers: a user ID, a product ID and a rating, which can be either an explicit rating or
implicit feedback; see the discussion that follows. The requirement that each ID be a 32-bit
integer presents challenges for implementation. If your IDs are strings or larger numbers, it is
advisable to only use the hash code of each ID in ALS; even if two people or things map to the
same ID, the aggregate results can still be good. As an alternative, broadcast() a table of
product-ID to integer mappings to provide them unique IDs.
A MatrixFactorisationModel, which can be used to forecast() ratings for an RDD of (userID,
productID) pairs, represents the outcomes of ALS.7 Instead, utilise model to get the top
numProducts() products that are suggested for a specific user.userId, numProducts) in
recommendProducts. In contrast to other models in MLlib, the MatrixFactorisationModel is
enormous and contains one vector for each user and product. This means that it cannot be
loaded once during a program run and then saved to disk. Instead, you can save the
model.userFeatures and model.productFeatures RDDs of the feature vectors it produces to a
distributed filesystem.
Last but not least, ALS comes in two flavors: explicit ratings (the default) and implicit ratings
(which are enabled by calling ALS.trainImplicit() rather than ALS.train()). With explicit ratings,
each user must assign a score (such as 1 to 5 stars) to each product they review and the
anticipated ratings will also be scores. Each rating in implicit feedback is a measure of how
confident the user is that they will interact with a certain item (e.g., the rating may increase the
more times a user visits a website) and the predicted items are measured by confidence values.
In Hu et al., "Collaborative Filtering for Implicit Feedback Datasets," ICDM 2008, additional
information about ALS with implicit ratings is provided.
Dimensionality Reduction
Principal component analysis
There is a recurring interest in reducing the dimensionality of data points inside a dataset that
exists in a space with a high number of dimensions. This reduction enables the utilisation of
simpler analytical techniques for studying the data. In order to enhance the efficacy of model
training, one potential approach could involve the utilisation of two-dimensional point plotting or
the reduction of feature dimensions.
Principal component analysis (PCA) is widely regarded as the predominant technique utilised by
the machine learning community for the purpose of reducing dimensionality. The approach
described disregards dimensions that do not provide meaningful information by optimising the
mapping to a lower-dimensional space such that the variance of the data in the reduced
representation is maximised.
The construction of the normalised correlation matrix of the data is followed by the utilisation of
the singular vectors and values of this matrix to compute the mapping. A considerable
proportion of the variability in the initial dataset is captured by the singular vectors associated
with the largest singular values.
As ofMLlib 1.2, PCA is presently only supported in Java and Scala. To use it, you must first use
the mllib.linalg.distributed to represent your matrix.The class RowMatrix contains an RDD of
Vectors, one for each row.
Example: PCA in Scala
import org.apache.spark.mllib.linalg.Matrix
import org.apache.spark.mllib.linalg.distributed.RowMatrix
val points: RDD[Vector] = // ...
val mat: RowMatrix = new RowMatrix(points)
val pc: Matrix = mat.computePrincipalComponents(2)
// Project points to low-dimensional space
val projected = mat.multiply(pc).rows
// Train a k-means model on the projected 2-dimensional data
val model = KMeans.train(projected, 10)
The projected RDD in this illustration comprises a two-dimensional representation of the original
points RDD and can be used for graphing or running other MLlib algorithms, including K-means
clustering.
It should be noted that the utility class mllib.linalg.Matrix, which is identical to Vector in that it
represents dense matrices, is what computePrincipalComponents() produces. With toArray, you
can access the underlying data.
Singular value decomposition
The lower-level singular value decomposition (SVD) primitive is also made available by MLlib.
An m×n matrix A is factorised by the SVD into three matrices A ≈ UΣV , where:
·
The matrix U is an orthonormal matrix, with its columns referred to as left singular vectors.
·
The matrix Σ is a diagonal matrix where the diagonal elements are nonnegative and
arranged in descending order. These diagonal elements are commonly referred to as singular
values.
·
The matrix V is an orthonormal matrix, with its columns referred to as the right singular
vectors.
We typically just require the top singular values and their related singular vectors for big
matrices, not the entire factorisation. This can reduce storage requirements, remove noise and
restore the matrix's low-rank structure. The size of the generated matrices will be if we maintain
the top ksingular values. U: m × k, Σ : k × k and V : n × k.
For unsupervised machine learning applications, where the objective is to cluster data points
based on their similarities, Spark MLlib offers a number of clustering techniques. Customer
segmentation, anomaly detection and pattern recognition are just a few examples of the many
applications that clustering is employed in. In Spark MLlib, the following clustering techniques
are frequently used:
·
K-Means: One of the most well-liked clustering techniques is K-Means. K is a user-defined
parameter and it divides the data into K clusters. The algorithm updates the centroids based on
the mean of the data points assigned to each cluster and iteratively assigns data points to the
nearest cluster centroid. The goal of K-Means is to reduce the sum of squared distances
between each data point and its associated centroid.
·
Bisecting K-Means: It employs a top-down strategy to iteratively divide clusters into subclusters until the desired number of clusters is obtained. This algorithm is an extension of the KMeans algorithm. Due to the fact that it eliminates the need to repeat the algorithm numerous
times for various K values, it may be more effective for huge datasets.
·
Gaussian Mixture Model (GMM):A probabilistic model called a GMM posits that data points
are produced by combining several Gaussian distributions. In order to increase the likelihood of
the data given the model, it assigns probabilities of data points belonging to each cluster and
iteratively changes the model parameters.
·
Latent Dirichlet Allocation (LDA):A probabilistic generative model called LDA is employed
in topic modelling. Although its main application is topic modelling, it may also be used to group
text data according to how the words are distributed in the documents.
·
Power Iteration Clustering (PIC):The power iteration method is used by PIC, a scalable
graph clustering algorithm, to find clusters in graphs or network data.
·
Streaming K-Means:By adding the ability to handle streaming data—data that comes in a
continuous stream—this method expands K-Means. Without having to reprocess old data, it
effectively updates cluster centroids when new data points are received.
The clustering algorithms in Spark MLlib take advantage of Apache Spark's parallel processing
capabilities to operate effectively on vast amounts of data in distributed settings. It is simple to
incorporate clustering with other Spark components and data processing jobs because they are
a part of the DataFrame-based API.
When utilising clustering algorithms in Spark MLlib, it's critical to undertake feature engineering
if necessary, preprocess the data and carefully choose the right algorithm and hyperparameters
based on the dataset's features and the particular clustering task. In addition, the quality of the
clusters and the ideal number of clusters (K) for a specific dataset can be assessed using
assessment measures like the Silhouette Score or Davies-Bouldin Index.
Here is the answer to Question 20, based on the provided document:
Question 20: DBSCAN does not require:
Predefined number of clusters
The document states that DBSCAN has "No requirement for specifying the number of classes
(numClasses) in advance, providing flexibility in clustering."
Sources
WhatsApp I...3_0de1c6bd
PDF
GIVE QUESTIONS FOLLOWED BY SHORT ANS
Analysis
Here are the questions from the provided images, followed by their short answers:
Question 1: PCA stands for:
Answer: Principal Component Analysis
Question 1 (repeated): A retail company has collected customer purchasing patterns and wants
to group customers for targeted marketing. They use clustering techniques on features such as
[incomplete sentence, likely "frequency of purchases, basket size, and other factors"].
Answer: This scenario describes an application of clustering for customer segmentation.
Question 2: What is the primary goal of clustering?
Answer: To group similar data points into distinct sets or clusters.
Question 1 (likely a continuation or rephrasing of previous questions, incomplete): Which
function in R is used to read a CSV file?
Answer: read.csv() is commonly used.
Question 4: What is regex used for?
Answer: Pattern matching in text.
Question 4 (repeated): DBSCAN is best suited for:
Answer: Identifying clusters of varying shapes and handling noise/outliers.
Question 6: Factor Analysis is mainly used to:
Answer: Reduce a large number of variables into a smaller number of factors or latent variables.
Question 7: The first step in data wrangling is:
Answer: Data collection/preparation.
Question 8: The dplyr package in R is used [incomplete].
Answer: For data manipulation.
Question 9: Web scraping is primarily used to:
Answer: Extract information/data from websites.
Question 10: Which of the following is not a dimensionality reduction technique?
Answer: The options are not visible, but common dimensionality reduction techniques include
PCA, SVD, and Factor Analysis.
Question 11: Which distance metric is commonly used in clustering?
Answer: Euclidean distance.
Question 11 (repeated): Which library is commonly used for text mining in R?
Answer: The tm package is commonly used.
Question 12: Hierarchical clustering results are visualized using:
Answer: Dendrograms.
Question 12 (repeated): In clustering, the silhouette score is used to:
Answer: Evaluate how well data points fit within their assigned clusters, indicating cluster
separation.
Question 13: What does "batch effect" refer to in data analysis?
Answer: Technical variation arising from external factors during sample handling, which can
confound biological signals.
Question 14: A heatmap in clustering is useful for:
Answer: Visualizing patterns and relationships within clustered data.
Question 15: Which of the following is a density-based clustering method?
Answer: DBSCAN is a density-based clustering method.
Question 17: The mutate() function in dplyr is used to:
Answer: Add new columns or modify existing ones.
Question 20: What is the role of group_by() in dplyr?
Answer: To group data for summary or other grouped operations.
Question 19: Clustering is a type of:
Answer: Unsupervised learning.
� Big Data Analytics Study Guide
� Data Structures & Data Science Concepts
Data Structures
●
●
●
●
●
Array: Linear data structure.
Stack:
○ Used for recursive algorithm implementation.
○ Push: Inserting an element.
○ Stock Span Problem: Application of stack.
Array Memory:
○
✅ Advantage: Easy access to elements.
○
❌ Limitation: Fixed memory allocation.
Queue:
Single-ended Queue is not a standard type.
Data Science Concepts
●
●
●
Workflow: Starts with Data Cleansing after data acquisition.
Modeling:
○ MARS: Models deterministic equation sets.
○ Inference: Gold standard of data analysis.
Unstructured Data: Queried via Information Retrieval.
●
●
Categorical Outcomes: Example — Causal relationships.
KDD: Knowledge Discovery in Databases.
ETL & Warehousing
●
●
●
ETL: Extract, Transform, Load.
Staging Area: Validates data before entry to warehouse.
Purpose: Used to build Data Warehouses.
� Database Management Systems (DBMS) & SQL
DBMS Fundamentals
●
●
Both system and application software.
FAT: File Allocation Table structure.
SQL Essentials
●
●
●
●
DDL (Data Definition Language):
○ CREATE, TRUNCATE
DML (Data Manipulation Language):
○ DELETE
Relational Algebra: Procedural.
Metadata: Data about data.
Transactions & Recovery
●
COMMIT: Saves changes permanently.
●
●
ROLLBACK: Auto-executed on crash.
Durability: Prevents data loss.
Data Manipulation
●
●
●
LIKE: Pattern matching.
Domains: Allowed attribute values.
Joins:
○ Right Outer Join: Includes right table non-matching entries.
●
Relationships: One-to-Many (e.g., one teacher → many classes).
●
Filtering: Done via Data Mining.
� MySQL, Cloud Computing & Apache Spark
MySQL Functions
●
COUNT: Aggregate function.
●
GROUP BY: Groups by column.
●
●
●
HAVING: Filters grouped data.
Can have multiple expressions.
Optional GROUP BY inclusion.
Cloud Computing
●
●
●
●
Planning Phases: 5 total.
PAC: Probably Approximate Correct.
Virtualization: Pooling shared resources.
Abstraction: Core principle of cloud architecture.
Apache Spark
●
●
●
●
●
●
●
●
●
●
Catalyst Optimizer: Rule-based.
Data Reading: HTML is invalid format.
Program Counter (PC): Instruction pointer.
Instruction Execution: Copies placed in cache.
Launch: $ spark-shell
Streaming: Uses Spark Core.
RDDs: Resilient Distributed Datasets.
Joins: Use join() between DataFrames.
Limitations:
○ ML algorithms not directly supported in Spark SQL.
Rounding Values: Use round().
●
Query Execution: sqlContext.sql().
●
Temporary Tables: df.registerTempTable().
� General Computing Concepts
●
●
●
●
●
●
●
●
●
●
Invalid SQL Type: FRACTION
JDBC:
○ Packages: sql, javax.sql
○ Components: DriverManager, Driver, Connection, Statement, ResultSet
Column Format: Vertically represented.
SQL Date Format: yyyy/mm/dd
Pandas: Open-source Python data library.
Dimensionality Reduction: Reduces collinearity.
Communication Building: Not a step in data science lifecycle.
UMA (Uniform Memory Access): Equal memory access time.
Bus: Connects multiple devices.
Collective Communication: Data sharing across group tasks.
Big Data Analytics: Understanding the Core Concepts
Big Data Analytics is a field that helps us make sense of the enormous and complex amounts of
data generated every day. It involves collecting, processing, analyzing, and visualizing this data
to gain valuable insights and make better decisions. This document provides a comprehensive
overview of the fundamental building blocks of this exciting domain.
Module I: Data Engineering Basics
Data engineering is the foundation upon which big data analytics is built. It involves getting data
ready for analysis.
What is Data? At its most basic, data refers to raw facts, statistics, or information. It can be
gathered from various sources like databases, files, sensors, web applications, or other ongoing
data streams.
Types of Data Data isn't all the same; it comes in different forms based on its structure and
properties.
●
●
●
●
●
●
Structured Data: This is highly organized data that fits into a fixed format, like tables in
a traditional database.
Unstructured Data: This type of data doesn't have a predefined format or organization.
Examples include text documents, images, audio, and video.
Semi-structured Data: This data has some organizational properties but isn't as rigidly
structured as traditional databases. XML and JSON files are good examples.
Time Series Data: This is data collected over a period of time, often at regular intervals.
Geospatial Data: This data relates to a specific geographic location.
Big Data: This refers to the massive volume of data that is too complex and vast for
traditional processing methods.
File Formats File formats are common methods for encoding and storing data in computer files
so that software applications can easily read, write, and process them. Different file formats are
used for diverse types of data, each with its own advantages and uses.
●
●
●
Delimited Files (CSV, TSV): These are text-based files where values in each line
(record) are separated by a special character called a delimiter. A comma is used for
CSVs (Comma-Separated Values) , and a tab character is used for TSVs (TabSeparated Values). They are simple and widely supported by applications.
XLSX (Microsoft Excel Open XML Spreadsheet): This is an XML-based spreadsheet
file type developed by Microsoft. It can contain multiple worksheets and is considered
secure as it cannot save harmful code.
XML (Extensible Markup Language): This is a markup language with rules for
expressing data. It is readable by both humans and machines and is platformindependent, facilitating data exchange across different systems.
●
●
PDF (Portable Document Format): Developed by Adobe, PDF is used to deliver
documents that are independent of software, hardware, and operating systems. It's
commonly used for forms and legal/financial documents.
JSON (JavaScript Object Notation): An open standard for transmitting structured data
over the internet. It's programming language-agnostic and widely recognized for
efficiently transmitting various data sizes and structures, including audio and video.
Data Processing Data processing is the essential process of converting raw, unprocessed data
into usable information. This typically involves steps like data cleaning, integration, analysis, and
visualization to produce crucial insights for decision-making.
Basic Concepts of Different Types of Data Structures A data structure is an intelligently
designed format for organizing, processing, accessing, and storing data. Its main purpose is to
arrange data in a way that is easy for both machines and people to understand and use
effectively.
●
Why are data structures important? Standard data types like integers or numbers are
often not enough to express the logical intent for complex data processing. Data
structures logically combine data pieces, enabling efficient use, persistence, and sharing
of data. They form the basis for more complex applications. Choosing the right data
structure is crucial because a poor choice can lead to slow performance.
Classification of Data Structures: Data structures are broadly classified into:
●
●
Linear Data Structures: In these structures, data items are arranged sequentially or
linearly, with each element connected to its neighbors. Examples include arrays, stacks,
queues, and linked lists.
○ Static Data Structure: A type of linear structure with a fixed memory size.
Arrays are an example.
○ Dynamic Data Structure: A type of linear structure whose size is not fixed and
can be updated while the program is running. Queues and stacks can be
examples.
Non-linear Data Structures: Data items in these structures are not arranged
sequentially. Examples include trees and graphs. They cannot be fully examined in a
single operation.
Common Data Structures Explained:
●
●
Arrays: A linear data structure that stores a set of elements close to one another in
memory, typically for easy access of similar items. Arrays allow for quick processing of
large amounts of data. Operations include searching, sorting, inserting, and deleting.
Linked List: A linear data structure where elements are not stored at successive
memory positions but are connected using pointers. Linked lists are flexible in size and
allow for easy insertion and deletion of elements.
●
●
●
●
Stack: A linear data structure that follows the "Last In, First Out" (LIFO) order. Data
entry and retrieval occur only at one end, known as the "top". Operations are called
"push" (inserting) and "pop" (withdrawing). Real-life examples include a stack of dinner
plates or browser history.
Queue: A linear data structure that follows the "First In, First Out" (FIFO) order. Data is
added at one end (rear) and removed from the other (front). Real-life examples include a
line at a ticket booth or people riding an escalator.
Tree: A non-linear, hierarchical data structure where items are organized in a tree-like
manner. The highest node is called the "root node". Trees offer enhanced efficiency in
data retrieval, especially for hierarchical data. Applications include game development
and database indexing.
Graph: A non-linear data structure consisting of vertices (nodes) and edges connecting
pairs of nodes. Graphs are used to solve complex programming challenges. Google
Maps and social networks are common real-world applications.
Different Types of Data Repositories Data repositories are systems for storing, managing,
and organizing various forms of data for easy access and retrieval.
●
●
●
●
●
Relational Database Management Systems (RDBMS): These are structured data
repositories that store data in tables with predefined schemas. They use SQL
(Structured Query Language) for data management and retrieval and are frequently
used for transactional applications and business analytics. Examples include MySQL
and Oracle. RDBMS ensures data integrity through features like Entity Integrity,
Referential Integrity, User-defined Integrity, and Domain Integrity. They offer advantages
like ease of management, security, flexibility, and good scalability. However, they can
have high costs, complexity with large data, and structured boundaries that may lead to
data loss.
NoSQL Databases: These are non-relational data repositories offering scalability and
flexible schema design. They are suitable for managing large amounts of unstructured or
semi-structured data, real-time analytics, and content management systems. They are
"not only SQL" and can handle large-scale data analytics and internet applications.
Advantages include ease of design, horizontal scaling, and high availability through
auto-replication. Disadvantages include limited functionality, less robust transaction
management compared to RDBMS, and a lack of standardized interfaces.
Data Warehouses: Centralized, integrated repositories for large volumes of historical
and current data from multiple sources. They support business intelligence, reporting,
and data analysis by providing a single source of truth for data analysis. Key
characteristics include being subject-oriented, integrated, time-variant (storing historical
data), and non-volatile (read-only).
Data Lakes: Centralized repositories capable of storing data of any scale, both
structured and unstructured, in its raw format. They allow for big data processing, realtime analytics, and machine learning without upfront data organization. They offer
flexibility, scalability, and cost-effectiveness.
Data Marts: Focused data repositories designed to cater to a specific subject matter or
department within the context of a data warehouse. They expedite data access for
designated teams and provide a centralized area for relevant data, leading to improved
decision-making and increased efficiency.
ETL Process and Big Data Platforms The
ETL (Extract, Transform, Load) process is a fundamental data engineering methodology for
collecting data from diverse sources, transforming it into a reliable resource, and then
integrating it into systems for end-user access and business problem-solving.
●
●
●
Extract: The first step involves getting data from various sources like business systems,
APIs, and databases.
Transform: In this phase, the raw extracted data is cleaned, mapped, and changed into
a standardized format compatible with various applications.
Load: The final step involves moving the converted data from a staging area to a target
database or data warehouse.
Big Data Platforms are specialized software and hardware ecosystems designed to handle,
process, and analyze massive volumes of structured, semi-structured, and unstructured data.
They provide fault-tolerance, high performance, and scalability for big data applications.
●
●
●
●
●
●
Hadoop: An open-source framework for storing and analyzing massive datasets across
computer clusters. Its core components are HDFS (Hadoop Distributed File System) for
storage and MapReduce for processing.
Spark: An open-source, distributed computing system that allows for in-memory data
processing, making it very fast. It's used for real-time data streaming, machine learning,
and graph processing.
Hive: An open-source data warehousing software built on top of Hadoop that allows
users to access, modify, and administer large-scale datasets using SQL-like queries
(HiveQL).
HBase: A NoSQL database built on HDFS that provides random read and write access
to vast amounts of structured data, suitable for real-time processing.
Pig: A high-level data flow language (Pig Latin) and execution framework used for
handling and analyzing big datasets, simplifying complex data transformations in
Hadoop.
Kafka: A distributed streaming platform for real-time data streaming and message
queuing, often used for stream processing and data ingestion.
Module II: Relational Database Concepts
RDBMS Concepts A Relational Database Management System (RDBMS) is a software system
that facilitates the creation, organization, and manipulation of relational databases. It is based
on the relational model proposed by Dr. E.F. Codd in 1970, which stores data in tables with
rows and columns. Each row represents a distinct record, and each column denotes a different
property of that record. RDBMS uses primary keys to uniquely identify records and foreign keys
to create relationships between tables. SQL is used to interact with the database.
Relational Database Structure A relational database is viewed as a collection of tables used to
edit data using the relational model. It includes system catalog tables outlining data
organization, a configuration file, and a recovery log. Logical data structures (tables, views,
indexes) are separated from physical storage structures.
MySQL MySQL is a popular, fast, and straightforward relational database management system
(RDBMS) used by many organizations. It's open-source, powerful, uses a standard version of
SQL, and supports various operating systems and programming languages.
PostgreSQL PostgreSQL is an open-source object-relational DBMS (ORDBMS) with academic
roots. It's known for being robust, feature-rich, and compliant with ANSI-SQL 92 (and much of
ANSI 99). It supports user-defined database functions, custom data types, and table structure
inheritance.
Module III: SQL for Data Science
SQL Basics SQL (Structured Query Language) is the language used to interact with databases.
It's a comprehensive database language used to run non-procedural queries on relational
databases.
Basic SQL Language SQL was standardized by ANSI in 1986 and ISO in 1987. It is used to
access, modify, and retrieve data from database objects like tables, views, and functions.
●
●
Data Definition Language (DDL): Used to define and manage the structure of
database objects. Commands include
CREATE, ALTER, DROP, TRUNCATE, and COMMENT.
Data Manipulation Language (DML): Used to interact with and manipulate data stored
in the database. Commands include
SELECT, INSERT, UPDATE, DELETE, and MERGE.
Advantages of SQL: SQL is easy to learn, is a universal language supported by most RDBMS,
excels at data retrieval and manipulation, enforces data integrity, offers scalability, supports
transaction management, provides robust security features, and is used for reporting and data
modeling.
Operations of SQL SQL uses operators to perform logical and mathematical operations on
data.
●
●
Arithmetic Operators: Perform mathematical calculations like addition (+), subtraction
(-), multiplication (*), and modulus (%). SQL implementations typically do not provide a
direct division operator but it can be represented using other procedures.
Comparison Operators: Used to compare two values, resulting in TRUE, FALSE, or
UNKNOWN (if NULL is involved). Examples include equal to (=), greater than (>), less
than (<), greater than or equal to (>=), less than or equal to (<=), and not equal to (<>).
Creating a Database in the Cloud A cloud database is hosted on a cloud computing platform
(public, private, or hybrid). It functions similarly to an on-premises database, but the cloud
provider manages the platform and infrastructure, reducing the burden on internal IT teams.
Businesses can choose self-managed databases (IaaS) or managed database services
(DBaaS).
Using String Patterns and Ranges to Query Data SQL allows for flexible querying using
string patterns and ranges.
●
●
The
LIKE operator enables pattern matching in character columns using wildcard
characters: % (zero, one, or many characters) and _ (a single character).
Comparison operators are used for numerical ranges.
Sorting and Grouping Data in Result Sets
●
Sorting (ORDER BY): The ORDER BY clause in SQL is used to sort data in ascending
(ASC) or descending (DESC) order based on one or more columns.
●
Grouping (GROUP BY): The GROUP BY clause groups the result set based on common
values in one or more columns. It is often used with aggregate functions (like
SUM(), COUNT()) to summarize data for each group. The
HAVING clause can then be used to filter these grouped results.
Analyze Data Using Python Python is an excellent programming language for data analysis
due to its rich collection of data-centric packages.
●
●
●
NumPy: This package provides high-performance multidimensional array objects and
functions for manipulating and analyzing them. It is fundamental for scientific computing
in Python.
Pandas: Built on NumPy, Pandas offers powerful data structures like Series (onedimensional labeled array) and DataFrame (two-dimensional tabular data with labeled
rows and columns) for manipulating relational or labeled data. It's widely used for filtering
and grouping data.
Matplotlib: A user-friendly Python library for data visualization. It creates various
graphical representations like line plots, bar charts, scatter plots, and histograms.
Pyplot is a Matplotlib package offering a MATLAB-like interface.
Module IV: Big Data Architectures
What is Big Data? Big data refers to the massive volume of structured, semi-structured, and
unstructured data generated at an unprecedented rate from diverse sources like social media,
sensors, and web applications. It's too complex and vast for traditional data processing and
storage techniques.
Impact of Big Data Big data provides insightful information and business intelligence, offering a
competitive edge to organizations by enabling data-driven decisions, trend spotting, and pattern
identification. It has applications in business intelligence, personalization, healthcare, IoT, and
social media analysis.
The Three Vs of Big Data:
●
●
●
Volume: Enormous amounts of data, often in terabytes, petabytes, or even exabytes.
Velocity: Data is generated, gathered, and processed at an unheard-of rate.
Variety: Big data includes diverse formats and types like structured, semi-structured,
and unstructured data (text, images, video, audio, logs).
Challenges of Big Data Managing big data comes with several challenges:
●
●
●
●
●
●
●
●
●
●
Managing large volumes: Merging incredibly large datasets from various systems into
a coherent architecture is difficult.
Data quality issues: Poor data quality can lead to mediocre results from analytics and
AI systems.
Data preparation and integration: Collecting and integrating heterogeneous data from
numerous sources can be challenging.
Cost-effectively scaling: Without a plan, organizations risk significant financial loss
when scaling huge data systems.
Assessing and choosing technologies: The wide range of big data technologies
available can make selection complex.
Developing commercial insights: The focus can sometimes be too much on
technology and not enough on extracting meaningful business insights.
Finding and keeping experts: A significant challenge is finding and retaining skilled
workers with big data expertise.
Preventing out-of-control spending: Underestimating computational resources
needed can lead to unexpected cloud bills.
Governing ecosystems: Data governance issues become harder as big data
applications proliferate across more systems.
Ensuring clear use cases and context: Organizations often focus on technology
without fully understanding the data's context and practical uses.
Parallel Processing, Scaling, and Data Parallelism Big data analytics heavily relies on
parallel processing, which enables fast and effective analysis of massive amounts of data by
spreading data and computations across multiple processors or nodes.
●
●
Distributed Data Storage: Data is distributed across nodes (e.g., HDFS), allowing
concurrent retrieval and processing.
MapReduce Paradigm: A programming model that relies on parallel processing,
dividing data processing into a Map phase (concurrent processing) and a Reduce phase
(compiling intermediate results).
●
●
●
Parallel Data Transformation: Allows data transformations, filtering, and aggregation to
be performed concurrently across multiple nodes, speeding up data preparation.
Parallel Machine Learning: Distributes training and assessment tasks across nodes to
speed up model training and increase scalability.
Real-Time Stream Processing: Enables simultaneous processing of incoming data
across nodes for low-latency analysis.
Scaling refers to a system's ability to handle increased data quantities and computational
needs without compromising performance.
●
●
Vertical Scaling (Scaling Up): Upgrading hardware (e.g., adding CPU, memory) to a
single system to handle more workloads. It's simpler to manage but has hardware limits.
Horizontal Scaling (Scaling Out): Adding more machines or nodes to a distributed
computing system to manage greater workloads. This is the main technique for big data,
offering high scalability and fault tolerance by using commodity hardware.
Data Parallelism: A key idea in big data analytics focusing on partitioning large datasets into
smaller ones and processing them concurrently across multiple nodes. This method is crucial in
distributed computing frameworks like Apache Hadoop and Apache Spark.
Tools of Big Data The Big Data ecosystem includes various tools for handling, processing, and
analyzing massive quantities of data.
●
●
●
●
●
●
Apache Hadoop: Provides a distributed storage (HDFS) and processing (MapReduce)
platform for massive datasets. It's scalable, fault-tolerant, and cost-effective.
Apache Spark: An open-source, distributed computing system for in-memory data
processing, offering high speed, machine learning, and graph processing capabilities. It
has high-level APIs in various languages (Scala, Java, Python, R).
Apache Hive: A data warehousing and SQL-like querying tool built on Hadoop, enabling
users to interact with and analyze data stored in HDFS using SQL-like queries (HiveQL).
Apache HBase: A distributed, scalable, and consistent NoSQL database built on HDFS,
providing random read and write access to huge amounts of structured data for real-time
processing.
Apache Pig: A high-level data flow language (Pig Latin) and execution framework that
simplifies programming complex data transformations on Hadoop.
Apache Kafka: A distributed streaming platform for real-time data streaming and
message queuing, used for data pipelines and event processing.
Beyond the Hype Big data's true value comes from its ability to influence decision-making.
Organizations need efficient ways to convert vast, fast-changing data into meaningful findings.
Moving beyond the hype means using data analytics practically and pragmatically to benefit
businesses. This involves defining clear objectives, ensuring quality data, setting realistic
expectations, addressing privacy and ethics, fostering collaboration, adopting an agile
approach, measuring success, balancing automation with human expertise, ensuring scalability,
and promoting continuous learning.
Big Data Use Cases Big data analytics has transformed various industries:
●
●
●
●
●
●
●
●
●
●
Banking and Finance: Fraud detection, risk management, asset management.
Accounting: Identifying outliers, focusing on high-risk areas, visualizing data,
connecting financial and non-financial data.
Aviation: Optimizing operations, resource allocation, flight safety, loyalty programs,
route optimization.
Agriculture: Precision farming, climate change predictions, pesticide use, equipment
optimization, supply chain streamlining.
Automotive: Design and production process improvement, vehicle maintenance,
manufacturing monitoring, market trend prediction, logistics analysis.
Biomedical Research and Healthcare: Enhancing patient outcomes, developing better
medications, forecasting outbreaks, personalized medicine, genomic research.
Business and Management: Streamlining processes, digitization, automation,
overcoming data silos and incompatibility.
Cloud Computing: Performing big data workloads on the cloud, outsourcing operational
overloads.
Cybersecurity: Identifying anomalies to detect suspicious actions, automating threat
responses.
Telecom: Optimizing network capacity, addressing customer churn, developing new
product offerings.
The Hadoop Ecosystem and Common Components The Hadoop ecosystem is a collection
of tools, libraries, and frameworks built around Apache Hadoop, providing resources for data
storage and analysis. It enables massive parallelism with low latency and high throughput.
●
●
HDFS (Hadoop Distributed File System): A massive data storage system running on
inexpensive, networked hardware. It stores data across multiple servers in a distributed
way, allowing concurrent access and ensuring fault tolerance by replicating file blocks.
○ Namenodes and Datanodes: An HDFS cluster consists of a single master
(namenode) and multiple workers (datanodes). The namenode manages the
filesystem namespace and metadata, while datanodes store and retrieve blocks.
○ HDFS Federation: Allows a cluster to scale by adding multiple namenodes, each
controlling a portion of the filesystem namespace.
○ HDFS High Availability (HA): Hadoop 2 introduced HA with active-standby
namenode pairs to prevent the namenode from being a single point of failure,
ensuring continuous operation.
Hive, HBase, and Spark: These tools are integral parts of the Hadoop ecosystem, as
described earlier.
Module V: Apache Spark for Data Engineering and Machine Learning
Apache Spark and its Techniques Apache Spark is an open-source, distributed computing
platform designed for large data processing and analytics. It offers a unified engine for batch
processing, real-time streaming, machine learning, and graph processing.
●
Key Techniques: Distributed Computing, Resilient Distributed Datasets (RDDs),
DataFrames and Datasets (higher-level abstractions over RDDs), Transformations and
Actions, Lazy Evaluation, Spark SQL, Spark Streaming, Machine Learning Library
(MLlib), GraphX, Spark GraphFrames, Spark Streaming + Kafka Integration, and
MLflow.
Spark Structured Streaming Structured Streaming is a high-level API in Spark for continuous
processing of unending streams of data. It treats a stream as an unbounded, continually
appended table. This allows developers to use familiar batch-like DataFrame queries on
streaming data.
●
●
Micro-Batch Processing: Structured Streaming uses a micro-batch approach, where
streaming computation is a continuous sequence of small batch processing operations.
This offers advantages like efficient recovery from failures and exactly-once processing
guarantees.
Output Modes:
○ Append mode: Only newly appended rows are written to the output sink.
○ Update mode: Only rows that have changed since the last trigger are updated in
the external storage.
○ Complete mode: The entire updated result table is written to external storage at
the end of each trigger.
GraphFrames on Apache Spark GraphFrames is a module for Apache Spark that offers
DataFrame-based Graphs. It provides high-level APIs for manipulating and analyzing graph
data, including features like finding motifs (structural patterns), DataFrame-based serialization,
and powerful graph queries. It's designed for analytics and can handle large workloads.
ETL Workloads Apache Spark plays a crucial role in modern ETL (Extract, Transform, Load)
processes for big data. It allows organizations to automate data pipelines, combine, cleanse,
and prepare data from diverse sources into a reliable and uniform format for analysis and
reporting.
Spark ML Fundamentals Spark MLlib (Machine Learning Library) is a part of Apache Spark
that provides a collection of machine learning tools and methods for processing massive
amounts of data and distributed machine learning.
●
●
Machine Learning Types:
○ Supervised Learning: Aims to make predictions about output labels based on
input data that has been previously labeled. It includes:
■ Classification: Categorizing inputs into a discrete set of classes or labels
(e.g., spam or not spam).
■ Regression: Predicting a continuous numerical value (e.g., house price).
○ Unsupervised Learning: Helps understand the structure of unlabeled data by
finding hidden patterns or groupings. Clustering is a prime example.
Key Concepts in Spark ML:
○
○
○
Transformer: A function that takes a DataFrame and produces a new
DataFrame with additional columns, performing rule-based transformations or
generating predictions.
Estimator: Learns parameters from a DataFrame using the .fit() method and
returns a Model (which is a Transformer).
Pipeline: Assembles multiple Estimators and Transformers into a single
workflow.
Spark ML Regression and Classification Spark ML Regression offers techniques to predict a
dependent variable's value based on independent variables using statistical methods. It
includes algorithms like linear regression, decision tree regression, and random forest
regression.
●
●
●
●
●
Linear Regression: Predicts the output as a linear combination of features.
Logistic Regression: A binary classification method that finds a linear separation plane
between positive and negative cases (labels 0 or 1).
Support Vector Machines (SVMs): Another binary classification method using linear
separation planes.
Naive Bayes: A multiclass classification technique that determines the degree of
belongingness of each data point to different classes.
Decision Trees and Random Forests: Versatile models for both classification and
regression, representing a tree of binary decisions leading to a prediction. Random
forests are ensembles of multiple decision trees.
Spark ML Clustering Clustering is an unsupervised learning problem that involves grouping
similar items together. It's used for anomaly detection and data exploration.
●
●
K-Means: A widely used clustering algorithm that divides data into K clusters (where K
is a user-defined parameter) by iteratively assigning data points to the nearest cluster
centroid and updating centroids.
Collaborative Filtering and Recommendation: A method for recommender systems
that uses user feedback and interactions (explicit or implicit) to suggest new products.
○ Alternating Least Squares (ALS): A scalable collaborative filtering approach
that computes a feature vector for each user and product to approximate their
interaction score.
Dimensionality Reduction This involves reducing the number of dimensions in a dataset while
retaining important information, making data simpler to analyze.
●
●
Principal Component Analysis (PCA): A predominant technique for dimensionality
reduction that optimizes mapping to a lower-dimensional space by maximizing the
variance of data in the reduced representation.
Singular Value Decomposition (SVD): A lower-level primitive that factorizes a matrix
into three matrices, often used to reduce storage requirements, remove noise, and
restore low-rank structure.
This explanation covers the key concepts presented in the "Big Data Analytics.docx" document,
aiming for clarity and simplicity.
Sources