Flashcards: Google Cloud Professional Data Engineer

Automates data movement from SaaS applications to Google BigQuery on a scheduled, managed basis. Your analytics team can lay the foundation for a data warehouse without writing a single line of code.

Supports Google Ads, Campaign Manager, Google Ad Manager, and YouTube Content and Channel Owner Reports. Through BigQuery Data Transfer Service, users also gain access to data connectors that allow you to easily transfer data from Teradata and Amazon S3 to BigQuery.

You can use Cloud Dataflow IAM roles to limit access for users within a project or organization, to just Cloud Dataflow-related resources, as opposed to granting users viewer, editor, or owner access to the entire Cloud Platform project.

Minimal role for creating and managing dataflow jobs.

Provides the permissions necessary to execute and manipulate Cloud Dataflow jobs.

Provides read-only access to all Cloud Dataflow-related resources.

Provides the permissions necessary for a Compute Engine service account to execute work units for a Cloud Dataflow pipeline.

- The developer who creates and examines jobs will need the roles/dataflow.admin role. - For more sophisticated permissions management, the developer interacting with the Cloud Dataflow job will need the roles/dataflow.developer role. - They will need the roles/storage.objectAdmin or a related role in order to stage the required files.

- For debugging and quota checking, they will need the project roles/compute.viewer role. - Absent other role assignments, this will allow the developer to create and cancel Cloud Dataflow jobs, but not interact with the individual VMs or access other Cloud services. - The controller service account needs the roles/dataflow.worker role to process data for the Cloud Dataflow service. It will need other roles (such as roles/storage.objectAdmin in order to access job data.

Cloud Dataflow roles can currently be set on organizations and projects only.

When an identity calls a Google Cloud Platform API, Google BigQuery requires that the identity has the appropriate permissions to use the resource. You can grant permissions by granting roles to a user, a group, or a service account.

- Primitive roles include the Owner, Editor, and Viewer roles that existed prior to the introduction of Cloud Identity and Access Management. - Predefined roles provide granular access for a specific service and are managed by GCP. Predefined roles are meant to support common use cases and access control patterns. - Custom roles provide granular access according to a user-specified list of permissions.

- The gcloud iam roles describe command - The roles.get() method in the Cloud IAM API

The permissions granted are a union of each role's permissions.

You provide permission to run BigQuery jobs or to manage all of a project's BigQuery resources.

In the IAM policy hierarchy, BigQuery datasets are child resources of projects. Tables and views are child resources of datasets — they inherit permissions from their parent dataset.

You automatically have the equivalent of the bigquery.jobs.get and bigquery.jobs.update permissions for that job.

Provides permissions to manage all resources within the project. Can manage all data within the project, and can cancel jobs from other users running within the project.

When applied to a dataset, dataEditor provides permissions to: - Read the dataset's metadata and to list tables in the dataset. - Create, update, get, and delete the dataset's tables. When applied at the project or organization level, this role can also create new datasets.

When applied to a dataset, dataOwner provides permissions to: - Read, update, and delete the dataset. - Create, update, get, and delete the dataset's tables. When applied at the project or organization level, this role can also create new datasets.

When applied to a dataset, dataViewer provides permissions to: - Read the dataset's metadata and to list tables in the dataset. - Read data and metadata from the dataset's tables. When applied at the project or organization level, this role can also enumerate all datasets in the project. Additional roles, however, are necessary to allow the running of jobs.

Provides permissions to run jobs, including queries, within the project. The jobUser role can enumerate their own jobs and cancel their own jobs.

When applied at the project or organization level, metadataViewer provides permissions to: - List all datasets and read metadata for all datasets in the project. - List all tables and views and read metadata for all tables and views in the project. Additional roles are necessary to allow the running of jobs.

Access to create and use read sessions

Provides permissions to run jobs, including queries, within the project. The user role can enumerate their own jobs, cancel their own jobs, and enumerate datasets within a project. Additionally, allows the creation of new datasets within the project; the creator is granted the bigquery.dataOwner role for these new datasets.

A partitioned table is a special table that is divided into segments, called partitions, that make it easier to manage and query your data. By dividing a large table into smaller partitions, you can improve query performance, and you can control costs by reducing the number of bytes read by a query.

- Tables partitioned by ingestion time: Tables partitioned based on the data's ingestion (load) date or arrival date. - Partitioned tables: Tables that are partitioned based on a TIMESTAMP or DATE column.

BigQuery automatically loads data into daily, date-based partitions that reflect the data's ingestion or arrival date. Pseudo column and suffix identifiers allow you to restate (replace) and redirect data to partitions for a specific day.

The partitions have the same schema definition as the table. If you need to load data into a partition with a schema that is not the same as the schema of the table, you must update the schema of the table before loading the data. Alternatively, you can use schema update options to update the schema of the table in a load job or query job.

Partitioned tables allow you to bind the partitioning scheme to a specific TIMESTAMP or DATE column. Data written to a partitioned table is automatically delivered to the appropriate partition based on the date value (expressed in UTC) in the partitioning column.

- The __NULL__ partition — represents rows with NULL values in the partitioning column - The __UNPARTITIONED__ partition — represents data that exists outside the allowed range of dates

You can shard tables using a time-based naming approach such as [PREFIX]_YYYYMMDD. This is referred to as creating date-sharded tables. Using either standard SQL or legacy SQL, you can specify a query with a UNION operator to limit the tables scanned by the query.

When you create date-named tables, BigQuery must maintain a copy of the schema and metadata for each date-named table. Also, when date-named tables are used, BigQuery might be required to verify permissions for each queried table. This practice also adds to query overhead and impacts query performance.

A wildcard table represents a union of all the tables that match the wildcard expression. For example, the following FROM clause uses the wildcard expression gsod* to match all tables in the noaa_gsod dataset that begin with the string gsod.

The wildcard table functionality does not support views. If the wildcard table matches any view in the dataset, the query returns an error. This is true whether or not your query contains a WHERE clause on the _TABLE_SUFFIX pseudo column to filter out the view. Currently, cached results are not supported for queries against multiple tables using a wildcard even if the Use Cached Results option is checked. If you run the same wildcard query multiple times, you are billed for each query.

Wildcard tables support native BigQuery storage only. You cannot use wildcards when querying an external table or a view. Queries that contain Data Manipulation Language (DML) statements cannot use a wildcard table as the target of the query. For example, a wildcard table may be used in the FROM clause of an UPDATE query, but a wildcard table cannot be used as the target of the UPDATE operation.

Wildcard tables are useful when a dataset contains multiple, similarly named tables that have compatible schemas.

To query a group of tables that share a common prefix, use the table wildcard symbol (*) after the table prefix in your FROM statement.

To restrict the query so that it scans an arbitrary set of tables, use the _TABLE_SUFFIX pseudo column in the WHERE clause. The _TABLE_SUFFIX pseudo column contains the values matched by the table wildcard.

To scan a range of tables, use the _TABLE_SUFFIX pseudo column along with the BETWEEN clause.

To scan a range of ingestion-time partitioned tables, use the _PARTITIONTIME pseudo column with the _TABLE_SUFFIX pseudo column.

To scan all tables in a dataset, you can use an empty prefix and the table wildcard, which means that the _TABLE_SUFFIX pseudo column contains full table names.

Cloud Spanner automatically creates an index for each table's primary key column. You can also create secondary indexes for other columns. Adding a secondary index on a column makes it more efficient to look up data in that column. For example, if you need to quickly look up a set of SingerId values for a given range of LastName values, you should create a secondary index on LastName, so Cloud Spanner does not need to scan the entire table.

Every Cloud Firestore (in Datastore mode query) computes its results using one or more indexes which contain entity keys in a sequence specified by the index's properties and, optionally, the entity's ancestors. The indexes are updated to reflect any changes the application makes to its entities, so that the correct results of all queries are available with no further computation needed.

Built-in indexes - By default, a Datastore mode database automatically predefines an index for each property of each entity kind. These single property indexes are suitable for simple types of queries. Composite indexes - Composite indexes index multiple property values per indexed entity. Composite indexes support complex queries and are defined in an index configuration file (index.yaml).

However, it does not support some kinds of query common in other database technologies: in particular, joins and aggregate queries aren't supported within the Datastore mode query engine.

• Each table has only one index, the row key. • Rows are sorted lexicographically by row key, • Columns are grouped by column family and sorted in lexicographic order within the column family. • All operations are atomic at the row level.

• Ideally, both reads and writes should be distributed evenly • In general, keep all information for an entity in a single row. • Related entities should be stored in adjacent rows, • Cloud Bigtable tables are sparse.

- User information: Do you need quick access to information about connections between users (for example, whether user A follows user B)? - User-generated content: If you show users a sample of a large amount of user-generated content, such as status updates, how will you decide which status updates to display to a given user? - Time series data: Will you often need to retrieve the most recent N records, or records that fall within a certain time range? If you're storing data for several kinds of events, will you need to filter based on the type of event?

Hive is declarative and Pig is imperative.

Specifies exactly how to perform the data analysis, which limits the flexibility of the underlying systems. They can't decide how to use the available resources to get the results.

Makes execution plans but then requests the underlying systems determine how to process the data. So Pig can fit better into the pipeline paradigm of processing data.

The way it does this is by maintaining metadata to define a schema on top of the data. This is one way to work with a large amount of distributed data in HDFS using familiar SQL syntax.

It ingests data into a data warehouse format requiring a schema. It does not support real-time queries, row-level updates, or unstructured data. Some queries may run much slower than others due to the underlying transformations Hive has to implement to simulate SQL.

Pig can also deal with semi-structured data, such as data having partial schemas, or for which the schema is not yet known. For this reason it is sometimes used for Extract Transform Load (ETL). It generates Java MapReduce jobs. Pig is not designed to deal with unstructured data.

A messaging service that decouples senders and receivers, enabling asynchronous messaging for ingesting data. This is useful for ingesting streaming data. Publishers publish messages to a topic, and subscribers subscribe to the topic and receive the messages. Messages are stored until they're delivered and acknowledged by the subscribers.

A service that provides a unified method for both batch and streaming data processing. Dataflow builds on the Apache Beam SDK.

A data warehouse service with a highly scalable interactive data exploration interface. It can capture and analyze data in real time. So it makes a great partner for data that's been ingested through Pub/Sub and processed though Dataflow.

You create a pipeline job in Python or Java using Apache Beam. Data is read in batch from Cloud Storage or as a stream through Pub/Sub, it is processed in Dataflow. The results are stored in BigQuery for interactive analysis or in Bigtable for use by applications, there's nothing to maintain.

Among them is "class" which declares the frequency of anticipated access of the object. This allows the Cloud Storage service to distinguish between types of objects and internally manage them to service levels

● Avoid small reads ● Use large block/file sizes where possible ● Avoid iterating over many nested directories in a single job

Throughput for processing data in Cloud storage is higher than throughput for HDFS on Persistent disk in the cluster.

Colossus also contains data structures called Tablets that are used to identify and manage the data. And metadata about the Tablets is what is stored on the VMs in the Bigtable cluster itself.

It can manipulate the actual data. It can manipulate the Tablets that point to and describe the data. Or it can manipulate the metadata that points to the Tablets. Rebalancing tablets from one node to another is very fast, because only the pointers are updated.

It detects "hot spots" where a lot of activity is going through a single Tablet and splits the Tablet in two. It can also rebalance the processing by moving the pointer of a tablet to a different VM in the cluster. So its best use case is with big data -- above 300 GB

Recovery is fast because only the metadata needs to be copied to the replacement node. Colossus provides better durability than the default 3 replicas provided by HDFS.

That index is called the Row Key. There are no alternate indexes or secondary indexes. And when data is entered, it is organized lexicographically by the Row Key.

(Reduced Instruction Set Computing). Simplify the operations. And when you don't have to account for variations, you can make those that remain very fast.

Writing a query that controls the amount of data processed. In general, this is done with SELECT by choosing subsets of data at the start of a job rather than by using LIMIT which only omits data from the final results at the end of a job.

-Step one, copy the data to Cloud Storage. -Step two, update the file prefix in your application scripts from HDFS to GS. You just change hdfs:// to gs://. -Step three, create a Dataproc cluster and run your job. You can also stage your scripts in Cloud Storage. In most cases, that's all you have to do to get your current job running in Dataproc.

It makes scaling decisions based on Hadoop YARN Metrics. It is designed to be used only with off-cluster persistent data, not on-cluster HDFS or HBase. It works best with a cluster that processes a lot of jobs or that processes a single large job.

1) A fixed amount of time after the cluster enters the Idle state. 2) Set a timer. (You give it a timestamp) The count starts immediately once the expiration has been set. 3) Set a duration. Time in seconds to wait before deleting the cluster. (Range is from 10 minutes minimum to 14 days maximum, with a granularity of 1 second)

Only when a request is submitted that requires output, the Spark actually process the data. The benefit of this strategy is that Spark can look at all the requests and the intermediate results and construct parallel pipelines based on the resources that are available in the cluster at that time.

When it is passed to Spark in this way, Spark understands the multiple steps and that the results of one transformation are to be passed to the next transformation. This allows Spark to organize the processing in any way it decides based on the resources available in the cluster.

You deploy your code to the Cloud Functions service and set it up to be triggered by a class of events.

However, for data processing there are tools such as Cloud Dataproc Workflow Templates and Cloud Composer that are designed to manage workflows without having to code the service yourself.

At that time you specify the trigger that will cause the Cloud Function to run, such as the trigger bucket for Cloud Storage or the trigger topic for Cloud Pub/Sub.

Map Reduce, Hadoop, Spark Scala, and Spark Pyspark

To overcome this problem, Cloud Dataproc's BigQuery Connectors use Cloud Storage. The workers are able to write their shards as objects in Cloud Storage. When the work is complete, as part of the connector shutdown process, a single load job is issued in BigQuery.

It can create a new cluster, select from an existing cluster, submit jobs, hold jobs for submission until dependencies can complete, and it can delete a cluster when the job is done.

Cloud Composer can be used to automate Cloud Dataproc jobs and to control clusters.

So, you don't really need to choose. You can store your data if it's structured data, it's tabular data, store it in BigQuery and you will get the same sort of discounts that you get in cloud storage as well. So if you have some data and you haven't edited it in a few weeks, you start getting automatically the discounted rates for older data.

The answer for that would be something like Cloud SQL or something like Spanner. But for ad hoc analysis of very large datasets, for data warehousing, for business intelligence, those kinds of operations, BigQuery is a great choice.

You put your data onto BigQuery, you're not managing any clusters beyond that point. So you want to run a query on BigQuery, you just run the query on BigQuery. You don't need to create a cluster ahead of time. You only pay when you query the data.

A dataset is basically a collection of tables. Dataset is a collection of tables belong to your organization, and you basically do access control on a dataset basis not at the table basis because you will want to join tables together typically within a dataset. So you don't do access control on a single table, you do access control on a dataset that consists of multiple tables.

A view is a live view of a table. So essentially you can think of a view as a query. That query returns our result set and then you can write a query to process that view just like you're writing something on a table. So you can use views as a cool way to actually restrict control to your dataset, because a view basically is a select ready make.

One of the ways that you can fix this is by probably removing the tail, for example with the Having clause, filtering them out so that you are not processing those tail things.

So, typical pipeline goes from a source to a sink, involves branching, and involves a number of transforms The input to that transform is a parallel collection. It's a PCollection. So, each transform on the pipeline, it's input is a PCollection.

Well, when you replace a running pipeline, you don't lose any data. So, some of the data gets processed by the old pipeline and any data that wasn't process by the old pipeline will get processed by the new pipeline. You can obviously see the advantage of your processing, streaming data right. But in order for that replacement to work, your transforms have to have names, unique names.

The grep command processes text line by line, and prints any lines which match a specified pattern. Grep, which stands for "global /regular expression /print," is a powerful tool for matching a regular expression against text in a file, multiple files, or a stream of files.

The templates help separate the development activities and the developers from the execution activities and the users.

Dataprep is an interactive graphical system for preparing structured or unstructured data for use in analytics such as BigQuery, visualization like, Data Studio and to train machine learning models.

The elements are divided into datasets, recipes and output. A dataset roughly translates into a Dataflow pipeline read, a recipe usually translates into multiple pipeline transformations, and an output translates into a pipeline action.

This is especially useful for data quality tasks and for Master Data task, combining data from multiple sources where programming may not be required.

The label is a correct output for an input. (the true answer - either known or to be determined)

The input is a thing that you will know and that you can provide at the time of prediction. These are things, for example, if they're images, the image itself as an input.

An example is a combination of the label and the input. An input and its corresponding label together form an example.

A model is a mathematical function that takes an input and creates an output that approximates a label for that input.

Training is this process of adjusting the weights of a model in such a way that it can make predictions given an input.

A prediction is this process of taking an input and applying the mathematical model to it. So, to get an output, that is hopefully the correct output for that input.

Supervised and Unsupervised ML models. The key difference is that with supervised models, we have labels, or in other words, the correct answers to whatever it is that we want to learn to predict. In unsupervised learning, the data does not have labels.

is a linear model…

If we wanted to predict the gender of the person making a payment, it then becomes a classification problem because in this dataset, there happened to be only two categories of people.

So four numbers. So, what an image ultimately is, is that it's a three-dimensional array of numbers, a 1D array as vector, a 2D array as a matrix, a three dimensional-four dimensional array we don't have a good name for, in general, we call an n-dimensional array, a tensor. And that's where the name, Tensor Flow comes from.

A generalized vector with more than three components each of which is a function of the coordinates of an arbitrary point in space of an appropriate number of dimensions.

So, you would go ahead and pick one of those word2vec things, and you will use it to convert your text inputs into vectors. From then on, all you have are numbers and you go ahead and do your machine learning on those numbers.

What we do is, because we decide for spam was going to be one and not spam was going to be zero, we'll interpret that number as a probability, that if this probability is one, if it's 100% likely to be spam, and if the probability is zero, it is 100% likely to be not spam, and if it's a number between zero and one, the number is 0.8, then we say that it is 80% likely that this email is spam. So, we interpret the output of the model, this output that's between zero and one, we interpret it as a probability of it being, label equals one.

A neuron is a way to combine all of the inputs and make a decision on those inputs.

When the learning rate is low, training takes a very long time. If you try to speed things up by increasing the learning rate, we might miss the minimum altogether. So, that's not good either. What we need is a perfect learning rate. The best learning rate unfortunately varies from problem to problem. So, you will have to experiment.

So, let's say we have a dataset of 100,000 samples, and just to make the math easy, let's say our batch is 100 samples. This means that every 1,000 steps is one epoch.

The weights are the things that you get to change so that your model captures your data.

The batch size is the number of points over which we try out the changes in weights. When we start with an arbitrary value for weight and then we increase the weight and we decrease the weight and we calculate the loss, we don't do this over the entire dataset. We do this on only a few points. Those few points that we do it on is called a batch size.

We change the weights so that the error gets lower.

every once in a while where we say, okay, now that we've changed a bunch of weights, let's go back and figure out how well our model does over an entire dataset, and we do this so that we can decide whether or not to stop the training.

The process of taking your data, dividing it into batches, carrying out gradient descent, and every few epochs doing an evaluation and deciding whether to stop or not.

A neuron is essentially a weighted sum of its inputs, followed by what is called an activation function. There are different activation functions. Things that you can do to that weighted sum. But, at its core, the simplest neuron does a weighted sum of its inputs.

a layer of neurons is a combination of neurons, all of which share the same set of inputs.

These are the things that you feed into a neuron. So, the inputs to the first layer of your neural network will be your original raw data.

Things that you calculate from the raw data. So, we have our inputs and then we have things that we calculate from those inputs.

a process of coming up with the features that you're going to calculate from your original inputs.

BigQuery, DataFlow, and machine learning processing, SDK to organize the data to get different types of organization, TensorFlow to create the model, Cloud ML Engine to train and deploy the model.

is how "wrong" or "far away" your prediction is from the true distribution. Cross-entropy is commonly used to quantify the difference between two probability distributions. Usually the "true" distribution (the one that your machine learning algorithm is trying to match) is expressed in terms of a one-hot distribution.

In predictive analytics, a table of confusion (sometimes also called a confusion matrix), is a table with two rows and two columns that reports the number of false positives, false negatives, true positives, and true negatives. This allows more detailed analysis than mere proportion of correct classifications (accuracy).

is the error that you use for optimizing a regression model.

is the error measure that you use as a loss function to optimize a classification model.

is the fraction correct, it's the way you report the performance of a classification model when datasets tend to be balanced.

is a measure of skill that looks at the accuracy when a machine learning model says, it is off the category that you're trying to find.

is the accuracy in the machine learning model, it's the accuracy in terms of how well the machine learning model can find the things that you're trying to find.

If it's balanced, then you use accuracy. So, if it's unbalanced and you tend to have lots of cats in your images, you tend to use precision, if you have very few cats in your images, you will tend to use recall.

is a high-performance library for numerical computation. So, it's a library that lets you work with numbers in an efficient way. It's open-source, and follows a graph processing idea in a way that's similar to Apache Beam and Dataflow.

First step, setup your feature columns. Step two, create a model passing in the feature columns. When you create the model, you will specify whether it's a regression problem or a classification problem. Then, you will write an input function to get your training data, and that input function is going to return the features, and the corresponding labels, where the features is a dictionary of all of the input features. Then, you will train your model using estimator.train. Once you have trained the model, you can use that trained model to predict.

In this case, we're creating a three-layer network, three hidden layers. So, depending on the way you count, you have a total of five layers. One input, three hidden and one output. Three hidden layers are 128 notes in the first layer, 64 nodes in the second layer, and 32 nodes in the third layer.

The train_spec consists of the things that used to be passed into the train method. So, that train_spec takes the training input function that's used to read in the training data set but notice, that also takes a parameter that's measured in steps. The eval_spec controls the evaluation and the checkpointing of the model, since they happen at the same time. The way it works is that the training loop saves the model into a checkpoint and the eval loop restores the model from the checkpoint and uses it to evaluate the model.

The levels are debug, info, warn, error, and fatal.

A collection of visualization tools that are especially designed to help you visualize TensorFlow models and the training of transfer from models.

Then you call a GCloud command or click on the GCP Web console to deploy the model as a microservice. The model is then available for access through a REST API. When you're deploying though, you don't just put the original model, you export it along with what is called a serving input function.

Kubeflow Pipelines is a platform for composing, deploying, and managing end-to-end machine learning workflows

A user interface (UI) for managing and tracking experiments, jobs, and runs. An engine for scheduling multi-step ML workflows. An SDK for defining and manipulating pipelines and components. Notebooks for interacting with the system using the SDK. Key benefits are reusability and portability

You're going to create a vocabulary of keys. So, that's your first step and that's the pre processing, you have to do this before you do your training. Then, you would create a new dataset where the pre processed values can be used. So, before you ever get to training the model you need to create a vocabulary and this vocabulary also needs to be available at prediction time.

A one hot encoding is a representation of categorical variables as binary vectors. This first requires that the categorical values be mapped to integer values. Then, each integer value is represented as a binary vector that is all zero values except the index of the integer, which is marked with a 1.

We take our inputs, we do our pre-processing. The pre-processing is where we create a vocabulary, we do our feature crosses, we do our scaling, we do our bucketizing. Then, you're going to do your feature creation, which is like feature crosses of all your categorical features. Then, you're going to train your model.

The dense dimensions define how large each block will be in size (the number of rows, columns and the depth of the Z axis). The sparse dimensions define the number of possible blocks the database may hold. In numerical analysis and scientific computing, a sparse matrix or sparse array is a matrix in which most of the elements are zero. By contrast, if most of the elements are nonzero, then the matrix is considered dense.

is about systematically trying out various neural network architectures and different processing methods.

improves through feature engineering, hyperparameter tuning, and using lots of data.

step number one, upload the images. Step number two, labeled images. Step number three, train the images, and of course, evaluate the model and serve the model in an application.

Binary logistic regression models identify one of two classes. Multiclass logistic regression models are used for classification.

This is also known by other names such as "discretizing," "chopping data," or "binning". Specific methods sometimes used include "median split" or "extreme third tails". (putting continuous variables in buckets)

#1: Variable volumes require ability of ingest to scale and be fault-tolerant #2 : Latency is to be expected #3 : Need instant insights

(1. Number one, what results are you calculating?) (2. The second is where in event time are you computing?) (3. When are the results actually materialized? When are they saved, when are they sent out?) (4. Finally, if a new message comes in that's late, how should you refine it? How should you change an already computed result?)

The BigQuery lets you do this. With BigQuery you can ingest streaming data and run queries as the data flies by.

BigQuery and Bigtable are about user generated queries, ad-hoc queries, queries that you have that you do once in a long while.

BigQuery is SQL, Bigtable is no SQL. But all of these are fully managed, no-ops services.

Overfitting results when a model performs well on the training set, generating only a small error, but struggles with new or unknown data. In other words, the model overfits itself to the data. Instead of training a model to pick out general features in a given type of data, an overtrained model learns only how to pick out specific features found in the training set.

Cloud IoT Core, using Cloud Pub/Sub underneath, can aggregate dispersed device data into a single global system that integrates seamlessly with Google Cloud data analytics services.

When you run your pipeline with the Cloud Dataflow service, the runner uploads your executable code and dependencies to a Google Cloud Storage bucket and creates a Cloud Dataflow job, which executes your pipeline on managed resources in Google Cloud Platform.

Cloud Pub/Sub Notifications are the recommended way to track changes to objects in your Cloud Storage buckets because they're faster, more flexible, easier to set up, and more cost-effective.

When you create the view, it must be created in a dataset separate from the source data queried by the view. Because you can assign access controls only at the dataset level, if the view is created in the same dataset as the source data, your users would have access to both the view and the data.

Both Wide and Deep learning model can help build good recommendation engine. It's useful for generic large-scale regression and classification problems with sparse inputs (categorical features with a large number of possible feature values), such as recommender systems, search, and ranking problems.

Mission-critical, relational database service with transactional consistency, global scale, and high availability. (Global scale, low latency and the ability to scale horizontally)

In supervised machine learning, you feed the features and their corresponding labels into an algorithm in a process called training. During training, the algorithm gradually determines the relationship between features and their corresponding labels. This relationship is called the model.

In unsupervised learning, the goal is to identify meaningful patterns in the data. To accomplish this, the machine must learn from an unlabeled data set. In other words, the model has no hints how to categorize each piece of data and must infer its own rules for doing so.

Multi-Regional Storage is geo-redundant. The geo-redundancy of Nearline Storage data is determined by the type of location in which it is stored: Nearline Storage data stored in multi-regional locations is redundant across multiple regions, providing higher availability than Nearline Storage data stored in regional locations.

Cloud Vision API enables you to derive insight from your images with our powerful pretrained API models or easily train custom vision models with AutoML Vision. The API quickly classifies images into thousands of categories (such as “sailboat” or “Eiffel Tower”), detects individual objects and faces within images, and finds and reads printed words contained within images.

You can now search every moment of every video file in your catalog. It quickly annotates videos stored in Google Cloud Storage, and helps you identify key entities (nouns) within your video; and when they occur within the video. Separate signals from noise, by retrieving relevant information within the entire video, shot-by-shot, -or per frame.

Cloud Composer is a "fully managed workflow orchestration service" that empowers you to author, schedule, and monitor pipelines that span across clouds and on-premises data centers.

When you create the view, it must be created in a dataset separate from the source data queried by the view. Because you can assign access controls only at the dataset level, if the view is created in the same dataset as the source data, your data analysts would have access to both the view and the data.

As an example, you might use this feature to export audit log entries from an organization's projects to a central location.

Cloud Dataprep automatically detects schemas, datatypes, possible joins, and anomalies such as missing values, outliers, and duplicates so you get to skip the time-consuming work of profiling your data and go right to the data analysis. Cloud Dataprep automatically identifies data anomalies and helps you to take corrective action fast.

So the best solution is to use HBase with Dataproc which can be installed using initialization actions.

"User" only allows users to run queries across projects. "Owner" gives more privileges to the users. "Viewer" does not give user permissions to run jobs.

In streaming mode, lookup tables need to be accessible by your pipeline. If the lookup table never changes, then the standard Cloud Dataflow SideInput pattern reading from a bounded source such as BigQuery is a perfect fit.

Cloud DLP helps you better understand and manage sensitive data. It provides fast, scalable classification and redaction for sensitive data elements like credit card numbers, names, social security numbers, US and selected international identifier numbers, phone numbers and GCP credentials.

App Engine is a Platform-as-a-Service. It means that you simply deploy your code, and the platform does everything else for you. For example, if your app becomes very successful, App Engine will automatically create more instances to handle the increased volume. Compute Engine is an Infrastructure-as-a-Service. You have to create and configure your own virtual machine instances. It gives you more flexibility and generally costs much less than App Engine. The drawback is that you have to manage your app and virtual machines yourself.

Is communication that can be sent at any time without need for the sender and receiver to be time-synchronized. Asynchronous communications can be sent at any time without regard to whether or not the receiver is ready.

a better option if your application involves real-time person-to-person communications such as for gaming, chat, and streaming of activities.

For pushing, the subscriber needs to be an HTTPS web application. It needs to have a web hook, that is accessible via HTTPS. Because what pubsub is going to do, is that it's going to call that URL. So, in other words, in a push subscription, your subscriber needs to be a server application.

This is for two reasons: Storing one event per row makes it easier to run queries against your data. Storing many events per row makes it more likely that the total row size will exceed the recommended maximum.

Cloud data flow is an execution framework that runs Apache Beam pipelines but there are others as well. There's runtimes for Spark, there's runtimes for Flink and so on.

So the idea is that you write Beam code, you submit it to Dataflow, and Dataflow will run this Beam code on your behalf. And it will keep up with your data stream adding new workers of necessary, removing those workers of necessary. Right, so it gives you a fully managed auto scaling execution environment.

including watermarks, triggers, and lag monitoring of delayed messages. By default, the timestamp is set during the publishing process and represents the real time at which the message is published to PubSub which is a system time.

In this case, rather than use process context.output, you can use process context.output with timestamp. In the example shown, the process context instance is called c, so it shows c.outputWithTimestamp.

So the watermark is DataFlow tracking how far behind the processing time, or how far behind the processing time is from the event time. Dividing the event time up into discrete groups of events is called windowing.

With windows, you're saying where in "event time" you want to process the data. The watermark is a "heuristic or working definition" of how Dataflow receives the data that needs to be computed. And the "trigger" represents when the computation was executed. So, the watermark helps DataFlow decide when it's a good time to close the window.

Windowing can be useful for computations over both bounded and unbounded key collections. But is required, when creating aggregations on infinite data. To capture bursty data such as user activity, a common method is called session based windows.

Once you capture your data onto the Google Transfer Appliance, ship the appliance to the Google upload facility for rehydration. Data rehydration is the process by which you fully reconstitute the files so you can access and use the transferred data.

Cloud SQL cannot support multi-petabyte data. Storage limit for Cloud SQL is 10TB. Cloud Storage does not provide SQL interface. Cloud Datastore does not provide a SQL interface and is a NoSQL solution.

[For BigQuery] You can load data: (1) From Cloud Storage; (2) From other Google services, such as Google Ad Manager and Google Ads; (3) From a readable data source (such as your local machine); (4) By inserting individual records using streaming inserts; (5) Using DML statements to perform bulk inserts; (6) Using a Google BigQuery IO transform in a Cloud Dataflow pipeline to write data to BigQuery

Column filtering cannot be applied to Table, but it can be done through Views.

Wildcard tables enable you to query multiple tables using concise SQL statements. Wildcard tables are available only in standard SQL. The wildcard character, "*", represents one more characters of a table name. The wildcard character can appear only as the final character of a wildcard table name. The wildcard table name contains the special character (*), which means that you must enclose the wildcard table name in backtick (`) characters.

You can query data in a view by adding the dataset containing the view as a data source.

the results of the query are in table format, which becomes the field definition (schema) for your data source.

Data Studio uses your SQL as an inner select statement for each generated query to BigQuery.

Cloud Spanner is the first horizontally scalable, globally consistent database. It's proprietary, not open source.

Here's some example, mission critical use cases. Powering customer authentication and provisioning for multinational businesses. Building consistent systems for transactions and inventory management and the financial services in retail industries. Supporting high volume systems that require low latency and high throughput in the advertising and media industries.

Cloud SQL hits a wall at around 8,000 queries per second.

Spanner distributes work easily. It distributes globally, if that's needed and it provides consistent performance. To support more throughput in Spanner, you just add more nodes.

Bigtable is big, it's fast, it's auto-scaling, and it's no SQL. The data can be semi-structure or it can be structured. It"s built for data that's very fast changing that has a very high throughput and you use it when you don't need transactions, when you don't need strong relational semantics.

Unlike Pub-Sub, unlike BigQuery, it is not cluster free. So, you're thinking in terms of a cluster of nodes with Bigtable. Bigtable like everything else on GCP, your data is encrypted at flight, at rest, on the wire, and you get full control of the data, you get the identity access management that's common to the platform.

So, the clusters consist of nodes, and these nodes contain the metadata; the data itself remains on Google Cloud Storage.

So, that it can do auto-scaling. If we get a new cluster that comes up, it only needs pointers to the data, it doesn't need to have the data.

a narrow table is great for sparse data.

That most common query that you need to support, essentially helps you figure out what your row key needs to be. Your row key needs to reflect this most common query.

Bigtable only stores things in ascending order of the row key. So, you would need to put in the reverse timestamp, so that the latest records are first.

Remember that if you add nodes, it could take Cloud Bigtable about 20 minutes to rebalance the load before it shows a change in performance.

In general, smaller rows offers higher throughput, and therefore is better for streaming performance. Cloud Bigtable takes time to process cells within a row. So if there are fewer cells within a row, it will generally provide better performance than more cells

Is to avoid creating hotspots when writing, which would cause Cloud Bigtable to have to split tablets and adjust loads. To accomplish that, you want the data to be as evenly distributed as possible.

Also, to get valid results your test needs to perform enough actions over a long enough period of time to give Cloud Bigtable the time and conditions necessary to learn the usage pattern and perform its internal optimizations.

Bigtable essentially looks at access patterns. The way that you're accessing it, figures out how many nodes it needs to keep running, figures out where the tablets need to be, how the tablets need to be split up, which machines need to process what data, all of these kinds of things, it learns and looks at the access pattern and reconfigures itself.

If you use HDFS, then you have to keep the Cloud Dataproc cluster running at all times just to persist the storage. But, if you put all the data on Cloud Storage, you can only use HDFS for working files, and shut the cluster down when it's not actually processing a job.

It's common in data applications to simply increase the size of the local disks. That creates dependencies in lift and shift, where you might need to use custom instances. Also, that might mean using persistent disk, but it changes the IOPS. So, you need to plan for throughput capacity and not just storage capacity.

Spinnaker is an open-source, multi-cloud continuous delivery platform that helps you release software changes with high velocity and confidence. Spinnaker provides two core sets of features: (1) application management; (2) application deployment

Setting up Jenkins as a Continuous Integration (CI) system within Spinnaker lets you trigger pipelines with Jenkins, add a Jenkins stage to your pipeline, or add a Script stage to your pipeline.

Telemetry is the automatic recording and transmission of data from remote or inaccessible sources to an IT system in a different location for monitoring and analysis. Telemetry data may be relayed using radio, infrared, ultrasonic, GSM, satellite or cable, depending on the application (telemetry is not only used in software development, but also in meteorology, intelligence, medicine, and other fields).

A data lake is primarily unstructured data from logs, click-streams, and other kinds of dynamic or streaming data such as IoT.

it's essential to design a schema that makes it possible to distribute reads and writes evenly across each table.

A safer approach is to use a reversed version of the user's numeric ID, which spreads traffic more evenly across all of the nodes for your Cloud Bigtable table.

Any data in Cloud Storage persists after your cluster is shut down. This is one of the reasons to choose Cloud Storage over HDFS since HDFS data is removed when a cluster is shut down (unless it is transferred to a persistent location prior to shutdown).

a Dataproc cluster cannot be with only preemptible instances. It needs to have two non-preemptible worker nodes.

This disk space is used for local caching of data and is not available through HDFS. You can override the default disk size with the `gcloud dataproc clusters create --preemptible-worker-boot-disk-sizecommand at cluster creation`

Using the Drain option to stop your job tells the Cloud Dataflow service to finish your job in its current state. Your job will immediately stop ingesting new data from input sources. However, the Cloud Dataflow service will preserve any existing resources, such as worker instances, to finish processing and writing any buffered data in your pipeline. Cancel does not handle in-flight messages and it might result in data loss.

You create instances, which contain up to 4 clusters that your applications can connect to. Each cluster contains nodes, the compute units that manage your data and perform maintenance tasks.

The choice between SSD and HDD is permanent, and every cluster in your instance must use the same type of storage, so make sure you pick the right storage type for your use case.

For instances that use replication, app profiles control how your applications connect to the instance's clusters.

Each cluster is located in a single zone. You can create an additional cluster in any zone where Cloud Bigtable is available. For example, if the first cluster is in us-east1-b, you can choose a different zone in the same region, such as us-east1-c, or a zone in a separate region, such as europe-west2-a.

Tablets are stored on disk, separate from the nodes but in the same zone as the nodes. Each node is responsible for keeping track of specific tablets on disk; handling incoming reads and writes for its tablets; and performing maintenance tasks on its tablets, such as periodic compactions.

Otherwise, the cluster might not be able to handle incoming requests, and latency could go up. You should monitor your clusters' CPU and disk usage, and add nodes if you exceed the recommendations and limits.

Also, if you try to remove nodes from a cluster, and the change would cause the cluster to exceed the hard limit on storage, Cloud Bigtable will deny the request.

If you are using more than the recommended percentage of the storage limit, add nodes to the cluster. You can also delete existing data, but deleted data takes up more space, not less, until a compaction occurs.

Spark is important because it does part of its pipeline processing in memory rather than copying from disk. This allows Spark to choose the best way to distribute the work and run the pipeline. When Spark sees a transformation, it registers it in the directed graph and then it waits. An action triggers Spark to process the pipeline, the output is usually a result format, such as a text file.

Resilient Distributed Datasets (RDD) is a fundamental data structure of Spark. It is an immutable distributed collection of objects. Each dataset in RDD is divided into logical partitions, which may be computed on different nodes of the cluster.

Any function that returns an RDD is a transformation, elaborating it further we can say that Transformation is functions which create a new data set from an existing one by passing each data set element through a function and returns a new RDD representing the results. Actions return final results of RDD computations. Actions trigger execution using lineage graph to load the data into original RDD and carry out all intermediate transformations and return the final results to the Driver program or writes it out to the file system.

DML's are limited by quotas. Maximum number of combined UPDATE, DELETE, and MERGE statements per day per table is 200.

Instead, break your query into stages where each stage materializes the query results by writing them to a destination table. Querying the smaller destination table reduces the amount of data that is read and lowers costs. The cost of storing the materialized results is much less than the cost of processing large amounts of data.

Cloud SQL provides the ability to replicate a master instance to one or more read replicas. A read replica is a copy of the master that reflects changes to the master instance in almost real time. Cloud Storage would not be ideal if the data is relational.

Partner Interconnect provides connectivity between your on-premises network and your VPC network through a supported service provider. A Partner Interconnect connection is useful if your data center is in a physical location that can't reach a Dedicated Interconnect colocation facility or if your data needs don't warrant an entire 10 Gbps connection.

Cloud Router uses Border Gateway Protocol (BGP) to exchange routes between your Virtual Private Cloud network and your on-premises network. On Cloud Router, you configure an interface and a BGP peer for your on-premises router. The interface and BGP peer configuration together form a BGP session.

For layer 2 connections, four virtual circuits are required, split between two metros. Layer 2 also requires you to add four BGP sessions to the on-premises router, one for each Cloud Router.

If your applications aren't mission-critical and can tolerate some downtime, you can use the 99.9% availability configuration.

Serverless services remove more of the IT responsibility so managing the underlying servers is not part of your overhead and the individual instances are not visible.

Cloud Dataprep is used to prepare or condition data and to prepare pipelines before processing data. Cloud Datalab is a notebook, that is a self-contained workspace that holds code, executes the code, and displays results. Dialogflow is a service for creating chatbots. It uses AI to provide a method for direct human interaction with data.

Then it can be transmitted and stored and when it's retrieved, the data needs to be de-serialized to restore the structure into a meaningful data object.

Its primary use is in Apache Hadoop where it can provide both a serialization format for persistent data and a wire format for communication between Hadoop nodes and from client programs to the Hadoop services. It helps to understand the data type supported in different representations systems.

Numeric is very good at storing common fractions associated with money. Numeric avoids the rounding error that occurs in a full floating point representation. So, it's used primarily for financial transactions.

An authorized view allows you to share query results with particular users and groups without giving them access to the underlying tables. You can also use the view's SQL query to restrict the columns (fields) the users are able to query.

Cloud SQL is not NoOps because it requires SQL usage.

The only difference between the traditional programming and neural network is that you let the computer determine the parameters (weights and bias) by learning from training datasets.

With BigQuery, you’ll get great performance on your data, while knowing you can scale seamlessly to store and analyze petabytes more without having to buy more capacity.

(1) Linear regression: These models can be used for predicting a numerical value. (2) Binary logistic regression: These models can be used for predicting one of two classes (such as identifying whether an email is spam). (3) Multiclass logistic regression for classification: These models can be used to predict more than two classes such as whether an input is "low-value", "medium-value", or "high-value".

You can use Natural Language API to extract information about people, places, events, and much more mentioned in text documents, news articles, or blog posts. You can also use it to understand sentiment about your product on social media or parse intent from customer conversations happening in a call center or a messaging app.

Synchronous speech recognition returns the recognized text for short audio (less than ~1 minute) in the response as soon as it is processed. To process a speech recognition request for long audio, use Asynchronous Speech Recognition.

Cloud Dataproc is a fast, easy to use, managed Spark and Hadoop service for distributed data processing, but it does not help easy orchestration. Cloud Scheduler is a fully managed enterprise-grade cron job scheduler, however it is not an orchestration tool.

- Can read, query, copy or export tables in the dataset - Can call 'get' on the dataset - Can call 'get' and 'list' on tables in the dataset - Can call 'list' on table data for tables in the dataset

- Same as READER, plus: - Can edit or append data in the dataset - Can call insert, insertAll, update or delete data in the dataset - Can use tables in the dataset as destinations for load, copy or query jobs

- Same as WRITER, plus: - Can call 'update' on the dataset - Can call 'delete' on the dataset

A user with the OWNER role can't remove their own OWNER role.

Process diverse datasets — structured and unstructured. Transform data stored in CSV, JSON, or relational table formats. Prepare datasets of any size, megabytes to terabytes, with equal ease.

Choose App Engine to capture and insert the data into BigQuery using streaming inserts. The data can then be analyzed and visualized using DataStudio.

Slots Allocated to the project and Slots Available for the project.

For example, a data stream representing user mouse activity may have long periods of idle time interspersed with high concentrations of clicks. If data arrives after the minimum specified gap duration time, this initiates the start of a new window.

But if you were to issue the Drain command instead, Cloud Dataflow immediately closes any in-process windows and fires all triggers. The system does not wait for any outstanding time-based windows to finish.

"gsutil rsync" command makes the contents under dst_url the same as the contents under src_url, by copying any missing files/objects (or those whose data has changed), and (if the -d option is specified) deleting any extra files/objects. src_url must specify a directory, bucket, or bucket subdirectory.

Without access to the Internet, you can enable Private Google Access, and place job dependencies in Cloud Storage; cluster nodes can download the dependencies from Cloud Storage from internal IPs.

Network address translation (NAT) takes your private IP addresses and translates them into public IP addresses. This helps conserve valuable public addresses while at the same time allows hosts in your network to access services and remote hosts across the Internet (or other public network).

(1) Creating an instance from an image; (2) Creating an instance from a public image; (3) Creating an instance from a custom image; (4) Creating an instance with an image shared with you; (5) Creating an instance from a snapshot; (6) Creating an instance from a container image

However, BigQuery is more of an a scalable, fully managed enterprise data warehousing solution and not ideal fast changing data.

Then you can shut your cluster down when you're not actually processing a job.

First, trying to tweak all of its settings so it can run efficiently with multiple different kinds of jobs, and second, trying to cost justify utilization.

are API calls that reference the functions you want them to perform.

They're defined inline, making the sequence of the code easier to read and understand. And because the code is used in only one place, the function doesn't need a name and it doesn't clutter the namespace.

when the system tries to process the data as soon as it's received

Sometimes the answer is to extract the data from BigQuery into Cloud Dataproc and let Spark run the analysis. Also, if you needed to alter or process the data, you might read from BigQuery into Cloud Dataproc, process the data, and write it back out to another dataset in BigQuery.

You may want to run a Spark job on the data.

And the alternate to HBase is BigTable; and the alternate to HDFS is Cloud Storage.

is controlled by access to the cluster as a resource.

You can use the open source Apache Beam API to define the pipeline and submit it to Cloud Dataflow. Then Cloud Dataflow provides the execution framework.

and the same code does real-time streaming and batch processing.

Cloud Dataflow supports side inputs. That's where you can take data and transform it in one way and transform it in a different way in parallel so that the two can be used together in the same pipeline.

to limit access to only dataflow resources not just the project.

than an application running on an instance.

Dataflow Templates open up new options for separation of work. That means better security and resource accountability.

SQL, transactions, and strong consistency at scale

a fully managed database service that makes it easy to set up, maintain, manage, and administer your relational PostgreSQL, MySQL, and SQL Server databases in the cloud.

Low latency, massively scalable, NoSQL

NoSQL database built for global apps

The Firebase Realtime Database is a cloud-hosted NoSQL database that lets you store and sync data between your users in realtime. (Real time syncing for JSON data)

Cloud Memorystore for Redis provides a fully managed in-memory data store service built on scalable, secure, and highly available infrastructure managed by Google. Use Cloud Memorystore to build application caches that provides sub-millisecond data access.

Cloud Datastore is a highly-scalable NoSQL database for your web and mobile applications. Cloud Datastore provides a myriad of capabilities such as ACID transactions, SQL-like queries, and indexes.

Log queries are labelled as "filters" in the user interface, since they let you select a particular set of logs entries.

Stackdriver aggregates metrics, logs, and events from infrastructure, giving developers and operators a rich set of observable signals that speed root-cause analysis and reduce mean time to resolution (MTTR).

connects your application’s production data to your source code by inspecting the state of your application at any code location in production without stopping or slowing down your requests.

analyzes and aggregates the errors in your cloud applications. Notifies you when new errors are detected.

provides latency sampling and reporting for App Engine, including per-URL statistics and latency distributions.

provides you with the ability to filter, search, and view logs from your cloud and open source application services. Allows you to define metrics based on log contents that are incorporated into dashboards and alerts. Enables you to export logs to BigQuery, Cloud Storage, and Cloud Pub/Sub.

provides endpoint checks to web applications and other internet-accessible services running on your cloud environment. You can configure uptime checks associated with URLs, groups, or resources, such as instances and load balancers.

provides continuous profiling of resource consumption in your production applications, helping you identify and eliminate potential performance issues.

In its outer layer, the Redis data model is a dictionary which maps keys to values. One of the main differences between Redis and other structured storage systems is that Redis supports not only strings, but also abstract data types.

Cassandra on the other hand is a recent open source fork of a standalone database system initially coded by Facebook, which while implementing the BigTable data model, uses a system inspired by Amazon’s Dynamo for storing data.

BigTop also features vagrant recipes for spinning up "n-node" hadoop clusters, and the bigpetstore blueprint application which demonstrates construction of a full stack hadoop app with ETL, machine learning, and dataset generation.

It enables users to configure, manage and orchestrate data motion, pipeline processing, disaster recovery, and data retention workflows. Instead of hard-coding complex data lifecycle capabilities, Hadoop applications can now rely on the well-tested Apache Falcon framework for these functions.

Workflow scheduler system for MR jobs using DAGs (Direct Acyclical Graphs). Oozie Coordinator can trigger jobs by time (frequency) and data availability

The Hadoop Distributed File System (HDFS) offers a way to store large files across multiple machines.

It includes a distributed key-value in-memory store, SQL capabilities, map-reduce and other computations, distributed data structures, continuous queries, messaging and events subsystems, Hadoop and Spark integration.

The current Apache MapReduce version is built over Apache YARN Framework. YARN stands for “Yet-Another-Resource-Negotiator”. It is a new framework that facilitates writing arbitrary distributed processing frameworks and applications.

With its ability to integrate with Hadoop and inbuilt tools for interactive query analysis (Shark), large-scale graph processing and analysis (Bagel), and real-time analysis (Spark Streaming), it can be interactively used to quickly process and query big data sets.

Is a distributed real-time computation system for processing fast, large streams of data. Storm is an architecture based on master-workers paradigma. So a Storm cluster mainly consists of a master and worker nodes, with coordination done by Zookeeper.

Flink is a data processing system and an alternative to Hadoop's MapReduce component. It comes with its own runtime, rather than building on top of MapReduce.

It provides a simple API that enables users to write or re-use generic Java code, thereby lowering the expertise needed to write big data applications.

Apache REEF™ (Retainable Evaluator Execution Framework) is a library for developing portable applications for cluster resource managers such as Apache Hadoop™ YARN or Apache Mesos.

Twill uses a simple thread-based model that Java programmers will find familiar. YARN can be viewed as a compute fabric of a cluster, which means YARN applications like Twill will run on any Hadoop 2 cluster.

Many data analysis techniques such as machine learning and graph algorithms require iterative computations, this is where Bulk Synchronous Parallel model can be more effective than "plain" MapReduce.

Tez generalizes the MapReduce paradigm to a more powerful framework based on expressing computations as a dataflow graph. Tez is not meant directly for end-users – in fact it enables developers to build end-user applications with much better performance and flexibility.

DataFu also provides Hadoop jobs for incremental data processing in MapReduce. DataFu is a collection of Pig UDFs (including PageRank, sessionization, set operations, sampling, and much more) that were originally developed at LinkedIn.

Apache Beam is an open source, unified model for defining and executing data-parallel processing pipelines, as well as a set of language-specific SDKs for constructing pipelines and runtime-specific Runners for executing them.

Random, real-time r/w operations in column-oriented very large tables (BDDB: Big Data Data Base). It’s the backing system for MR jobs outputs. It’s the Hadoop database. It’s for backing Hadoop MapReduce jobs with Apache HBase tables

Distributed key/value store is a robust, scalable, high performance data storage and retrieval system. Accumulo is software created by the NSA with security features.

Distributed, columnar, relational data store optimized for analytical use cases requiring very fast reads with competitive write speeds.

Columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model or programming language.

Document-oriented database system. It is part of the NoSQL family of database systems. Instead of storing data in tables as is done in a "classical" relational database, MongoDB stores structured data as JSON-like documents

Data Warehouse infrastructure developed by Facebook. Data summarization, query, and analysis. It’s provides SQL-like language.

Drill is the open source version of Google's Dremel system which is available as an infrastructure service called Google BigQuery.

Tajo is designed for low-latency and scalable ad-hoc queries, online aggregation, and ETL (extract-transform-load process) on large-data sets stored on HDFS (Hadoop Distributed File System) and other data sources.

It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms.

Large scale log aggregator, and analytics.

Distributed publish-subscribe system for processing large amounts of streaming data.

Thrift manages serialization of data to and from a service, as well as the protocol that describes a method invocation, response, etc. Instead of writing all the RPC code -- you can just get straight to your service logic.

Zookeeper is for building distributed systems, simplifies the development process, making it more agile and enabling more robust implementations.

Avro data is described by a schema, and one interesting feature is that the schema is stored in the same file as the data it describes, so files are self-describing.

When a patch is uploaded to the bug tracking system an automated process downloads the patch, performs some static analysis, and runs the unit tests. These results are posted back to the bug tracker and alerts notify interested parties about the state of the patch.

An independent security module that integrates with open source SQL query engines Apache Hive and Cloudera Impala, Sentry delivers advanced authorization controls to enable multi-user applications and cross-functional processes for enterprise data sets.

delivers comprehensive approach to central security policy administration across the core enterprise security requirements of authentication, authorization, accounting and data protection.

Mesos is a cluster manager that provides resource sharing and isolation across cluster applications.

Apache Helix is a generic cluster management framework used for the automatic management of partitioned, replicated and distributed resources hosted on a cluster of nodes.

It analyzes data activities, yarn applications, jmx metrics, and daemon logs etc., provides state-of-the-art alert engine to identify security breach, performance issues and shows insights.

A Web crawler is an Internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing. Web crawlers can copy all the pages they visit for later processing by a search engine that indexes the downloaded pages so that users can search them much more quickly.

Apache Giraph is an iterative graph processing system built for high scalability.

Toolkit detects and extracts metadata and structured text content from various documents using existing parser libraries.

Zeppelin is a modern web-based tool for the data scientists to collaborate over large-scale data exploration and visualization projects. It is a notebook style interpreter that enable collaborative analysis sessions sharing between users.

offers a large range of analytical functions, a highly functional semantic layer often absent in other open source platforms and projects, and a respectable set of advanced data visualization features including geospatial analytics

The Apache-licensed Impala project brings scalable parallel database technology to Hadoop, enabling users to issue low-latency SQL queries to data stored in HDFS and Apache HBase without requiring data movement or transformation.

These networks of models are called feedforward because the information only travels forward in the neural network, through the input nodes then through the hidden layers (single or many layers) and finally through the output nodes. In MLN there are no feedback connections such that the output of the network is fed back into itself.

A recurrent neural network is a class of artificial neural networks where connections between nodes form a directed graph along a temporal sequence. This allows it to exhibit temporal dynamic behavior. Unlike feedforward neural networks, RNNs can use their internal state to process sequences of inputs.

Cloud Bigtable scales in direct proportion to the number of machines in your cluster. No more managing masters or regions; just design your table schemas, and Cloud Bigtable will handle the rest for you.

-Keep keys in the cloud, for direct use by cloud services -Manage your keys in a cloud-hosted solution -You can create, rotate, automatically rotate and destroy symmetric encryption keys

-Keep keys on-premises, and use them to encrypt your cloud services -Use your own encryption keys as part of services on Google Cloud Platform -Google uses the key in memory and does not write it to storage -You provide the keys as part of API service calls

Google Cloud Storage is not suitable to handle real time streaming data. It also needs to be used with BigQuery for analytics.

Remember that the billing needs to be in a single project, the jobs need to be executed from the billing project with data access to the dataset from different projects.

However, it does not provide access to view the data.

After you configure a data transfer, the BigQuery Data Transfer Service automatically schedules and manages recurring data loads from the source application into BigQuery.

Online predictions are returned in the response message. Batch predictions are optimized to handle a high volume of instances in a job and to run more complex models. Online predictions do not store data in Cloud Storage and batch predictions do not provide low latency.

You can avoid exploding indexes by avoiding queries that would require a custom index using a list property. As described above, this includes queries with multiple sort orders or queries with a mix of equality and inequality filters.

In comparison to a "Pipeline" that encapsulates an entire series of computations.

Also wildcards do not support views.

Queries against time-partitioned tables can restrict the data read by supplying _PARTITIONTIME filters that represent a partition's location. All the data in the specified partition is read by the query, but the _PARTITIONTIME predicate filter restricts the number of partitions scanned.

Use if the requirement is that the file cannot be decrypted in the cloud, so encrypt it before it is uploaded and after it is downloaded adds a layer of encryption.

Encryption that occurs after Cloud Storage receives your data, but before the data is written to disk and stored.

Cloud VPN is over the internet through IPSec VPN at a low cost for your data bandwidth needs up to 3.0 Gbps; and Cloud Router helps only in dynamic routing.

Cloud Memorystore for Redis provides a fully managed in-memory data store service built on scalable, secure, and highly available infrastructure managed by Google; but it does not support consistent transactions.

Denormalization requires more storage, and also increased the amount of data processed.

The ultimate goal, assuming a large enough dataset, is to help a machine mimic the human brain’s feature extraction and abstract association capabilities from new uncategorized data.

Multi-Regional and Coldline storage classes provide multi-region geo-redundant deployment, which can sustain regional failure.

During training, the agent receives a reward when it performs this task, which is called a reward function.

You feed the features and their corresponding labels into an algorithm in a process called training. During training, the algorithm gradually determines the relationship between features and their corresponding labels. This relationship is called the model.

The goal is to identify meaningful patterns in the data. To accomplish this, the machine must learn from an unlabeled data set. In other words, the model has no hints how to categorize each piece of data and must infer its own rules for doing so.

Your data source can be an Amazon Simple Storage Service (Amazon S3) bucket, an HTTP/HTTPS location, or a Cloud Storage bucket. Your data sink (the destination) is always a Cloud Storage bucket.

Synchronize content of two buckets/directories

Copy files and objects

Update to the latest gsutil release

The rsync -d option is very useful and commonly used, because it provides a means of making the contents of a destination bucket or directory match those of a source bucket or directory.

means you are making a copy in memory of the actual parameter's value that is passed in, a copy of the contents of the actual parameter. Use "pass by value" when when you are only "using" the parameter for some computation, not changing it for the client program.

(also called pass by address), a copy of the address of the actual parameter is stored. Use pass by reference when you are changing the parameter passed in by the client program.

Cloud Datalab is a powerful interactive tool created to explore, analyze, transform, and visualize data and build machine learning models on Google Cloud Platform. Cloud Datalab is built on Jupyter, which boasts a thriving ecosystem of modules and a robust knowledge base.

Offline prediction. This is when your ML model is used in a batch scoring job for a large number of data points, where predictions are not required in real-time serving. Online prediction. This is when your ML system is used to serve real-time predictions, based on online requests from the operational systems and apps.

is the process of reducing the number of random variables under consideration, by obtaining a set of principal variables. It can be divided into feature selection and feature extraction.

You cannot include both compressed and uncompressed files in the same load job.

If you have CSV files with data encoded in ISO-8859-1 (also known as Latin-1) format, you should explicitly specify the encoding when you load your data so it can be converted to UTF-8.

If you are loading data from Cloud Storage, you also need permissions to access to the bucket that contains your data.

-bigquery.tables.create -bigquery.tables.updateData -bigquery.jobs.create

-bigquery.dataEditor -bigquery.dataOwner -bigquery.admin

-bigquery.user -bigquery.jobUser -bigquery.admin

bigquery.dataOwner access gives the user the ability to create and update tables in the dataset via a load job.

With the launch of BigQuery 2.0, BigQuery released support for standard SQL, and renamed BigQuery SQL to legacy SQL. Standard SQL is the preferred SQL dialect for querying data stored in BigQuery.

In standard SQL, you escape such keywords and identifiers using backticks ` .

Cloud APIs provide similar functionality to Cloud SDK and Cloud Console, and allow you to automate your workflows by using your favorite language.

is a software architectural style that defines a set of constraints to be used for creating Web services. Web services that conform to the REST architectural style, called RESTful Web services (RWS), provide interoperability between computer systems on the Internet .

So, a 404 means a requested resource wasn't found; a 401 code means the request wasn't authorized; a 200 code means everything is OK; and a 500 means there was an unrecoverable application error on the server.

the reported schema will be inaccurate until the view is updated. Even though the reported schema may be inaccurate, all submitted queries produce accurate results.

And you cannot reference query parameters in views.

Disable the trigger to prevent new traffic while debugging; change the pipeline from save to restore and then hold off any other testing while changes are being made.

Set "debug" to "true". When debug is true, the branch statement will do a restore pipeline and load the file that had been run before.

That's because the most efficient Cloud Bigtable queries use the row key, a row key prefix, or a row range to retrieve the data. Other types of queries trigger a full table scan, which is much less efficient. By choosing the correct row key now, you can avoid a painful data-migration process later.

Using a reverse domain name is an especially good idea if each row's data tends to overlap with adjacent rows. In this case, Cloud Bigtable can compress your data more efficiently. This approach works best when your data is spread across many different reverse domain names.

However, because new users are more likely to be active users, this approach is likely to push most of your traffic to a small number of nodes. A safer approach is to use a reversed version of the user's numeric ID, which spreads traffic more evenly across all of the nodes for your Cloud Bigtable table.

Each table has only one index, the row key. There are no secondary indices.

1. Export the Big Table content into Cloud Storage 2. Transform the row keys into the new format 3. Import the adjusted data into the new Big Table

Instead of using a self-join, use a window (analytic) function to reduce the number of additional bytes that are generated by the query.

To avoid performance issues that result from data skew: - Use an approximate aggregate function such as APPROX_TOP_COUNT to determine if the data is skewed. - Filter your data as early as possible.

To avoid performance issues that are associated with unbalanced joins: - Pre-filter rows from the table with the unbalanced key. - If possible, split the query into two queries.

If you need OLTP-like behavior (single-row updates or inserts), consider a database designed to support OLTP use cases such as Cloud SQL.

Add another attribute (maybe Order_Id) to the message payload to mark the unique check-out order across all terminals. Make sure that messages whose "orderId" and "prodSku" values match corresponding rows in the BigQuery table are discarded.

Increase the complexity of your model by, e.g., introducing an additional layer or increasing the size of vocabularies or n-grams used to avoid underfitting.

Use Cloud Machine Learning to host your model. Monitor the status of the Jobs object for 'failed' job states.

Call the Cloud Natural Language API from your application. Process the generated Entity Analysis as labels. This is correct because it provides a managed service and a fully trained model, and the user is pulling the entities, which is the right label.

Given these requirements, the recommended approach will be to write the data to BigQuery for #1 and to Cloud Bigtable for #2. A PCollection is immutable, so you can apply multiple transforms to the same one.

Use the Cloud Dataflow Counting source transform to emit a value daily, beginning on the day you create the pipeline. - Pass this value into a global window via a data-driven trigger that activates on each element. - In a DoFn, use this process as a trigger to pull data from your bounded source (such as BigQuery). - Create your SideInput for use in downstream transforms.

You design the pipeline to use a session window with a gap time duration of 60 minutes, because it will send a message per user after that user is inactive for 60 minutes. A Global Window is not correct because it will cause messages to be sent out every 60 minutes to all users regardless of where they are in their current session.

Store and update the data in a regional Google Cloud Storage bucket and create a federated data source in BigQuery. This is correct because regional storage is cheaper than BigQuery storage.

Cloud Dataproc also collects logs and console output and puts them in the Cloud Storage staging bucket associated with the cluster you run the job on.

Logs are captured in Cloud Storage by default, using the staging bucket that you specify when you create the cluster.

Grouping your data into buckets that correspond to your business structure or functional areas can also make it easier to manage permissions.