Google Cloud Professional Data Engineer

Automates data movement from SaaS applications to Google BigQuery on a scheduled, managed basis. Your analytics team can lay the foundation for a data warehouse without writing a single line of code.

Supports Google Ads, Campaign Manager, Google Ad Manager, and YouTube Content and Channel Owner Reports. Through BigQuery Data Transfer Service, users also gain access to data connectors that allow you to easily transfer data from Teradata and Amazon S3 to BigQuery.

You can use Cloud Dataflow IAM roles to limit access for users within a project or organization, to just Cloud Dataflow-related resources, as opposed to granting users viewer, editor, or owner access to the entire Cloud Platform project.

Minimal role for creating and managing dataflow jobs.

Provides the permissions necessary to execute and manipulate Cloud Dataflow jobs.

Provides read-only access to all Cloud Dataflow-related resources.

Provides the permissions necessary for a Compute Engine service account to execute work units for a Cloud Dataflow pipeline.

- The developer who creates and examines jobs will need the roles/dataflow.admin role. - For more sophisticated permissions management, the developer interacting with the Cloud Dataflow job will need the roles/dataflow.developer role. - They will need the roles/storage.objectAdmin or a related role in order to stage the required files.

- For debugging and quota checking, they will need the project roles/compute.viewer role. - Absent other role assignments, this will allow the developer to create and cancel Cloud Dataflow jobs, but not interact with the individual VMs or access other Cloud services. - The controller service account needs the roles/dataflow.worker role to process data for the Cloud Dataflow service. It will need other roles (such as roles/storage.objectAdmin in order to access job data.

Cloud Dataflow roles can currently be set on organizations and projects only.

When an identity calls a Google Cloud Platform API, Google BigQuery requires that the identity has the appropriate permissions to use the resource. You can grant permissions by granting roles to a user, a group, or a service account.

- Primitive roles include the Owner, Editor, and Viewer roles that existed prior to the introduction of Cloud Identity and Access Management. - Predefined roles provide granular access for a specific service and are managed by GCP. Predefined roles are meant to support common use cases and access control patterns. - Custom roles provide granular access according to a user-specified list of permissions.

- The gcloud iam roles describe command - The roles.get() method in the Cloud IAM API

The permissions granted are a union of each role's permissions.

You provide permission to run BigQuery jobs or to manage all of a project's BigQuery resources.

In the IAM policy hierarchy, BigQuery datasets are child resources of projects. Tables and views are child resources of datasets — they inherit permissions from their parent dataset.

You automatically have the equivalent of the bigquery.jobs.get and bigquery.jobs.update permissions for that job.

Provides permissions to manage all resources within the project. Can manage all data within the project, and can cancel jobs from other users running within the project.

When applied to a dataset, dataEditor provides permissions to: - Read the dataset's metadata and to list tables in the dataset. - Create, update, get, and delete the dataset's tables. When applied at the project or organization level, this role can also create new datasets.

When applied to a dataset, dataOwner provides permissions to: - Read, update, and delete the dataset. - Create, update, get, and delete the dataset's tables. When applied at the project or organization level, this role can also create new datasets.

When applied to a dataset, dataViewer provides permissions to: - Read the dataset's metadata and to list tables in the dataset. - Read data and metadata from the dataset's tables. When applied at the project or organization level, this role can also enumerate all datasets in the project. Additional roles, however, are necessary to allow the running of jobs.

Provides permissions to run jobs, including queries, within the project. The jobUser role can enumerate their own jobs and cancel their own jobs.

When applied at the project or organization level, metadataViewer provides permissions to: - List all datasets and read metadata for all datasets in the project. - List all tables and views and read metadata for all tables and views in the project. Additional roles are necessary to allow the running of jobs.

Access to create and use read sessions

Provides permissions to run jobs, including queries, within the project. The user role can enumerate their own jobs, cancel their own jobs, and enumerate datasets within a project. Additionally, allows the creation of new datasets within the project; the creator is granted the bigquery.dataOwner role for these new datasets.

A partitioned table is a special table that is divided into segments, called partitions, that make it easier to manage and query your data. By dividing a large table into smaller partitions, you can improve query performance, and you can control costs by reducing the number of bytes read by a query.

- Tables partitioned by ingestion time: Tables partitioned based on the data's ingestion (load) date or arrival date. - Partitioned tables: Tables that are partitioned based on a TIMESTAMP or DATE column.

BigQuery automatically loads data into daily, date-based partitions that reflect the data's ingestion or arrival date. Pseudo column and suffix identifiers allow you to restate (replace) and redirect data to partitions for a specific day.

The partitions have the same schema definition as the table. If you need to load data into a partition with a schema that is not the same as the schema of the table, you must update the schema of the table before loading the data. Alternatively, you can use schema update options to update the schema of the table in a load job or query job.

Partitioned tables allow you to bind the partitioning scheme to a specific TIMESTAMP or DATE column. Data written to a partitioned table is automatically delivered to the appropriate partition based on the date value (expressed in UTC) in the partitioning column.

- The __NULL__ partition — represents rows with NULL values in the partitioning column - The __UNPARTITIONED__ partition — represents data that exists outside the allowed range of dates

You can shard tables using a time-based naming approach such as [PREFIX]_YYYYMMDD. This is referred to as creating date-sharded tables. Using either standard SQL or legacy SQL, you can specify a query with a UNION operator to limit the tables scanned by the query.

When you create date-named tables, BigQuery must maintain a copy of the schema and metadata for each date-named table. Also, when date-named tables are used, BigQuery might be required to verify permissions for each queried table. This practice also adds to query overhead and impacts query performance.

A wildcard table represents a union of all the tables that match the wildcard expression. For example, the following FROM clause uses the wildcard expression gsod* to match all tables in the noaa_gsod dataset that begin with the string gsod.

The wildcard table functionality does not support views. If the wildcard table matches any view in the dataset, the query returns an error. This is true whether or not your query contains a WHERE clause on the _TABLE_SUFFIX pseudo column to filter out the view. Currently, cached results are not supported for queries against multiple tables using a wildcard even if the Use Cached Results option is checked. If you run the same wildcard query multiple times, you are billed for each query.

Wildcard tables support native BigQuery storage only. You cannot use wildcards when querying an external table or a view. Queries that contain Data Manipulation Language (DML) statements cannot use a wildcard table as the target of the query. For example, a wildcard table may be used in the FROM clause of an UPDATE query, but a wildcard table cannot be used as the target of the UPDATE operation.

Wildcard tables are useful when a dataset contains multiple, similarly named tables that have compatible schemas.

To query a group of tables that share a common prefix, use the table wildcard symbol (*) after the table prefix in your FROM statement.

To restrict the query so that it scans an arbitrary set of tables, use the _TABLE_SUFFIX pseudo column in the WHERE clause. The _TABLE_SUFFIX pseudo column contains the values matched by the table wildcard.

To scan a range of tables, use the _TABLE_SUFFIX pseudo column along with the BETWEEN clause.

To scan a range of ingestion-time partitioned tables, use the _PARTITIONTIME pseudo column with the _TABLE_SUFFIX pseudo column.

To scan all tables in a dataset, you can use an empty prefix and the table wildcard, which means that the _TABLE_SUFFIX pseudo column contains full table names.

Cloud Spanner automatically creates an index for each table's primary key column. You can also create secondary indexes for other columns. Adding a secondary index on a column makes it more efficient to look up data in that column. For example, if you need to quickly look up a set of SingerId values for a given range of LastName values, you should create a secondary index on LastName, so Cloud Spanner does not need to scan the entire table.

Every Cloud Firestore (in Datastore mode query) computes its results using one or more indexes which contain entity keys in a sequence specified by the index's properties and, optionally, the entity's ancestors. The indexes are updated to reflect any changes the application makes to its entities, so that the correct results of all queries are available with no further computation needed.

Built-in indexes - By default, a Datastore mode database automatically predefines an index for each property of each entity kind. These single property indexes are suitable for simple types of queries. Composite indexes - Composite indexes index multiple property values per indexed entity. Composite indexes support complex queries and are defined in an index configuration file (index.yaml).

However, it does not support some kinds of query common in other database technologies: in particular, joins and aggregate queries aren't supported within the Datastore mode query engine.

• Each table has only one index, the row key. • Rows are sorted lexicographically by row key, • Columns are grouped by column family and sorted in lexicographic order within the column family. • All operations are atomic at the row level.

• Ideally, both reads and writes should be distributed evenly • In general, keep all information for an entity in a single row. • Related entities should be stored in adjacent rows, • Cloud Bigtable tables are sparse.

- User information: Do you need quick access to information about connections between users (for example, whether user A follows user B)? - User-generated content: If you show users a sample of a large amount of user-generated content, such as status updates, how will you decide which status updates to display to a given user? - Time series data: Will you often need to retrieve the most recent N records, or records that fall within a certain time range? If you're storing data for several kinds of events, will you need to filter based on the type of event?

Hive is declarative and Pig is imperative.

Specifies exactly how to perform the data analysis, which limits the flexibility of the underlying systems. They can't decide how to use the available resources to get the results.

Makes execution plans but then requests the underlying systems determine how to process the data. So Pig can fit better into the pipeline paradigm of processing data.

The way it does this is by maintaining metadata to define a schema on top of the data. This is one way to work with a large amount of distributed data in HDFS using familiar SQL syntax.

It ingests data into a data warehouse format requiring a schema. It does not support real-time queries, row-level updates, or unstructured data. Some queries may run much slower than others due to the underlying transformations Hive has to implement to simulate SQL.

Pig can also deal with semi-structured data, such as data having partial schemas, or for which the schema is not yet known. For this reason it is sometimes used for Extract Transform Load (ETL). It generates Java MapReduce jobs. Pig is not designed to deal with unstructured data.

A messaging service that decouples senders and receivers, enabling asynchronous messaging for ingesting data. This is useful for ingesting streaming data. Publishers publish messages to a topic, and subscribers subscribe to the topic and receive the messages. Messages are stored until they're delivered and acknowledged by the subscribers.

A service that provides a unified method for both batch and streaming data processing. Dataflow builds on the Apache Beam SDK.

A data warehouse service with a highly scalable interactive data exploration interface. It can capture and analyze data in real time. So it makes a great partner for data that's been ingested through Pub/Sub and processed though Dataflow.

You create a pipeline job in Python or Java using Apache Beam. Data is read in batch from Cloud Storage or as a stream through Pub/Sub, it is processed in Dataflow. The results are stored in BigQuery for interactive analysis or in Bigtable for use by applications, there's nothing to maintain.

Among them is "class" which declares the frequency of anticipated access of the object. This allows the Cloud Storage service to distinguish between types of objects and internally manage them to service levels

● Avoid small reads ● Use large block/file sizes where possible ● Avoid iterating over many nested directories in a single job

Throughput for processing data in Cloud storage is higher than throughput for HDFS on Persistent disk in the cluster.

Colossus also contains data structures called Tablets that are used to identify and manage the data. And metadata about the Tablets is what is stored on the VMs in the Bigtable cluster itself.

It can manipulate the actual data. It can manipulate the Tablets that point to and describe the data. Or it can manipulate the metadata that points to the Tablets. Rebalancing tablets from one node to another is very fast, because only the pointers are updated.

It detects "hot spots" where a lot of activity is going through a single Tablet and splits the Tablet in two. It can also rebalance the processing by moving the pointer of a tablet to a different VM in the cluster. So its best use case is with big data -- above 300 GB

Recovery is fast because only the metadata needs to be copied to the replacement node. Colossus provides better durability than the default 3 replicas provided by HDFS.

That index is called the Row Key. There are no alternate indexes or secondary indexes. And when data is entered, it is organized lexicographically by the Row Key.

(Reduced Instruction Set Computing). Simplify the operations. And when you don't have to account for variations, you can make those that remain very fast.

Writing a query that controls the amount of data processed. In general, this is done with SELECT by choosing subsets of data at the start of a job rather than by using LIMIT which only omits data from the final results at the end of a job.

-Step one, copy the data to Cloud Storage. -Step two, update the file prefix in your application scripts from HDFS to GS. You just change hdfs:// to gs://. -Step three, create a Dataproc cluster and run your job. You can also stage your scripts in Cloud Storage. In most cases, that's all you have to do to get your current job running in Dataproc.

It makes scaling decisions based on Hadoop YARN Metrics. It is designed to be used only with off-cluster persistent data, not on-cluster HDFS or HBase. It works best with a cluster that processes a lot of jobs or that processes a single large job.

1) A fixed amount of time after the cluster enters the Idle state. 2) Set a timer. (You give it a timestamp) The count starts immediately once the expiration has been set. 3) Set a duration. Time in seconds to wait before deleting the cluster. (Range is from 10 minutes minimum to 14 days maximum, with a granularity of 1 second)

Only when a request is submitted that requires output, the Spark actually process the data. The benefit of this strategy is that Spark can look at all the requests and the intermediate results and construct parallel pipelines based on the resources that are available in the cluster at that time.

When it is passed to Spark in this way, Spark understands the multiple steps and that the results of one transformation are to be passed to the next transformation. This allows Spark to organize the processing in any way it decides based on the resources available in the cluster.

You deploy your code to the Cloud Functions service and set it up to be triggered by a class of events.

However, for data processing there are tools such as Cloud Dataproc Workflow Templates and Cloud Composer that are designed to manage workflows without having to code the service yourself.

At that time you specify the trigger that will cause the Cloud Function to run, such as the trigger bucket for Cloud Storage or the trigger topic for Cloud Pub/Sub.

Map Reduce, Hadoop, Spark Scala, and Spark Pyspark

To overcome this problem, Cloud Dataproc's BigQuery Connectors use Cloud Storage. The workers are able to write their shards as objects in Cloud Storage. When the work is complete, as part of the connector shutdown process, a single load job is issued in BigQuery.

It can create a new cluster, select from an existing cluster, submit jobs, hold jobs for submission until dependencies can complete, and it can delete a cluster when the job is done.

Cloud Composer can be used to automate Cloud Dataproc jobs and to control clusters.

So, you don't really need to choose. You can store your data if it's structured data, it's tabular data, store it in BigQuery and you will get the same sort of discounts that you get in cloud storage as well. So if you have some data and you haven't edited it in a few weeks, you start getting automatically the discounted rates for older data.

The answer for that would be something like Cloud SQL or something like Spanner. But for ad hoc analysis of very large datasets, for data warehousing, for business intelligence, those kinds of operations, BigQuery is a great choice.

You put your data onto BigQuery, you're not managing any clusters beyond that point. So you want to run a query on BigQuery, you just run the query on BigQuery. You don't need to create a cluster ahead of time. You only pay when you query the data.

A dataset is basically a collection of tables. Dataset is a collection of tables belong to your organization, and you basically do access control on a dataset basis not at the table basis because you will want to join tables together typically within a dataset. So you don't do access control on a single table, you do access control on a dataset that consists of multiple tables.

A view is a live view of a table. So essentially you can think of a view as a query. That query returns our result set and then you can write a query to process that view just like you're writing something on a table. So you can use views as a cool way to actually restrict control to your dataset, because a view basically is a select ready make.

One of the ways that you can fix this is by probably removing the tail, for example with the Having clause, filtering them out so that you are not processing those tail things.

So, typical pipeline goes from a source to a sink, involves branching, and involves a number of transforms The input to that transform is a parallel collection. It's a PCollection. So, each transform on the pipeline, it's input is a PCollection.

Well, when you replace a running pipeline, you don't lose any data. So, some of the data gets processed by the old pipeline and any data that wasn't process by the old pipeline will get processed by the new pipeline. You can obviously see the advantage of your processing, streaming data right. But in order for that replacement to work, your transforms have to have names, unique names.

The grep command processes text line by line, and prints any lines which match a specified pattern. Grep, which stands for "global /regular expression /print," is a powerful tool for matching a regular expression against text in a file, multiple files, or a stream of files.

The templates help separate the development activities and the developers from the execution activities and the users.

Dataprep is an interactive graphical system for preparing structured or unstructured data for use in analytics such as BigQuery, visualization like, Data Studio and to train machine learning models.

The elements are divided into datasets, recipes and output. A dataset roughly translates into a Dataflow pipeline read, a recipe usually translates into multiple pipeline transformations, and an output translates into a pipeline action.

This is especially useful for data quality tasks and for Master Data task, combining data from multiple sources where programming may not be required.

The label is a correct output for an input. (the true answer - either known or to be determined)

The input is a thing that you will know and that you can provide at the time of prediction. These are things, for example, if they're images, the image itself as an input.

An example is a combination of the label and the input. An input and its corresponding label together form an example.

A model is a mathematical function that takes an input and creates an output that approximates a label for that input.

Training is this process of adjusting the weights of a model in such a way that it can make predictions given an input.

A prediction is this process of taking an input and applying the mathematical model to it. So, to get an output, that is hopefully the correct output for that input.