Skip to main content

How do I create Google BigQuery data feed metrics?

Updated

Ready to visualize and track your data that's stored in Google BigQuery? Get started by setting up a connection between PowerMetrics and your Google BigQuery account and adding one (or more) data feeds.

When you're finished adding data feeds, you'll use them to power your custom metrics.

This article includes:

Notes:

  • When you create a Google BigQuery data feed you can either choose a pre-built data source or, if the pre-built options don't include what you're looking for, create a custom one.
  • If you choose to create a custom Google BigQuery data source (instead of a pre-built one), you first need to create an SQL query using the Google BigQuery Query editor. You'll copy and paste that query into Klipfolio when choosing the data to retrieve for your data feed.

Should I create data feeds or connect directly to my BigQuery warehouse?

In PowerMetrics you can connect to your BigQuery warehouse and create either data warehouse metrics or data feed metrics. Which solution is better for you and your data?

  • Data warehouse metrics are the recommended solution for database and data warehouse infrastructures. They may be right for you if you have a lot of data and want scalability (the amount of data that can be handled is only limited by the capacity of your warehouse), have stringent security policies (data is stored and managed in your data warehouse) and demand higher performance (performance can be optimized by tuning your warehouse).
     
  • Data feed metrics are primarily used to connect to services using REST API to get data. They may be right for you if you process small amounts of data (the maximum data feed size is 10 MB. Each time the raw data is queried it needs to fit within this limit after the transformations are defined in the feed) and want a mixed storage model (metric history and data are stored in the metric layer in PowerMetrics and in the warehouse).

Creating a Google BigQuery data feed

When you create a data feed, you:

Connecting PowerMetrics to your Google BigQuery account

The first step in creating a Google BigQuery data feed is to connect PowerMetrics to your account.

Here are a couple things to note:

Your data is safe with us. The first time you connect your Google BigQuery account to PowerMetrics, you'll be prompted to enter your Google login credentials and give permission to allow access to the data within your Google account. We are serious about protecting your data privacy.

Managing your connections. By default, your connection name looks like this: yourname_<date and time created>. You can modify the default connection name when you create a new connection or from your list of connections, accessed by clicking the account and settings button in the left navigation sidebar and selecting Connections. Each time you connect to Google BigQuery, you can use the same account (or switch to a different account if needed). If you have trouble accessing an existing connection, your OAuth token may have expired. If that happens to you, go here for help.

To connect PowerMetrics to your Google BigQuery account:

  1. In the left navigation sidebar, click the + button beside Data Feeds.
    add data feeds button
  2. Click Select data.
  3. On the Where is your data? page, select Google BigQuery. (See below.)
    Google Big Query tile
  4. Select a pre-built or custom connection to use for your data feed.
  5. If this is your first time connecting to Google BigQuery:
    • Click Connect new account. Enter your Google login credentials and click Next. If prompted, click Allow to enable Klipfolio to securely access your Google Big Query data. Click Continue.
  6. If you've connected to Google BigQuery before:
    • Select an existing connection from the drop-down list and click Continue.

Next step - Choosing data for the data feed.

Choosing data for the data feed

To choose data for the data feed:

  1. On the Configure data service page:
    • If you selected the List of Projects pre-built option: Click Get data.
    • If you selected the Sample Query - Life Expectancy by Country pre-built option:

Select a Project from the drop-down menu (or, optionally, manually enter the Project name/ID in the text field).

At Request Option, select either BigQuery SQL or Raw data (JSON), depending on the desired output format.

Click Get data.

  • If you selected Create a custom Google BigQuery data source:

Select a Project from the drop-down menu (or, optionally, manually enter the Project name/ID in the text field).

At Request Option, select BigQuery SQL, paste the SQL query you copied from the Google BigQuery Query Editor into the SQL Query text box.
Note: Be sure to include #standardSQL as the first line in your query, otherwise you may receive a date format error.

Click Get data.

For tips on working with the Google BigQuery API, see "More about Google BigQuery's API and queries".

 The image below displays a Google BigQuery SQL query and its results in Klipfolio:
Google Big Query SQL query example

  1. Ensure this is the data you're looking for and, then, click Continue.
  2. You're taken to the data feed editor, where you can either save the data as-is or modify it.
    If you want to modify the data feed, go to this article for detailed editing information.
  3. Click Save data feed.

The data feed is added to your account and is ready to use for custom metrics. To see a list of all your data feeds, click Data Feeds in the left navigation sidebar.

More about Google BigQuery's API and queries

Klipfolio connects to hundreds of services in the cloud that have a REST API. If you take the time to learn your service’s API to create queries for the data you want, your possibilities are endless. Here’s a quick overview video on APIs and Klipfolio:

  Klipfolio API's 101

If you need assistance using APIs, enlist a data analyst or developer.

See the reference table below for API related information and sample queries.

API Documentation Google BigQuery API documentation
Authentication Method OAuth Token Authentication
Response Format CSV or JSON
Sample Queries

Life Expectancy by Country in Raw data (query in JSON format):

{
"query": "#standardSQL\r\nSELECT\r\n age.country_name,\r\n age.life_expectancy,\r\n size.country_area\r\nFROM (\r\n SELECT\r\n country_name,\r\n life_expectancy\r\n FROM\r\n `bigquery-public-data.census_bureau_international.mortality_life_expectancy`\r\n WHERE\r\n year = 2016) age\r\nINNER JOIN (\r\n SELECT\r\n country_name,\r\n country_area\r\n FROM\r\n `bigquery-public-data.census_bureau_international.country_names_area` where country_area > 25000) size\r\nON\r\n age.country_name = size.country_name\r\nORDER BY\r\n 2 DESC"
} 

Next steps - Creating Google BigQuery custom metrics

Now that you've created one or more Google BigQuery data feeds, you can return to them and use them to make custom metrics. A single data feed can be used to create a single or multiple custom metrics.

To create custom metrics using a data feed:

  1. In the left navigation sidebar, click Data Feeds to open your list of data feeds.
  2. Select the data feed you want to use for your custom metric.
  3. Click either the +Add metric button in the top right corner of the window or open the Metrics tab and click +Add metric. (See below.)
    add metric options
  4. Choose settings for your custom metric and click Save metric. If you need help, go here to learn more.

The metric opens in its overview page and is added to your list of metrics (accessed by clicking Metrics in the left navigation sidebar).

Now you're ready to start interacting with and learning from your metric data!

Learn more

Related articles