Gateway Documentation

Application Description

The Gateway is an application that allows users to download data from the HData system, en masse, into simple formats like csv. Any term, from any dataset, for any company can be downloaded for any number of years. Gateway is a product being continually updated and new developments integrated into it, so check back often to see what new features are introduced.

  • Industry: Electric, Oil, Gas
  • Sub-Industry: NA
  • Data Timeline: Quarterly, Annual
  • Geography: US States

Data Sources

FERC
- Electric: 1, 1-3/Q, 1-F
- Gas: 2, 2-A, 2-3/Q
- Oil: 6, 6-Q
- In development: 714, EQR, 549B

EIA
- In development

Gateway Overview

initial page view with numbers
Page Overview

There are six components to the Gateway Application:

1. Term Search Bar: text-based search on terms from the data
2. Industry Filter: narrow the list of terms and forms based on industries
3. Form Filter: narrow the list of terms based on form type
4. Term Browse/Selection: selection area of terms to include in the download
5. Terms Selected: running list of terms already selected for download
6. Download: page to specify the company, time period, years, and footnotes to include in the download into .csv

1. Term Search Bar

search bar results w tooltip
  1. The search bar works very simply to give a result based on the name of the term, matching most closely with what you type in the search bar. There are many terms in the dataset that have similar names or the exact same name between different forms and filings. It is recommended to narrow the list of terms by industry and/or form, and then typing a term in the search bar. The resulting list of terms will filter below the search bar and may be selected.

2. and 3. Industry and Form Filters

initial page with form filter dropdown

The industry and form filters are useful tools to narrow the scope of terms visible within the view. There are over 8000 terms at the current count of the dataset, meaning finding one by scrolling is a bit like a needle in a haystack. Utilizing these filters will be key to finding the correct terms for your analysis.

The industry filters follow the energy types of oil, gas, and electric, and also map onto the various forms and reports HData pulls data from. For example, selecting electric will narrow the list of terms to only show terms from electric forms. In this case, it will include the FERC Forms 1, 1-3/Q, and 1-F. You may narrow this range even further by using the Form filter to only select terms from the Form 1-3/Q. You may also go straight to the Form filter if you prefer without specifying an industry.

4. Term Browse/Selection

initial page with dropdown selection
 

You may also browse the list of terms by form and schedule. As seen in the screenshot above, the forms act like a tree structure. Clicking on the arrow right of the form or schedule will open the list of terms underneath it. Selecting the box next to a form or schedule will select all the terms under it. Note that there is a 1000 term limit to the data export, so selecting an entire form will not allow you to download it.

You can narrow the list of forms available in the view via the Form and Industry filters at the top of the screen. Given the similarity or identical names of terms based on similar form types, many terms and forms are collapsed. In the case of Form 1 in the snapshot above, terms are shared between the Form 1, 1-3/Q, and 1-F. The differentiating factor of the forms is the time periods they report on. Selecting "accrued utility revenues" will pull that term for all forms where specifying in the download between quarterly, quarterly cumulative, or annual will pull that term from the respective form (1 for annual, 1-3/Q for quarterly and quarterly cumulative). For more information on downloading, see the section 5 below called "Download."

5. Terms Selected

clicking in form from right pane
 

The pane at the right of the screen (number 5 in the screenshot at the beginning of this article) hosts the list of terms you have already selected for download. Clicking on the terms by form will open a pop-up window showing the list you've already selected, and the ability to modify your selection. Click on a new term will include it in the list, while clicking on an already-checked term will de-select it from the list.

6. Download

download file pop up
 

Once you have selected the terms you want, click on the download icon at the top-right corner of the screen. This will show a pop-up with several dimensions you can include in the data download. The text-search bar allows you to select a list of companies to include in the download. Time Frame focuses on the time period of data to include. In the snapshot above, quarterly is selected, meaning the as-reported number for the quarter will be included. Quarterly cumulative is the running sum of quarterly numbers for each quarter (i.e. Q2 = actual Q1 + actual Q2). In many datasets HData manages, the source data defualts to reporting the quarterly cumulative number. In those cases, HData calculates the actual quarterly number if "quarterly" is selected meaning Q2 actual = Q2 cumulative - Q1 actual. Annual data is the as-reported number for the entire year.

The Year Range narrows the selection to include the years needed for the download. If quarterly is selected between the years 2018-2022, that means the output will includes quarters one through four for each year between 2018 and 2022. The checkbox at the bottom of the pop-up is an optional selection to include footnotes in the output. If selected, the output will be downloaded as a .zip file with one folder being the csv of data, and another folder hosting .html files of the footnotes. Hyperlinks are included in the csv file connecting the numbers to the footnotes, so the correct footnotes may be accessed through the csv.

Once you have made your selection, click "Download File" at the bottom of the pop-up window.

Output File

snapshot_gateway_doc
 

A download will occur automatically with your new data as a .zip file. If you did not select footnotes, your folder will contain a basic excel sheet and a PDF file with a Terms of Use and simple documentation of the data. If you selected footnotes, an additional folder will house the footnotes as .html files.

Opening the excel sheet, you will see two tabs: 1) HData Gateway Data and 2) Summary. The first tab is the data you specified in gateway, while the second tab is a short summary of the data sources and the last time there were updated. If you selected footnotes to be included in the folder, you may see some numbers in the first tab in a blue color with an associated hyperlink to them indicating it as an attached footnote. Clicking on the number will open your browser with the footnote content. There are steps you must take to ensure that this hyperlink is correctly associated with the number, this is explained below.

Analyzing Footnotes via Windows

 

If you are using windows, you can find your data under the "downloads" folder on your device. Once you have located your folder, you will need to extract the files into a new folder. Unless files are unzipped successfully, links between data points and the footnotes will be broken. To unzip the file, right click on the .zip file and click on "extract all."

extract_all_screenshot
 

Clicking on "extract all" will give you a new screen option of where to place the unzipped folder and give it a new name. In this example, I extracted the folder into my "downloads" folder and renamed it "extracted_gateway_data." Then clicking "extract" will correctly extract the data and ensure the hyperlinks between the footnotes and data in the excel file are maintained. You may now treat the new folder as explained at the beginning of the data output section.

extract_local

Analyzing Footnotes via Mac/Apple

 

If you are using Mac, you can find your data under the "downloads" folder on your device or as it pops up under "download". Once you have located the .zip file, click on the .zip file to open it.

mac_download_gateway
 

You will see the folder with a folder holding the footnotes, the excel file, and a PDF description and terms of use for the data. Clicking on the excel file will show you the data with certain numbers highlighted in blue. This indicates that the number has an associated footnote. Clicking on one of these numbers will give you a popup screen below.

mac_folder_gateway
 

This will show the pathway from the number to the footnote folder in the associated .zip file. Click on the green "select" button.

mac_folder_gateway_2
 

The correct footnote will automatically be selected, all you need to do is click "Grant Access." This will link the number with the footnote and also open the footnote on your browser.