All About Data

MCast™ runs on data. Data is used to build our models, data is used to continually run our models, and data is output to you when the models generate new forecasts.

• If you are looking for information about the load data needed to build forecasting models, click here
• If you are looking for information about the weather data needed to build forecasting models, click here
• If you are looking for information about the load data needed to generate forecasts in MCast™, click here
• If you are interested in sending load data corrections to MCast™, click here
• If you are looking for information about the weather data needed to generate forecasts in MCast™, click here
• If you are looking for information about the forecast data you receive from MCast™, click here
• If you are interested in the MCast™ API, its capabilities, and how to get started with it, click here
• If you are interested in the SFTP site used in conjunction with MCast™, click here
• If you want to jump to some FAQ's about data surrounding MCast™, click here

Historical Data

To license MCast™, Marquette Energy Analytics requires two different sets of input data: Load Data and Weather Data.

Getting Started with Load Data

We will work with you to organize data representing your service territories into Operating Areas. From there, you can collect and send us the historical data we require to build forecasting models and perform analysis.

The following are some common questions we receive about the initial parts of the load data preparation process.

What is an Operating Area?

An Operating Area (often shortened to "op-area" or "OpArea") is a natural gas or electric distribution region comprised of a specific set of customers. This region will experience typical growth or decline in customers and energy usage over time. Energy supply for each operating area is typically nominated on a daily basis, and the daily and/or hourly forecasts are often the basis for those nominations.

We know that individual demand models for multiple smaller op-areas perform better than a single model for a larger aggregate op-area. The advantages of smaller geographic operating areas are:

• Better weather representation over the customer base, especially on days with fast-moving changes in weather.
• Better local customer growth modeling.

Once we understand how your service territories can be organized into operating areas, we will work with you to assign names to these areas and establish an order in which forecasts and subtotals appear on reports.

Historical data needed to build models

Marquette Energy Analytics requires a history of daily or hourly load that represents your customer base's actual consumption of natural gas or electricity. This historical data must represent actual, measured energy that is consistent with the load to be forecasted:

• Some Local Distribution Companies (LDCs) have us forecast the total load, while others have us forecast just sales or transport.
• Some LDCs have us forecast just firm (core) load, while others have us forecast firm plus interruptible load.
• Some LDCs will have us forecast sales and transportation loads separately. Even though these typically occupy the same geographic areas, they are treated as separate "op-areas" by the software.

The more data we can get, the better. We have built models on as little as 2 or 3 years of historical data, but longer data sets produce better models. Typically, our models perform best when trained on at least 8 to 10 years of data.

At Marquette Energy Analytics, we welcome data submissions in various units, including Dth, Bcf, and others. It is essential that the initial data upload and all subsequent data updates consistently use the same unit of measurement. To facilitate smooth communication, please inform us of the specific units you intend to use when sending your data.

A Note on Interruptible Loads

MCast™ is designed to forecast demand, even on days when total demand cannot be met and customers must be interrupted. To do this, we must make sure the historical data we use to build your forecasting models contain adjustments to account for interrupted demand. To do this, we ask two questions:

1. Do we know the dates when interruptions occurred?
2. Can you provide us with an estimate of the interrupted volume?
   • If you don’t have those estimates, we can use our own models to make them.

Note that, once you begin sending data to MCast™ regularly, we still want you to send data that estimates your demand when possible (even if an interruption occurs). For further reading on this, see the FAQ section "How do I handle data in MCast™ that doesn't reflect my area's actual demand?"

Format requirements for historical load data

We require daily or hourly measured natural gas or electric load for each hour or day, going back as far as possible. Data can be sent to us for each operating area in one of two ways:

1. (Preferred) As the measured historical load for each gate/meter serving an operating area. We roll up the data by gates/metering, including any subtracting/deducting of specified gates. We have tools that can detect anomalies and inconsistencies in low-level readings to get cleaner data for each op-area
2. As total measured load for each operating area. This is often the case for areas that are served by a single supply gate.

For files representing hourly load, the date and hour can be either in one column or two separate columns.

Formats that work best are Microsoft Excel files (.xls or .xlsx) or comma-separated variable (.csv) files. Text files or databases can be acceptable. The data can be in different files or on different tabs in a workbook and can be in different formats. The following are common ways files are formatted.

Gate and date per row

Date per row for several gates

Note

Column headers do not have to be exactly like the ones specified above, but they should be unique. Extra headers and columns are nice to be avoided if possible, but if not they can be ignored by our system. If you have another file format, send us a sample and we will let you know if it meets our requirements.

Getting Started with Weather Data

Marquette Energy Analytics requires weather station data to build forecasting models and to generate daily and/or hourly short-term demand forecasts. Daily use of MCast™ requires hourly forecasted and observed weather data. Most Marquette Energy Analytics customers obtain weather forecasts and recent weather measurements from a weather data vendor that supports electronic transmission of current data on demand.

Selecting Weather Stations

Ideally, we want a centrally located weather station. When the operating area is geographically large, we prefer to use multiple weather stations to better represent the weather across the entire area. We want to use well-sited, well-maintained weather stations with long histories of hourly data to provide you with the most accurate forecasts possible.

We can help you identify NOAA weather stations in and around your service territories, determine the best combinations of weather stations to use, and obtain the historical weather from NOAA. Marquette Energy Analytics requires weather stations with hourly weather forecasts and observations, which enables us to align the weather data with the gas day.

We can work with your weather data vendor to set up the observed and forecasted weather file(s) in an agreed upon format and configure electronic transmission.

Weather Station Optimization (WSO)

Our team will perform a Weather Station Optimization study (or WSO) prior to modeling the forecasts for any new operating area to determine the optimal weighted combination of stations for forecasting your natural gas demand. We have found combinations of weather stations that have reduced gas load estimation error by as much as 20% over the existing methods.

For a detailed breakdown of what a WSO is, see the FAQ section "What is a Weather Station Optimization study?

Ongoing Load Data

Input Data Webinar

In the video below, our team talks all about load data - also commonly referred to as "input data," "sendout," or "flow data." They cover how to upload it, how to correct it, and more!

• 0:00 - Introductions
• 1:38 - Input data
  • 3:33 - Weather data overview
  • 4:33 - Load data overview
  • 6:31 - Gate, transport, and aggregated load data
  • 7:43 - Uploading via the SFTP site
  • 10:39 - Manually entering load data
  • 14:06 - Once a month update files
• 15:14 - Correcting input data
  • 16:28 - Correct it in the file
  • 17:40 - Correct it on the MCast™ website
  • 19:24 - Lock the values on the MCast™ website
  • 22:04 - Correct it through a once-a-month update file

Daily Data Required to Run a Forecast

The best model performances are obtained by providing daily and/or hourly measured load. The application gets this information by either reading in a file uploaded to an SFTP site or by uploading data via the MCast™ API. Whatever option you choose, most users choose to automate this data upload.

• You can jump to the SFTP section for more information about what SFTP is and how Marquette Energy Analytics uses it.
• You can jump to the API section for more information about the MCast™ API and how to use it.

Data File Formats

Acceptable file formats are .csv, .xlsx, or .txt. Load data can also be entered manually into the application. The data within the file must be organized in 1 of the following 2 formats.

Date per row

In this format, all data for a single date is contained in a single row. Multiple operating areas may be represented as different columns in the file.

Record per row

In this format, each row contains a single flow value, specific to one operating area and time period.

Updating daily load data

You may need to provide updates to load data in MCast™ outside of the standard protocol you establish for daily data updates. In those cases, you have a few options listed below. For further reading, see the FAQ section for "Incorrect data got uploaded by mistake."

Correcting data through normal SFTP or API routes

If you are sending MCast™ daily data via either a file upload to our SFTP site or through a call to the MCast™ API, the process is built to handle regular short-term data updates! Each time that MCast™ parses a new load file or API call, it will check to see if any of the uploaded data differs from what is currently stored on the system. Once it has been determined what data is new or updated, then that new/updated data gets read in to MCast™ and a new forecast begins generating from the first day of new data forward.

Whether you're uploading a file to the SFTP site or sending an API call, we encourage you to upload your daily data on a rolling 7-14 day window. That way, the past couple weeks of load observations can be regularly updated as your flow numbers are finalized, and the data can get into your forecasts seamlessly.

If you need to update data older than 14 days ago, note that the forecast may take longer than expected to generate. Read on to see if any of the other data update options make more sense for you.

Correcting data through the web interface

Built into the MCast™ website is a feature called "Correct Load Values". Using this tool, you can view and manually update the database records for each date + operating area combination. You can also lock values using the lock icon next to any of the entry forms. Locking a value will ensure that no data sent to MCast™ in the future can overwrite the value (this can be helpful if, for example, an automated process is sending an incorrect load value to MCast™ multiple times)

• If you are a "Manual Entry" customer, then this form will appear as "Manually Enter Values" on your MCast™ website.

Monthly data updates

Many customers require regular large data updates, such as finalized load numbers at the end of each month. In a case like this, you have 2 options:

1. Upload a file containing a larger-than-normal set of data (this is sometimes referred to as a "monthly file")
2. Upload a larger-than-normal data set using the MCast™ API

When larger data sets like this are uploaded to MCast™, it is expected that forecasts will take noticeably longer to complete than normal. This is because the models will generate new forecasts from the first date of new data onwards (for example, if I upload a data correction from 30 days ago, then MCast™ will generate forecasts for 30 days ago, for 29 days ago, for 28 days ago, etc. until the models catch up to the present).

If you are interested in setting up regular long-range data updates, please reach out to your MCast™ support contact(s) to discuss the situation and decide on the best option.

MCast™ supports uploading no more than 2 months (or roughly 60 days) of data. For data updates that are larger than 2 months, we request that you reach out to the MCast™ support team. We have tools available to upload large data sets directly to the backend.

Annual data updates for model training

Once a year Marquette Energy Analytics will re-train forecasting models for all customers licensing a short-term daily load forecast. As part of this process, you may receive an email from the MEA team asking if you have any notable data updates to share with us before re-training and optimizing your models for the upcoming heating season.

If you do have data to provide at this time, we recommend that you provide the data in one of the acceptable formats for historical operating area data.

Non-standard data updates

Sometimes you may need to update a significant chunk of data (perhaps you learned that a meter was failing to report for a length of time, or the customer base in a delivery area has experienced a notable change).

If the data required for this update can fit easily into one of the protocols listed above, such as a normal file upload process or through the "Correct Load Values" feature of the website, then feel free to make the correction yourself! However, if the data update is more sizable than what you normally upload to MCast™, we recommend you reach out to the Marquette Energy Analytics support team at support@marquetteenergyanalytics.com so that we can evaluate the situation and determine the best path forward together.

Specific File Requirements

There are some specific requirements for flow files that apply regardless of which file format you use.

• Unique header titles are required for files (but they do not need to match the header titles from the examples above)
• There is no strict requirement on file naming, but the name needs to be consistent. Some find that file names containing some form of a date stamp can be useful like \<yyyymmdd>_load.csv where \<yyyymmdd> is the date that the file was generated. Others prefer to have the same filename every day such as load.csv
• Load files only require whatever new data is necessary to generate new forecasts (this is often 1 new day of load data). However, we recommend including a few previous days and/or hours of measurements. This is helpful to "catch up" a configuration in the event a forecast run is missed; this is also helpful if you wish to input corrections to previously entered load data.

Hourly Load Files

Files representing hourly load have the same requirements as daily files with a few additional hourly-specific requirements to ensure that we can read in your data correctly.

• The date and hour can be either in one column or two separate columns
• You must communicate if your file is configured to log data as "Hour Beginning" (meaning the hour listed is at the start of the collection period), or "Hour Ending" (meaning the hour listed is at the end of the collection period) !!! example An "Hour Beginning" file that has an entry for 7:00 indicates that the value was measured from 7:00 - 7:59. An "Hour Ending" file that has an entry for 7:00 means that the value was measured from 6:00-6:59.

• You must format your file to be in either "Clock Time" or "Standard Time" (see the warning below for more)

  Warning

  An hourly load file MUST be properly formatted as "Clock Time" or as "Standard Time" in order to ensure that your data is correctly read-in to the product. An hourly load file incorrectly configured for Clock Time when the data corresponds to Standard Time (or vice versa) can lead to the hourly load being read in for the wrong hour. This can have substantial effects on forecast accuracy. For more information on Clock Time and Standard time, read the section Reporting your data for Daylight Savings Time

Ongoing Weather Data

Weather data is a required input to MCast™. When any of our load forecasting products are run, they read in the most recently available weather forecasts and observations.

All weather stations must have hourly weather forecasts and observations in order to be used as an input to MCast™.

There are two options for getting this weather data into the product:

1. Sourcing through our preferred weather vendor, AccuWeather for Business
2. Providing weather data to us through a weather vendor or meteorology department

Weather Station Optimization (WSO)

Our team will perform a Weather Station Optimization study (or WSO) prior to modeling the forecasts for any new operating area to determine the optimal weighted combination of stations for forecasting your natural gas demand. We have found combinations of weather stations that have reduced gas load estimation error by as much as 20% over the existing methods.

For a detailed breakdown of what a WSO is, see the FAQ section "What is a Weather Station Optimization study?

AccuWeather for Business - our preferred weather vendor

We are proud to partner with AccuWeather for Business (AFB) to provide forecasted and observed weather data to Marquette Energy Analytics customers at a special discounted rate! Years of collaboration have resulted in an efficient, reliable, and automated data transfer pipeline between the AccuWeather for Business system and MCast™. By utilizing AFB's XML Query Tool, MCast™ can make API calls to receive all needed data automatically, removing the need for any flat file transfers or manual weather data intake.

Additional advantages to choosing AccuWeather for Business:

• AFB licensing costs can be included in your MCast™ invoice for easy billing
• Your organization can use AccuWeather for Business weather data in any other systems that utilize weather. We'd be happy to discuss what your organization's weather data needs are and how best to meet them.

If you're ready to further automate your forecasting process with AccuWeather for Business, send us an email at info@marquetteenergyanalytics.com

Working with a weather vendor or meteorology department

If you already source weather data through another 3rd party vendor or a meteorology department at your company, you can use this data in MCast™. Speak with a Marquette Energy Analytics support team member, and they'll be able to work directly with the team responsible for that data, ensuring that it's compatible with our application.

If you are a weather provider new to our requirements

Weather instruments that are required for MCast™

MCast™ requires hourly weather data for a number of weather instruments, depending on which product you license.

The following table lists all of the weather instrument requirements for the 3 types of load forecasting products.

Weather Instrument	Daily	Hourly Gas	Hourly Electric
Temperature (TEMP)	YES	YES	YES
Wind Speed (WIND)	YES	YES	YES
Dew Point (DPT)	YES
Global Horizontal Irradiance (GHI)			YES
Wet Bulb Temperature (WET)			YES

API options

When possible, our team prefers to load data via APIs rather than through flat file transfer. If you are able to provide access to your weather data via API calls, let us know! We'd love to evaluate your API to see if we can easily integrate it with our systems.

Data File Formats

When loading via flat files, MCast™ requires weather data in .csv or .txt files. Excel file formats are not supported for weather data.

You can upload either a single file containing all weather station data, or you can upload a different file for each weather station's data.

You can include observed and forecasted data in the same file, or you can split them into separate files.

Specific File Requirements

• File(s) must be retrievable via either SFTP or FTP
• File(s) must be in .csv or .txt format
• The file(s) name must be consitent
  • We recommend something like \<yyyyMMdd>_weather.txt where \<yyyyMMdd> is the date that the file was generated.
• File(s) must contain hourly forecasts out as far as the GasDay models go, which is 8 days

Note

We can accept files with less than 8 days of weather forecasts, but this will limit how far out the forecast models can go (For example: if your weather files only go out 5 days, our models can only forecast out 5 days)

• File(s) must contain at least 1 day's worth of retrospective data
  • We recommend including a full week
• Any and all missing observed weather reads must be patched with estimated values

Accessing Your Data

The Downloads Tab

You can access, configure, and download all analysis files via the "Downloads" button on the right side of the header. Note that the contents of the "Downloads" tab will be different depending on what page of the website you are on (for example, if you are on the "Daily" page, then the downloads will only pertain to the data related to your daily forecasts; if you are on the "Hourly" page, then the downloads will only pertain to data related to your hourly forecasts.)

Current Forecast

What is the Current Forecast file?

Users can download the output file for the currently-displayed forecast here. If you wish to modify the file contents and organization of the output file that you download, you can make those edits here using the "Advanced Settings" section.

Note

Modifications to the file contents from the Current Forecast panel will NOT modify the file contents of any output files you receive automatically after a forecast completes. To modify those file configurations, you must go to the "Configuration" section of the "Settings" tab.

Current Forecast Weather

What is the Current Forecast Weather file?

Users can download the weather data used in the currently-displayed forecast here. If you wish to modify the file contents and organization of the weather file that you download, you can make those edits here using the "Advanced Settings" section.

Note

Modifications to the file contents from the Current Forecast Weather interface will NOT modify the file contents of any weather file(s) you receive automatically after a forecast completes. To modify those file configurations, you must go to the "Configuration" section of the "Settings" tab.

Daily Model Performance

What is the Daily Model Performance file?

This allows you to compare forecasted and observed values for each day and intraday period. Each row represents a GasDay forecast for the day and intraday period given, and includes the forecasted weather that was used in that forecast. Each row also includes the observed load and weather for comparison. Only one forecasted value is displayed from each forecast, and it is set to be a specific number of days (horizons) out.

• By default the "Day Number" is set to be 0, meaning it is the forecast 0 days out.
  • If the file were to be set to "Day Number" = 1, then for each date in the file it will find the GasDay forecast generated when that date was 1 day out.
• These day numbers from 0 to 7 correspond with the day numbers found along the top of the summary table on the Daily page.

Under "Advanced Settings", you can select the days out (or "Day Number") that you wish to be in the file, the forecasts you wish to include in the file, which columns you wish to be included in the file, and you can transpose the data within the file.

This download will only show up if you license a daily forecasting product.

Hourly Model Performance

What is the Hourly Model Performance file?

This allows you to compare hourly forecasted values to the observations on a rolling window. Each row represents an hourly forecast and includes the forecasted weather that was used in that forecast, as well as the observations for that hour. The rolling window is set to be some number of hours out from the forecast time.

• By default the "Hour Number" is set to be 0, meaning it is the forecast 0 hours out.
  • If you set the "Hour Number" = 4, then the forecast is 4 hours out. In the file, this would mean that the 10:00 AM observed data will be compared to the forecast made at 6:00 AM for 10:00 AM, the 11:00 AM observed data will be compared to the forecast made at 7:00 AM for 11:00 AM, and so forth.

For this analysis to generate a full data set, you must run your model every hour. If you don't run every hour, there will be missing data in this file.

Under "Advanced Settings", you can select the hours out (or "Hour Number"), that you wish to be in the file, which columns you wish to be included in the file, and you can transpose the data within the file.

This download will only show up if you license an hourly forecasting product.

Hourly Perfect Weather

What is the Hourly Perfect Weather file?

This allows you to re-generate a full hourly forecast using only observed weather data. Using only observed (or "perfect") weather removes any forecasting errors that resulted from inaccurate weather forecasts, so that the only error remaining would be reflective of modeling error only.

The date you choose will be used as the "Day 0" of the Perfect Weather forecast that is created when clicking "Download CSV."

You may generate an Hourly Perfect Weather forecast for any date in the past on which you have observed weather.

This download will only show up if you license an hourly forecasting product.

Normal Weather

What is the Normal Weather file?

This allows you to download the normal temperature for the calendar year, either by weather station or operating area.

By default, normal temperature values are generated relative to 30-year normals, and they are regenerated each time you are given a new model update. Speak to your support specialist if you would like this changed at all to fit your company's needs.

Planner Daily Summary

What is the Planner Daily Summary file?

This allows you to view the MCast™ Planner estimates for the average daily loads for a variety of scenarios for a range of months.

These scenarios are reported in terms of percent chance and 1-in-N-years and reflect the probability of the "total load" for the time period. For instance, if you select "20% chance" or "1-in-5" while the month selected is "February", then "Cold 20% Chance" is the average daily load of a February scenario with a total load that there is a "20% chance" of exceeding, while "Warm 20% Chance" is the average daily load of a February scenario with a total load that there is an "80% chance" of exceeding. Likewise, "Expected" is the average daily load of a February scenario with a total load that there is a "50% chance" of exceeding.

Note

Whether these probabilities are displayed in percent chance notation or 1-in-N notation is determined by clicking the settings gear on the MCast™ Planner page.

This download will only show up if you license Planner.

Planner Monthly Summary

What is the Planner Monthly Summary file?

This allows you to view the MCast™ Planner estimates for the monthly total demand probabilities for a range of months.

These probabilities are reported in terms of percent chance and 1-in-N-years. For instance, "Cold 20% Chance" or "Cold 1-in-5" refers to a monthly total load that there is a 20% chance of exceeding, while "Warm 20% Chance" or "Warm 1-in-5" refers to a monthly total load that there is an 80% chance of exceeding.

Note

Whether these probabilities are displayed in percent chance notation or 1-in-N notation is determined by clicking the settings gear on the MCast™ Planner page.

This download will only show up if you license Planner.

The MCast™ API

An Application Programming Interface (or "API") is a way for software to talk to other software. An API defines a common language to share information between different applications.

MCast™ has its own secure API that defines how other software can get data from and post data to the application.

APIs are often used by IT and software developers to integrate different processes and services. At Marquette Energy Analytis, we try to perform data transfers using APIs whenever possible (such as when loading Weather Data into MCast™). To ensure the integrity and confidentiality of your data, our REST APIs are secured with industry-standard authentication mechanisms. We continuously monitor and update our API security to safeguard against unauthorized access and potential vulnerabilities.

If you are working on any new automation or improvements to old processes, let your IT support contact know that MCast™ provides an API for them to utilize!

Getting Started with the MCast™ API

Before you can use any of the functionality in the API, you'll first need an API key for authentication. An Admin or User of MCast™ can easily generate an API key within the MCast™ website in the Settings menu.

Treat this API key the same as you would a password to a website. It grants a user full access to see all of your data, to upload new data, and to generate new forecasts all using the API. Best practice when authenticating with API keys is to create a new API key for each application that would use the API instead of having just a single API for your entire company. This way it's easy to track what the API is being used for in your organization, as well as to revoke access to a single application using the API without having to change the key in all other applications.

Once you have an API key setup, you can start accessing the API via HTTPS requests. You can see the full API Reference documentation on the different API functionality available through your MCast™ web portal, by clicking on the "API Reference" button in the same menu you used to generate an API key.

Within the API documentation, you can click the "Authorize" button and paste your API key in the popup window to enable trying out API requests without leaving the documentation page. Each API endpoint has a "Try it out" button that will let you enter your own parameters and see what the API returns. Use this to quickly develop your HTTPS requests prior to writing the code to do it.

Warning

This runs real requests against your live MCast™ instance, so make sure you set the dryRun parameter to true on any requests that upload data or run a forecast. If dryRun is set to false, then your API request will execute on your production database.

If you want to write as little code as possible when working with the API, you can copy-paste from our code samples below, or you can try out the Postman API Client. Postman is an application that lets you try out API requests similar to the MCast™ API documentation page, but will also output code to run that request in many different programming languages.

What can I use the MCast™ API for?

The MCast™ API allows you to integrate the MCast™ forecasting platform with other applications and automated processes within your organization. There are three main uses of the MCast™ API:

Accessing data

Through the MCast™ API you can access load forecasts that MCast™ produces, load observations that your company uploads to us, and weather data that is used when generating forecasts.

Uploading load data

You can use the API to upload a data file that was configured by your MEA support specialist, or you can upload a formatted JSON containing your load data. Jump to the Ongoing Load Data section for more.

You can also use the API to make corrections to data you've previously sent to MCast™. Jump to the section Correcting data through normal SFTP or API routes for more.

Generating an MCast™ forecast

You can use the API to launch a new MCast™ forecast, either after uploading formatted JSON load data, or after your weather provider has made new weather forecasts available. See the Generating Demand Forecasts section for more.

Note

If you have a file "watcher" set up that triggers a forecast upon upload of a file, that will still work if you upload a file via the API.

If you don't know what a file "watcher" is, see the information listed under "SFTP File Transfer" in the section Generating Demand Forecasts.

Some more specific examples of how you might utilize the MCast™ API:

• Import MCast™ data into your own flow management system
• Send MEA automated data updates as new data becomes available
• Run your own analytics on MCast™ data
• Schedule MCast™ to run at times when new weather forecasts become available

Samples

Check our API samples repository for examples of using our API in any of the following languages:

• PowerShell
• Bash
• C#
• Python
• JavaScript
• Power Query M

API Intro Webinar for Developers

Hear our team talk through the MCast™ API - including a couple of code-heavy examples of using the API - in the video below!

• 0:00 - Introductions
• 1:10 - What is an API
• 3:10 - Why use the MCast™ API
  • 4:02 - More specific uses
• 6:19 - Documentation on the MCast™ API
• 8:23 - Demo introduction
• 9:55 - Demo part 1: Upload data with the API (using PowerShell)
  • 13:20 - Generating an API Key
  • 15:13 - Starting with a sample repository
  • 17:47 - Editing the code for your needs
• 24:02 - Demo part 2: Download forecast data with API, and load it into Power BI (using C#)
  • 26:40 - Starting with some pre-written C# code
  • 29:45 - Going to the sample repository for more
  • 31:44 - Using the "API Reference" page
  • 37:30 - Duplicating the request in the C# code
  • 47:56 - Parse the result into database records
• 56:52 - Reminder of how to find the API Documentation used in the demo

API "Low-Code" Example with Power Query M

Hear our team talk through using Microsoft Excel's built-in query language, Power Query M, to quickly generate an API call and load it into Excel in the video below!

Recorded on 8/15/2023

• 0:00 - Overview and setup
• 0:35 - Generating an API Key
• 1:38 - Accessing the Help documentation
• 2:16 - Setting up the query in Excel
• 2:45 - Customizing the query
• 4:37 - Getting the results

API "No-Code" examples for Excel and Power BI

Hear our team talk through a detailed, no-code use case for the MCast™ API. The goal of this talk was to build dynamic API calls for loading data into Microsoft Excel and Power BI without writing a single line of code.

• 0:00 - Presentation overview and goals
• 1:00 - Demo part 1: Setting up your API
  • 1:56 - Access necessary API information
  • 6:00 - Create an API Key
  • 7:32 - Authorize your key to use the API Reference page
  • 8:43 - Generate a "Daily Observed Load" query
• 11:25 - Demo part 2: API calls in Excel
  • 12:53 - Get observed load into an Excel workbook
  • 15:20 - Reformat the load data in Excel
  • 19:17 - Edit so that Excel calls a dynamic range (grabbing the past 90 days of observed load)
  • 28:06 - Generate a "Daily Forecasted Load" query
  • 31:00 - Get forecasted load into an Exel workbook
  • 32:17 - Reformat the forecast data in Excel
  • 35:32 - Edit so that Excel calls a dynamic range (grabbing the past 90 days of forecasted load)
• 37:54 - Demo part 3: API calls in Power BI
  • 38:53 - Brief overview of Power BI
  • 39:35 - Get observed load and forecasted load into Power BI
  • 40:22 - Merge the observed and forecasted data sets together
  • 42:10 - Reformat the merged data set
  • 43:31 - Add new columns to calculate a forecast percent error
  • 45:30 - Graph the data

MCast™ SFTP sites

SFTP stands for SSH File Transfer Protocol, which is a network protocol that is commonly used for file transfers. Transferring files through SFTP is a secure, encrypted way to transfer data between your company and Marquette Energy Analytics.

MCast™ uses SFTP to collect new ongoing load data before generating a load forecast. You can transfer load data by uploading files to the Marquette Energy Analytics SFTP site set up for your web instance. Your instance can be configured to automatically generate a forecast whenever you upload a file.

You can also use the SFTP site to make corrections to data you've previously sent to MCast™. Jump to the section Correcting data through normal SFTP or API routes for more.

Your SFTP site can also be configured to receive ongoing weather data, if it is not provided on a third-party site.

MCast™ also uploads data to the SFTP site. At the end of each forecast generation, any output CSV files you have enabled will be uploaded to a folder on the SFTP server called “output.”

Connecting to your MCast™ SFTP Site?

There are multiple ways to connect to your SFTP site.

Manually Connecting: The most common way individuals connect to the site is through an SFTP client. Software such as Filezilla and WinSCP are popular, easy-to-use clients to connect to SFTP. Due to the increased security inherent in SFTP, you will not be able to connect to an SFTP site through your file explorer or an internet browser.

Through Automation/Scripts: For daily operations, most companies prefer to script a process that automatically uploads and/or downloads the needed files. Most scripting languages have extensive SFTP libraries to connect through. When using a script to connect and upload to the SFTP site, it is important to ensure that preserving timestamps are disabled. The tool MCast™ uses to start a new forecast relies on the file's timestamps matching the time the file was uploaded.

If you need credentials or are having connection issues with the SFTP site, see this section about accessing the SFTP site.

Running Tests to Connect, Upload, and Download

We are happy to accommodate any tests that you need to run regarding the ability to send and receive data via SFTP. Please reach out to a member of our support team to discuss further.

Data Retention Policy on SFTP Sites

At Marquette Energy Analytics, we do not perform automatic cleaning or file wiping from the SFTP site. However, it’s essential to consider your usage of the SFTP site. Accumulated data can lead to several challenges, including confusion, clutter, and longer download times. To maintain efficiency, we recommend periodically deleting files that are over a month old.

Our team reserves the right to remove data of any kind from the SFTP site if it is more than a month old and is causing any issues with ongoing MCast™ performance.

Frequently Asked Questions

Incorrect data got uploaded by mistake. What can I do to fix it?

If you are using Manual Entry for uploading flow data...

Just navigate to the day(s) that you have updated data for and correct the value. Once you hit "Save", a new forecast will begin generating from the earliest date of new data up through the latest date of data.

If you are uploading a file to run forecasts every day...

You have two options for correcting flow data

1. Within the file: Update the values within your load file, then upload the file (either directly to the SFTP site or by attaching the file to an API call). Any new values in the file will be read in as the most up-to-date data, and a new forecast will begin generating from the earliest date (up to 12 days prior) of new data up through the latest date of data.
2. Using the manual correction window: If you click the sidebar on the top left of the page and select "Correct load values", you will be able to manually overwrite the data in the product database. After clicking "Save", the data corrections will be committed to the database and a new forecast will begin generating from the earliest date of new data up through the latest date of data.

Tip

When using the manual correction window, you can choose to "lock" any value. This will ensure that the value entered does not get overwritten by any future load files. For more information about locking values, read the section for "Correct Load Values" in the description of The Header.

If you are using the API to upload data as a formatted JSON...

Submit the updated values and trigger a new forecast to begin. The new forecast will begin generating from the earliest date of new data up through the latest date of data.

For further reading around data updates, see the "Updating daily load data" section.

How do I handle data in MCast™ that doesn't reflect my area's actual demand?

Users often find themselves in a situation where the demand of a load area is inaccurately represented in that load area's data. It may be because you issued curtailments to certain customers during a cold event, or a cold event led to stopping delivery to interruptible customers, or system issues led to energy not flowing to some customers. Whatever the case, for any of these interruption events it means that the "true" data point for your load data does not reflect the unimpeded flow of energy into your entire territory.

MCast™ is a product built to forecast energy demand on your system. It is not meant to represent "true" gas or electricity sendout, but rather it is meant to predict what the sendout would be if all customer needs were met without interruption. Given that, and because our forecasting models tune to the load data you upload over time, when your system experiences interruption events that are reasonably large it is recommended that you clean your data to reflect demand (or, at the very least, predicted demand) in MCast™.

To upload or correct the data when one of these events occurs, it's recommended that you follow one of the protocols listed under "Updating daily load data".

If you have questions about the data, the way you're handling a data update in MCast™, or anything related to an interruption event, we'd be happy to assist you! Don't hesitate to reach out to our team at support@marquetteenergyanalytics.com

Note

There is no threshold amount at which Marquette Energy Analytics can say you must enter a load correction (in other words, we cannot simply say "any data point that is greater than X% off from the 'true' demand needs to be corrected"). That is why we simply recommend correcting data points when interruption events are "reasonably large."

If you are uncertain if a given interruption event warrants corrections, just reach out to data@marquetteenergyanalytics.com and our data scientists would be happy to let you know what they think!

How do I grab historical forecast data from MCast™?

At any given time, MCast™ contains the load forecast data from the previous three to four years. Those previous forecasts are available to view and download through your MCast™ website.

• If using the Daily forecasting tool, you can navigate to older day-0 forecast dates by clicking the "Forecast Date" dropdown in the top left of the page, and then clicking a forecast from the "Forecast Timestamp" dropdown directly next to it.
• If using the Hourly forecasting tool, you can view and navigate to older forecasts by clicking the forecast timestamp dropdown found on the right side of the page below the time series graph.

If you'd like to evaluate the accuracy of prior forecasts, the best way to do that is by comparing forecasted data to observed data. Your options for gathering the observed data (both for weather observations and load observations) vary, depending on which forecasting product(s) you license.

For those licensing a Daily forecast:

1. Use the Performance page of MCast™, found under the "Performance" button in the page's header, to utilize built-in tools for understanding the performance of your forecasting models. You may also download the data behind this page using the CSV button found near the data selectors at the top of the page.
   • Note that the Performance page is powered by Daily Perfect Weather forecasts, which are a helpful tool in understanding the performance of your forecasting models. Those forecasts are updated automatically, on a once-a-week schedule. For further reading, see "What is 'Perfect Weather'?"
2. Use the "Daily Model Performance" file, found in the "DOWNLOADS" tab of the Daily page on MCast™, to download a host of data including the load observations, weather observations, load forecasts, and weather forecasts.

For those licensing an Hourly forecast,

1. Use the "Hourly Perfect Weather" file, found in the "DOWNLOADS" tab of the Hourly page on MCast™, to generate. For further reading, see "What is 'Perfect Weather'?"
2. Use the "Hourly Model Performance" file, found in the "DOWNLOADS" tab of the Hourly page on MCast™, to download a host of data including the load observations, weather observations, load forecasts, and weather forecasts.
   • Note that this file currently only generates correctly if you are running a forecast once every hour throughout the day. If you do not run at that frequency but want to produce this data for forecast analysis, reach out to our team at support@marquetteenergyanalytics.com

Our data retention policy

Marquette Energy Analytics is proud to provide easy access to previous forecast data. Using the MCast™ website or API, you will always be able to access data from at least 3 years prior to today's date. However, due to data storage and infrastructure requirements, we cannot guarantee maintenance of all historical MCast™ forecasts. We actively encourage our customers to save off any forecasting data that they may want available for future access.

Marquette Energy Analytics reserves the right to clear any forecasting data from our servers with no notice to the customer if the data in question was generated three or more years ago.

How do I gain access to upload data to MCast™?

If you don't have access to upload or modify load data from within the MCast™ website, then that means you are on a Read-Only account level. To modify data via the user interface, you must be granted either a User account level or an Admin account level by whoever is in charge of account administration at your organization. If you are an admin attempting to grant this access to a user(s), see the Admin Actions & Questions page for more.

If you are seeking access to your company's SFTP site for uploading files to and downloading files from MCast™, then you will need login credentials for that SFTP site. Talk to whoever is in charge of account administration at your organization to ascertain the necessary credentials.

If you are seeking access to your company's MCast™ API for data uploading and downloading, then you will need to authenticate a unique API key for your organization's data to be accessible. A User or Admin account level can generate a key for your API at any time through the "SETTINGS" tab of the website. Talk to whoever is in charge of API key management at your organization to ascertain the necessary authentication.

If you need further assistance, please reach out to our support team at support@marquetteenergyanalytics.com

For further information on user options, see the User Management section of our "Admin Info" page.

What is a "Weather Station Optimization" study?

As part of onboarding a new customer or area, Marquette Energy Analytics completes a Weather Station Optimization (WSO) study to determine the weather stations that produce the most accurate possible load forecasts. It is done because a geographical area (and the energy needs within that area) very rarely maps directly to a single weather station. There are typically a handful of weather stations in and around an area whose weather data can be used to create an effective forecast.

When performing a WSO, our team runs a variety of modeling scenarios to understand what combination of the weather stations as an input to our forecasting models will yield the most accurate model performance.

Example

If we were performing a WSO on an area called "Metropolis" and there were 3 weather stations (A, B, C) near Metropolis, the results of the WSO might look something like this:

Weather Station	Percent
Station A	23%
Station B	58%
Station C	19%

This means that weather data representing the "Metropolis" area at a given time will be equal to:

• ((Station A weather) x 0.23) + ((Station B weather) x 0.58) + ((Station C weather) x 0.19)

This study is a standard first step for most work that Marquette Energy Analytics does, and as such is typically lumped into the costs of purchasing a study or licensing a SaaS product. However, it can also be run independently of other work.

If you are interested in having a WSO performed (or re-performed) on your data, reach out to data@marquetteenergyanalytics.com and we can discuss the details of your request.

If you need to access the weather station weights applied to your MCast™ instance, reach out to support@marquettenergyanlytics and someone will get back to you with the configuration that is currently in place.

Special Cases when it comes to Weather Weighting

If you have reason to use a weather weighting different from what is determined by our WSO, just let us know. We are happy to modify the weights as necessary, and the results of our WSO are in no way binding.

If you have any total areas in your MCast™ instance, note that WSO studies are not run to produce a total area's weather. Instead, the weather for the total area is calculated using the same formula that calculates the load for that total area

Example

If I have a Total area called "Total City" and it is comprised of 2 areas, "Metropolis" (which is weighted as half of the Total City) and "Smallville"(which is weighted as half of Total City), then

• the load representing Total City will be (Metropolis + Smallville)
• and the weather representing Total City will be ((Metropolis weather) x 0.5) + ((Smallville weather) x 0.5)

I am interested in switching to license weather data through Marquette Energy Analytics' preferred vendor. What may be some implications of that change to the weather we are sourcing?

If you decide to switch your weather source to AccuWeather for Business via Marquette Energy Analytics' partnership with them, here are the main implications

• Licensing costs: Our partnership with AccuWeather for Business allows us to source the data for you at a discounted rate. Your weather licensing costs would also be tacked onto your normal MCast licensing quote, so you can pay it all together. Our team would be happy to generate a quote for you to see what the additional cost of your weather data licensing would come out to!
• Weather Data in MCast: Upon switching to AccuWeather or Business, we would shift your instance of MCast™ to source data from AccuWeather using their XML Query Tool, an API that allows us to ensure the most efficient data transfers and the fastest forecasting possible.
• Weather Data for other uses: All of the AccuWeather for Business data could be automatically output from MCast, via either file uploads to your SFTP site or calls to our MCast™ API. If you’re interested in alternatives, we’d be happy to discuss other data access possibilities as well (such as setting you up with a connection to an SFTP site hosting weather data alone).

If you wish to discuss this further or have more questions, reach out to support@marquetteenergyanalytics.com!