Skip to content

BIS - Daily

Introduction

The BIS publishes a wide range of datasets related to international banking, debt securities, foreign exchanges, global liquidity, credits, property prices and other similar indicators.

BIS – Daily consists of 3 Sub-DPs.

1) Policy rates Daily 1) US dollar exchange rates daily 1) Effective exchange rates daily.

This includes timeseries and respective metadata with additional fields such as currency, collections, baskets depending upon the datasets.

In case of US dollar exchange rates, we have ‘currency’ and ‘collection’ and in case of Effective exchange rates, we have ‘type’ and ‘basket’

Source: BIS

Tags: Time-series, Risk, Daily

Modules

### Scrapping:

All the datasets under BIS can be found on this link as csv file format. https://www.bis.org/statistics/full_data_sets.htm

Using BeautifulSoup library we are scraping all the links of these datasets and then storing them into a list. We are using a regex pattern match to get the index of the list which has the link of the datasets that we need. The reason of using regex is not having a fixed index number every time we scrap the site for collecting the link.

After getting the links that we download, extract and load the file into a pandas data frame. We then extract the required values from the data frame and the save it as raw data and timeseries for the next stages of the pipeline.

Cleaning:

Duplicate and additional columns are removed from the data. Location names are rectified and country names are formatted correctly.

Geocoder:

Coordinates are added to the metadata for the country. Region and region code are also appended. Geocoder library is used for getting coordinates. We also have a separate JSON file for country’s coordinates to avoid calling third party library to make geocoding process more efficient and faster.

Standardization:

The raw data that we scraped in the Scrapping stage is loaded and additional information like sample frequency, units, source and description are included in the metadata. Function for fetching ISO country code and appending it is present in standardization. Predefined domain and subdomain are added in this step.

MetaData:

Timeseries reference id (ts_ref_id) is added to the timeseries data and final timeseries is stored in the bucket. Metadata format is finalized and also stored in the s3 bucket.

Ingest:

Metadata and timeseries data are ingested in the mongoDB and latest timestamp id (mongoDB id for latest timestamp) is appended to metadata for decreasing search for latest data point.

Data Format

Timeseries Attributes

Attributes Descriptions
ts_ref_id Id used to connect timeseries data to the metadata.
value Timeseries information stored for BIS datasets.
timestamp standard timestamp used for the timeseries

Metadata Attributes

Attributes Descriptions
ts_ref_id Id used to connect metadata to the timeseries
coordinates Latitude and Longitude of the station location (geojson format).
country country of the timeseries data.
country_code ISO 3-letter country code
description description of the indictors
domain Predefined domain by Taiyo.
name name of the BIS datasets
units type of value stored in Timeseries
original_id orginal id defined by BIS (in this case its only {BIS_datasets}_{country iso2 code})
region region for a country according to World Bank Standards
region_code region code for a region according to World Bank Standards.
sample_frequency frequency in which data gets updated on the source.
sub_domain Predefined subdomain by Taiyo.
time_of_sampling time of data collection
timezone Timezone for the time and date
url url for the each of the datasets under BIS.
latest_timestamp_id mongoDB id for latest timestamp in the timeseries.
type (only for Effective exch. Rate) N for Nominal
basket (only for Effective exch. Rate) B for Broad (60 economies), N for Narrow(27 economies)
currency (only for US dollar exch. Rate) currency of the respective country
collection (only for US dollar exch. Rate) A for Average of observations through period

Data Flow

The above data pipeline runs on Argo and it will be executed on a periodic frequency.

DAGs:

  • BIS - Policy rate: Total No of DAGs file is 1
  • BIS - US dollar Exchange rate: Total No of DAGs file is 1
  • BIS – Effective Exchange rate: Total No of DAGs file is 1

Taiyo Data Format

Entity BIS Policy Rate
Frequency Daily
Updated On 24-04-2022 UTC 03:24:30 PM
Coverage Covering about 38 countries
Uncertainties Some older data might not be available for every country.
Entity BIS US dollar exchange rate
Frequency Daily
Updated On 24-04-2022 UTC 03:24:30 PM
Coverage Covering about 81 countries
Uncertainties Some older data might not be available for every country.
Entity BIS Effective exchange rate
Frequency Daily
Updated On 24-04-2022 UTC 03:24:30 PM
Coverage Covering about 86 countries
Uncertainties Some older data might not be available for every country.
## Scope for Improvement

Following can be improved in the next version of the data product:

  • Every time Argo Workflow run, we over write existing data on the S3 bucket.

  • In future we might want to improve it to only scrap the data that we don’t already have.

  • There are Monthly and yearly data published by BIS, We want to include all those as DPs as well.