Skip to content

BIS - Policy Rate Daily

Introduction

The BIS publishes wide range of datasets related to international banking, debt securities, foreign exchanges, global liquidity, credits, property prices and other similar indicators. The policy interest rate is the rate at which the central bank will pay or charge commercial banks for their deposits or loans. This rate will consequently affect the interest rates that commercial banks apply with their customers, both borrowers and depositors.

Source: BIS

Tags: Time-series, Risk, Daily

Modules

Scrapping:

All the datasets under BIS can be found on this link as csv file format. https://www.bis.org/statistics/full_data_sets.htm

Using BeautifulSoup library we are scraping all the links of these datasets and then storing them into a list. We are using a regex pattern match to get the index of the list which has the link of the datasets that we need. The reason of using regex is not having a fixed index number every time we scrap the site for collecting the link.

After getting the links that we download, extract and load the file into a pandas data frame. We then extract the required values from the data frame and the save it as raw data and timeseries for the next stages of the pipeline.

Cleaning:

Duplicate and additional columns are removed from the data. Location names are rectified and country names are formatted correctly.

Geocoder:

Coordinates are added to the metadata for the country. Region and region code are also appended. Geocoder library is used for getting coordinates. We also have a separate JSON file for country’s coordinates to avoid calling third party library to make geocoding process more efficient and faster.

Standardization:

The raw data that we scraped in the Scrapping stage is loaded and additional information like sample frequency, units, source and description are included in the metadata. Function for fetching ISO country code and appending it is present in standardization. Predefined domain and subdomain are added in this step.

MetaData:

Timeseries reference id (ts_ref_id) is added to the timeseries data and final timeseries is stored in the bucket. Metadata format is finalized and also stored in the s3 bucket.

Ingest:

Metadata and timeseries data are ingested in the mongoDB and latest timestamp id (mongoDB id for latest timestamp) is appended to metadata for decreasing search for latest data point.

Data Format

Timeseries Attributes

Attributes Descriptions
ts_ref_id Id used to connect timeseries data to the metadata.
value Timeseries information stored for BIS datasets.
timestamp standard timestamp used for the timeseries

Metadata Attributes

Attributes Descriptions
ts_ref_id Id used to connect metadata to the timeseries
map_coordinates Latitude and Longitude of the station location (geojson format).
country country of the timeseries data.
country_code ISO 3-letter country code
description description of the indictors
domain Predefined domain by Taiyo.
name name of the BIS datasets
units type of value stored in Timeseries
original_id orginal id defined by BIS (in this case its only {BIS_datasets}_{country iso2 code})
region region for a country according to World Bank Standards
region_code region code for a region according to World Bank Standards.
sample_frequency frequency in which data gets updated on the source.
sub_domain Predefined subdomain by Taiyo.
time_of_sampling time of data collection
timezone Timezone for the time and date
url url for the each of the datasets under BIS.
latest_timestamp_id mongoDB id for latest timestamp in the timeseries.
supplemental_information_and_breaks additional information about the data
publication_source source of publications
compilation

Data Flow

The above data pipeline runs on Argo and it will be executed on a periodic frequency.

DAGs:

  • BIS - Policy rate: Total No of DAGs file is 1
  • BIS - US dollar Exchange rate: Total No of DAGs file is 1
  • BIS – Effective Exchange rate: Total No of DAGs file is 1

Taiyo Data Format

Entity BIS Policy Rate
Frequency Daily
Updated On 24-04-2022 UTC 03:24:30 PM
Coverage Covering about 38 countries
Uncertainties Some older data might not be available for every country.
Entity BIS US dollar exchange rate
Frequency Daily
Updated On 24-04-2022 UTC 03:24:30 PM
Coverage Covering about 81 countries
Uncertainties Some older data might not be available for every country.
Entity BIS Effective exchange rate
Frequency Daily
Updated On 24-04-2022 UTC 03:24:30 PM
Coverage Covering about 86 countries
Uncertainties Some older data might not be available for every country.
## Scope for Improvement

Following can be improved in the next version of the data product:

  • Every time Argo Workflow run, we over write existing data on the S3 bucket.

  • In future we might want to improve it to only scrap the data that we don’t already have.

  • There are Monthly and yearly data published by BIS, We want to include all those as DPs as well.