ADB – Asian Development Bank
Introduction
The Asian Development Bank is a regional development bank established on 19 December 1966, which is headquartered in the Ortigas Center located in the city of Mandaluyong, Metro Manila, Philippines. The company also maintains 31 field offices around the world to promote social and economic development in Asia.
Source: ADB Projects
Tags: Multilateral, Government Announcement, Public Procurement, Infrastructure, Construction
Modules
Scrapping:
Scrapper uses ADB Projects page to get a list of all projects listed there. Scrapper goes through all the pages for each project, collecting all the data and stores it in a single CSV. This CSV is stored in the bucket.
Cleaning:
Currency conversion, handling null values, removing duplicate entries, timestamp formatting, and sector & subsector cleaning is done.
Geocoder:
Coordinates are added to the metadata for the country. Region and region codes are also appended. The Geocoder library is used for getting coordinates. We also have a separate JSON file for the country’s coordinates to avoid calling a third-party library to make the geocoding process more efficient and faster.
Standardization:
Additional information like sample frequency, units, source and description are included in the metadata. Predefined sectors and subsectors are added in this step.
MetaData:
Metadata contains timestamp(approval date) range, sectors, subsectors and country codes. And is also stored in the s3 bucket.
Ingest:
Metadata and project data are ingested into an Elastic Search cluster. Index created based on fields - sector, subsector, map coordinates.
Metadata
Metadata Attributes
| Attributes | Descriptions |
|---|---|
| original_id | ID given by ADB.org |
| name | Name of the Project represented in the data point |
| project_number | Unique number assigned to a project |
| country | Place of performance - Country |
| project_status | Status of Project provided by ADB |
| project_type_or_modality_of_assistance | Project type as categorised by ADB and its modality |
| impact | Impact of project |
| description | Project description |
| project_rationale_and_linkage_to_country_or_regional_strategy | Context and reason for the project |
| gender_equity_and_mainstreaming | Project's gender equity. Projects involving gender equity directly address gender equality and women's empowerment |
| strategic_agendas | Strategic agenda of the project as specified by ADB |
| drivers_of_change | ADB identified drivers of change in the project |
| sector_and_subsector | Sector and Subsector for the project as categorized by ADB |
| sector | Sector based on ADB categorization |
| subsector | Subector based on ADB categorization |
| budget | Total Amount or Investment Amount |
| url | Link to project page |
| environment | The manner in which the project will affect the environment |
| involuntary_resettlement | Whether or not the project requires the involuntary resettlement of people |
| indigenous_peoples | The manner in which the project will affect indigenous peoples |
| summary_of_environmental_and_social_aspects | Summary of all environmental and social aspects of the project |
| environmental_aspects | Summary of all environmental aspects of the project |
| stakeholder_communication_participation_and_consultation | Details regarding how people affected by the project will be/were consulted during the project |
| concept_clearance | Concept clearance for the project |
| credit_committee_meeting | Credit Committee Meeting |
| approval | Approval date |
| pds_creation_date | Project Data Sheet creation date |
| last_pds_update | Date when the PDS was last updated |
| description_of_outcome | Description of project outcomes |
| progress_toward_outcome | Progress towards project outcomes |
| description_of_project_outputs | Description of project outputs |
| status_of_implementation_progress_outputs_activities_and_issues | Summary of project Outputs, Activities and Issues |
| geographical_location | The location in which the project is being implemented |
| during_project_design | Details regarding how people affected by the project will be/were consulted during the design |
| during_project_implementation | Details regarding how people affected by the project will be/were consulted during the implementation |
| consulting_services | Consulting entities, such as individual consultants, companies, universities and non-government organizations engaged by ADB for the project |
| procurement | Procurement notices |
| responsible_adb_officer | ADB officer responsible for the project |
| responsible_adb_department | ADB departmenet responsible for the project |
| responsible_adb_division | ADB division responsible for the project |
| executing_agencies | government/non-government agencies that are working with ADB to implement the project or program |
| mrm | Management Review Meeting date |
| status | Status of the project as identified by ADB |
| approval_date | Approval date |
| approval_year | Approval year |
| country_name | Place of performance - Country Name |
| country_code | Place of performance - Country Code (ISO3) |
| region_name | Place of performance - Region Name |
| region_code | Place of performance - Region Code (ISO3) |
| project_or_tender | Label for Project or Tender |
| source | Source Name Abbreviation |
| aug_id | Unique ID generated for identification of the project |
| map_coordinates | Map Coordinates in decimal degrees |
| identified_sector | Identified Sector |
| identified_subsector | Identified Subsector |
| identified_sector_subsector_tuple | Identified Subsector |
| identified_status | Identified Status |
| |
Data Flow
The above data pipeline runs on Argo and it will be executed on a biweekly frequency.
DAGs:
- ADB: Total No of DAGs file is 1
Scope for Improvement
The following can be improved in the next version of the data product: - In future, we might want to improve it to only scrap the data that we don’t already have.
Useful Links
- https://www.adb.org/projects
- https://www.adb.org/projects/glossary