Data Ingestion - Clinical Release Notes 2.1
FHIR Converter
Identified 6 characters present in the raw data for Clinical upload that were getting HTML encoded as below: < = <
= > & = & ‘ = ' “ = " = –> =
Note: This issue is fixed in the CSV to FHIR JSON Conversion part of the main ADF pipeline. Only double quotes cannot be represented as such as the required output FHIR JSON uses double quotes to represent string values. Hence double quotes, if present, is being converted to single quotes
Audit and Log
Audit and Log incorporated as part of Main ADF from preprocessing, this would automatically generate the report Date parameters for the report. Staged Copy - To handle the environments which deny public database access to ADF (connects thru only Private Endpoints).
Archival
Added source file Data Lake archiving to the following location: ingestion > YYYY > mm > DD Added folder clearing functionality for Clinical preprocess folder: ingestion > preprocess > USCore > Payor Name > CSV Added folder clearing functionality for Clinical landing folder: ingestion > landing > Payor Name > Clinical
[]: