Creating a Pipeline

What is a pipeline?

A set of methods that include path information and conditions for DFINERY to export data to partner storage. The types of pipelines provided by DFINERY are divided into periodic and one-time.

Notes on Pipeline Types

Periodic Pipeline

• Use this when exporting data according to a set cycle.

• Up to 5 settings can be set.

• Supports only event data.

One-time pipeline

• Use this when exporting data in one go.

• There is no maximum creation limit.

• Supports both event data and profile data.

• For user profile data, there are the following extraction restrictions: Only one user profile with an 'Active' status as of midnight can be generated per day.

• New pipelines cannot be created during extraction.

Creating a Pipeline

Path

Additional Settings > Partner Integration > Partner Sync List > Partner Sync Details

Go to Partner Details Page

Click the '⁝' button on the right edge of the Partner Sync list to go to each partner's detail page.

Click the Create Pipeline button and select the type

Click the 'Create Pipeline' button in the upper right corner, select one from the expanded list, and go to the creation page.

Complete creation after entering the required information

Required information may vary by partner.

Amazon S3 예시

Check pipeline status

You can check the specific status of the pipeline on the Partner Sync details page. Columns can be customized through 'Column Settings' in the upper right of the list area.

Amazon S3

Provided Column

Item	Note
Number of Generations (Periodic)	All pipeline counts (regardless of state)
Active water (Periodic)	Number of pipelines in 'Active' state
Number of completions (One-time)	Number of extractions completed
Number of errors (In the last 7 days)	Provides the number and details of errors that occurred.
Last sync date and time	Last sync date and time of data
Situation	[One-time] • Completed: One-time file creation completed • Extracting: Creating a one-time file • Failure: Failure (S3 authentication error, etc.) [Periodicity] • End: End of repeat cycle • Active: If there is a continuous cycle • Pause: Pauses syncing. • Inactive: Synchronization is disabled (* cannot be reactivated) • Failure: Failure (S3 authentication error)
Administrator name	Pipeline Management Name
S3 Path	Bucket name + bucket folder
Data	Data type selected when creating the pipeline • event • User Profile
To give	Set cycle
Constructor	Original creator
Creation date	First created date
Modifier	Last modified by
Date of modification	Last modified date

Pipeline Error Case

Division	Error message	How to solve	Exposure text
unknown error	All system exceptions	Check the system log to analyze the cause and resolve the problem.	unknown error An unexpected error occurred within the system.
ddb update fail	DynamoDB update failed	Check DynamoDB table status and permissions, and retry.	ddb update fail DynamoAn error occurred while updating data in a DynamoDB table.
query error	Query execution failed	Check query syntax or database connection status.	failed_to_run_query The requested database query failed and was not executed.
invalid_parameter	Invalid extract request parameter	Review the format and values ​​of the request parameters and modify them correctly.	invalid_parameter The provided request parameter is invalid or malformed.
invalid arn	Invalid ARN	Verify that the ARN is correct and try again providing the correct ARN.	invalid arn Failed to retrieve role (Role) through provided ARN (Role).
invalid region or bucket	There is no bucket matching the region and bucket name set in S3	Double check the region and bucket name, and modify them to an existing bucket.	invalid region or bucket The requested S3 region and bucket name do not match or do not exist.
invalid bucket folder format	The format of the bucket folder set in S3 is incorrect	Review the format of the folder path, correct it, and try again.	invalid bucket folder format The S3 bucket folder path is not in the correct format or does not exist.
missing s3 policy	S3 policy is not set for the transferred role	Add a policy for S3 access to the role and reset permissions.	missing s3 policy The required S3 access permissions for the role are not set.
missing kms policy	KMS policy is not set for the transferred role	Add a policy for KMS access to the role and reset permissions.	missing kms policy The required KMS access permissions for the Role are not set.