# Creating a Pipeline

## **What is a pipeline?**

A set of methods that include path information and conditions for DFINERY to export data to partner storage. The types of pipelines provided by DFINERY are divided into **periodic** and **one-time.**

## **Notes on Pipeline Types**

### **Periodic Pipeline**

* Use this when exporting data according to a set cycle.
* Up to 5 settings can be set.
* Supports only event data.<br>

### **One-time pipeline**

* Use this when exporting data in one go.&#x20;
* There is no maximum creation limit.
* Supports both event data and profile data.
* For user profile data, there are the following extraction restrictions: Only one user profile with an 'Active' status as of midnight can be generated per day.
* New pipelines cannot be created during extraction.

## **Creating a Pipeline**

#### Path

Additional Settings > Partner Integration > Partner Sync List > Partner Sync Details

### **Go to Partner Details Page**

Click the '⁝' button on the right edge of the Partner Sync list to go to each partner's detail page.

### **Click the Create Pipeline button and select the type**

Click the 'Create Pipeline' button in the upper right corner, select one from the expanded list, and go to the creation page.

<figure><img src="https://content.gitbook.com/content/KGOeGww72vu5In7SeJ8c/blobs/ZH1IYCLoODELW1PMr9Pm/image.png" alt=""><figcaption></figcaption></figure>

## **Complete creation after entering the required information**

Required information may vary by partner.

<figure><img src="https://content.gitbook.com/content/KGOeGww72vu5In7SeJ8c/blobs/f0tbl8xryldx0ymTx7p3/image.png" alt=""><figcaption><p>Amazon S3 예시</p></figcaption></figure>

## **Check pipeline status**

You can check the specific status of the pipeline on the Partner Sync details page. Columns can be customized through 'Column Settings' in the upper right of the list area.

### **Amazon S3**

#### **Provided Column**

<table><thead><tr><th width="162">Item</th><th>Note</th></tr></thead><tbody><tr><td><strong>Number of Generations</strong> (Periodic)</td><td>All pipeline counts (regardless of state)</td></tr><tr><td><strong>Active water</strong> (Periodic)</td><td>Number of pipelines in 'Active' state</td></tr><tr><td><strong>Number of completions</strong> (One-time)</td><td>Number of extractions completed</td></tr><tr><td><p><strong>Number of errors</strong> </p><p>(In the last 7 days)</p></td><td>Provides the number and details of errors that occurred.</td></tr><tr><td><strong>Last sync date and time</strong></td><td>Last sync date and time of data</td></tr><tr><td><strong>Situation</strong><br></td><td><p><strong>[One-time]</strong></p><p>• Completed: One-time file creation completed</p><p>• Extracting: Creating a one-time file</p><p>• Failure: Failure (S3 authentication error, etc.)</p><p><strong>[Periodicity]</strong></p><p>• End: End of repeat cycle</p><p>• Active: If there is a continuous cycle</p><p>• Pause: Pauses syncing.</p><p>• Inactive: Synchronization is disabled (* cannot be reactivated)</p><p>• Failure: Failure (S3 authentication error)</p></td></tr><tr><td><strong>Administrator name</strong></td><td>Pipeline Management Name</td></tr><tr><td><strong>S3 Path</strong></td><td>Bucket name + bucket folder</td></tr><tr><td><strong>Data</strong></td><td><p>Data type selected when creating the pipeline</p><p>• event</p><p>• User Profile</p></td></tr><tr><td><strong>To give</strong></td><td>Set cycle</td></tr><tr><td><strong>Constructor</strong></td><td>Original creator</td></tr><tr><td><strong>Creation date</strong></td><td>First created date</td></tr><tr><td><strong>Modifier</strong></td><td>Last modified by</td></tr><tr><td><strong>Date of modification</strong></td><td>Last modified date</td></tr></tbody></table>

## **Pipeline Error Case**

<table><thead><tr><th width="154">Division</th><th width="182">Error message</th><th>How to solve</th><th>Exposure text</th></tr></thead><tbody><tr><td>unknown error</td><td>All system exceptions</td><td>Check the system log to analyze the cause and resolve the problem.</td><td><mark style="color:red;"><strong>unknown error</strong></mark><br>An unexpected error occurred within the system.</td></tr><tr><td>ddb update fail</td><td><p>DynamoDB update failed</p><p><br></p></td><td>Check DynamoDB table status and permissions, and retry.</td><td><mark style="color:red;"><strong>ddb update fail</strong></mark><br>DynamoAn error occurred while updating data in a DynamoDB table.</td></tr><tr><td>query error</td><td>Query execution failed</td><td>Check query syntax or database connection status.</td><td><mark style="color:red;"><strong>failed_to_run_query</strong></mark><br>The requested database query failed and was not executed.</td></tr><tr><td>invalid_parameter</td><td>Invalid extract request parameter</td><td>Review the format and values ​​of the request parameters and modify them correctly.</td><td><mark style="color:red;"><strong>invalid_parameter</strong></mark><br>The provided request parameter is invalid or malformed.</td></tr><tr><td>invalid arn</td><td>Invalid ARN</td><td>Verify that the ARN is correct and try again providing the correct ARN.</td><td><mark style="color:red;"><strong>invalid arn</strong></mark><br>Failed to retrieve role (Role) through provided ARN (Role).</td></tr><tr><td>invalid region or bucket</td><td>There is no bucket matching the region and bucket name set in S3</td><td>Double check the region and bucket name, and modify them to an existing bucket.</td><td><mark style="color:red;"><strong>invalid region or bucket</strong></mark><br>The requested S3 region and bucket name do not match or do not exist.</td></tr><tr><td>invalid bucket folder format</td><td>The format of the bucket folder set in S3 is incorrect</td><td>Review the format of the folder path, correct it, and try again.</td><td><mark style="color:red;"><strong>invalid bucket folder format</strong></mark><br>The S3 bucket folder path is not in the correct format or does not exist.</td></tr><tr><td>missing s3 policy</td><td>S3 policy is not set for the transferred role</td><td>Add a policy for S3 access to the role and reset permissions.</td><td><mark style="color:red;"><strong>missing s3 policy</strong></mark><br>The required S3 access permissions for the role are not set.</td></tr><tr><td>missing kms policy</td><td>KMS policy is not set for the transferred role</td><td>Add a policy for KMS access to the role and reset permissions.</td><td><mark style="color:red;"><strong>missing kms policy</strong></mark><br>The required KMS access permissions for the Role are not set.</td></tr></tbody></table>