Azure Synapse
Create a source to monitor data from Azure Synapse.
Prerequisites for Adding an Azure Synapse Source
To connect to Azure Synapse, you need existing Microsoft Entra ID or Microsoft SQL Server accounts.
These accounts need to have the permissions listed in the following table to access databases, schemas, and tables within your Synapse workspace. For more information, see Synapse Database Objects and Microsoft Azure documentation.
| Authorization Type | Permissions Needed | Synapse Server-less (On-demand) | Synapse with Dedicated Pool |
|---|---|---|---|
| Microsoft Entra ID | Synapse SQL Administrator Synapse Administrator Storage Blob Data Reader (see Note) | All Validio features (Some limitations with Lineage and Catalog, see Note) | All Validio features |
| Microsoft SQL Server | On all databases: SELECT VIEW DATABASE STATE VIEW DEFINITION | None | All Validio features (Without VIEW DATABASE STATE permissions, no Lineage) |
Note: The Storage Blob Data Reader permission is needed on the blob storage services containing external tables.
Note: For authorization with Microsoft Entra ID for Synapse workspaces on a server-less pool, you will be able to view all catalog assets in Lineage and relationships between table assets and view assets. However, you will not see relationships between two table assets. Also, the utilization metrics in Catalog will only show the count of reads, because writes are handled outside of the Synapse (updating the files in the blog store).
Add Azure Synapse Credentials
To monitor Azure Synapse Analytics in Validio, add valid credentials for either Microsoft Entra ID or Microsoft SQL Server. Refer to the following credential parameters tables. For more information, see About Credentials.
Recommendation For Synapse workspaces on a server-less pool, the Microsoft SQL Server account may not have access to the data. We recommend creating your credential with Microsoft Entra ID authorization.
Microsoft Entra ID Credential Parameters
| Field | Required? | Description |
|---|---|---|
| Credential Type | y | Azure Synapse Entra ID Credential |
| Name | y | Identifier for the credentials. Used when accessing sources. |
| Client ID | y | Application (client) ID which you can find in your Azure Portal. |
| Client Secret | y | Client secret value for your application, which you can find in your Azure Portal. |
| Host | y | DNS hostname or IP address to the database server. |
| Port | y | Port number of the database server. |
Microsoft SQL Server Credential Parameters
| Field | Required? | Description |
|---|---|---|
| Credential Type | y | Azure Synapse SQL Credential |
| Name | y | Identifier for the credentials. Used when accessing sources. |
| Username | y | SQL Server username. |
| Password | y | SQL Server password. |
| Host | y | DNS hostname or IP address to the SQL server. |
| Port | y | Port number of the SQL server. |
Add an Azure Synapse Source
To add a source for Azure Synapse,
- Navigate to Observability > Sources and click + New Source.
- Under Source type, select Azure Synapse.
- Under Credentials,
- Select Use existing credential.
- Select the valid credential from the list.
- Under Config,
- Enter a Name for the source.
- Enter the Database, Schema, and Table to specify the data to read for the source. For more information, see the Configuration Parameters table.
- Select the Cron preset to specify the polling frequency for the data.
- Under Schema,
- Select the Fields to validate the new source.
- (Optional) Specify a timestamp field to use as the Cursor field and a Lookback time to indicate how far back to read data from the source. For more information, see Data Warehouses.
- Under Window,
- Select the Window type.
- Specify the Date-time field.
- Specify the Window size and Unit to define the period of time to validate the source.
- Click Continue to create the source.
Azure Synapse Configuration Parameters
| Field | Required? | Description |
|---|---|---|
| Name | y | Identifier for the source. Used when setting up validators. |
| Database | y | Name of the Azure Synapse database. |
| Schema | y | Name of the schema that contains the table to access. |
| Table | y | Name of the table to read data from. |
| Cron preset | Defines the polling interval, or how often to query the bucket for new data. Select Custom to use your own cron expression. | |
| Cron expression | Automatically assigned based on the selected Cron preset. You can customize the expression. |
Synapse Database Objects
Validio queries specific database objects to populate Catalog and Lineage pages after the source is added.
Catalog:
- sys.databases
- sys.schemas
- sys.objects
- sys.tables
- sys.columns
- sys.extended_properties
- sys.types
- INFORMATION_SCHEMA.TABLES
Lineage:
- sys.dm_exec_requests_history
- sys.dm_pdw_exec_requests
- sys.dm_pdw_exec_sessions
Updated 8 months ago