HomeDocumentationRecipesChangelog
HomeRequest DemoContact
Documentation
HomeRequest DemoContact

Azure Synapse

Create a source to monitor data from Azure Synapse.

Suggest Edits

Prerequisites for Adding an Azure Synapse Source

To connect to Azure Synapse, you need existing Microsoft Entra ID or Microsoft SQL Server accounts.

These accounts need to have the permissions listed in the following table to access databases, schemas, and tables within your Synapse workspace. For more information, see Synapse Database Objects and Microsoft Azure documentation.

Authorization TypePermissions NeededSynapse Server-less
(On-demand)		Synapse with Dedicated Pool
Microsoft Entra IDSynapse SQL Administrator
Synapse Administrator
Storage Blob Data Reader (see Note 1)		All Validio features
(Some limitations with Lineage and Catalog,
see Note 2)		All Validio features
Microsoft SQL ServerOn all databases:
SELECT
VIEW DATABASE STATE
VIEW DEFINITION		NoneAll Validio features
(Without VIEW DATABASE STATE permissions, no Lineage)

📘

Notes

  1. The Storage Blob Data Reader permission is needed on the blob storage services containing external tables.
  2. For authorization with Microsoft Entra ID for Synapse workspaces on a server-less pool, you will be able to view all catalog assets in Lineage (as nodes) and table-view relationships (as edges), but not table-table relationships (as edges). Also, the utilization metrics in Catalog will only show the count of reads, because writes are handled outside of the Synapse (updating the files in the blog store).

Add Azure Synapse Credentials

To monitor Azure Synapse Analytics in Validio, add valid credentials for either Microsoft Entra ID or Microsoft SQL Server. Refer to the following credential parameters tables. For more information, see About Credentials.

👍

Recommendation

For Synapse workspaces on a server-less pool, the Microsoft SQL Server account may not have access to the data. We recommend creating your credential with Microsoft Entra ID authorization.

Microsoft Entra ID Credential Parameters

FieldRequired?Description
Credential TypeyAzure Synapse Entra ID Credential
NameyIdentifier for the credentials. Used when accessing sources.
Client IDyApplication (client) ID which you can find in your Azure Portal.
Client SecretyClient secret value for your application, which you can find in your Azure Portal.
HostyDNS hostname or IP address to the database server.
PortyPort number of the database server.

Microsoft SQL Server Credential Parameters

FieldRequired?Description
Credential TypeyAzure Synapse SQL Credential
NameyIdentifier for the credentials. Used when accessing sources.
UsernameySQL Server username.
PasswordySQL Server password.
HostyDNS hostname or IP address to the SQL server.
PortyPort number of the SQL server.

Add an Azure Synapse Source

To add a source for Azure Synapse,

  1. Navigate to Observability > Sources and click + New Source.
  2. Under Source type, select Azure Synapse.
  3. Under Credentials,
    1. Select Use existing credential.
    2. Select the valid credential from the list.
  4. Under Config,
    1. Enter a Name for the source.
    2. Enter the Database, Schema, and Table to specify the data to read for the source. For more information, see the Configuration Parameters table.
    3. Select the Cron preset to specify the polling frequency for the data.
  5. Under Schema,
    1. Select the Fields to validate the new source.
    2. (Optional) Specify a timestamp field to use as the Cursor field and a Lookback time to indicate how far back to read data from the source. For more information, see Data Warehouses.
  6. Under Window,
    1. Select the Window type.
    2. Specify the Date-time field.
    3. Specify the Window size and Unit to define the period of time to validate the source.
  7. Click Continue to create the source.

Azure Synapse Configuration Parameters

FieldRequired?Description
NameyIdentifier for the source. Used when setting up validators.
DatabaseyName of the Azure Synapse database.
SchemayName of the schema that contains the table to access.
TableyName of the table to read data from.
Cron presetDefines the polling interval, or how often to query the bucket for new data.

Select Custom to use your own cron expression.
Cron expressionAutomatically assigned based on the selected Cron preset. You can customize the expression.

Synapse Database Objects

Validio queries specific database objects to populate Catalog and Lineage pages after the source is added.

Catalog:

  • sys.databases
  • sys.schemas
  • sys.objects
  • sys.tables
  • sys.columns
  • sys.extended_properties
  • sys.types
  • INFORMATION_SCHEMA.TABLES

Lineage:

  • sys.dm_exec_requests_history
  • sys.dm_pdw_exec_requests
  • sys.dm_pdw_exec_sessions

Updated 5 months ago