# Welcome

Welcome to our docs site!  Here you can read all about our Data Privacy Apps.  Click below to find docs for a specific app!

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-cover data-type="files"></th><th data-hidden></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Synthetic Data Generator</strong></td><td>Docs for our Snowflake Native APp</td><td></td><td></td><td><a href="/spaces/T3mz67BmqxnN8PCFRX83">/spaces/T3mz67BmqxnN8PCFRX83</a></td></tr></tbody></table>


# About

The Synthetic Data Generator app from DataMynd is available as a [Native App](/) on the [Snowflake Marketplace](https://app.snowflake.com/marketplace/listing/GZT1Z1MSVYE/datamynd-synthetic-data-generator?search=synthetic).

The app provides users with a simple workflow to easily generate high-quality synthetic data based on one or multiple tables in a real production schema.

The application works by training one of several available ML models on real data before the trained model is used to generate synthetic data.

The newly generated synthetic data is saved to a new schema and the app can also be used to quickly preview, analyze and evaluate the fit relative to the real data.

<figure><img src="/files/jgGI8YqHSRPApDUj3l4T" alt=""><figcaption><p>Conceptual architecture for Synthetic Data Generator Native App (orange) deployed within a single Snowflake account (blue).</p></figcaption></figure>

All logic is performed on the consumer's Snowflake account. No external API calls are made at any point either during installation or use. No third-party services are used at any point by the application. No Data or metadata are shared outside of the consumer's account.


# Requirements And Setup

The application must be deployed using a role with sufficient privileges (like ACCOUNTADMIN).  Alternatively, see sample SQL below to create a role called CONSUMER\_ADMIN that can be used to install the app.

```
CREATE ROLE IF NOT EXISTS CONSUMER_ADMIN;
GRANT ROLE CONSUMER_ADMIN TO ROLE ACCOUNTADMIN;
```

See below sample SQL to grant the proper privileges to the CONSUMER\_ADMIN role created with the SQL shown above:

```
GRANT CREATE DATABASE, IMPORT SHARE ON ACCOUNT TO ROLE CONSUMER_ADMIN WITH GRANT OPTION;
GRANT CREATE APPLICATION ON ACCOUNT TO ROLE CONSUMER_ADMIN;
GRANT EXECUTE TASK, EXECUTE MANAGED TASK ON ACCOUNT TO ROLE CONSUMER_ADMIN WITH GRANT OPTION;
GRANT USAGE ON WAREHOUSE <insert warehouse name> TO ROLE CONSUMER_ADMIN WITH GRANT OPTION;
GRANT CREATE WAREHOUSE ON ACCOUNT TO ROLE CONSUMER_ADMIN WITH GRANT OPTION;
```

Note, if your account is set up to use [PrivateLink](/synthetic-data-generator/private-link-information), you may have some additional steps to follow.  Please see here or reach out to <support@datamynd.ai> for more information.

Upon running the app for the first time, a 'first time setup' page will launch asking the user to grant certain privileges to the app as well giving a name to a new database that will be created to store the output synthetic data.  You will also notice an optional ‘Training Warehouse’ reference can be configured.  This warehouse will be used for certain asynchronous tasks including model training, and can improve speed and performance of the app.

<figure><img src="/files/1iO1ilzY5wj47wZafdB3" alt=""><figcaption><p>First Time Setup page (as seen in dark mode) after all privileges have been granted.  Click 'Continue to APP' to continue.</p></figcaption></figure>

The application can only use and profile data that it has been granted access to.  When creating a new project in the app, the user will be instructed to grant several privileges to ensure the app can access the appropriate data.  See [Object Privileges](/other/object-privileges) for more details.


# Private Link Information

Since the DataMynd Synthetic Data Generator is a Snowflake Native App with a Streamlit front end, you may require a bit more setup to work with accounts that use PrivateLink (see below).

Solution References:

{% embed url="<https://docs.snowflake.com/en/developer-guide/streamlit/privatelink>" %}
This page offers some information about Streamlit network considerations with PrivateLink
{% endembed %}

{% embed url="<https://docs.snowflake.com/en/developer-guide/streamlit/troubleshooting>" %}
This page offers basic solutions for resolving network (and other) issues encountered when trying to run Streamlit
{% endembed %}

Private Link Documentation:

{% embed url="<https://docs.snowflake.com/en/user-guide/admin-security-privatelink>" %}

{% embed url="<https://docs.snowflake.com/en/user-guide/privatelink-azure>" %}

{% embed url="<https://docs.snowflake.com/en/user-guide/private-service-connect-google>" %}
Note, Streamlit is currently not supported for Google Cloud Private Service Connect
{% endembed %}


# Project Setup

Manage synthetic data projects on this page, and view details on existing projects.  Start by creating a new project or loading an existing one.  Shown as 'Start Page' in app.

**Creating New project:**

1. Click "Create New Project" button.

   <figure><img src="/files/II3rLqxkvKWpyYzsIbqs" alt=""><figcaption></figcaption></figure>
2. Enter desired project name in the empty text box followed and select whether the project will contain one or multiple tables.  We recommend starting with a single table for your first project.
3. It is very important to run the Grant statements (outside of the app) provided when creating a project. Failure to do so will results in errors when trying to select data during the next step.  See [Object Privileges](/other/object-privileges) for more details.
4. Click "Create Project" to continue.

   <figure><img src="/files/CqjLCvpzvWKx6smU2ioc" alt=""><figcaption></figcaption></figure>

**Resuming an existing project:**

1. If this is not your first time using the app, you will see a list of existing projects.  Click an existing project to view details about that project.  Details include high level stats like the schema/tables/chosen, the learning model and whether the model has been trained successfully
2. Click "Load Project".

   <figure><img src="/files/mxqi1OulolvjzC8FgU3K" alt=""><figcaption></figcaption></figure>
3. After the project is loaded you can click "Resume Project" to pick up where you left off. If you've gone all the way through a project, this will load the Data Exploration page.

Deleting an existing project:

1. Once a project has been loaded, you will have the option to delete the project.

<mark style="color:red;">**WARNING**</mark>: Deleting a project will also drop any synthetic data generated using that project.  Please copy the data if needed before doing this.


# Select ML Model

(Model Type) page in app.  This page allows you to select and configure your model and parameters used for training and synthesis.

Depending on your choice of "Single Table" or "Multiple Tables" when creating the project, you will have different models (Synthesizers) available to choose from.

**Single-table** synthesizers include GaussianCopula, CTGAN, TVAE, and CopulaGAN (all based on models from [Synthetic Data Vault](https://sdv.dev/SDV/user_guides/single_table/models.html)).  In the future, this may include other models like those specific to certain industries.

**Single-Table Synthesizer Selection:**  These options vary in speed and data quality. We recommend starting with the "GaussianCopula" model (fastest but least accurate) for single table before moving onto one of the others to improve accuracy of results.

<figure><img src="/files/dZiz62xFzWeT7jQr40kq" alt=""><figcaption><p>Single-table synthesizer details.  CTGAN and TVAE deliver accurate results for a single table</p></figcaption></figure>

**Multi-table** synthesizers currently only include the DataMynd Premium Synthesizer (DmSynthMT1).  The DataMynd synthesizer is built to optimize performance and accuracy running on Snowflake. &#x20;

Note: DmSynthMT1 is only available after selecting multiple tables, but it can be used for a single table as well (just add a single table).

**Advanced Parameters:**

1. Here you will also be able to select optional parameters for the selected model. The defaults are recommended for most users (especially when starting out), and should be adjusted with caution.
2. If using a neural-net based model, you will see an 'Epochs' parameter. We recommend starting with a low number (e.g. 5) and increasing once you've validated the model is working as expected.

<mark style="color:red;">**Warning**</mark>: Most of these parameters' default settings will work for most starting scenarios.  Changing these can cause errors.  We recommend only changing these after you have a good familiarity for using the app.

<figure><img src="/files/tgQ2zyY1GOUwC9uchTyT" alt=""><figcaption><p>See SDV documentation (eg. <a href="https://sdv.dev/SDV/user_guides/single_table/ctgan.html#how-to-modify-the-ctgan-hyperparameters">CTGAN parameters</a>) for details  on single table parameters.</p></figcaption></figure>

**DmSynthMt1 Parameters:**

* epochs: Number of training epochs.  Raise this for more accurate results.  We recommend starting with a low number (\~5) while determining initial fit.
* batch\_size: Number of records (root table) per training batch.  Raise this number to improve performance.  <mark style="color:red;">**Warning**</mark>: too high a number may cause training errors.
* optimize\_batch\_size: Runs an optimization step at the start of the training process.  10-15 minute overhead.  Useful for large or complex datasets, or when epochs is high.
* compile: Enables model to run much more efficiently.  5-10 minute overhead.  Useful for large or complex datasets, or where epochs is high.

<figure><img src="/files/0b3L9J65wLK3T74g14wZ" alt=""><figcaption></figcaption></figure>


# Data Config

Configure your training data, relationships, field parameters, privacy options, and additional constraints.

**Select Your Source Table(s):**

1. Here you will add tables to your project, one at a time.
2. Use the dropdowns to select a database, schema and table.  Remember: you must have run the correct grants to add tables or other objects to a project.  If you do not see your source data here, please see [Object Priviliges](/other/object-privileges).
3. Click the '+' icon to the left of the dropdowns to add the table to your project.

   <figure><img src="/files/IuG99HAhWmtVE8EsKAgh" alt=""><figcaption></figcaption></figure>
4. Once you have added your tables, it's time to configure each table. Click 'Config' to do this.
5. You can also click the 'x' button to the left of a table you've added to remove it.

**Field Details:**

1. First you will configure each field in your table on the 'Field Details' tab.
2. We recommend starting with the "Type" column.  Set all of your ID fields to 'ID'.
3. Check that all your text fields are set to 'Categorical'.
4. Remove any fields that you do not want to include in the model by clicking the 'X' icon to the left of the field.
5. One primary key should be selected (using the P-Key checkbox) if available for each table. "ID" must be selected as the type for the primary key.  This is important, especially when working with multiple tables.&#x20;
6. Click the 'Anonymize' checkbox for any fields that contain PII. These fields will not be included in the training and the results will be generated from scratch.
7. Adjust the 'Format' field where needed.  Only available for ID and datetime fields.  Accepts regex and strftime formats.

   <figure><img src="/files/WKLX4uvzUc61yxbG0Ouu" alt=""><figcaption></figcaption></figure>

**Anonymize:**

**Note**: The anonymize checkbox is only available for categorical fields since continuous fields (dates, numeric fields) are not generated with discrete values from the source data.

When the user selects anonymize for a field, the user must also select the type of output the user expects to replace the field values for that column. E.g. selecting 'name' for the type (person category) will randomly generate full names to populate the field values when generating.

1. Set the anonymization options for each field that contains sensitive data.
2. First, select the Category and Type of anonymization.
3. This will result in random values being generated of that type (using the [Faker ](https://faker.readthedocs.io/en/master/)library). For example, if you have a field of full names, you should select 'person' then 'name' for these two fields.
4. Some types support localization and extra parameters. When this is the case, you will see a dropdown for Locales and a text box for the extra parameters. See tooltips for more details.

   <figure><img src="/files/Mejo4IpvDrh7c2a7RMXF" alt=""><figcaption><p>Anonymization types are listed by category, with a short list of common types shown under the popular category</p></figcaption></figure>

**Tip**: Categorical fields can cause the model training to take longer, along with the number of distinct values for each categorical field. Therefore, the user should be mindful of the number of categorical fields for each table in the model. We also recommend starting small with a smaller number of categorical fields before advancing to different models and more categorical fields / records. This restriction will be lifted in future versions of the app.

**Constraints:**

1. Constraints are rules that the synthetic data must follow. For example, you can set a constraint that the 'age' field must be greater than 18. You would do this by selecting 'Scalar Inequality', then selecting the age field from the dropdown, and finally entering 18 into the input box.
2. Don't forget to click the '+' button to add the constraint to the table.

   <figure><img src="/files/KfE5lSbcaFLdEiym6o9T" alt=""><figcaption><p>This inequality constraint would cause the output publish date field to always be greater than the public version date field.</p></figcaption></figure>

**Relationships:**

1. After you've configured and saved your tables and Field Details (IDs), you must configure the relationships between the tables when working with multiple tables.
2. You must select your parent and child tables, and their respective PK and FK fields.
3. The parent table is the one whose primary key in the relationship is referenced by the child table's foreign key. In other terms, the parent table would be synonymous with the dimension table, while the child table would be synonymous with the fact table.
4. Don't forget to click the '+' button to add the relationship to the project.

   <figure><img src="/files/PgwgFuEwihVYERdwRRp0" alt=""><figcaption></figcaption></figure>


# Train and Generate

This page handles training the synthetic data model and using the trained model to generate synthetic data. Shown as 'Execute' page in the app.

**Model Training:**

1. By default the model will sample "10000" rows of the real data to train the configured model. Increasing this number can improve quality of results but may also increase training time.
2. Select a root table for projects with multiple tables.  This should be your most central fact table in your model.
3. After clicking "Train Model" please allow some time for the model to be generated. Depending on the amount of data being used to train the model this can run for a few minutes to a couple of hours.
4. You may leave this page while the model trains (and even start or edit another model!).  All training and generation tasks run in the background.
5. The Status should automatically update once the model has been trained.

   <figure><img src="/files/Tcoi7gWQCPaLpkUonRL6" alt=""><figcaption></figcaption></figure>

**Data Generation:**

1. After training, the last step is to generate data. Enter the number of rows to be generated, and whether or not to overwrite any existing data in the new synthetic data table.  For projects with multiple tables, you must select the number of rows to generate for each.
2. Click "Generate Data" to generate your synthetic data and move onto the next page to view the results after a few moments! The speed of this step is also dependent on data size and model complexity.

   <figure><img src="/files/6bHlaBPRp4UDRq0wUo9b" alt=""><figcaption></figcaption></figure>
3. Once complete, you will get a notification showing the location that the new data was stored.


# Quality Report and Preview

The Data Explorer page provides several tools for evaluating results.  Shown as 'Explore' page in app.

**Explore Your Data**: This page is intended to show the user how well the new synthetic data matches the original data. The layout option on the left will allow the user to select what types of exploration options to view.

The **Quality Report** includes a histogram comparison of the real and synthetic data as well as several quality metrics that show how well the synthetic data represents the real data.

**Histogram View**: Generates a histogram for each field for the real data (green line) and the synthetic data (blue line). The grey bars indicate the variance % between histograms. Does not currently react to the filters (as of the first release of the app). Large variations indicate that you may need to tweak training parameters (like # of epochs). If you see 100% variance for all bins, you likely have an issue with types (see Configure Page – Column Types)

<figure><img src="/files/RbUVMY7SAMZE31ha08Zy" alt=""><figcaption></figcaption></figure>

The **Table View** allows you to preview your new synthetic data. Filters in the left sidebar can be used for quick exploration (table + histograms).

<figure><img src="/files/1L9LR1FTNDxe5bKv7XRO" alt=""><figcaption></figcaption></figure>

Sidebar Options:

* **Layout**: provides several views of the data and options for toggling certain fields to be showin in histogram comparisons (ids and anonymized fields)

  <figure><img src="/files/NG1T3LX276KHxtAErB7H" alt=""><figcaption></figcaption></figure>
* **Select Table**: For multi-table projects, use this selector to choose which table to view metrics for.

  <figure><img src="/files/nrIM6SKadpWppWycHkd3" alt=""><figcaption></figcaption></figure>
* **Filters**: Filters affect both the Table view as well as the Histogram view. Use these to explore your new synthetic data.
* Note: Currently, the filters do not affect the quality or fit scores & metrics.

  <figure><img src="/files/GrIvyvR8hyMU5KIdgvv7" alt=""><figcaption></figcaption></figure>


# Object Privileges

Upon setting up a project, certain privileges must be granted to the application to provide access to the source data.  The individual privileges are shown below but are also referenced in the app when creating a new project (see [project setup](/application-workflow/project-setup)).

These database and schema level privileges must be granted for all source data:

`GRANT USAGE ON DATABASE`` `<mark style="color:orange;">`<database>`</mark>` ``TO APPLICATION SYNTHETIC DATA GENERATOR;`

`GRANT USAGE, MONITOR ON SCHEMA`<mark style="color:orange;">`<database>.<schema>`</mark>` ``TO APPLICATION SYNTHETIC DATA GENERATOR;`

Depending on the type of source object, you must also run the following grant statements for each object you would like to train using the app.

**Tables:**

`GRANT REFERENCES, SELECT ON TABLE`` `<mark style="color:orange;">`<database>.<schema>.<table>`</mark>` ``TO APPLICATION SYNTHETIC_DATA_GENERATOR;`

**Views:**

`GRANT SELECT ON VIEW`<mark style="color:orange;">`<database>.<schema>.<view>`</mark>` ``TO APPLICATION SYNTHETIC_DATA_GENERATOR;`

**Tables:**

`GRANT SELECT ON MATERIALIZED VIEW`` `<mark style="color:orange;">`<database>.<schema>.<materialized_view>`</mark>` ``TO APPLICATION SYNTHETIC_DATA_GENERATOR;`

**Tables:**

`GRANT SELECT ON DYNAMIC TABLE`<mark style="color:orange;">`<database>.<schema>.<dynamic_table>`</mark>` ``TO APPLICATION SYNTHETIC_DATA_GENERATOR;`

**Tables:**

`GRANT SELECT ON ICEBERG TABLE`` `<mark style="color:orange;">`<database>.<schema>.<iceberg_table>`</mark>` ``TO APPLICATION SYNTHETIC_DATA_GENERATOR;`


# Performance and Accuracy Tuning

Several notes and tips on tuning performance and accuracy for the app.

* Categorical fields can cause the model training to take longer, along with the number of distinct values for each categorical field. Therefore, the user should be mindful of the number of categorical fields for each table in the model. We also recommend starting small with a smaller number of categorical fields before advancing to different models and more categorical fields / records. This restriction will be lifted in future versions of the app.
* Training time may vary based on several factors including data complexity, number of fields (especially categorical fields), and amount of data.
* •For neural net-based models, the # of training epochs will determine the accuracy of the model.  An ideal value can require a fair bit of testing to determine.  Feel free to send an email to <support@datamynd.io> if you need help.  We have found ideal values between 300 and 2000 depending on the dataset (which can take some time to train)