Open Source Test Data Ops

A developer-first way to create safe, anonymized test data and sync it across all environments for high-quality local, stage and CI testing

Built for Security and Scalability

Neosync is built for teams of all sizes and ships with features that put security and compliance teams of all sizes at ease. Whether you're an enterprise who needs an air-gapped deployment or a startup looking to get started quickly, Neosync can help.

Complete referential integrity

Neosync automatically handles and preserves your data's referential integrity.

Retries

Neosync handles automatically handles retries in the case of an error during the sync process.

Scheduling

Run jobs ad-hoc, pause existing jobs or schedule jobs to run on any cadence you decide.

Scalable

Neosync's async pipeline is horizontally scalable making it a great choice for bigger data sets.

Security

Neosync comes with RBAC out of the box and is designed to be multi-tenant for teams deploying on-prem.

Audit

Neosync ships with audit controls so security and compliance teams have a full audit trail over the system.

Test Data Ops Built for Developers

Automatically sync all of your data stores, from local databases to s3 buckets with anonymized, production data to safely build and test your applications and services.

APIs to automatically sync local databases with prod to keep them up to date.

Locally develop against safe,anonymized, production-like data

Subset your data using a custom query to shrink it to fit locally.

Use the Neosync CLI to back up to an old copy or pull the latest

fill_local_db.py

1...
2  schedule = "0 23 * * *"
3  haltOnNewColAdd = True
4  jobRes, err = jobclient.CreateJob(ctx, connect.NewRequest({
5      'AccountId': accountId,
6      'JobName': 'prod-to-stage',
7      'ConnectionSourceId': prodDbResp['Msg']['Connection']['Id'],
8      'DestinationSourceIds': [
9          stageDbResp['Msg']['Connection']['Id'],
10          s3Resp['Msg']['Connection']['Id'],
11      ],
12      'CronSchedule': schedule,
13      'HaltOnNewColumnAddition': haltOnNewColAdd,
14      'Mappings': [
15          {
16              'Schema': 'public',
17              'Table': 'users',
18              'Column': 'account_number',
19              'Transformer': JobMappingTransformer.custom_account_number,
20          },
21          {
22              'Schema': 'public',
23              'Table': 'users',
24              'Column': 'address',
25              'Transformer': JobMappingTransformer.address_anonymize,
26          },
27      ],
28  }))
29  if err:
30      raise Exception(err)
31...

1...
2  schedule = "0 23 * * *"
3  haltOnNewColAdd = True
4  jobRes, err = jobclient.CreateJob(ctx, connect.NewRequest({
5      'AccountId': accountId,
6      'JobName': 'prod-to-stage',
7      'ConnectionSourceId': prodDbResp['Msg']['Connection']['Id'],
8      'DestinationSourceIds': [
9          stageDbResp['Msg']['Connection']['Id'],
10          s3Resp['Msg']['Connection']['Id'],
11      ],
12      'CronSchedule': schedule,
13      'HaltOnNewColumnAddition': haltOnNewColAdd,
14      'Mappings': [
15          {
16              'Schema': 'public',
17              'Table': 'users',
18              'Column': 'account_number',
19              'Transformer': JobMappingTransformer.custom_account_number,
20          },
21          {
22              'Schema': 'public',
23              'Table': 'users',
24              'Column': 'address',
25              'Transformer': JobMappingTransformer.address_anonymize,
26          },
27      ],
28  }))
29  if err:
30      raise Exception(err)
31...

Use GitOps to hydrate your CI databases with safe data

Automatically hydrate your CI database with safe, production-like data

Declaratively define a step in your CI pipeline with Neosync

Protect sensitive data from your CI pipelines

Subset your database to reduce your

Fully customizable data transformations

Anonymize, mask or generate data using a transformer from our library of pre-built tranformers or write your own transformation logic in code.

Pre-built Transformers

Choose from our library of pre-built transformers

Custom Transformers

Create your own transformer for when you need bespoke logic

Powerful subsetting

Subsetting allows you to filter your source database so that you can easily reproduce bugs and data errors and shrink your production database so that it fits locally. Neosync maintains full referential integrity of your data automatically.

Test Data Management meets GitOps

Neosync is built with DevOps and infrastructure teams in mind. Use frameworks you know like terraform to manage your Neosync infrastructure and even create new jobs.

Manage your test infrastructure in code

Easily create jobs, connections, mappings and more

Audit and track changes across teams

Centralize your configurations in one place

terraform.tf

1resource "neosync_job" "staging-sync-job" {
2    name = "prod-to-stage"
3
4    source_id = neosync_postgres_connection.prod_db.id
5    destination_ids = [
6      neosync_postgres_connection.stage_db.id,
7      neosync_s3_connection.stage_backup.id,
8    ]
9
10    schedule = "0 23 * * *" # 11pm every night
11
12    halt_on_new_column_addition = false
13
14    mappings = [
15      {
16        "schema" : "public",
17        "table" : "users",
18        "column" : "account_number",
19        "transformer" : "custom_accout_number",
20      },
21      {
22        "schema" : "public",
23        "table" : "users",
24        "column" : "address",
25        "transformer" : "address_anonymize"
26      },
27    ]
28  }

1resource "neosync_job" "staging-sync-job" {
2    name = "prod-to-stage"
3
4    source_id = neosync_postgres_connection.prod_db.id
5    destination_ids = [
6      neosync_postgres_connection.stage_db.id,
7      neosync_s3_connection.stage_backup.id,
8    ]
9
10    schedule = "0 23 * * *" # 11pm every night
11
12    halt_on_new_column_addition = false
13
14    mappings = [
15      {
16        "schema" : "public",
17        "table" : "users",
18        "column" : "account_number",
19        "transformer" : "custom_accout_number",
20      },
21      {
22        "schema" : "public",
23        "table" : "users",
24        "column" : "address",
25        "transformer" : "address_anonymize"
26      },
27    ]
28  }

Join our Community

Backed by a passionate group of early enthusiasts, contributors, and advocates.

Nucleus Cloud Corp. 2024