Data sources are often evolving with the introduction of new features and retiring old ones. This causes the schema (i.e. data fields) to change over time, so ensuring that your data pipelines are kept up to date Jepto automatically updates your pipelines to take advantage of these new fields.
These changes can happen quite often and are not a distraction that you should be concerned with, which is why Jepto will always conduct the maintenance behind the scenes. This article just highlights what happens when this occurs.
At Jepto, we automatically make schema changes from the source to your data warehouse and the update depends on the situation:
If your source adds a new field, we will detect the change and add the field in your warehouse, backfilling where applicable.
If a field is removed, your existing data is maintained and it's ignored in all future syncs so that the old data remains accessible.
If a fields datatype has changed, we attempt to accommodate both the old and new data by retaining the old field and creating a new field with descriptive names.
If your source adds a new table, we will determine if we can add it into an existing table (de-normalized structure) or add it as a new table.
If a table is deleted in your source, no changes will be made to your Data Warehouse and it's ignored on all future syncs.