Upserts and Dog-Walking: What’s New with Change Detection in FME 2019

I discussed database updates最近，并提到有两种一般情况。

One way is when you receive a changelog – a list of updates to be made – and apply them to your control database. That’s a simple one, because you know in advance what has changed and therefore which features need updates.

更新的第二种方法是，当您收到一个全新的数据集时，没有迹象表明什么是新的或更改的。为此，您需要做change detection. Previously an FME Hub transformer by the name ofUpdatedEtector通常使用。但是在2019年，我们对ChangeDetectortransformer that should make it your go-to transformer from now on…

Change Detection: What’s New?

In 2018更早的更改器变压器将“原始”数据集与“修订”版本进行了比较，并将这些功能分离为添加，删除和不变。

• 添加: Records in the revised dataset that weren’t in the original
• Deleted: Records in the original dataset that aren’t in the revised
• Unchanged: Records in the revised dataset that matched a feature in the original dataset

However, what it didnotdo is identify records that had changed. If a record existed in both the original and revised, but one of its attributes was now a different value, then that counted as a new (Added) feature. This made it hard to carry out “upserts” (database updates where a record already exists) because it was harder to tell whether a feature was truly new or not.

In 2019, however, the transformer is capable of handling updated records. In fact the new transformer design shows this through an Updated output port:

So this is very important. It vastly widens the scope of change detection for this transformer.

However, there is the issue of matching original features to their revised counterpart. FME can’t decide a revised record has been updated, without an original record to compare against. This is done using a key attribute value, meaning that the 2019 ChangeDetector has new parameters to handle that:

We’ll look at an example shortly. For now just notice that the parameters dialog has a parameter called Update Detection Key Attributes, through which to select an ID or key value.

Anyway, if that functionality sounds familiar, it might be because we already included it in an FME Hub transformer called the UpdateDetector…

替换FME Hub UpdatedEtector

The UpdateDetector was created to fill gaps in the ChangeDetector’s functionality. Judging by the number of downloads, it was very popular transformer. But now it is deprecated on the FME Hub:

新的ChangeDetector不仅取代这个中心transformer, it exceeds it in both functionality and performance. The UpdateDetector will still function in an existing workspace, but we advise you to replace it with a new ChangeDetector.

Now let’s take a look at an example of how the new ChangeDetector works…

ChangeDetector Example

假设我有一个地址数据库：

除此之外，我还得到了新版本的数据。现在，我必须确定哪些地址记录已更改，以便可以将这些更改推向数据库。我通过简单地为每个数据集添加读者并将其传递给更改轨道来做到这一点：

从那里，我可以看到原始数据集和修订后的数据集之间发生了变化，并添加了2个新记录。此外，修订中没有原始数据中的13个记录。大多数记录都没有改变。让我们看一下我使用的参数：

1. GlobalID is the update detection key. This is the attribute that tells FME how to match records in the Original data with records in the revised data.
2. The selected attributes are the ones I am checking for changes. i.e. where two records have a matching GlobalID, check these attributes for differences.
3. 该标志告诉变压器还检查空间数据集中的几何形状。几个高级参数控制着这些精确的检查（见下文）。
4. This parameter defines a list in which to store the changes that occurred; for example which attribute values differed and how.

正如参数a little different, so can the output be…

新的变更检测输出

Because the ChangeDetector transformer now checks for matches – but not necessarily on every attribute – it’s possible that Original and Revised might count as a match, and yet not be totally identical. For example, attributesa,b, andcare a match, butdis different. The features are still a match though because you didn’t pickdin the Selected Attributes.

为了处理该方案，参数允许您输出匹配的功能（或两者）：

如果要输出这两个功能，则添加匹配ID属性，以便您可以识别哪些功能算作匹配。

Additionally, by setting a List Name, the output features record the differences between Original and Revised. This record (for example) has two differences:

该列表告诉我，两个属性（ownernm1和ownernm2）已修改为新值，而此列表：

…tells me that the geometry of the feature was modified.

Also – just as the UpdateDetector did – this transformer sets thefme_db_operationattribute. Here is an example for a deleted record (was in the original, but not the revised):

This means that I can simply pass features to a database writer, specify the Feature Operation (fme_db_operation）和匹配列（再次在这里globalid）…

…并且我的地址数据库会自动修改，并根据需要进行更新，添加和删除。

新的公差算法

You may have noticed one of the advanced geometry parameters calledVector Tolerance:

Despite the name, this is not the same asthe tolerance setting I discussed previously对于FME2018。为什么这种容忍度有所不同？因为它不是试图找到两行相交的位置，并且没有试图调整现有点。相反，它发现了使用称为的东西，是否有两个功能Fréchet distance.

Fréchet距离是使用两个特征之间距离得出的两个空间特征（通常是“曲线”）的相似性的度量。

The common – and simpler – explanation is that of walking a dog. Say you walk along a path with your dog on a lead (leash). You walk in a relatively straight line (the red line, A, below), but your dog moves from side to side, in order to sniff at trees (the blue line, B):

问题是，你们每个人都需要多长时间才能走多长时间？在上图中，标记了狗和沃克之间的最大差距F. If my dog lead is at leastFlength, then we can walk our respective path without pulling at each other.

This concept makes a great solution for change detection. In FME terms a feature is unchanged if the Fréchet distance between Original and Revised is less than the specified tolerance value.

We feel that this algorithm is an improvement over the past method. It works on more geometries and it also allows applying tolerance in lenient matching mode, which wasn’t possible before.

Fréchet? Which Fréchet?

Please feel free to skip past this part if you aren’t into computational geometry. However, for you connoisseurs of Fréchet距离，您会注意到这是True Fréchet, not the离散的fréchet(which only calculates distance between vertex points). There is also aWeak Fréchetand FME uses that when the Lenient Geometry Matching parameter is active:

A Weak Fréchet is when you say that the walker or the dog is allowed to backtrack their steps. In a True Fréchet each has to keep moving forward.

In this example the dog walker (A) keeps walking in a straight line. The dog walks straight at first (tob1) but then veers around and to the right (tob2). They are still moving forward along their path, but their path is in a different direction:

In a True Fréchet the walker cannot reverse their path to account for this deviation. The most they can do is stop at their current position. For example here the walker minimizes the lead length by stopping to wait at point a1, while the dog follows their meandering path:

That, perhaps, is where the analogy breaks down a bit. Frechet distances are calculated knowing the path in advance, whereas it’s impossible in real life to know what course a dog will take!

Anyway, in a Weak Fréchet, the walker is allowed to reverse their course. When the dog starts heading from b1 to b2, the walker can double-back to say a2, to make for a shorter Fréchet:

If you don’t understand, you really shouldn’t worry. Just look at the two paths and remember that the Lenient Matching option means two features are more likely to be classed as a match.

More Change Detection Information

上面的示例研究了新的变更检测行为。但是，在某些情况下，您可能需要执行与旧更换器相同的添加/删除/不变的过程。即，您不一定需要寻找修改的记录。

如果是这样，请简单地将更新检测键参​​数空为空：

然后，您的功能被分配给插入或删除，根据它们是原始或修订的输入：

And – of course – if you don’t have 2019, then you can still download the UpdateDetector. The term “UpdateDetector” now only appears in Workbench as an alias for the ChangeDetector though; so access the transformer through the FME Hub, being sure to check the option for Show Deprecated in the interface.

Additionally, theMatchertransformer got a small makeover for 2019. Its parameters dialog was refreshed and gained the same tolerance parameter and algorithm as the ChangeDetector.

Summary

So that’s what’s coming up for Change Detection. Now you’ve read this article you won’t be surprised to see the entirely new ChangeDetector dialog when you upgrade to FME2019.

我相信这是一个非常有用的更新，我期待着使用它。哦，如果您在挣扎（！），这是犬斑差异图像的答案：

Happy FMEing,

马克爱尔兰

Mark, aka iMark, is the FME Evangelist (est. 2004) and has a passion for FME Training. He likes being able to help people understand and use technology in new and interesting ways. One of his other passions is football (aka. Soccer). He likes both technology and soccer so much that he wrote an article about the two together! Who would’ve thought? (Answer: iMark)