upserts和狗步行：2019年FME中的改变检测有什么新鲜事

I discussed database updatesrecently, and mentioned that there are two general scenarios.

One way is when you receive a changelog – a list of updates to be made – and apply them to your control database. That’s a simple one, because you know in advance what has changed and therefore which features need updates.

The second way of updating is when you receive a whole new dataset, with no indication of what is new or changed. For that you’ll need to doChange detection。Previously an FME Hub transformer by the name ofUpdateDetectorwas commonly used. But in 2019 we have a great update to the换档板transformer that should make it your go-to transformer from now on…

Change Detection: What’s New?

In 2018一种nd earlier the ChangeDetector transformer compared an “original” dataset against a “revised” version and separated the features into Added, Deleted, and Unchanged.

• Added：在未在原件中的修改数据集中的记录
• 删除了：在未修订的原始数据集中的记录
• 不变的：Records in the revised dataset that matched a feature in the original dataset

但是，它做了什么notD.o is identify records that had changed. If a record existed in both the original and revised, but one of its attributes was now a different value, then that counted as a new (Added) feature. This made it hard to carry out “upserts” (database updates where a record already exists) because it was harder to tell whether a feature was truly new or not.

In 2019那however, the transformer is capable of handling updated records. In fact the new transformer design shows this through an Updated output port:

所以这非常重要。它极大地扩展了这种变压器的变化检测范围。

但是，存在将原始功能与其修订后的对应物匹配。FME无法确定已更新修订的记录，而无需原始记录以比较。这是使用密钥属性值完成的，这意味着2019 ChangeRetector有新的参数来处理：

我们很快就会看一个榜样。目前仅注意参数对话框具有名为Update Detection键属性的参数，通过该参数来选择ID或键值。

无论如何，如果该功能听起来很熟悉，可能是因为我们已经在一个名为UpdateTetector的FME集线器变压器中包含它......

Replacing the FME Hub UpdateDetector

The UpdateDetector was created to fill gaps in the ChangeDetector’s functionality. Judging by the number of downloads, it was very popular transformer. But now it is deprecated on the FME Hub:

新的ChangeDetector不仅取代这个中心transformer, it exceeds it in both functionality and performance. The UpdateDetector will still function in an existing workspace, but we advise you to replace it with a new ChangeDetector.

现在让我们来看看新的Changetetector如何工作的示例......

换档板Example

Let’s say I have an address database:

Besides that I have been given a new version of the data. I must now determine which address records have changed, so that I can push those changes to the database. I do that by simply adding a reader for each dataset, and passing it to a ChangeDetector:

From there I can see that 35 records changed between the original and revised dataset, with 2 new ones added. Also 13 records from the original data are absent from the revision. The majority of records are unchanged. Let’s look at the parameters I used:

1. GlobalID是更新检测密钥。这是一个属性，告诉FME如何在原始数据中使用修订的数据中的记录匹配记录。
2. The selected attributes are the ones I am checking for changes. i.e. where two records have a matching GlobalID, check these attributes for differences.
3. This flag tells the transformer to also check the geometry in a spatial dataset. Several advanced parameters control those exact checks (see below).
4. This parameter defines a list in which to store the changes that occurred; for example which attribute values differed and how.

正如参数一种little different, so can the output be…

New Change Detection Output

因为ChangeTetector变压器现在检查匹配项 - 但不一定在每个属性上 - 原始和修改可能会将其视为匹配，但尚未完全相同。例如，属性一种那B.那一种ndC一种re a match, butD.是不同的。功能仍然是一个匹配，因为你没有挑选D.in the Selected Attributes.

To handle that scenario a parameter allows you to output either – or both – of the matched features:

If you do output both features, then a Match ID attribute is added, so that you can identify which features counted as a match.

Additionally, by setting a List Name, the output features record the differences between Original and Revised. This record (for example) has two differences:

The list tells me that two attributes (OWNERNM1 and OWNERNM2) were modified with new values, whereas this list:

...告诉我该功能的几何图形已被修改。

同样 - 就像UpdateTetector所做的那样 - 这个变形金刚设置了fme_db_operation一种ttribute. Here is an example for a deleted record (was in the original, but not the revised):

这意味着我可以简单地传递给数据库编写器的功能，指定功能操作（fme_db_operation）一种nd match column (here GlobalID again)…

…and my address database is automatically modified with updates, additions, and deletions as necessary.

新的公差算法

You may have noticed one of the advanced geometry parameters calledVector Tolerance：

尽管名称，这与之不一样我以前讨论过的公差设置for FME 2018. Why is this tolerance different? Because it’s not trying to find where two lines intersect and it’s not trying to adjust existing points. Instead it finds whether two features are within tolerance, using something called theFréchet distance。

The Fréchet distance is a measure of the similarity of two spatial features (usually “curves”) derived using the distance between the two features.

The common – and simpler – explanation is that of walking a dog. Say you walk along a path with your dog on a lead (leash). You walk in a relatively straight line (the red line, A, below), but your dog moves from side to side, in order to sniff at trees (the blue line, B):

The question is, how long does the lead need to be for each of you to walk your respective path? In the above diagram the widest gap between dog and walker is markedF。如果我的狗铅至少是F长度，我们可以走一定的路径而不互相拉动。

这个概念为改变检测提供了一个很好的解决方案。在FME中，如果原始和修改之间的FRéchet距离小于指定的公差值，则功能不变。

我们觉得这种算法是过去过去的方法的改进。它适用于更多几何形状，它还允许在延伸匹配模式下应用公差，这是之前不可能的。

Fréchet? Which Fréchet?

Please feel free to skip past this part if you aren’t into computational geometry. However, for you connoisseurs of FréchetD.istances, you’ll notice this is aTrue Fréchet那not theDiscrete Fréchet（仅计算顶点之间的距离）。还有一个Weak Fréchet一种nd FME uses that when the Lenient Geometry Matching parameter is active:

A Weak Fréchet is when you say that the walker or the dog is allowed to backtrack their steps. In a True Fréchet each has to keep moving forward.

In this example the dog walker (A) keeps walking in a straight line. The dog walks straight at first (toB1）B.ut then veers around and to the right (toB2）。They are still moving forward along their path, but their path is in a different direction:

In a True Fréchet the walker cannot reverse their path to account for this deviation. The most they can do is stop at their current position. For example here the walker minimizes the lead length by stopping to wait at point a1, while the dog follows their meandering path:

That, perhaps, is where the analogy breaks down a bit. Frechet distances are calculated knowing the path in advance, whereas it’s impossible in real life to know what course a dog will take!

无论如何，在一个弱的Fréchet中，沃克被允许扭转他们的课程。当狗从B1开始前往B2时，助行器可以双回来说A2，以便更短的FRéchet：

如果你不明白，你真的不应该担心。只需查看两条路径，并记住延伸匹配选项意味着两个功能更有可能被归类为匹配。

More Change Detection Information

The above example looked at the new change detection behaviour. However, in some cases you might want to carry out the same Added/Deleted/Unchanged process that the old ChangeDetector did. i.e. you don’t necessarily need to look for modified records.

If that’s the case, then simply leave the Update Detection Key parameter empty:

Then your features are assigned to Inserted or Deleted, according to whether they were Original or Revised input:

And – of course – if you don’t have 2019, then you can still download the UpdateDetector. The term “UpdateDetector” now only appears in Workbench as an alias for the ChangeDetector though; so access the transformer through the FME Hub, being sure to check the option for Show Deprecated in the interface.

此外，这是匹配transformer got a small makeover for 2019. Its parameters dialog was refreshed and gained the same tolerance parameter and algorithm as the ChangeDetector.

Summary

So that’s what’s coming up for Change Detection. Now you’ve read this article you won’t be surprised to see the entirely new ChangeDetector dialog when you upgrade to FME2019.

I believe this is a really useful update, and I look forward to using it. Oh, and in case you were struggling(!) here’s the answers to the canine spot-the-difference image:

Happy FMEing,

Mark Ireland

Mark，Aka Imark，是FME Evangelist（EST.2004），并对FME培训有热情。他喜欢能够以新的和有趣的方式帮助人们理解和使用技术。他的其他一个激情是足球（又名。足球）。他非常喜欢技术和足球，以至于他在一起写了一篇关于这两者的文章！谁会想到？（答案：imark）