关于并行处理

每个FME翻译通常在您的计算机上作为单一进程运行。这通常意味着,FME顺序地处理指定的每组特征通过...分组parameter. Versions of FME 2012+ can use multiple-core processors, which, on modern personal computers, allows multiple tasks to be performed in parallel. FME also uses hyper-threading, a technology used to make each physical core appear as two logical processors to the host operating system. By splitting the workload between cores/processors, FME performance can improve.

In transformers that support this feature, parallel processing lets you run a transformation as several simultaneous processes. The Group-By parameter allows you to assign features to processes. The Parallel Processing parameter allows you to define different levels of processing, from没有平行极端.

In the example here, the Group By parameter is set toSTATE基于读者数据集。通过设置并行处理级别,FME可以将每个状态运行为单独的同时进程。

并行处理水平

参数 过程数量
没有平行 1
Minimal coresThe processor, or CPU, is the physical part of the computer that performs mathematical calculations. It is the most important part of a computer system. Traditional processors have only one core on the processor, meaning that at any given time, only one set of calculations is being performed. If a processor is dual-core, this means the single chip contains hardware for two processors, now called cores to distinguish them from the single chip, running simultaneously, side by side. (Source: http://www.ehow.com/facts_5730257_computer-core-processors_.html)/ 2
中等 确切数量的核心
Aggressive 核心x 1.5
极端 cores x 2

例如,在四核机器上,最小的并行性将导致两个同时的FME进程。8核机上的极端并行性将导致16个同时流程。您可以尝试此功能,并在Windows任务管理器和Workbench日志窗口中查看信息。有关更多信息,请参阅Parallel Processing Level Parameter如下。

Note:FME License:There is a limit to the number of processes available for FME licenses: Base Edition: 4; Professional Edition: 8; All other editions: 16.

任务管理器进程

启用并行处理时,FME通过产生新的并行处理功能组fmeworker.exeinstance for each group of features.

  • 在任务管理器中,流程如额外可见fmeworker.exeinstances:

工作台日志

在Workbench日志中,信息消息显示许可证限制(如果适用),请求,处理每个“Worker”的过程内存使用情况,并识别有关每个Worker_Key的信息:

11:25:16 |2.0 |0.0 |通知|目前的FME许可证有限公司的人数限制为“16”

11:25:16 |2.0 |0.0|INFORM|AreaOnAreaOverlayer: The Parallel processing level of `MODERATE' has requested `8' workers

11:25:23| 0.0| 0.0|INFORM|11> START - ProcessID: 6852, peak process memory usage: 26384 kB, current process memory usage: 26384 kB

11:25:23| 0.0| 0.0|INFORM|11> FME Configuration: Command line arguments are `C:\apps\FME_2013\fmeworker' `C:\Users\\AppData\Local\Temp\childProcMap1353957917240_7552.fme' `LOG_STANDARDOUT' `YES' `-WORKER_CAPABILITY' `215L804U4L92U1L10U1L5U1' `-WORKER_KEY' `4120345835'

Note:当FME产生额外的进程时,它需要将输入功能发送到新进程并从过程中接收输出功能。与单进程模式相比,这增加了额外的CPU开销。

Parallel Processing Parameter

To use parallel processing, the workflow should have several groups of features that can each be processed independently. Each group will become a separate (parallel) process. Some grouping techniques are discussed below.

The level of parallelism (how many processes can be executed at a single time) depends on theParallel Processingparameter, which has five modes:

  • 没有平行
  • Minimal
  • 中等
  • Aggressive
  • 极端

根据所执行的操作,一种模式可以比另一个模式更有利,并且侵略性或极端并不总是提供最佳性能:在某些工作区中,并行处理不提供任何优势;在其他工作区中,最小或中等的并行性是最佳选择(例如,与表面相关的变压器如surfacemodeller.要么TINGeneratorusually work best with these options).

Parallel processing is incorporated into some transformers, and you can also expose its functionality through custom transformers.

To use Parallel Processing on a custom transformer, click the Transformer Parameters in the Navigator pane:

具有并行处理的自定义变压器不必限于其内的单个变压器:您可以使用多个变压器。

For more information on using parallel processing with custom transformers, seeThe FME Evangelist.

Usage Notes

Parallel processing can improve FME performance; but it can also degrade it or have very little effect. When using parallel processes, it is important that the processing (CPU) time for each group is anticipated to be significantly more than the overhead of launching a new process and sending the features back and forth between processes. If this is not the case, then enabling parallel processing will be slower than using no parallelism.

Trying a small subset of your data in multi-processing mode will help you determine whether there is an advantage to using it on an entire dataset.

Many, Small Groups

Parallel processing is not recommended when you have many groups, each with a small number of features. Each group spawns an FME process and that takes time. For example, with 10,000 groups of 10 features, you might find it costs more performance to start and stop FME 10,000 times than you save in parallel processing. Conversely, 10 groups of 10,000 features might be more worthwhile.

Data Volumes

Parallel processing only provides an advantage when data volumes are large enough: for smaller datasets, the overhead of running multiple processes can easily make the translation slower than a single process.

Other System Resources

You need to ensure other system resources such as memory are adequate for the task. Firing up eight processes to do heavy polygon dissolving when you have eight cores is fine, but if you only have 2GB of memory then you may actually slow down a translation.

当任务卸载到其他地方时,并行处理非常有效。例如,如果您有多个请求通过该请求httpcaller., it might be worth using parallel processing because the impact on system resources is small.

写信给磁盘

当任务涉及令状ing to disk, spawning multiple processes will not speed up the task.

附加信息和示例

FME Community: Parallel Processing

The FME Evangelist