斯潘8
span4
每一个FME翻译通常是你计算机上的一个过程。然而,FME can be set up to take advantage of multiple-core processors and improve parallelization of computations (doing multiple tasks at once).FME还利用了超线程技术,一种用于使每个物理核心在主机操作系统中显示为两个逻辑处理器的技术。
By using parallel processing,performance may be improved significantly over a single process.
As a brief introduction,注意,FME中的每个并行过程都使用自己的数据集,and data cannot be passed between processes.因此,必须使用group by参数将数据分组,and set each group to be handled by a different process.
用户正在计算温哥华市公园游客数量的统计数据,使用FME的统计计算器变压器。每个公园都有一个属性来定义它所居住的社区。That neighborhood attribute is used to group the data and by setting a Parallel Processing Level,每组由一个单独的过程处理,potentially improving performance.
The processing level determines how many processes run in parallel.最小创建的进程最少。极端创造最多。确切数量取决于所用计算机上的核心和处理器数量:
然而,FME将创建的过程数量是有限制的。此限制与FME许可级别有关。FME Base Edition allows a maximum of 4 processes;专业版:8;All other editions: 16.
Here are some general tips for parallel processing:
In theory,parallel processing should produce results faster than a single process.然而,there are instances where that might not be the case:
Rather than have a single transformer carry out parallel processing,可以对一组变压器进行并行处理。
This is done by creating a Custom Transformer from that group.A custom transsformer has its own parameters for parallel processing,and it does not have to be limited to a single transformer within it.
使用这样的自定义转换器还意味着“分组依据”和“并行处理依据”设置可能不同(例如,我可以按邻居将我的公园分组,但在城市的基础上进行并行处理)。
This is a set of examples where Parallel Processing was of use.For a tutorial to carry out yourself,seethis page.
All examples here were conducted on a Quad Core (8 virtual processors) machine with 4Gb of RAM on a 64-bit Windows platform.Keep in mind that results may vary depending on hardware configuration and FME version.
Because surface modeling is such an intense process,使用并行处理可能非常有益。
此示例从点云输入生成DEM:
The RasterDEMGenerator group-by is set tofme_basename将每个点云作为自己的组进行处理。
Moderate is the best result here,比没有并行化快两倍多。最小并行速度较慢,因为它不使用完整的处理能力。攻击性和极端模式的速度较慢,因为它们使用的是完全的处理能力,而彼此的代价是。
在第二次测试中,the point cloud files are grouped according to the first character of their name (this is what the SubstringExtractor transformer is for).
使用更大的测试数据集可以显示结果与数据大小的比例关系:
tingenerator是SurfaceModeler的另一个子集,类似于RasterDemGenerator。
A single TINGenerator in this example takes five minutes to produce a surface.然而,even before parallelism,we can use a trick where one TINGenerator makes small surfaces (from each source LAS) with a second TINGenerator to combine these small surfaces into a single surface.
Coupling this double surface generation with parallel processing gives excellent results:
Based on the results of the tests above we can decisively conclude that parallel processing allows faster surface modelling and can be recommended for the machines supporting multi-threading.
此示例使用一个shapefile数据集,其中包含美国主要道路,目的是用25 m的缓冲区缓冲每条道路。该进程将缓冲转换器包装在自定义转换器中,以便group by参数可以对parallel process by参数使用不同的属性。
This is useful because it lets us create the optimal number of groups,这里是8点到16点之间。The ModuloCounter transformer is used to do this,它的“count maximum”参数是创建的组数:
The numbers for buffering of 450,000 original road segments are the following:
As we can see,the smaller sizes of the groups in the last test do not compensate the multiprocessing overhead (firing up FME sessions and sending features between FME instances).如果每个小组都有更多的特性,这就不算什么问题了。
并行处理可以在工作区中的任何变压器上使用。In the above example,parallel processing the LineJoiner transformer (wrapped up in a custom transformer) gives the following:
We could conclude that parallel processing has no advantages over normal line joining.然而,when the dataset is approximately 5 times as large,the results are quite different:
Without parallel processing,one single process can hog resources,在FME优化内存使用并将数据缓存到磁盘时使计算机瘫痪。
Clipping is another operation where multiprocessing can be beneficial.在这个例子中,we read US major roads,已经加入国家内部,and clip them to the county boundaries.Again,we use the second digit of the FIPS number for making processing groups:
The results on a large dataset (~450,000 features) gives the following:
使用更大的数据集(~2250000个特性),结果更加明显:
Clipping point clouds in 3D can be useful for a simple surface filtering:
The results on a relatively small point cloud look as follows:
For more information,请看Parallel Processing documentationor theParallel Processing section of the Desktop Advanced Training.
© 2019 亚搏在线Safe Software Inc |合法的