span8
span4
span8
span4
Each FME translation is usually a single process on your computer.However,FME can be set up to take advantage of multiple-core processors and improve parallelization of computations (doing multiple tasks at once).FME also makes use of hyper-threading,a technology used to make each physical core appear as two logical processors to the host operating system.
By using parallel processing,performance may be improved significantly over a single process.
As a brief introduction,note that each parallel process in FME uses its own set of data,and data cannot be passed between processes.Therefore you must divide data into groups using a Group-By parameter,and set each group to be handled by a different process.
Here a user is calculating statistics about the number of visitors to parks in the city of Vancouver,using FME's StatisticsCalculator transformer.Each park has an attribute that defines which neighborhood it resides in.That neighborhood attribute is used to group the data and by setting a Parallel Processing Level,each group is handled by a separate process,potentially improving performance.
The processing level determines how many processes run in parallel.Minimal creates the fewest processes.Extreme creates the most.The exact amount depends on the number of cores and processors on the computer being used:
However,there is a limit to the number of processes that FME will create.This limit is tied to the FME license level.FME Base Edition allows a maximum of 4 processes;Professional Edition: 8;All other editions: 16.
Here are some general tips for parallel processing:
In theory,parallel processing should produce results faster than a single process.However,there are instances where that might not be the case:
Rather than have a single transformer carry out parallel processing,it's possible to enable parallel processing on a whole group of transformers.
This is done by creating a Custom Transformer from that group.A custom transsformer has its own parameters for parallel processing,and it does not have to be limited to a single transformer within it.
Using a Custom Transformer like this also means that the "group by" and "parallel process by" settings can be different (for example I might group my parks together by neighborhood,but parallel process them on the basis of city).
This is a set of examples where Parallel Processing was of use.For a tutorial to carry out yourself,seethis page.
All examples here were conducted on a Quad Core (8 virtual processors) machine with 4Gb of RAM on a 64-bit Windows platform.Keep in mind that results may vary depending on hardware configuration and FME version.
Because surface modeling is such an intense process,using parallel processing can be very beneficial.
This example generates a DEM from a Point Cloud input:
The RasterDEMGenerator group-by is set tofme_basenameto process each point cloud as its own group.
Moderate is the best result here,over twice as fast as no parallelization.Minimal parallelism is slower because it does not use the full processing power.Aggressive and Extreme modes are slower because they are using full processing power at each others expense.
In a second test,the point cloud files are grouped according to the first character of their name (this is what the SubstringExtractor transformer is for).
Using a larger test dataset shows the results do scale with data size:
TINGenerator is another subset of SurfaceModeller similar to the RasterDEMGenerator.
A single TINGenerator in this example takes five minutes to produce a surface.However,even before parallelism,we can use a trick where one TINGenerator makes small surfaces (from each source LAS) with a second TINGenerator to combine these small surfaces into a single surface.
Coupling this double surface generation with parallel processing gives excellent results:
Based on the results of the tests above we can decisively conclude that parallel processing allows faster surface modelling and can be recommended for the machines supporting multi-threading.
This example uses a Shapefile dataset containing major US roads where the intention is to buffer each road with a 25 m buffer.The process will wrap the Bufferer transformer inside a custom transformer so that the Group-By parameter can use a different attribute to the Parallel Process By parameter.
This is useful because it lets us create the optimal number of groups,here between 8 and 16.The ModuloCounter transformer is used to do this,its "Count Maximum" parameter being the number of groups created:
The numbers for buffering of 450,000 original road segments are the following:
As we can see,the smaller sizes of the groups in the last test do not compensate the multiprocessing overhead (firing up FME sessions and sending features between FME instances).This would be less of an issue were each group to have a much larger number of features.
Parallel processing can be used on any transformer in a workspace.In the above example,parallel processing the LineJoiner transformer (wrapped up in a custom transformer) gives the following:
We could conclude that parallel processing has no advantages over normal line joining.However,when the dataset is approximately 5 times as large,the results are quite different:
Without parallel processing,one single process can hog resources,paralyzing the computer while FME optimizes memory use and caches data to disk.
Clipping is another operation where multiprocessing can be beneficial.In this example,we read US major roads,that are already joined within states,and clip them to the county boundaries.Again,we use the second digit of the FIPS number for making processing groups:
The results on a large dataset (~450,000 features) gives the following:
With an even larger dataset (~2,250,000 features) the results are even more marked:
Clipping point clouds in 3D can be useful for a simple surface filtering:
The results on a relatively small point cloud look as follows:
For more information,please seeParallel Processing documentationor theParallel Processing section of the Desktop Advanced Training.
How To Use Parallel Processing in FME
Converting Point Clouds to Surface Models Using the PointCloudLASClassifier
How to Read and Translate all Feature Classes from Multiple ESRI Geodatabases
In what order are features processed when there are parallel transformers
Maximum concurrent FME processes error
Creating Boundary and Point Features from a Point Cloud
Clipping and Tiling Point Cloud Data
Passing a Published Parameter to a Workspace from the Command Line
© 2019 亚搏在线Safe Software Inc |Legal