span8
span4
I'm wondering if anyone can think of a way where I can control the number of jobs being run on FME.
I have a process whereby I need to run some complex analysis on FME Server.要做到这一点,I need to split the process into a large number of jobs (current process has 250,000 jobs).这是唯一一种不会因为需要保存在内存中的特性而导致FME崩溃的方法。
When on desktop,you can control the number of processes in WorkspaceRunner,however,this doesn't seem to exist with the FMEServerJobSubmitter.我还尝试使用RESTAPI,但这有同样的问题。If I run the process in sequence,it would take way too long.If I run in parallel,it tries to load all 250,000 jobs to the FME Serve Queue and crashes the application.
所以,I'm looking for a away to tell FME Server to queue a certain number of jobs in batches.
Once the first batch is done,it would release the second batch.This would enable me to run in parallel but without crashing the application server.
Any suggestions welcome.Multiple JobSubmitters sn't the answer as it still runs one at a time.
What version of FME Server are you using?
在2018/2019年,您应该能够发送更多的作业,队列应该自动使用系统上的可用内存…We do have a customer that regularly sends 30,000 jobs but this is 2018.Jobs do vary and size and this can impact the memory used by the new queue (and old).
The only caveat is If there is some sort of restart on the system only 50,000 jobs would be re-queued to 亚搏在线safeguard against memory issues on restart.This is configurable if you were finding the queue/memory/system could take more jobs in the queue 亚搏在线safely.
我知道另一种方法可以做到这一点,但这并不比分块和使用减速器分批发送作业更漂亮。I think my approach would also be more of a hack approach than the suggested ideas below,这意味着你可能会去一些我们通常不推荐的地方——FME服务器系统数据库。Thus,I'll not bother sharing it!:)
As a temporary hack could you have an interim workspace that handles subbatches?
Let's say you can queue 5000 jobs at a time without issue.The controller workspace calls the interim workspace 50 times (250000/5000),with wait for job to complete set to yes.The interim workspace calls the production workspace 5000 times with wait for job to complete set to yes.
If the parameterization is too difficult to pass through the interim workspace (unique,non-incremental parameters on the production workspace),in the controller workspace you can write the parameters to a file (FeatureWriter) or database and the interim workspace would read that file,parse it,and pass the parameters to the production workspace.
I thought about this and think this is the way I will do it.Just hoping there was a better way.
出于好奇,which application crashes when loading 250'000 jobs in the server queue?FME Server?
© 2019 亚搏在线Safe Software Inc |Legal