Hello everyone.I've been challenged with providing an FME Server workspace that parses all or some of the the Summary Stats info from the jobs log file in order to post it in a Slack and email notification.

My specific question is how can I ensure that the correct job_id log file is called once the workspace completes on the server?

These server workspaces are several that run independent of one another from the same repository to update a PostGIS table with updated records.

I currently have a workspace that successfully parses the Features Written and Number Records Added Summaries,but it's being ran against an older log file from the Desktop.I'd like to have Server recognize a new job id log file,call that file into this published workspace,parse the necessary info,write it to a MSWord template or templated email message,and send out as an update notification.

*I understand that Python is probably the best practice to perform this 亚搏在线workflow,but outside of Python and using only my FME Pro Workbench,how can I achieve this using what's available in the Desktop and published to Server (FME Cloud) instance.

**My versions of all are 2018.1 Build 18520.

Introduction

In FME Server 2018,a move was made to deprecate Job Priority and replace it with Job Queues.The reason behind this was driven by necessary architectural changes as FME Server moves towards a more robust fault tolerant model.The option was still available in 2018.0 to specify a Job Priority when running a workspace but behind the scenes,a Job Queue was dynamically created instead.

Now for FME 2018.1,the option to specify a Job Priority has been removed entirely.Previous Job Priorities can still be retrieved using REST API,and theFMEServerJobSubmittertransformer is backward compatible.To learn more about Job Queues,please read thedocumentation.

How to Prioritize Jobs in FME Server 2018.0+

To prioritize jobs in FME Server 2018.0+,a Job Queue can be created with the Priority set from 1 (highest) to 10 (lowest).Job Queues are set on the Engines and Licensing Page,and can then be assigned to any workspace,repository,schedule or subscription.For detailed instructions on how to set up a Job Queue,see theJob Queuedocumentation orQueues and Data Processing Jobs: FME Server and the Art of Queuingblog post.

Workaround for FMEServerJobSubmitter

In FME Desktop 2018.1+,when using theFMEServerJobSubmittertransformer and connecting to FME Server 2018.0+,the Job Priority parameter is no longer available.To set a priority,a Job Queue will need to be assigned.When connecting to an FME Server older than 2018.0,the Job Priority parameter will be available to maintain backward compatibility.

If using a version of FME Desktop older than 2018.1,connecting a FMEServerJobSubmitter to FME Server 2018.1+ with a Job Priority set,nothing will happen.If you would like to set a priority,you will need to create a Job Queue and assign the workspace to it on runtime.

Workaround when Upgrading to FME Server 2018.1+

After restoring a backup from a previous version of FME when upgrading to FME 2018.1,the Job Priority field can be retrieved using REST API.

Select one of the three ways below and enter your server information into the URL.Then paste the URL into a browser window,this will return a JSON script.In the JSON script look for priority: which will be under the heading TMDirectives,this is the priority that was originally set in the older version of FME Server.To overwrite this priority in 2018.1+,assign the schedule,subscription or workspace to a Job Queue.To pretty-print your JSON for readability,enter the returned JSON into thisweb app.

3 main ways of retrieving the job priority using REST API:

1) Schedule with a priority set

http://
       
        /fmerest/v3/schedules/
        
         /
         
        
       

Results:

2) Subscription with a priority set

http://
       
        /fmerest/v3/notifications/subscriptions/
        
         /properties
        
       

Results:

3) Job ID (of a workspace running on an older schedule/subscription)

http://
       
        /fmerest/v3/transformations/jobs/id/
        
       

Results:

If you have any additional questions regarding Job Priority,post aquestionin our 亚搏国际在线官网community or contact亚搏在线Safe Support.

Have you ever had a job complete successfully on FME Server but didn't realize there are warnings in the log?

Warnings can range from must read to simple FYI's which the log displayed in FME Server's UI shows in blue text.Great!What would be even better is if the job completion page let me know if the job had any warnings or within the job history table there was a warning section.

This would bring more awareness to the user that the job may have something to double check without explicitly looking through the log.

What does everyone think of this idea?Let me know in the comments.

I suggest to insert a column "Execution time" in the jobs Completed and a "Last execution time" in the Scheduler.That would be an easy way to manage the time/day we want a process scheduled running.No Dashboards needed.

Maybe it would be cool to be able to choose which column we want to see.If I right click on the bar column names,I could delete or add some columns.Like in Windows Explorer.

Troubleshooting Tips

A good place to start when trying to identify why a workspace that runs without a problem on FME Desktop fails when it is run on FME Server is the jobs log file.Check there to see if there are any error or warning messages that may help get a better idea of what is causing the failure on FME Server.

Also,keep in mind any differences between the system where your installation of FME Desktop is running versus your installation of FME Server.

• Is one running a 32-bit FME vs a 64-bit FME?
• Is the FME Server system able to access where your data is stored (either through a UNC path or over the network to your database)?
• Are all the required applications and client libraries (Oracle clients,JDBC,etc) correctly installed on the FME Server system?

Note: If you are working with a distributed installation of FME Server,ensure that the machines where your FME Engines are installed have been correctly configured as above.

Error Message in the Job Log or Web Interface

Custom transformer can't be found?

• If you are using a linked custom transformer inside your workspace,you'll need to make sure the transformer definition file (.fmx) has been published where the FME Server Engine can find it.
  See:Publishing Custom Resources to FME Server

Error related to a Web or Database Connection?

• Take a look at this page for troubleshooting ideas:
  FME Server Troubleshooting: Connections (Web and Database)

Can't find source or destination dataset?

If you are working with data located on your local system,you'll need to ensure that it is placed in a location where the FME Server Engines are able to access it.If you are working with a Database,ensure that it allows connections from the machine(s) where your FME Engines are installed.

• Workspace fails when reading/writing data on a Mapped Drive
• Workspace fails when reading/writing data through a UNC Path
• Error about missing client libraries when connecting to a database
• Verify FME Server Engine permissions to access data and other files on the system

Error similar to "This FME edition does not recognize the factory."

This error will most commonly occur if you created a workspace using a newer version of FME Desktop than your FME Server version and the workspace uses a transformer or reader/writer that was added or was significantly updated in the newer FME version.

• Workspace was created with a newer version of FME Desktop than FME Server

Reprojection error when converting between NAD27 and NAD83 datums

You may encounter an error like this one: "Could not reproject the geometry of the preceeding feature: Reprojector: Could not create reprojection engine"

• Could not create reprojection engine error

Error running workspace that contains a Python script

• No module named arcpy
• Could not import external python module

For more information about working with Python on FME Server,see:Using Python with FME Server

Job fails due to missing client libraries

This can happen if the client libraries required for connecting to certain data formats with FME have either not been installed on the system where the FME Server Engines are installed or if the wrong bit version was installed on the system (i.e.32-bit client libraries on a system with 64-bit FME Engines).

• Workspace fails due to missing client libraries

Issue Running Job Under One of the Services

Problems Running a Job Through the Data Download Service

Your job returns a success message,but the link to download your results either isn't returned at all or throws an error when you try to download your data.

• Job Runs,but no zip file is created
• Zip file is empty no matter how much data is returned
• Zip file is empty when job returns a large amount of data (more than 10GB)
• Data Download link returns 404 error message
• Errors running job through Data Download service in a distributed install of FME Server using a mix of Linux and Windows machines

Problems Running a Job Through the Data Streaming Service

• The Data Streaming service unexpectedly returns a zip file
• Returns error saying no output data was created

Are you still experiencing issues?

Please consider posting to theFME 亚搏国际在线官网Community Q&Aif you are still experiencing issues that are not addressed in this article.也有different support channelsavailable.

Have ideas on how to improve this?

You can add ideas or product suggestions to ourIdeas Exchange.

I was chatting with a customer and they said they'd love to be able to let their users run scripts that access data and directories they are already entitled to using the Active Directory account.

Currently,the jobs all run as the single account specified as the Log On user for the FME Engine windows service).That account needs permission to access ANY data that might be processed by ANY job on FME Server.

I'd prefer if the jobs ran as the user that creates them.

We have data intense rasterization workspace (which we can share via dropbox) using Mapnik.The workspace is run with Red hat 7 / FME 2017.1.1.1.The server has sufficient memory and cpu capabilities and it is working fine with smaller data sets.

The desktop is also FME 2017.1.1.1 version and the fmeserver user has sufficient rights (root + https://knowledge.亚搏在线safe.com/articles/718/fme-server-troubleshooting-guide-testing-permissio.html ).

Checked the logs and it reports "No message length returned in FME Engine reponse" and restarts.

Can anyone help?

Are you encountering issues submitting jobs with the FMEServerJobSubmitter.Please read below for some common troubleshooting tips,questions and resources.

Troubleshooting Tips

• Confirm you can access the FME server from your machine by testing the URL,password and username from a web browser.
• Confirm you have engines licensed and available.
• Can you run one of the sample workspaces?If so,the problem may be with the helper (worker/child) workspace or the parameters you are passing in.
• Check the job log files \engine\current|old\jobs\ \job_ .log

Common Issues

"FME Server hangs when I start the FMEServerJobSubmitter from a workspace uploaded on the server"

• In FME 2017.1+ the helper workspace will automatically run a subprocess to initiate the helper workspace if you have only one engine available,however,you can still have the job hang if you have parallel processing = No and wait to complete = Yes.Older versions of FME will always hang when there is only one engine available.
• If you have job queues specified in the FMEServerJobSubmitter,you will need to remove this configuration as it will cause a conflict.

"FME Server submits a second job when ‘Wait to Complete' is set to ‘Yes'"

• This can happen if the FME Server URL Connection Parameter that is specified in the FMEServerJobSubmitter transformer does not match the FME Server Core host name.Please seethis articlefor more information.

"I can't submit more than 1 helper process to run at a time when I have "Submit Jobs" set to In Parallel"

• You can only have one helper workspace run on an engine at a time.In parallel,each helper workspace will be submitted at the same time,but running still depends on available engines.Please seethis articlefor more information.

"What are the differences between 2016 and 2017.1.1"

• Below is a chart which should help with the differences.The big thing to note is a new parameter called "Submit Jobs"

Common Questions from FME 亚搏国际在线官网Community Q&A

1. Error: output_datasets{0}.path' cannot be located
2. Why does my workspace on FME Server hang when using the FMEServerJobSubmitter
3. How to use FME Jobsubmitter with one token without username and password
4. FME Server Job Submitter hanging
5. Where can I find my job logs
6. I've upgraded FME Server to 2018 and my FMEServerJobSubmitter Job priority settings are being ignored
7. Is it possible to pass attributes or variable values from child workspace back to parent?

Known issues with the FME Server Job Submitter

1. FME Server unable to process user parameters in the FMEServerJobSubmitter
2. My child workspace is reported as Job #0 and fails to run,why?
3. In FME Desktop 2017 I upgraded the FMEServerJobSubmitter to Version 3 and it isn't accepting attributes as parameter values
   
   

Popular Ideas by the FME 亚搏国际在线官网Community

1. FME Server Job Resubmitter
2. Create a Replace Workspace Runner with FMEServerJobSubmitter tool

Other Resources

1. FME Server 亚搏在线Workflow Management
2. FMEServerJobSubmitter: Using Multiple Engines

Are you still experiencing issues?

Please consider posting to theFME 亚搏国际在线官网Community Q&Aif you are still experiencing issues that are not addressed in this article.也有different support channelsavailable.

Have ideas on how to improve this?

You can add ideas or product suggestions to ourIdeas Exchange.

Is there a way to export a list of scheduled jobs from FME Server to a txt file or some other format?

Thanks in advance.

Tried to cancel jobs,getting: Error: java.sql.SQLException: COM.亚搏在线safe.fmeserver.database.FMEServerDBException: org.postgresql.util.PSQLException: Connection refused.Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

What to do?

The jobs list is empty (running,queued and completed).When a job has finished running (it is possible to see it in the UI) it sits for a few minutes in the interface and then gets automatically deleted.I disabled all cleanups logs for jobs.It seems to be only something related to the interface because if I access C:\ProgramData\亚搏在线SafeSoftware\FME Server\resources\logs\engine\current\jobs the actual logs are there but they are simply not showing on the web interface under the "Manage -> Jobs" tab.

Has anyone already experienced this issue?

Related to the FME Server 2017 "running job expiry time" (tm_ttc) parameter that cancels a job if it runs beyond a specified length of time:

https://knowledge.亚搏在线safe.com/idea/18984/configure-time-limits-for-a-job-and-alertcancel-if.html

-------

I'd like a time limit per workspace.

Related to the FME Server 2017 "running job expiry time" (tm_ttc) parameter that cancels a job if it runs beyond a specified length of time:

https://knowledge.亚搏在线safe.com/idea/18984/configure-time-limits-for-a-job-and-alertcancel-if.html

-----

I'd very much like to have the ability to set a global time limit.

Would the same/similar functionality be available as system wide defaults (e.g.max threshold of like 12 hours).