Is there a way to make FME Server move emails to a certain folder or even delete them once they have been read by the IMAP publisher?

There is no such option in the publisher settings.Is there maybe a trick to do the same using a custom transformer?

Hello,

I have several publications in FME Server that are monitoring for incoming emails.The publications are subscribed to different topics.I am making a master log (in SQL) of all incoming emails for all the topics.

When I start the monitor for a topic in FME Server,I get this sample JSON notification output:

{ "email_publisher_subject": "[this is the subject]","email_publisher_sent": "Fri Jun 30 14:41:19 EDT 2017","email_publisher_content_type{1}": "text/html;charset=\"iso-8859-1\"","email_publisher_from": "testemail@gmail.ca","email_publisher_content_type{0}": "text/plain;charset=\"iso-8859-1\"","ws_topic": "Document_Incoming (EMAIL)","email_publisher_received": "Fri Jun 30 14:41:32 EDT 2017","email_publisher_to": "destination_email@gmail.ca","email_publisher_content{0}": "[email content goes here]","fns_type": "email_publisher" }

I want to capture "ws_topic" as an attribute.If I save this notification to a text of JSON file and run it via a FME script by using a JSON reader,the ws_topic is saved in the database with the value "Document_Incoming (EMAIL)".But when I upload the same script to FME Server and map the source to the Notification Message,ws_topic comes up NULL.All other parameters are logging fine.

I don't understand what I am doing wrong here.Any help will be appreciated.

Are you encountering issues with the Notification Service in FME Server?Please read below for some common troubleshooting tips,questions and resources.

Troubleshooting

Topic Monitoring

• Have you pressed the ‘Start Monitoring' button?
• If you test the topic,does a notification arrive?
• Does topic monitoring work on the same server that FME Server is installed on?
  • Test on the server,和一个远程机器。
• Is port 7078 open (default port for WebSocket communication,which Topic Monitoring uses)?
• Do you see any errors in the Developer Information window (In Google Chrome right click > Inspect > Network > refresh page)?

• Have you checked the log files?
  • fmenotification.log (Logs > Service > current)
• Even though topic monitoring is not working,are the correct notifications being triggered?
• Have you configured for SSL?
  • If yes,did this work beforehand?Have you correctly configured thepropertiesFile.properties?

Publications or Subscriptions

• Are the rightportsopen for the right protocol?
• If using Email as a protocol,has your email account been configured to allow other applications to use it?
• Check the log file for the Publication/Subscription being used (Logs > core > current >)
• Are you seeing expected topic messages and content in the Topic Monitoring window?

Existing Questions in the FME 亚搏国际在线官网Community Q&A

1. Distributed engine environment causes data to be written to the wrong location
2. Topic Monitoring doesn't connect
3. Topic Monitoring doesn't work in Mozilla Firefox
4. Topic Monitoring isn't working after configuring for HTTPS/SSL

For questions relating to specific publisher or subscriber protocols,I would suggest searching or posting on ourFME 亚搏国际在线官网Community Q&A

Other Resources

Notifications documentation

FME Server Authoring Course

Ultimate Real-Time: Monitor Anything,Update Anything Webinar

Integrating the Enterprise with Event-Driven Messaging Webinar

Are you still experiencing issues?

Please consider posting to theFME 亚搏国际在线官网Community Q&Aif you are still experiencing issues that are not addressed in this article.There are alsodifferent support channelsavailable.

Have ideas on how to improve this?

You can add ideas or product suggestions to ourIdeas Exchange.

We can configure the S3Watch publisher to poll every x minutes/hours/etc.,but we cannot specify at what time of day the poll occurs.It'd be great to set up the publisher to poll at specific times e.g.noon and midnight.

Ability to disable a publisher.

This would be nice if you want to disable it on a test server when you move the project over to a production server and don't need the test one to keep running,but only to be there for future testing.

As announced by Google on December 11,2018, Google Fusion Tables and the Fusion Tables API will be turned off on December 3,2019.Some users may start to see errors in August 2019.Fusion Tables support will be removed from FME after the service has been shut down.For ways to migrate your data out of Google Fusion Tables using FME,see the blog post 7+ Google Fusion Tables Alternatives.

This article is specific to FME Server 2017+.For earlier versions of this article please refer to:Tutorial: Directory Watch

Introduction

Note: This tutorial requires aGoogleaccount and Microsoft Excel.

Using a standard Directory Watch configuration,FME Server will trigger a notification every time a watched file (or folder) is modified.This can create an overload of processing on the server.This tutorial presents a method to optimally handle these migrations using an idle time delay.

In the following example,an excel spreadsheet is being stored in a watched folder.When the spreadsheet is modified,a notification is generated by FME Server and the contents (i.e.filename,timestamp) are stored in a database table.The table is checked by a workspace running on a schedule and when a desired amount of time has elapsed for each modified file,the data is automatically backed up to a Google Fusion Table in the cloud.

Downloads

directory-watch-idle-delay.zip

Part 1: Watch a File Directory

1.Create the directory to watch

Navigate to the file system on the machine where FME Server is installed,or your own machine provided that FME Server is located on the same network.Create a new folder called ‘Resource' and copy the Project_Inventory.xlsx file from the attached download into the new folder.Right click anywhere in the Resource folder and click Properties.In the Sharing tab click,Advanced Sharing then enable Share This Folder,ensure the permissions are set to write,then click ok.Copy the UNC Network Path (e.g.\\KC-LS-WINDOWS\Resource),we will need this in the next step and close the Properties window.Additionally,depending on your sharing settings within your computer,you might have to click the Share...button on the Resource Properties dialog and click Share.

Create a shared folder containing Project Inventory.xlsx,ensure permissions are set to full

2.Create the Directory Watch publication

Log into your FME Server and on the side menu bar under Notifications,select Publications.Click New,and give your publication the name 'sync_tool' and add a new topic to publish to named 'sync_tool'.Select Directory Watch protocol and for Directory to Watch choose 'Specify a location',and paste the UNC Network Path we copied in the last step.Monitor MODIFY operations (remove CREATE and DELETE).Finally,set the Poll Interval to 1 Minute,you can leave it as the default,but for testing purposes a lower interval will be quicker to debug.Click OK to create the publication.

Create the new Directory Watch publication in FME Server called sync_tool,set it to watch your Resource directory.

3.Setup the Data Migration Workspace

Open theproject_inventory_to_map.fmw (Workspace #1)in FME Workbench.This workspace writes Excel data to a Google Fusion table stored in the cloud.For more information on Google Fusion Tables refer to theuser documentation.

In the User Parameters in the Navigator pane,under Published Parameters,double click on the [SourceDataset_XLSXR],set the path to "\\YourUNCPath\Resource\Project Inventory.xlsx",make sure you include the quotations.

Double-click on the SQLExecutor to modify the parameters.We will need to login to our personal Google account and authorize FME.To do this,select Parameters in the SQLExecutor,in the Google Fusion Table Spatial Parameters dialog box,click on the (...).In the OAuth authentication window,log in using your own Google account credentials to manage Fusion Tables and click Allow.You now have a Refresh Token in the Google Fusion Tables Spatial Parameters dialog,copy this token to avoid re-authentication in later steps.Click OK to exit the SQLExecuter parameters

Double-click on the [GoogleToken] User Parameter and paste in the Fusion Table Refresh Token to connect to your own account.

project_inventory_to_map.fmw (workspace #1) 亚搏在线workflow

Publish to FME Server,create a new repository called Project_Inventory,then register with the Job Submitter Service.(Note: Ensure that you uncheck Upload Data Files so that the Excel file is not uploaded to the Server)

4.Setup the Directory Watch Receiver Workspace

In FME Workbench openproject_inventory_status.fmw (Workspace #2),this workspace runs when a Modify trigger is sent by the Directory Watch protocol.Details from the notification message (i.e.filename,timestamp) are stored in a database,which in this case will be a Google Fusion Table.

This workspace was set up with a User Parameter called GoogleToken to speed up the process of authentication in the SQLExecutor and the Writer,it is the same process as the last step.Click on the GoogleToken Published Parameter and paste your Refresh Token.

Publish the workspace FME Server,in the Project_Inventory repository,and register the workspace using the Notification Service.Click Edit and choose sync_tool for Subscribe to Topic.(Note: You may receive a warning message - this workspace uses custom formats,which will need to be provided to FME Server - click yes to accept.)

project_inventory_status.fmw (workspace #2) 亚搏在线workflow

In order for the workspace to run with a custom transformer on FME Server,we will need to upload the custom transformer.In FME Server,go to Resources and navigate to the Formats folder within the Engine folder.Click Upload and browse to the DIRECTORY_WATCH_READER.fds,which is located in the folder that was downloaded at the beginning of this tutorial.

Upload the DIRECTORY_WATCH_READER.fds to the Formats folder in FME Server

5.Setup the Directory Watch Receiver Workspace

Back in FME Workbench openproject_inventory_sync.fmw (工作空间# 3),this workspace is used to check the database(Google Fusion Table) for modified files and send them to be processed when necessary.

In the User Parameters,double click on [GoogleToken] and paste your refresh token.In the FMEServerJobSubmitter connect to your FME Server.Then select the Project_Inventory repository and select the project_inventory_to_map.fmw workspace,click the refresh icon in the lower left hand corner,to bring in the parameters from project_inventory_to_map.fmw.This will enable theproject_inventory_to_map.fmw(Workspace #1)to run onceproject_inventory_sync.fmw (工作空间# 3)has run.

Publish to FME Server in the same repository and register with the Job Submitter.

project_inventory_sync.fmw (workspace #3) 亚搏在线workflow

6.Create the Fusion Tables

In FME Workbench openDB_Loader.fmw (工作空间# 4),this workspace creates two tables in your Google Fusion Database.

In the Navigator pane under User Parameters,double click on [SourceDataset_XLSXR] and browse to the shared Resource folder and add the Project Inventory.xlsx file

Next,double click on [SourceDataset_CSV] and browse to the CSV file called project_inventory_file_status.csv that came with the file download.

Then update the [GoogleToken] just as before,then run the workspace.After running your translation,check in your Google Drive account to ensure the workspace ran correctly,you should see two files,Project Inventory and project_inventory_file_status.

DB_loader.fmw (workspace #4) 亚搏在线workflow.Creates two Google Fusion Tables

7.Setup a Schedule

The last step is to setup a schedule to check our tracking database.This can be set to run every few minutes.This schedule will runproject_inventory_sync.fmw (工作空间# 3),which checks the tracking database for the last modified time.If it is greater than the idle time set within the workspace,the data is migrated.

In FME Server,go to the Schedules page and click New to create a new schedule named project_inventory_sync and create a new Category called Project_Inventory.Check Run Immediately,then set it to Repeat Every 2 Minutes or however often you would like it to repeat.

Then set the Repository to Project_Inventory and select theproject_inventory_sync.fmw(工作空间# 3)and click OK to set up the schedule.A Published Parameters section will be exposed,you can adjust the Seconds to Wait if you like,this is the desired amount of time to wait after the Excel file has been updated and is idle.Once this condition is met,the data will be synced to the Google fusion table.Also,ensure that your Refresh Token is correct.Click OK to start the schedule.

Part 2: Verify the 亚搏在线Workflow

In FME Server,click on Notifications,then click the Topic Monitoring Tab.For Topics to Monitor,add sync_tool.This will send a notification each time the file is modified.Ensure this page is left open for Topic Monitoring to continue.

At this point,all the necessary steps are in place to have an optimized backup system for your spreadsheet data.To test this try the following steps:

Open Google Drive and open the two Fusion Table files you created,Project Inventory and project_inventory_file_status.To do this,right click on the file,then Open With > Google Fusion Tables.

Open the Excel document in Excel (project_inventory.xlsx) and enter a new row:

100007 repair  manhole replaced 49.2755567 -123.0576200

Save the excel file,ensure it is saved to the shared Resource folder and the file extension is .xlsx.This will trigger the Directory Watch protocol to post a Modify to the sync_tool topic.If you open the Jobs up in a new FME Server window,the project_inventory_status.fmw workspace will have run,then depending on your schedule,the project_inventory_sync.fmw will run,depending on how the schedule lined up to your update,you might have to wait for the schedule to run twice in order for the Google Fusion Table to update.Wait up to five minutes for this process to run through depending on how your schedule was set up and the Seconds to Wait parameter was set to.After the allotted time has passed refresh both Google Fusion Tables to see the updates.

Topic Monitoring in FME Server when the Project Inventory.xlsx file is modified in Excel

Conclusion

Using Idle Time Delay is useful for keeping a backup of your documents while you are updating them.The workspace only runs when it detects a change after a certain period of time in your local document,this frees up valuable server engines,only running the workspace when required instead of on a schedule.Creating a separate status table is also useful for ensuring that the updates to the local file were expected,especially when the file is shared among several users.

Additional Resources

• Google Fusion Tables

Tutorial Overview: Directory Watch| Previous: Directory Watch Publisher | Current: Directory Watch Publisher with Idle Time Delay (Advanced)

This article is specific to FME Server 2017.For earlier versions of this article please refer to:Tutorial: Directory Watch

Introduction

The Directory Watch publisher allows you to have FME Server watch a directory,and when a new file has been moved into the directory,a notification message containing the file path will be sent to a topic.From here a workspace can use the file in a translation.

In this example,we will be creating two directories in the Temp Shared Resource folder called "input" and "output".We will set-up FME Server to watch the "input" directory for new files,and when a new file is uploaded,a parent workspace will run.The parent workspace will pass the file to a child workspace which will convert it to a different format.The output from this translation will then be copied to the "output" directory,and will (optionally) send an email with a download link to a list of people.

Downloads

directorywatchpublisher2017.zip

Part 1: Watching a directory

1.Create the directory to watch

Open the FME Server Web Interface and log in.On the side menu bar click on Resources,then double-click on the Temp resource to open it.Click on the New Folder button and call the new folderconvert_tool.Within the convert_tool folder and create two new folders namedinputandoutput

convert_tool directory in FME Server with input and output folders

2.Create the Directory Watch publication

From the side menu bar,go to Notifications and select the Publications tab.Click New to create a new Publication with the nameConvert_Tool(title case).Select Directory Watch as the Protocol and then for Topics To Publish To create a new topic calledconvert_tool(all lowercase)

For the Directory to Watch,click on the [...],then select Temp > convert_tool > Input

For Filter,only monitor CREATE operations and change the Poll Interval to one minute.Click OK to create the publication.

Publication Convert Tool parameters in FME Server

3.Verify the directory watch publication

The Topic Monitoring page allows you to view the messages being published to a topic.Please note that you will need a browser which supports WebSockets in order to make use of this functionality.Check if your browser supports WebSockets.

Back in the Notifications,click on the Topic Monitoring tab.Click the Topics to Monitor drop-down menu,and select theconvert_tooltopic,once you've selected the topic,it will automatically start monitoring as long as this page is still open.

Using a different web browser tab,go to the Resources page,and upload two files to the input directory you created earlier (Temp > convert_tool > input).After a minute,on the topic monitoring page,you will see two messages,one for each file uploaded

Topic Monitoring convert_tool in FME Server

Part 2: Working with the incoming files

A common pattern when working with notifications that provide attachments (e.g.email,directory watch) is to use a FeatureReader transformer to read the data and then do something with it.Another pattern that fits well with the directory watch publication is to use a FMEServerJobSubmitter transformer to call a child workspace to convert the data;and when that workspace completes the parent workspace can copy the data to a directory,email a link to the data or more.

For this example,we will make use of the workspaces that are included within the zip file in the downloads section;namely,'easyTranslator2.fmw',and 'convert.fmw'.The 'easyTranslator2.fmw' workspace,(the child workspace),will be used to convert DWG files into SHAPE format and then create a zip for the newly created files.The 'convert.fmw' workspace,(the parent workspace),will then copy the zip file to a FME Server Resource directory for the user to access using a file copy writer.

Note: easyTranslator2.fmw differs from the original easyTranslator.fmw,which is provided with FME Server in the Samples repository.

1.Publish the child workspace

Open the easyTranslator2.fmw workspace and publish the workspace to FME Server in the Samples repository as a Job Submitter Service.

2.Publish the parent workspace

Open the convert.fmw workspace,and then open the properties of the FMEServerJobSubmitter transformer.Select your connection to FMEServer or Embed Connection Parameters.For Repository select Samples,and for Workspace select easyTranslator2.fmw.

Update the Job Parameters to match the image below:

FMEServerJobSubmitter parameters in convert.fmw in FME Workbench

Click OK to save the parameters.

Publish the workspace to FME Server,enter the Connection Parameters,click Next.Publish the Workspace to the Samples Repository,click Next.For Upload Connections,check your FME Server connection and click next.At the Register Services step,unselect Job Submitter and select the Notification Service.Configure the Notification service by clicking the "Edit..." button next to it:

• Set the Subscribed To Topic toconvert_tool
• Set the Parameter to Get Topic Message toSource Text File(s)

Publish convert.fmw to FME Server using the Notification Service and Convert_Tool Topic in FME Workbench

You may or may not see a warning box appear - New notification topics will be created - click yes to accept and continue

3.Use the convert tool

Back in FME Server go to the Resources page,and navigate to Temp > convert_tool > input.Upload one of the provided DWG files to the input directory.Wait for a minute or so and then confirm that the file is now in the Temp > convert_tool > output folder.You should see a zip file - download it and you will see a shapefile inside.

Note: If you don't see the zip file,check to see if the job was run and what the status was.Go to Jobs and click on the most recent convert.fmw job to read the job log.

Converted file in Output after uploading a file to Input in FME Server

Part 3: Sending a link to download the result (Optional)

You can go a step further by offering a link to the zip file in the FME Server Resource directory and sending that link via email.

1.Create the topic

Create a new topic by going to the Topics tab on the Notifications page.Name the topicemail_results

2.Create the email subscription

From the Subscriptions tab,click new to create a new email subscription.

• Name the subscriptionemail_results.
• Select the email_results topic for Topics Subscribed To
• Select the Email protocol
  • Fill in required SMTP information
  • Set Email Format to HTML
  • Click OK

Email subscription in FME Server,set up for use with gmail accounts.

Please see theSending Email Tutorialfor more information about setting up an email subscription.

Also,if you use a gmail account,you will likely need tocreate an app-specific password for FME Serverorallow less secure appsto access your Gmail account as outline inthis article.

3.Get an access token

In order to configure a workspace to send an email,you must first generate a valid FME Server token.In the top right-hand corner of FME Server click the drop down menu attached to your user name and select Manage Token.Copy the string of numbers and letters listed Current Token.

FME Server token from Manage Token menu

4.Re-publish the parent workspace

Back in FME Workbench in the convert.fmw workspace.Enable the objects in the Emailer Bookmark by right clicking on the bookmark and selecting Enable All Objects in Bookmark.

In the navigator panel,under User Parameters > Published Parameters,click on theHostNameand enter the hostname for your FME Server,and then click onTokento enter your FME Server Token.

Then click onEmailand enter the email you wish to receive notification updates to,every time a file is uploaded into the input folder.

Published Parameters to edit in convert.fmw in FME Workbench

Publish to FME Server,at the Register Services screen,click "Edit..." next to Notification service.Expand Notify on Job Completion,set Post to Topics on Success: toemail_resultsand set Post Data From: toDestination Text File.Finish publishing

Publish convert.fmw to FME Server using the Notification Service and email_results Topic in FME Workbench

5.Use the convert tool

Open the FME Server Web Interface > Resources > Temp > convert_tool > input.Upload one of the provided DWG files input directory.You should receive an email with a link to download the zip file.It takes a couple minutes for the email to be sent,if you haven't received an email after 5 minutes,double check the job logs as well as the email parameters.

Successful conversion emailed results

Conclusion

Using Directory Watch to convert files can be an efficient way to change file formats to a useful format for your organization if you regularly get different formats from outside sources.This way all the files will be converted in the same way and the original and new files will be contained in one spot.For receiving email updates,this can be useful to monitor file uploads,as well as download the files when do not have access to FME Server.

Data Attribution

The data used here originates from open data made available by theCity of Vancouver,British Columbia.It contains information licensed under the Open Government License - Vancouver.

Tutorial Overview: Directory Watch| Current: Directory Watch Publisher |Next: Directory Watch Publisher with Idle Time Delay (Advanced)

Introduction

The FME Server Notification Service has the ability to monitor file activity within a data storage resource.When a file has been created,modified or deleted,the server can trigger a notification that will publish a message to an FME Server Topic about the activity.This message will contain information about the file including the file path.From here,a workspace can take the file,translate/transform it,and send the results back to the user.

Currently,this includes the following types of storage:

• Directories (UNC path or local)
• Amazon S3
• FTP

FME Server 2017 Tutorials

• Directory Watch Publisher Tutorial: A step-by-step guide to setting up the Directory Watch publisher on FME Server.Specifically,this makes use of the ‘Create' filter in the Directory Watch settings.

• Directory Watch Publisher with Idle Time Delay: Advanced Tutorial: A step-by-step guide to using the ‘Modify' filter in order to keep data synced to a database in the cloud.

Earlier Version Tutorials

• Directory Watch Tutorial (2016)
• Directory Watch Tutorial (2015)
• Directory Watch Tutorial (2014)

Additional Resources

• Advanced Data Management (Directory Watch Publisher)
• FME Server Real-Time Data Processing
• FME Server Publishers Documentation
• FME Server Directory Watch Documentation
• Directory Watch Custom Reader

It would be very useful to have notifications that are able to be scheduled (i.e.they will only run for certain times of the day).

For example some of the functionality this would allow for:

Being able to set a email publisher to only watch an email outside of work hours,rather than the whole day

Running websockets as a notification process when engine demand may be higher for specific processes,but then allowing for websocketreceiver process to be always on when you know that engine demand won't be high.

This is an additional requirement to being able to enable/disable a notification

https://knowledge.亚搏在线safe.com/idea/38651/disable-a-notification.html

Hi all,

I am trying to set up an imap publisher,but everytime I try to save it it comes up with the error 'Could not find publisher'.The same error applies when I try to validate the publisher.All required fields have been filled in on the publisher.

I don't know what the error means as I can't find any reference to it in any documentation.

It looks like all my publishers aren't working as the directory watch publisher doesn't seem to pick anything up either.

Any ideas?

Thanks

When I publish a workspace to FME Server and set the job success and job failure topics for the service I don't see the resulting publication listed in the notifications page (publications) of the web UI.It would be good to list these as workspace publications when this happens.

This would also be great when a workspace is published and the Notification Service is set and topics are selected.This should also place an entry in the publications list.

Intro: FME Server Directory Watch Publisher|Previous Section: Directory Watch Publisher With Idle Time Delay

Introduction

There are special cases to be aware of when watching files and folders using FME Server.The following page describes these in detail and suggests some helpful tips.

Downloads

scheduled-watch-2016.zip

Using ‘Create' Triggers | Multi-File/Folder Datasets

The complexity of multi-file datasets (e.g.Shapefiles,Geodatabases) presents challenges when using the Directory Watcher.In these cases,you will want to ensure that all necessary files have been uploaded before processing the data.The following discusses the various approaches:

Zip Files

The easiest solution is to package your data as a.zip file.Since FME can read .zip directory for many formats,it makes sense to provide a packaged file into a watched location instead of multiple files.

Decelerator Transformer

If .zip is not an option or you are editing existing data in a watched resource,it makes sense to buffer the amount of time before processing.This can ensure that the data is complete before being utilized.A simple method for this process is to include aDeceleratortransformer at the start of the workspace.The "Create" trigger will occur,running the workspace which will then sit for an allotted amount of time before processing.There is a downside to this approach as the job will tie up an FME Engine while it waits.This method is best suited for smaller size multi-file datasets.

Creating a Schedule

An even better approach to watching multi-file datasets is to setup a scheduled job on FME Server.To do this you would capture the Modify trigger from a multi-file dataset,filter for the desired extension (i.e.‘.shp'),buffer the current time and then use the REST API to create a one-time schedule to process the workspace.The benefit of this approach is that this 亚搏在线workflow does not tie up an engine while waiting for the files to finish uploading.

The attached workspace (Scheduled_Watch.fmw) demonstrates this process.

Using ‘Create' Triggers | Processing Multiple Single File Datasets at Once

When working with single file datasets it is ideal to process many of them using one workspace/engine.For example,What if an image translation service is setup using FME Server and and the FTP Watch publication.When a GeoTIFF file is uploaded to the FTP directory,it gets reprojected and translated into a JPEG.Using this scenario,if 5 images are uploaded to the directory,then 5 separate jobs are created on the server.This can be made smarter with a little bit of intervention.

In theDirectory Watch with Idle Time Delay: Advanced Tutorialthere is a database used to store information from aModifynotification (i.e.filename).After a certain amount of time,a second workspace running by a scheduled task analyzes the database and processes appropriate files.We can use the same technique to process multiple single file datasets at one time using a single workspace.

To create this 亚搏在线workflow you must set up two workspaces,the first stores theCreatenotification details along with a timestamp into a database while the second workspace is designed to analyze the database and process the data.This workspace should decide which files have been stored for an appropriate amount of time,read them into the 亚搏在线workflow using a FeatureReader transformer and then process the entire set of files (i.e.reproject and translate).

Using ‘Modify' Triggers | Large Files

A large file may cause the watch protocol to report multipleModifytriggers when the file is changed.This is due to the length of time it takes to save changes to the file when it is stored on a local drive.There is a special parameter within the Directory Watch publisher calledMin Time Elapsed (Seconds).This setting will impedeModifytriggers from being reported for a file until a minimum amount of time has been reached (Note: this setting is not necessary when the directory is using a UNC path as the Modify trigger will occur after the file has completed saving).

As announced by Google on December 11,2018, Google Fusion Tables and the Fusion Tables API will be turned off on December 3,2019.Some users may start to see errors in August 2019.Fusion Tables support will be removed from FME after the service has been shut down.For ways to migrate your data out of Google Fusion Tables using FME,see the blog post 7+ Google Fusion Tables Alternatives.

Note: This article is specific to FME Server 2016,for an FME Server 2017+ version please refer toDirectory Watch Publisher with Idle Time Delay (Advanced) (2017)

Intro: FME Server Directory Watch Publisher|Previous Section: Directory Watch Publisher Tutorial|Next Section: Advanced Data Management (Directory Watch Publisher)

Introduction

Using a standard Directory Watch configuration,FME Server will trigger a notification every time a watched file (or folder) is modified.This can create an overload of processing on the server.This tutorial presents a method to optimally handle these migrations using an idle time delay.

In the following example,an excel spreadsheet is being stored in a watched folder.When the spreadsheet is modified,a notification is generated by FME Server and the contents (i.e.filename,timestamp) are stored in a database table.The table is checked by a workspace running on a schedule and when a desired amount of time has elapsed for each modified file,the data is automatically backed up to a Google Fusion Table in the cloud.

Downloads

Directory Watch Idle Delay

Requirements

• FME Server 2016
• FME Desktop 2016
• Google account
• Files from the attached 'Directory_Watch_Idle_Delay.zip'

Watch a File Directory

1.Create the directory to watch

• Navigate to the file system on the machine where FME Server is installed,or your own machine provided that FME Server is located on the same network.
• Create a new folder called ‘Resource'.(Note: Make sure the 'Resource' folder is shared,and that the appropriate user permissions have been applied.)
• Copy theProject_Inventory.xlsxfile from the attached download into the new folder.

2.Create the Directory Watch publication

• From the Manage drop-down menu,open Notifications,and select the Publications tab
• Click New,and give your publication the name 'sync_tool'
• Add a new topic to publish to named 'sync_tool'
• Select Directory Watch protocol
• For Directory to Watch,choose 'Specify a location',and enter the UNC path to the 'Resource' folder (e.g.) \\cyril\Shared\Resource
• Monitor MODIFY operations (remove CREATE and DELETE)
• Click OK to create the publication.

1.Setup the Data Migration Workspace

This workspace writes Excel data to a Google Fusion table stored in the cloud.For more information on Google Fusion Tables refer to theuser documentation.

• Open the workspace:project_inventory_to_map.fmw
• Modify the Excel Reader's Source Parameter to read the Excel file (saved to your shared folder).
• Modify the SQLExecuter to use your own Google account credentials to manage Fusion Tables.Open the transformer and select Parameters (…) From the Google Fusion Tables Spatial Parameters select (…),then enter your Google credentials.Allow the transformer to Manage your Fusion Tables.(Note: it is recommended that you copy the Fusion Table Refresh Token so that you do not need to re-authenticate in later steps)
• Right-click on the writer in the navigator and select edit parameters.Paste in the Fusion Table Refresh Token to connect to your own account.
• Publish to FME Server as a job submitter service.(Note: Ensure that you uncheck Upload Data Files so that the Excel file is not uploaded to the Server)

2.Setup the Directory Watch Receiver Workspace

This workspace runs when a Modify trigger is sent by the Directory Watch protocol.Details from the notification message (i.e.filename,timestamp) are stored in a database.

• Open the workspace:project_inventory_status.fmw
• Modify the SQLExecuter to use the Current Fusion Table Refresh Token.
• Modify the Google Fusion Table Refresh Token in the navigator.
• Publish to FME Server.(Note: You may receive a warning message - this workspace uses custom formats,which will need to be provided to FME Server - click yes to accept.)The workspace should be registered to the notification service and the sync_tool topic
• Add Directory Watch Custom Reader to FME Server.Open FME Server WebUI and from the Manage drop-down menu,choose Resources.Upload DIRECTORY_WATCH_READER.fds to the Engine > Formats folder.

3.Setup the Directory Watch Receiver Workspace

This workspace is used to check the database for modified files and send them to be processed when necessary.

• Open the workspace:project_inventory_sync.fmw
• Modify the Google Fusion Table Refresh Tokens in the navigator.
• Edit the FMEServerJobSubmitter to run project_inventory_to_map.fmw which was published in step 1.From the Edit Job Parameters dialog,set the Output Data Location = As Specified in Workspace and Parameter
• Publish to FME Server as a Job Submitter service.

4.Create the Fusion Tables

• Open the workspace template:DB_Loader.fmwt
• Modify the Google Fusion Table Refresh Tokens.
• Run the workspace.

5.Setup a Schedule

The last step is to setup a schedule to check our tracking database.This can be set to run every few minutes.This schedule will run project_inventory_sync.fmw which checks the tracking database for the last modified time.If it is greater than the idle time set within the workspace,the data is migrated.

• On the FME Server Web UI go to Manage drop-down menu,select Schedules,and create a New schedule.
• Create a new schedule named project_inventory_sync.Set its category to Dashboards.Select the Dashboards Repository,and the project_inventory_sync.fmw workspace.
• Edit the Seconds to Wait parameter to your desired length.This is the desired amount of time to wait after the Excel file has been updated and is idle.Once this condition is met,the data will be synced to the Google fusion table.
• Set the schedule to repeat every 1 minute,or to your desired repeat rate.

Verify the 亚搏在线Workflow

At this point all the necessary steps are in place to have a optimized backup system for your spreadsheet data.To test this try the following steps:

• Open the Google Fusion tables interface.(see:Google Fusion Tables).Create a Fusion Table,browse to the Excel file saved to your shared folder,and complete the Fusion Table creation process.
• Open the Excel document (project_inventory.xlsx) and enter a new row:

100007 repairman hole replaced 49.2755567 -123.0576200

Save the file.(This will trigger the Directory Watch protocol to post a Modify to the sync_tool topic.This in turn will run project_inventory_status.fmw and post the update to the Google Fusion tracking table.)

• enter a second new row:

100008 collect soil samples retrieved 49.2750005 -123.0572567

• Save the Excel file.
• Check both Fusion tables.You should see both your edits stored in the tracking database but they should not have been migrated to your Data Migration database yet.

Wait the allotted time specified in the Seconds to Wait parameter used when creating the schedule.Once that time has elapsed,you should see your edits migrated to the Data Migration table and the tracking database updated.

Additional Resources

• Google Fusion Tables

Continue to Part 3: Advanced Data Management (Directory Watch Publisher)

Note: This article is specific to FME Server 2016,for an FME Server 2017+ version please refer toDirectory Watch Publisher Tutorial (2017)

Intro: FME Server Directory Watch Publisher|Next Section: Directory Watch Publisher With Idle Time Delay

This article is specific to FME Server 2016.For earlier versions of this article please refer to:

• Directory Watch Tutorial (2015)
• Directory Watch Tutorial (2014)

Introduction

The Directory Watch publisher allows you to have FME Server watch a directory,and when a new file has been moved into the directory,a notification message containing the file path will be sent to a topic.From here a workspace can use the file in a translation.

In this example,we will be creating two directories in the Temp shared resource: "input" and "output".We will set-up FME Server to watch the "input" directory for new files,and when a new file is uploaded,a parent workspace will run.The parent workspace will pass the file to a child workspace which will convert it to a different format.The output from this translation will then be copied to the "output" directory,and will (optionally) send an email with a download link to a list of people.

Downloads

directory-watch.zip

Requirements

• FME Server 2016
• FME Desktop 2016
• Files from the attacheddirectory_watch.zip
• An email account for outgoing email(optional)

Watching a directory

1.Create the directory to watch

• Open the FME Server Web Interface and log in
• From the Manage drop-down menu,go to Resources
• Double-click on the Temp resource to open it,and then on the New Folder button - this creates a new folder in the Temp Resource
• Call the new folderconvert_tool
• Double-click the convert_tool folder and create two new folders namedinputandoutput

2.Create the目录表publication

• From the Manage drop-down menu,go to Notifications (or Click on the Manage Notifications button),and select the Publications tab
• Click New,and create a new Publication with the name "Convert_Tool".
• For Topics to Publish To: enter a new topic "convert_tool" (all lowercase).Select Click to Add to add the new topic to the available topics list.(Please note that Topics are case sensitive.)
• Under Protocol Settings,select Directory Watch protocol.
• From the Directory to Watch,choose Browse Resources.Navigate to and select the "input" folder you created in step 1;found under Temp > convert_tool > input.
• For Filter,only monitor CREATE operations.
• Click OK to create the publication.

3.Verify the directory watch publication

The Monitoring page allows you to view the messages being published to a topic.Please note that you will need a browser which supports WebSockets in order to make use of this functionality.Check if your browser supports WebSockets.

• Go to the Manage drop-down menu,and select Notifications (or click on the Manage Notifications button).Click on the Topic Monitoring tab
• Click the Topics to Monitor drop-down menu,and select the convert_tool topic
• Click the Start Monitoring button
• Using another web browser tab,go to the Resources page,and upload two files to the input directory you created earlier
• On the monitoring page,you will see two messages,one for each file uploaded

Working with the incoming files

A common pattern when working with notifications that provide attachments (e.g.email,directory watch) is to use a FeatureReader transformer to read the data and then do something with it.Another pattern that fits well with the directory watch publication is to use a FMEServerJobSubmitter transformer to call a child workspace to convert the data;and when that workspace completes the parent workspace can copy the data to a directory,email a link to the data or more.

For this example,we will make use of the workspaces that are included within the zip file in the downloads section;namely,'easyTranslator2.fmw',and 'convert.fmw'.The 'easyTranslator2.fmw' workspace,(the child workspace),will be used to convert DWG files into SHAPE format and then create a zip for the newly created files.The 'convert.fmw' workspace,(the parent workspace),will then copy the zip file to an FME Server Resource directory for the user to access using a file copy writer.

Note:easyTranslator2.fmw differs from the original easyTranslator.fmw,which is provided with FME Server in the Samples repository.

1.Publish the child workspace

• Open the easyTranslator2.fmw workspace
• Publish the workspace to FME Server in the Samples repository as a Job Submitter Service.

2.Publish the parent workspace

• Open the convert.fmw workspace
• Open the properties of the FMEServerJobSubmitter transformer
• Enter your host information
• Select the easyTranslator2.fmw workspace
• Set the parameters as follows:
  • Output Data Location: Temporary Folder
  • Source File: source_path
  • Source Format: source_format
  • Destination Format: output_format
• Click Finish to save the parameters

• Publish the workspace to FME Server
• Enter the Connection Parameters,click Next
• Publish the Workspace,click Next
• At the Register Services step,select the Notification Service
• Configure the Notification service by clicking the "Edit..." button next to it
  • Set the subscribed topic toconvert_tool
  • Set the Notification Data reader tosomefile [TEXTLINE]
• You may see a warning box appear -New notification topics will be created- click yes to accept and continue

3.Use the convert tool

• From the Manage drop-down menu,go to Resources,and navigate toTemp > convert_tool > input
  
• Upload one of the provided DWG files to the input directory
• Refresh the page and look atTemp > convert_tool > output
• You should see a zip file - download it and you will see two shapefiles inside.
• Note: If you don't see the zip file,check to see if the job was run and what the status was.Do this by navigating to the Jobs page;from the Manage drop-down menu,open Jobs.

Sending a link to download the result (Optional)

You can go a step further by offering a link to the zip file in the FME Server Resource directory and sending that link via email.

1.Create the topic

• Create a new topic by going to the Manage drop-down menu,selecting Notifications,and clicking on the Topics tab.
• Name the topicemail_results

2.Create the email subscription

• From the Subscriptions tab,click new to create a new email subscription
• Name the subscriptionemail_results
• Select the email_results topic
• Select the Email protocol
• Fill in required SMTP information
• Set Email Format to HTML
• Click OK

Please see theSending Email tutorialfor more information about this.

3.Get an access token

In order to configure a workspace to send an email you must first generate a valid FME Server token.A token can be easily generated by someone who has the FME Administrator role by following these steps:

• Open the FME token service: From the admin drop-down menu,select Manage Token,and New Token
• Enter the userid/password you want to use to offer file access.
• Copy the token and save it for later.

4.Re-publish the parent workspace

• Open FME Workbench
• Download the convert.fmw workspace for modification.
• In the navigator panel,under User Parameters > Private Parameters
  • Update the hostname with your FME Server hostname
  • Update the token to use the token you just created.
• Open the FMEServerEmailGenerator transformers and set the "To:" parameter to the email address(es) of the user(s) that you would like to notify when file conversion is completed
• Publish to FME Server
• At the Register Services screen,click "Edit..." next to Notification service.
• Click Notify on Job Completion
• Set "Post to Topics on Success:" toemail_results
• Set "Post Data from Writer:" tooutfile [TEXTLINE]
• Finish publishing

5.Use the convert tool

• Open the FME Server Web Interface > Manage > Resources > Temp > convert_tool > input
• Upload one of the provided DWG files input directory
• You should receive an email with a link to download the zip file

Continue to Part 2: Directory Watch Publisher With Idle Time Delay