help with StartProcessUsingManifest

[Tom Willis]

I'm having trouble with the StartProcessUsingManifest method through both the webservice and the Api dll.

And honestly I can not grasp from the documentation what the name attribute is supposed to refer to or how to override whatever was put in the xml project as the input.

When the report file exists that was used to setup the project and the job everything works fine through the Api dll, I have yet to get it to work through the webservice.In other words whatever I pass in as uris is ignored(probably cuz I have the name wrong).

Here's the xml project
<project version="XMLPRJ02">
<author>Tom Willis</author>
<description>
</description>
<model>
<location>F:datapump_intmodels88patdemo.mod</location>
</model>
<workflow>
<prerequisites />
</workflow>
<inputs process="individually" must_exist="all" order="xprj">
<main>
<report>
<location>F:datapump_intimport88*patdemo*.txt</location>
</report>
</main>
<distribution>
<action type="move">
<targetfolder>F:datapump_intdone88</targetfolder>
<targetname>&[yyyy]&[mm]&[dd]&[hh]&[mi]&[ss]&[source.name]&[jobid]&[counter].txt</targetname>
<overwrite>False</overwrite>
</action>
</distribution>
</inputs>
<state>
<view>
<window>table</window>
</view>
</state>
<exports>
<export>
<data_page view="table">
<name>88_resultscsv</name>
<table />
</data_page>
<file_page>
<files_in_folder>false</files_in_folder>
<location>F:datapump_intexport88patdemo.txt</location>
<when_exists>add</when_exists>
</file_page>
<distribution>
</distribution>
</export>
</exports>
</project>Here's the manifest

<manifest process="88_patdemo">
<inputs>
<item name="88_patdemo">
<uri>
file://F:/datapump_int/pre/88/bpatdemo.txt
</uri>
</item>
</inputs>
</manifest>job log when run through the Api dll

<joblog process="88_patdemo" jobID="1011">
<events>
<event time="2005-04-04T16:33:15">Created.</event>
<event time="2005-04-04T16:33:15">Process prerequisites were not met: Wildcard specification 'F:datapump_intimport88*patdemo*.txt' did not match any files.</event>
<event time="2005-04-04T16:33:16" alert="SystemAlertProcessFailed">Failed: DwchServer.NoRetryInfoException: Process prerequisites failed, but no retry information is available. Process will not be retried. ---> DwchServer.WildcardDidNotExpandPrerequisiteExcepti on: Wildcard specification 'F:datapump_intimport88*patdemo*.txt' did not match any files. at DwchServer.ProjectInputs.TestPrerequisites() at DwchServer.ProcessHandler.b(ServerDB A_0, JobRow A_1, JobLog A_2) --- End of inner exception stack trace --- at DwchServer.ProcessHandler.b(ServerDB A_0, JobRow A_1, JobLog A_2)</event>
<event time="2005-04-04T16:33:16" source="distributor">Distribution started</event>
<event time="2005-04-04T16:33:16" source="distributor" value="complete">Distribution completed</event>
</events>
<distribution />
</joblog>Through the webservice... I get a stack trace.


InboundLib.test.TestInbound.submitdataforextractio n : System.Web.Services.Protocols.SoapException : System.Web.Services.Protocols.SoapException: Server was unable to process request. ---> System.IO.FileLoadException: Access is denied: 'DwchServer.PumpAPI'.
File name: "DwchServer.PumpAPI"
at DwchServer.DwchServerWebService.StartProcessUsingM anifest(String strXml)

...

[Gareth Horton]

Tom

Using a manifest is really designed when you need to specify one or more files and any sort of wildcard spec does not really cut it, as the filenames are so variable, you only know them when you have them, so to speak.

In order to use a manifest, you need to set placeholders for them in the input files section of your XPRJ.

For example, instead of having a wildcard spec such as c:inbox*var*.txt you put in manifest.input1 for example.

You can put as many of these symbolic references in as you like, as many as you will need when you invoke the process.

The second part, the part that got you confused when you were creating the xml file for the manifest is the item name.

This refers to the symbolic name that you used in the project, so if your input was defined as manifest.input1, then your item name in the manifest is "item1"

I will check through that this exists in the documentation and if not, we will add it/make it more clear in the future.


Gareth

Originally posted by Tom Willis:
I'm having trouble with the StartProcessUsingManifest method through both the webservice and the Api dll.

And honestly I can not grasp from the documentation what the name attribute is supposed to refer to or how to override whatever was put in the xml project as the input.

When the report file exists that was used to setup the project and the job everything works fine through the Api dll, I have yet to get it to work through the webservice.In other words whatever I pass in as uris is ignored(probably cuz I have the name wrong).

Here's the xml project
<project version="XMLPRJ02">
<author>Tom Willis</author>
<description>
</description>
<model>
<location>F:datapump_intmodels88patdemo.mod</location>
</model>
<workflow>
<prerequisites />
</workflow>
<inputs process="individually" must_exist="all" order="xprj">
<main>
<report>
<location>F:datapump_intimport88*patdemo*.txt</location>
</report>
</main>
<distribution>
<action type="move">
<targetfolder>F:datapump_intdone88</targetfolder>
<targetname>&[yyyy]&[mm]&[dd]&[hh]&[mi]&[ss]&[source.name]&[jobid]&[counter].txt</targetname>
<overwrite>False</overwrite>
</action>
</distribution>
</inputs>
<state>
<view>
<window>table</window>
</view>
</state>
<exports>
<export>
<data_page view="table">
<name>88_resultscsv</name>
<table />
</data_page>
<file_page>
<files_in_folder>false</files_in_folder>
<location>F:datapump_intexport88patdemo.txt</location>
<when_exists>add</when_exists>
</file_page>
<distribution>
</distribution>
</export>
</exports>
</project>Here's the manifest

<manifest process="88_patdemo">
<inputs>
<item name="88_patdemo">
<uri>
file://F:/datapump_int/pre/88/bpatdemo.txt
</uri>
</item>
</inputs>
</manifest>job log when run through the Api dll

<joblog process="88_patdemo" jobID="1011">
<events>
<event time="2005-04-04T16:33:15">Created.</event>
<event time="2005-04-04T16:33:15">Process prerequisites were not met: Wildcard specification 'F:datapump_intimport88*patdemo*.txt' did not match any files.</event>
<event time="2005-04-04T16:33:16" alert="SystemAlertProcessFailed">Failed: DwchServer.NoRetryInfoException: Process prerequisites failed, but no retry information is available. Process will not be retried. ---> DwchServer.WildcardDidNotExpandPrerequisiteExcepti on: Wildcard specification 'F:datapump_intimport88*patdemo*.txt' did not match any files. at DwchServer.ProjectInputs.TestPrerequisites() at DwchServer.ProcessHandler.b(ServerDB A_0, JobRow A_1, JobLog A_2) --- End of inner exception stack trace --- at DwchServer.ProcessHandler.b(ServerDB A_0, JobRow A_1, JobLog A_2)</event>
<event time="2005-04-04T16:33:16" source="distributor">Distribution started</event>
<event time="2005-04-04T16:33:16" source="distributor" value="complete">Distribution completed</event>
</events>
<distribution />
</joblog>Through the webservice... I get a stack trace.


InboundLib.test.TestInbound.submitdataforextractio n : System.Web.Services.Protocols.SoapException : System.Web.Services.Protocols.SoapException: Server was unable to process request. ---> System.IO.FileLoadException: Access is denied: 'DwchServer.PumpAPI'.
File name: "DwchServer.PumpAPI"
at DwchServer.DwchServerWebService.StartProcessUsingM anifest(String strXml)

...

[Tom Willis]

Thanks for the clarification.

I'm having another issue now. Actually I don't know if it's an issue or not but Datapump is not behaving as I believed it would.

It seems that if I submit several jobs (same job name, different input file) the first one starts up fine, the sucessive ones will fail until the job that is running has completed.

- <joblog process="62_BPATDEMO" jobID="2423">
- <events>
<event time="2005-05-09T10:29:04">Created.</event>
<event time="2005-05-09T10:29:05" alert="SystemAlertProcessFailed">Failed: DwchServer.AnotherInstanceOfProcessIsStillActiveEx ception: Cannot start process '62_bpatdemo' because another instance of this process is still running. at DwchServer.ProcessHandler.b(ServerDB A_0, JobRow A_1, JobLog A_2)</event>
<event time="2005-05-09T10:29:06" source="distributor">Distribution started</event>
<event time="2005-05-09T10:29:07" source="distributor" value="complete">Distribution completed</event>
</events>
<distribution />
</joblog>Is there anyway to configure the job to queue up pending jobs?

[Gareth Horton]

Hi Tom

This is the way it is supposed to work.

Processes cannot be run concurrently in Datapump.

It is possible to run jobs concurrently, by setting the "Allow Parallel Jobs" option for a particular process.

What this means is say you have a process that contains 5 projects, if the setting is unchecked, then the 5 projects will execute in sequence. If it is checked, they will run in parallel.

I don't know what the most effective way of doing this would be, without knowing the pattern of your submission of files.

If you get multiple input files that you wish to process in a batch, but apply the project to them individually, e.g. you have File A, File B, File C and File D and they must all use the same model and export, but not loaded together, then use the "each in its own job" option for the input files in the "Grouping of multiple files" section.

This means when you create a manifest, you can pass a list of files, and if you have allow parallel jobs set to on, you will get 4 jobs running in parallel.

This parallel approach may speed up the operation, as the server is not waiting for each export to complete before starting another, which could buy you enough time before the next submission has to be made.

Gareth
Originally posted by Tom Willis:
Thanks for the clarification.

I'm having another issue now. Actually I don't know if it's an issue or not but Datapump is not behaving as I believed it would.

It seems that if I submit several jobs (same job name, different input file) the first one starts up fine, the sucessive ones will fail until the job that is running has completed.

- <joblog process="62_BPATDEMO" jobID="2423">
- <events>
<event time="2005-05-09T10:29:04">Created.</event>
<event time="2005-05-09T10:29:05" alert="SystemAlertProcessFailed">Failed: DwchServer.AnotherInstanceOfProcessIsStillActiveEx ception: Cannot start process '62_bpatdemo' because another instance of this process is still running. at DwchServer.ProcessHandler.b(ServerDB A_0, JobRow A_1, JobLog A_2)</event>
<event time="2005-05-09T10:29:06" source="distributor">Distribution started</event>
<event time="2005-05-09T10:29:07" source="distributor" value="complete">Distribution completed</event>
</events>
<distribution />
</joblog>Is there anyway to configure the job to queue up pending jobs?

[Tom Willis]

Thanks Gareth.

Unfortunately I have no way of knowing or anticipating the order of report files coming in. We could have reports coming in that would be processed by different jobs/templates, but often times we get a stream of reports that need to be processed by the same template. Order of processing matters to an extent and manual intervention when a job fails under these conditions is not ideal.

So I guess the queuing needs to happen in my program to avoid this condition.


Thanks for the clarification.

Tom Willis