DPF'S AND DATAPUMP

[Ontiveros]

I AM USING DATA PUMP AND MONARCH (BOTH V6). THE MODEL SET UP WITH A REPORT I HAVE WORKS FINE BUT WHEN I USE SET UP A DPF USING THE SAME REPORT AND MODEL, THE FILTER DOES NOT WORK EVEN THOUGH I SELECTED IT IN THE WORKFLOW-PROCESS OPTIONS. WHY IS THIS HAPPENING?

[Todd Niemi]

Could you provide some information as to what you're trying to do? Also, what error message (if any) are you receiving?

[ July 15, 2004, 02:17 PM: Message edited by: Todd Niemi ]

[Ontiveros]

I AM TRYING TO FILTER RECORDS FROM A FILE MAINTENANCE REPORT AND PUT THE RESULTS IN ANOTHER FOLDER FOR USERS TO RETREIVE WHEN THEY COME INTO THE OFFICE IN THE MORNING. DATA PUMP SHOULD BE ABLE TO DO THIS AT A SCHEDULED TIME. THE WEIRD THING IS THAT I DON'T GET AN ERROR MESSAGE. IN FACT, I CHECKED THE DPF.LOG FILE AND IT SAID THAT THE FILTER RAN.
IF I DON'T USE A FILTER, I GET APPROX. 50000 RECORDS. WITH THE FILTER, I GET 1000 OR SO. I CAN SEE NO LOGICAL REASON WHY THE FILTER ISN'T RUNNING USING THE DPF IN THE DATA PUMP ADMINISTRATOR.

[Todd Niemi]

If everything is configured properly, then I don't see why the filter wouldn't work. I suggest you contact Datawatch's tech support department, and walk them step-by-step through the process.

[Ontiveros]

I have contacted Rose Cavaretta in the Tech Support area and they are analyzing it. I was just trying to see if anyone else has had the problem or knows of why it would do that.

[Todd Niemi]

I personally haven't encountered this problem when using Data Pump (has anyone else?). Hopefully Datawatch's tech support department will come through for you. If you do get your problem solved, please post a message here to explain how you resolved it.

[Ontiveros]

Will Do.

[Jim Weeks]

When you say it the filter is not running, do you mean that you get an output file but it is empty or do you not get a file at all? Perhaps I have been experiencing a similar problem....I have some large filters running from .prj calls
in a .bat file. By large, I mean it takes a while for them to run. If I run them manually, that is open the project from Monarch and the use project export, it works fine. But from a .bat file, its almost like the project export kicks in before the open and filter has completed and it outputs a file with no records. Does that sound similar?

[Ontiveros]

THE ISSUE I AM HAVING IS THAT DATAPUMP BASICALLY IGNORES THE FILTER SET UP IN THE DP ADMINISTRATOR. I HAVE IT SET UP TO OPEN THE REPORT WITH A PARTICULAR MODEL. THE THE REPLY I RECEIVED FROM DATAWATCH WHEN i INQUIRED ABOUT THE PROBLEM IS :
There is a bug in version 6 of Data Pump that limits the expression in the filters and calculated fields to 1024 characters. In Monarch this limit is at 4095 so this is why your filter works in Monarch but not in Data Pump.
This issue does not exist in version 7, so you may want to consider an
upgrade to v7.

OUR COMPANY HAS ONLY HAD DATAPUMP AND MONARCH ON A SEPARATE SERVER FOR 6 MONTHS. MY QUESTION IS, "WHY WOULD THEY SELL OUR COMPANY PRODUCTS THAT THERE ARE KNOWN BUGS IN RATHER RECOMMEND THE UPGRADED VERSION.

[Darren]

Hey everyone,

I have experienced this problem as well when using Monarch 6 and MDP 6 when automating larger reports. I have seen this same scenario happen to me when I have the following events set in motion:

1). I have a filter in place - same as Ontiveros is talking about. Whereas the .dpf runs but misses the filter. When run manually everything is fine. When run automatically through MDP, the filter is missed.

2). I have some sort of formatting in place in the Monarch model (summary). Again, when all the automation is run from Momnarch to MDP, the formatting is missed.

This is only the case when really large files are executed through the process.

The resolution to my problem was that I had set my MDP Process Scheduler to look for the report every 1 minute. If I changed the Execution Frequency time frame from 1 minute to 5 minutes or longer, these problems went away.

I always kind of wondered about this scenario. Ontiveros, please let me know what your Execution frequency is for your .dpf's...

[Ontiveros]

MY EXECUTION FREQUENCY IS DAILY AT 3:00 A.M.

[Darren]

OK, fair enough. So if your Execution Frequency is set to run only once a day, then my scenario is not valid. Now that I understand your case better I think I've run into that scenario too. My workaround was to filter it elsewhere, like in Excel, but still this does not resolve the problem within MDP 6. Interesting to know that it's actually a documented bug in ver. 6.

[Ontiveros]

A solution I found was to break up the filter into groups. For example, say you have a filter that is over 1024 characters long. Use an IF statement in a calculated field. Here is an example of what I have made one of my calculated fields:
If([Field Desc].NotIn.("SERVICE CODE:*","QUANTITY:*","PER ITEM AMOUNT:*","TOTAL AMOUNT:*","TRANSACTION DATE:*","DESCRIPTION 1:*","INTEREST CYCLE:*","ACCOUNT STATUS CODE"),"1","0"). This will give the value of 1 in the calculated field if values in the field are not in the "not in" statement. In the actual filter of the model, you just need to specify whether you want the calculated field to equal a 1 or 0. It worked for me anyways.

[Grant]

I had this same problem. My issue was simply resolved, open your template in Monarch and ensure that under your summary options you select filter <DEFAULT> and save the model again. If no filter is selected here it overrides the filter specified in the MDP processing options.