PDA

View Full Version : Eliminating Duplicates in an export



Gareth Horton
July 12th, 2002, 01:17 PM
Joel McMullin wrote:
> Hi, all.
> One of Monarch's shortcomings is that it does not have a line by line comparator that allows you to eliminate duplicate records in tableview prior to export. This frequently makes it necessary to pass the exported table to a database first, where duplicate values are removed (through keying, for example) and then passing the results to the final destination (mailing list, for example).
> Here is a workaround so you can do it in one step. Many users have probably already figured this out, so my apologies if this is a redundant message.
> In your table view, set a calculated field called "constant" to a value of 1.
> Create a summary that includes all the fields you want to export, summarizing on the "constant" field.
> Your summary will have only one record for each line, no matter how often it was duplicated in the table. The "constant" field will tell you how many times it was duplicated in the original table. Now you can export the summary. The receiving application can be told to ignore the "constant" field since it has no meaning to the task at hand. Similarly, the final total can be ignoredor edited out of the resulting file.

This can also be done in the Summary Window using "count" as a Summary Field. This would eliminate creating a calculated field and as well as above show the duplicate records. If having a problem with the exported datatypes from the Summary Window, (All Key Fields are exported to database formats as text (i.e., character or general). The reason for this is that labels for unspecified values, such as "All Others" cannot be exported as dates or numbers. The values shown in the Summary Window represent a label more than a value. In the case of Summary Fields, the values are summarized, so do still represent "active" values that shoud be exported as such.), export as text. Bring the new text file in to Monarch, build a Model picking up all but the count field, adjust the datatypes in the Table Window and export as the proper database file.

Dee Moore
Datawatch Corporation
Technical Support
978.441.2200