Key Concepts for Using Flat Files

UDEs

The standard Connect implementation supports user data extracts (UDEs) as an input type. UDEs are flat files consisting of properly formatted, raw data to be read and used by Connect EDM campaigns. Such flat files can be categorized in one of the following UDE types:

Customer master data
Customer attribute data
Customer subscription interests

The UDE file format that is currently supported is the text-based, comma-separated value (CSV) format. You can use a CSV file to collect data from any table so that it can be conveyed as input to another table-oriented tool (such as Conduits for Connect) or enterprise application (such as a relational database).

Event-Based Scheduling

You can configure a session to start after an indicator file appears in a directory. You can use this feature to start a session if you do not know when the source data will become available.

To use event-based scheduling, you need a shell command, script, or batch file that creates an indicator file when all sources are ready for the session. The file must be created or sent to a directory local to the Informatica server. The file can be empty. It is the presence of the file that initiates the session.

If you manually start the session, the Informatica server waits for the indicator file to appear before running the session. If you schedule the session, the Informatica server waits for the first scheduled start time and then for the indicator file to appear before running the session.

When you monitor the Informatica server with an event-based scheduled session, the Server Manager displays the session status as File Wait.

As soon as the Informatica server starts the session, it deletes the indicator file. The Informatica server writes the time the file appears and the session start time in the session log.

The indicator file does not contain any data. The mere presence of the indicator file in the SrcFiles directory is sufficient to initiate the data integration process.

The indicator file must have a file name of indicator_*.dat, where the “*” represents the UDE type to be processed. For example, the data load for customer master (cm) records should be initiated by placing a file called indicator_cm.dat into the SrcFiles directory. The corresponding suffixes for the various data formats are:

UDE Type	Suffix	Indicator Examples
Customer Master data	cm	indicator_cm.dat
Customer Attribute data	at	indicator_at.dat
Customer Subscription interest	sb	indicator_sb.dat

Warning: Avoid specifying the source or target file name as the indicator file.

File List

You can create one session to run multiple source files. You might use this feature if, for example, your company collects data at several locations which you then want to move through the same session. When you create a mapping to use multiple source files, the properties of all files must exactly match the source definition.

To use multiple source files, you create a file containing the names and directories of each source file you want the Informatica server to use. This file is referred to as a file list. When you configure the session, you indicate that the source file is a file list and verify the location of the file list. When the session starts, the Informatica server reads the file list, then locates and reads the first file source in the list. After the Informatica server reads the first file, it locates and reads the next file in the list.

The Informatica server treats sessions using a file list as a single session, creating one log file and one reject file for the entire session. As a result, if the session performs incremental aggregation, the Informatica server performs incremental aggregation across all listed source files.

The file list is a list of all data files that are to be processed together. The file list file should contain one or more lines of text, such as:

                    <my_datafile1.txt> <my_datafile2.txt>
                    <my_datafileN.txt>

Where each <my_datafile*.txt> entry is one properly formatted ETL file containing the data records you want to load into the Connect database.

The file list file has the format file_list_*.dat, where the “*” represents the type of data to be processed. As with the indicator files, the corresponding suffixes for the various data formats are:

Customer Master – cm
Attributes – at
Subscriptions – sb

PMCMD

You can use the command-line utility pmcmd to communicate with the Informatica server. This utility does not replace the Server Manager, because there are many tasks that you can perform only with the Server Manager.

Pmcmd gives you versatility in running sessions. You can, for example, execute shell scripts or Perl programs that run sessions, or schedule sessions with operating system scheduling tools like cron.

Pmcmd returns zero if it works properly, and non-zero if it encounters difficulties executing the command.

Use the following syntax to start or stop a session or batch:

 pmcmd
                    {start, stop} {user_name, %user_env_var}{password ,
                    %password_env_var}{[TCP/IP:][hostname:]portno, IPX/SPX:ipx/spx_address}
                    [folder_name:]{session_name, batch_name} session_flag [wait_flag]

Output Files

If a session writes to target files, the Informatica server creates these files in the Informatica server’s variable directory, $PMTargetFileDir, by default. Output files are created based on file properties entered in the session property sheet and are named after the target table by default (for example, error_cust_attributes.out).