Efficient Log File Copying: A Clever Combination of Windows Commands

When it comes to copying large log files with new lines appended regularly, finding the right approach can be challenging. This article will explore a clever combination of Windows commands to accomplish this task without modifying the device that generates the logs.

The requirement is to copy only the new lines added after the previous copy from the source log file to the target log file. The target log file contains the latest details copied from the source log, and both files are available via SMB network share. Since the network connection is slow and congested, transferring the whole files each time is not feasible.

One possible solution is to use a combination of Windows commands that can resume partial downloads and skip certain lines when reading from the source file. Robocopy is a great candidate for this task, as it can resume partial downloads. However, getting it to work in this case proved challenging.

To overcome this hurdle, we need to find an alternative solution that leverages other Windows commands. Xcopy is another option worth exploring. By using xcopy with specific parameters, we can achieve the desired result.

Here's a possible approach:

1. Get the line count value from the target log file.

2. Use this line count value to skip the corresponding number of lines when reading from the source log file.

3. Append the new lines to the target log file using xcopy with the `/y` and `/a-d` parameters.

The command would look something like this:

```

xcopy /y /a-d +source_log_file.txt +target_log_file.txt

```

This command will append only the new lines added after the previous copy from the source log file to the target log file. The `/y` parameter suppresses prompting to confirm overwrite, and the `/a-d` parameter copies files that are newer on the source than in the destination.

In summary, by combining Windows commands like xcopy with specific parameters, we can efficiently copy only the new lines added after the previous copy from the source log file to the target log file. This approach avoids the need for modifying the device that generates the logs and is suitable for slow and congested network connections.

Copy log