Welcome to the new Woodmann RCE Messageboards Regroupment
Please be patient while the rest of the site is restored.

To all Members of the old RCE Forums:
In order to log in, it will be necessary to reset your forum login password ("I forgot my password") using the original email address you registered with. You will be sent an email with a link to reset your password for that member account.

The old vBulletin forum was converted to phpBB format, requiring the passwords to be reset. If this is a problem for some because of a forgotten email address, please feel free to re-register with a new username. We are happy to welcome old and new members back to the forums! Thanks.

All new accounts are manually activated before you can post. Any questions can be PM'ed to Kayaker.

Batch patcher for linux

RCE of Linux tools and programs.
Junior Member
Posts: 24
Joined: Sun May 30, 2004 2:01 pm

Batch patcher for linux

Post by sailor__eda »

I have a program that has several files that need to be patched. There are many files (over a 100) and I suspect the files themselves will change in the future so I wanted to see what the best way to do a batch binary patch in linux.

I've used diablo2oo dup2 for windows but, because I have so many files, I would prefer to make a generic search and replace script.

I've used basic text search and replace in linux under bash using perl or python but I've never handled binaries. Not that binaries will be that different, but before I started writing my script, I wanted to see if a patcher for linux like dup2 existed. I could not find one on google.

Does anybody have any suggestions how I can go about this? I would like to have s&r expression as in dup2 i.e the wildcards but any other suggestions are welcome.

Senior Member
Posts: 5329
Joined: Wed Apr 25, 2001 2:00 pm

Post by JMI »

Again, the FIRST thing YOU should try is to do, yourself, is SEARCH for what YOU want. :yay:

Have you actually put:

batch binary patch in linux

and various combinations of that criteria into YOUR favorite search engine and reviewed ANYTHING you found? Have you done ANY basic research on what YOU want?

I got 38,400 hits in google with that criteria.

So, how about you "start" which the basic requirements of this Fourm and at least "attempt" to help yourself.

And don't wine about my suggestion if you have done any searching, because your post does not indicate that you have done so, other than "look for something like dup2 for linux." ;)

You also might find some interesting items with:

batch binary patcher linux

I got 46,500 hits with that one.

Junior Member
Posts: 24
Joined: Sun May 30, 2004 2:01 pm

Post by sailor__eda »

Point well taken.

I should have specified what I did and I was lax about that - my mistake. I made that assumption since "patch" under linux gives a whole bunch of hits for the linux patch tool which works with source code, hence the large number of hits.

Let me attempt to rectify this.

I've looked at several native tools under linux that can do basic binary patching. vi also has a binary mode and can patch single files but multiple files is tedious. I found a scripting mode for vi that could actually do what I want and that was one of the paths I was going to proceed along.

In addition, there is another tool called sfk http://stahlworks.com/dev/index.php?tool=rep&back=dev

that does binary s+r but it doesn't handle wildcards so that makes it a bit useless for my purpose.

What I was really hoping for is not a dup2 tool for linux but rather a native linux tool that might serve the purpose.

Again, sorry for the lack of details on my ground work prior to posting.

Senior Member
Posts: 5329
Joined: Wed Apr 25, 2001 2:00 pm

Post by JMI »

Good second effort! :yay:

While I understand you are looking for an essentially ready made "tool" have you taken a closer look at the Linux command "patch" ?? Seems if you can script, you could "roll your own."

For example:

Code: Select all

patch - apply a diff file to an original 
patch [options] [originalfile [patchfile]] 
but usually just 

patch -pnum <patchfile 

patch takes a patch file patchfile containing a difference listing produced by the diff program and applies those differences to one or more original files, producing patched versions. Normally the patched versions are put in place of the originals. Backups can be made; see the -b or --backup option. The names of the files to be patched are usually taken from the patch file, but if there's just one file to be patched it can specified on the command line as originalfile. 
Upon startup, patch attempts to determine the type of the diff listing, unless overruled by a -c (--context), -e (--ed), -n (--normal), or -u (--unified) option. Context diffs (old-style, new-style, and unified) and normal diffs are applied by the patch program itself, while ed diffs are simply fed to the ed(1) editor via a pipe. 

patch tries to skip any leading garbage, apply the diff, and then skip any trailing garbage. Thus you could feed an article or message containing a diff listing to patch, and it should work. If the entire diff is indented by a consistent amount, or if a context diff contains lines ending in CRLF or is encapsulated one or more times by prepending "- " to lines starting with "-" as specified by Internet RFC 934, this is taken into account. 

With context diffs, and to a lesser extent with normal diffs, patch can detect when the line numbers mentioned in the patch are incorrect, and attempts to find the correct place to apply each hunk of the patch. As a first guess, it takes the line number mentioned for the hunk, plus or minus any offset used in applying the previous hunk. If that is not the correct place, patch scans both forwards and backwards for a set of lines matching the context given in the hunk. First patch looks for a place where all lines of the context match. If no such place is found, and it's a context diff, and the maximum fuzz factor is set to 1 or more, then another scan takes place ignoring the first and last line of context. If that fails, and the maximum fuzz factor is set to 2 or more, the first two and last two lines of context are ignored, and another scan is made. (The default maximum fuzz factor is 2.) If patch cannot find a place to install that hunk of the patch, it puts the hunk out to a reject file, which normally is the name of the output file plus a .rej suffix, or # if .rej would generate a file name that is too long (if even appending the single character # makes the file name too long, then # replaces the file name's last character). (The rejected hunk comes out in ordinary context diff form regardless of the input patch's form. If the input was a normal diff, many of the contexts are simply null.) The line numbers on the hunks in the reject file may be different than in the patch file: they reflect the approximate location patch thinks the failed hunks belong in the new file rather than the old one. 

As each hunk is completed, you are told if the hunk failed, and if so which line (in the new file) patch thought the hunk should go on. If the hunk is installed at a different line from the line number specified in the diff you are told the offset. A single large offset may indicate that a hunk was installed in the wrong place. You are also told if a fuzz factor was used to make the match, in which case you should also be slightly suspicious. If the --verbose option is given, you are also told about hunks that match exactly. 

If no original file origfile is specified on the command line, patch tries to figure out from the leading garbage what the name of the file to edit is, using the following rules. 

First, patch takes an ordered list of candidate file names as follows: 

If the header is that of a context diff, patch takes the old and new file names in the header. A name is ignored if it does not have enough slashes to satisfy the -pnum or --strip=num option. The name /dev/null is also ignored. 
If there is an Index: line in the leading garbage and if either the old and new names are both absent or if patch is conforming to POSIX, patch takes the name in the Index: line. 
For the purpose of the following rules, the candidate file names are considered to be in the order (old, new, index), regardless of the order that they appear in the header. 
Then patch selects a file name from the candidate list as follows: 

If some of the named files exist, patch selects the first name if conforming to POSIX, and the best name otherwise. 
If patch is not ignoring RCS, ClearCase, and SCCS (see the -g num or --get=num option), and no named files exist but an RCS, ClearCase, or SCCS master is found, patch selects the first named file with an RCS, ClearCase, or SCCS master. 
If no named files exist, no RCS, ClearCase, or SCCS master was found, some names are given, patch is not conforming to POSIX, and the patch appears to create a file, patch selects the best name requiring the creation of the fewest directories. 
If no file name results from the above heuristics, you are asked for the name of the file to patch, and patch selects that name. 
To determine the best of a nonempty list of file names, patch first takes all the names with the fewest path name components; of those, it then takes all the names with the shortest basename; of those, it then takes all the shortest names; finally, it takes the first remaining name. 

Additionally, if the leading garbage contains a Prereq: line, patch takes the first word from the prerequisites line (normally a version number) and checks the original file to see if that word can be found. If not, patch asks for confirmation before proceeding. 

The upshot of all this is that you should be able to say, while in a news interface, something like the following: 

| patch -d /usr/src/local/blurfl

and patch a file in the blurfl directory directly from the article containing the patch. 

If the patch file contains more than one patch, patch tries to apply each of them as if they came from separate patch files. This means, among other things, that it is assumed that the name of the file to patch must be determined for each diff listing, and that the garbage before each diff listing contains interesting things such as file names and revision level, as mentioned previously. 

-b or --backup
Make backup files. That is, when patching a file, rename or copy the original instead of removing it. When backing up a file that does not exist, an empty, unreadable backup file is created as a placeholder to represent the nonexistent file. See the -V or --version-control option for details about how backup file names are determined. 
Back up a file if the patch does not match the file exactly and if backups are not otherwise requested. This is the default unless patch is conforming to POSIX. 
Do not back up a file if the patch does not match the file exactly and if backups are not otherwise requested. This is the default if patch is conforming to POSIX. 
-B pref or --prefix=pref
Prefix pref to a file name when generating its simple backup file name. For example, with -B /junk/ the simple backup file name for src/patch/util.c is /junk/src/patch/util.c. 
Read and write all files in binary mode, except for standard output and /dev/tty. This option has no effect on POSIX-conforming systems. On systems like DOS where this option makes a difference, the patch should be generated by diff -a --binary. 
-c or --context
Interpret the patch file as a ordinary context diff. 
-d dir or --directory=dir
Change to the directory dir immediately, before doing anything else. 
-D define or --ifdef=define
Use the #ifdef ... #endif construct to mark changes, with define as the differentiating symbol. 
Print the results of applying the patches without actually changing any files. 
-e or --ed
Interpret the patch file as an ed script. 
-E or --remove-empty-files
Remove output files that are empty after the patches have been applied. Normally this option is unnecessary, since patch can examine the time stamps on the header to determine whether a file should exist after patching. However, if the input is not a context diff or if patch is conforming to POSIX, patch does not remove empty patched files unless this option is given. When patch removes a file, it also attempts to remove any empty ancestor directories. 
-f or --force
Assume that the user knows exactly what he or she is doing, and do not ask any questions. Skip patches whose headers do not say which file is to be patched; patch files even though they have the wrong version for the Prereq: line in the patch; and assume that patches are not reversed even if they look like they are. This option does not suppress commentary; use -s for that. 
-F num or --fuzz=num
Set the maximum fuzz factor. This option only applies to diffs that have context, and causes patch to ignore up to that many lines in looking for places to install a hunk. Note that a larger fuzz factor increases the odds of a faulty patch. The default fuzz factor is 2, and it may not be set to more than the number of lines of context in the context diff, ordinarily 3. 
-g num or --get=num
This option controls patch's actions when a file is under RCS or SCCS control, and does not exist or is read-only and matches the default version, or when a file is under ClearCase control and does not exist. If num is positive, patch gets (or checks out) the file from the revision control system; if zero, patch ignores RCS, ClearCase, and SCCS and does not get the file; and if negative, patch asks the user whether to get the file. The default value of this option is given by the value of the PATCH_GET environment variable if it is set; if not, the default value is zero if patch is conforming to POSIX, negative otherwise. 
Print a summary of options and exit. 
-i patchfile or --input=patchfile
Read the patch from patchfile. If patchfile is -, read from standard input, the default. 
-l or --ignore-whitespace
Match patterns loosely, in case tabs or spaces have been munged in your files. Any sequence of one or more blanks in the patch file matches any sequence in the original file, and sequences of blanks at the ends of lines are ignored. Normal characters must still match exactly. Each line of the context must still match a line in the original file. 
-n or --normal
Interpret the patch file as a normal diff. 
-N or --forward
Ignore patches that seem to be reversed or already applied. See also -R. 
-o outfile or --output=outfile
Send output to outfile instead of patching files in place. 
-pnum or --strip=num
Strip the smallest prefix containing num leading slashes from each file name found in the patch file. A sequence of one or more adjacent slashes is counted as a single slash. This controls how file names found in the patch file are treated, in case you keep your files in a different directory than the person who sent out the patch. For example, supposing the file name in the patch file was 

setting -p0 gives the entire file name unmodified, -p1 gives 


without the leading slash, -p4 gives 


and not specifying -p at all just gives you blurfl.c. Whatever you end up with is looked for either in the current directory, or the directory specified by the -d option. 

Conform more strictly to the POSIX standard, as follows. 
Take the first existing file from the list (old, new, index) when intuiting file names from diff headers. 
Do not remove files that are empty after patching. 
Do not ask whether to get files from RCS, ClearCase, or SCCS. 
Require that all options precede the files in the command line. 
Do not backup files when there is a mismatch. 
Use style word to quote output names. The word should be one of the following: 
Output names as-is. 
Quote names for the shell if they contain shell metacharacters or would cause ambiguous output. 
Quote names for the shell, even if they would normally not require quoting. 
Quote names as for a C language string. 
Quote as with c except omit the surrounding double-quote characters. 
You can specify the default value of the --quoting-style option with the environment variable QUOTING_STYLE. If that environment variable is not set, the default value is shell. 

-r rejectfile or --reject-file=rejectfile
Put rejects into rejectfile instead of the default .rej file. 
-R or --reverse
Assume that this patch was created with the old and new files swapped. (Yes, I'm afraid that does happen occasionally, human nature being what it is.) patch attempts to swap each hunk around before applying it. Rejects come out in the swapped format. The -R option does not work with ed diff scripts because there is too little information to reconstruct the reverse operation. 
If the first hunk of a patch fails, patch reverses the hunk to see if it can be applied that way. If it can, you are asked if you want to have the -R option set. If it can't, the patch continues to be applied normally. (Note: this method cannot detect a reversed patch if it is a normal diff and if the first command is an append (i.e. it should have been a delete) since appends always succeed, due to the fact that a null context matches anywhere. Luckily, most patches add or change lines rather than delete them, so most reversed normal diffs begin with a delete, which fails, triggering the heuristic.) 

-s or --silent or --quiet
Work silently, unless an error occurs. 
-t or --batch
Suppress questions like -f, but make some different assumptions: skip patches whose headers do not contain file names (the same as -f); skip patches for which the file has the wrong version for the Prereq: line in the patch; and assume that patches are reversed if they look like they are. 
-T or --set-time
Set the modification and access times of patched files from time stamps given in context diff headers, assuming that the context diff headers use local time. This option is not recommended, because patches using local time cannot easily be used by people in other time zones, and because local time stamps are ambiguous when local clocks move backwards during daylight-saving time adjustments. Instead of using this option, generate patches with UTC and use the -Z or --set-utc option instead. 
-u or --unified
Interpret the patch file as a unified context diff. 
-v or --version
Print out patch's revision header and patch level, and exit. 
-V method or --version-control=method
Use method to determine backup file names. The method can also be given by the PATCH_VERSION_CONTROL (or, if that's not set, the VERSION_CONTROL) environment variable, which is overridden by this option. The method does not affect whether backup files are made; it affects only the names of any backup files that are made. 
The value of method is like the GNU Emacs `version-control' variable; patch also recognizes synonyms that are more descriptive. The valid values for method are (unique abbreviations are accepted): 

existing or nil
Make numbered backups of files that already have them, otherwise simple backups. This is the default. 
numbered or t
Make numbered backups. The numbered backup file name for F is F.~N~ where N is the version number. 
simple or never
Make simple backups. The -B or --prefix, -Y or --basename-prefix, and -z or --suffix options specify the simple backup file name. If none of these options are given, then a simple backup suffix is used; it is the value of the SIMPLE_BACKUP_SUFFIX environment variable if set, and is .orig otherwise. 
With numbered or simple backups, if the backup file name is too long, the backup suffix ~ is used instead; if even appending ~ would make the name too long, then ~ replaces the last character of the file name. 

Output extra information about the work being done. 
-x num or --debug=num
Set internal debugging flags of interest only to patch patchers. 
-Y pref or --basename-prefix=pref
Prefix pref to the basename of a file name when generating its simple backup file name. For example, with -Y .del/ the simple backup file name for src/patch/util.c is src/patch/.del/util.c. 
-z suffix or --suffix=suffix
Use suffix as the simple backup suffix. For example, with -z - the simple backup file name for src/patch/util.c is src/patch/util.c-. The backup suffix may also be specified by the SIMPLE_BACKUP_SUFFIX environment variable, which is overridden by this option. 
-Z or --set-utc
Set the modification and access times of patched files from time stamps given in context diff headers, assuming that the context diff headers use Coordinated Universal Time (UTC, often known as GMT). Also see the -T or --set-time option. 
The -Z or --set-utc and -T or --set-time options normally refrain from setting a file's time if the file's original time does not match the time given in the patch header, or if its contents do not match the patch exactly. However, if the -f or --force option is given, the file time is set regardless. 

Due to the limitations of diff output format, these options cannot update the times of files whose contents have not changed. Also, if you use these options, you should remove (e.g. with make clean) all files that depend on the patched files, so that later invocations of make do not get confused by the patched files' times. 

diff(1), ed(1) 
Marshall T. Rose and Einar A. Stefferud, Proposed Standard for Message Encapsulation, Internet RFC 934 ftp://ftp.isi.edu/in-notes/rfc934.txt> (1985-01). 

There are several things you should bear in mind if you are going to be sending out patches. 
Create your patch systematically. A good method is the command diff -Naur old new where old and new identify the old and new directories. The names old and new should not contain any slashes. The diff command's headers should have dates and times in Universal Time using traditional Unix format, so that patch recipients can use the -Z or --set-utc option. Here is an example command, using Bourne shell syntax: 

LC_ALL=C TZ=UTC0 diff -Naur gcc-2.7 gcc-2.8

Tell your recipients how to apply the patch by telling them which directory to cd to, and which patch options to use. The option string -Np1 is recommended. Test your procedure by pretending to be a recipient and applying your patch to a copy of the original files. 

You can save people a lot of grief by keeping a patchlevel.h file which is patched to increment the patch level as the first diff in the patch file you send out. If you put a Prereq: line in with the patch, it won't let them apply patches out of order without some warning. 

You can create a file by sending out a diff that compares /dev/null or an empty file dated the Epoch (1970-01-01 00:00:00 UTC) to the file you want to create. This only works if the file you want to create doesn't exist already in the target directory. Conversely, you can remove a file by sending out a context diff that compares the file to be deleted with an empty file dated the Epoch. The file will be removed unless patch is conforming to POSIX and the -E or --remove-empty-files option is not given. An easy way to generate patches that create and remove files is to use GNU diff's -N or --new-file option. 

If the recipient is supposed to use the -pN option, do not send output that looks like this: 

diff -Naur v2.0.29/prog/README prog/README

--- v2.0.29/prog/README Mon Mar 10 15:13:12 1997

+++ prog/README Mon Mar 17 14:58:22 1997

because the two file names have different numbers of slashes, and different versions of patch interpret the file names differently. To avoid confusion, send output that looks like this instead: 

diff -Naur v2.0.29/prog/README v2.0.30/prog/README

--- v2.0.29/prog/README Mon Mar 10 15:13:12 1997

+++ v2.0.30/prog/README Mon Mar 17 14:58:22 1997

Avoid sending patches that compare backup file names like README.orig, since this might confuse patch into patching a backup file instead of the real file. Instead, send patches that compare the same base file names in different directories, e.g. old/README and new/README. 

Take care not to send out reversed patches, since it makes people wonder whether they already applied the patch. 

Try not to have your patch modify derived files (e.g. the file configure where there is a line configure: configure.in in your makefile), since the recipient should be able to regenerate the derived files anyway. If you must send diffs of derived files, generate the diffs using UTC, have the recipients apply the patch with the -Z or --set-utc option, and have them remove any unpatched files that depend on patched files (e.g. with make clean). 

While you may be able to get away with putting 582 diff listings into one file, it may be wiser to group related patches into separate files in case something goes haywire. 


You might also want to try the search:

how to batch patch linux files

which is where I got the code listing above.
Senior Member
Posts: 5329
Joined: Wed Apr 25, 2001 2:00 pm

Post by JMI »

Another possibility:

http://search.techrepublic.com.com/sear ... +tool.html

FLY Batch (exe)

FLY Batch is a versatile developer's tool which powers automated, centralized, server-side, high-volume and real-time batch processing. It uses a straightforward command-line interface to drive the required format conversion filters on Windows, Linux or UNIX. Input Formats: PDF, PostScript, EPS, WMF, EMF; Output formats: SVG, CGM, MIF, EPS, PostScript, DXF,...
Tags: PostScript, Microsoft Windows Metafile, SVG, ASCII, TIFF, JPEG, Unix, Adobe PDF, Linux, performance, tool

Senior Member
Posts: 5329
Joined: Wed Apr 25, 2001 2:00 pm

Post by JMI »

Another interesting concept:


http://www.netfilter.org/documentation/ ... WTO-2.html

Some of the discussion:

Patch-o-matic in fact, is mainly the `runme' shell script. If you run it without arguments, it will display its help message :

Usage: ./runme [--batch] [--reverse] [--exclude suite/patch-file ...] suite|suite/patch-file

--batch batch mode, automatically applying patches
--reverse back out the selected patches
--exclude excludes the named patches

The patches are contained in `patch-o-matic/pending/', `patch-o-matic/base', etc.. Here, `pending' and `base' are two suite names. ls the `patch-o-matic' directory to see all the suites. Example of `runme' commands :

./runme --batch pending
./runme --batch userspace/ipt_REJECT-fake-source.patch

The first command will attempt to apply all the patches from submitted suite, then the pending suite (we explain further why two suites). The second command will only apply the patch `ipt_REJECT-fake-source.patch' from the userspace suite.

Junior Member
Posts: 24
Joined: Sun May 30, 2004 2:01 pm

Post by sailor__eda »

Thanks for the posting.

From what I understand about patch, you generate the input to patch by doing a diff between the original and patched file. I want to avoid having to patch the files manually in the first place. I think for my purpose, what I really need is a binary search and replace tool with support for wildcards in the expressions like dup2.

I guess maybe I will have to write this one myself - it should be straight forward but I was being a lazy. Hey, one more tool in the RCE scene never hurt anybody :)

Junior Member
Posts: 24
Joined: Sun May 30, 2004 2:01 pm

Post by sailor__eda »

One of your links was helpful in an indirect way. I think I found what I'm looking for here


Thanks JMI

Senior Member
Posts: 5329
Joined: Wed Apr 25, 2001 2:00 pm

Post by JMI »

Think what you might learn by writing your own! :yay:

And just a note for saving space in the database, if you use the button on the far right at the bottom of the Thread, which looks like a page of text with a down arrow on it, you will open the "quick reply" window, which will reply without quoting the text. Unless you want to comment on something actually in the previous post, it takes less room in the database if you use that button. For example, we really don't need two copies of that listing of the Linux patch command, so I deleted that part of your reply.) Just a heads-up. :yay: