Configure Squid by opening /etc/squid/squid.conf using your favorite text editor. In the configuration file, search for the following directives and modify (or add, if they don't exist) as it follows: http_port 3128 - The port Squid will listen for connections. If your system has two or more interfaces, you can specify which IP address to use. Eg: http_port 192.168.0.1:3128 http_access deny all - Search for it in the config file, uncomment it (remove the # in front), and replace deny with allow so it becomes http_access allow all. Restart the Squid proxy with: CODE $ sudo /etc/init.d/squid restart Now you should have a fully functional HTTP proxy. To try it out, open a browser, open its preferences dialog and go to proxy settings. Here, enter the IP address of the machine running Squid and the port set in squid.conf. Now load a webpage. SETTING UP SQUID AUTHENTICATION AND WEB FILTERING This section will allow you to set up a web site filter for kids. The first time an address is entered in the browser's address bar, an authentication dialog will pop-up, prompting for a username and password. We will set-up two usernames, one with full and another with restricted access. First, open the /etc/squid/squid.conf and add the following line in the auth_param section: auth_param basic program /usr/lib/squid/ncsa_auth /etc/squid/passwd Now create the user accounts using htpasswd (use -c only for the first user): CODE $ sudo htpasswd -c /etc/squid/passwd dad Create the ACLs by adding the following lines in the ACCESS CONTROLS (acl) sections in Squid.conf: CODE acl dadUser proxy_auth dad Create the whitelist by opening a text editor, adding allowed domains like this: .google.com .kids-play.com .yahoo.com .msn.com and save it as /etc/squid/whitelist. Finally, search for http_access allow all in the Squid config file and modify it so it looks like this: http_access deny all This is how my Squid config sections look like: CODE # NETWORK OPTIONS Use deny all for squid with authentication and allow all for basic squid configuration. | |
Tuesday, August 25, 2009
Seting-Up a HTTP Proxy Server with Authentication and Filtering
create a shared and a static library with gcc
The code for the library
The header file
double mean(double, double);
Creating the static library
ar rcs libmean.a calc_mean.o
Creating the shared library
cc1: warning: -fPIC ignored for target (all code is position independent)
The programm using the library
#include#include "calc_mean.h" int main(int argc, char* argv[]) { double v1, v2, m; v1 = 5.2; v2 = 7.9; m = mean(v1, v2); printf("The mean of %3.2f and %3.2f is %3.2f\n", v1, v2, m); return 0; }
Linking against static library
Linking against shared library
Executing the dynamically linked programm
LD_LIBRARY_PATH=. ./dynamically_linked
Creating a library with the GNU archiver
Creating a library with the GNU archiver
The GNU archiver ar
combines a collection of object files into a single archive file, also known as a library. An archive file is simply a convenient way of distributing a large number of related object files together
To demonstrate the use of the GNU archiver we will create a small library ‘libhello.a’ containing two functions hello
and bye
.
The first object file will be generated from the source code for the hello
function, in the file ‘hello_fn.c’ seen earlier:
#include#include "hello.h" void hello (const char * name) { printf ("Hello, %s!\n", name); }
The second object file will be generated from the source file ‘bye_fn.c’, which contains the new function bye
:
#include#include "hello.h" void bye (void) { printf ("Goodbye!\n"); }
Both functions use the header file ‘hello.h’, now with a prototype for the function bye()
:
void hello (const char * name); void bye (void);
The source code can be compiled to the object files ‘hello_fn.o’ and ‘bye_fn.o’ using the commands:
$ gcc -Wall -c hello_fn.c $ gcc -Wall -c bye_fn.c
These object files can be combined into a static library using the following command line:
$ ar cr libhello.a hello_fn.o bye_fn.o
The option cr
stands for "create and replace".(33) If the library does not exist, it is first created. If the library already exists, any original files in it with the same names are replaced by the new files specified on the command line. The first argument ‘libhello.a’ is the name of the library. The remaining arguments are the names of the object files to be copied into the library.
The archiver ar
also provides a "table of contents" option t
to list the object files in an existing library:
$ ar t libhello.a hello_fn.o bye_fn.o
Note that when a library is distributed, the header files for the public functions and variables it provides should also be made available, so that the end-user can include them and obtain the correct prototypes.
We can now write a program using the functions in the newly created library:
#include "hello.h" int main (void) { hello ("everyone"); bye (); return 0; }
This file can be compiled with the following command line, as described in section 2.7 Linking with external libraries, assuming the library ‘libhello.a’ is stored in the current directory:
$ gcc -Wall main.c libhello.a -o hello
The main program is linked against the object files found in the library file ‘libhello.a’ to produce the final executable.
The short-cut library linking option -l
can also be used to link the program, without needing to specify the full filename of the library explicitly:
$ gcc -Wall -L. main.c -lhello -o hello
The option -L.
is needed to add the current directory to the library search path. The resulting executable can be run as usual:
$ ./hello Hello, everyone! Goodbye!
It displays the output from both the hello
and bye
functions defined in the library.
Debugging GCC
Examining core files
In addition to allowing programs to be run under the debugger, an important benefit of the -g
option is the ability to examine the cause of a program crash from a "core dump".
When a program exits abnormally (i.e. crashes) the operating system can write out a core file (usually named ‘core’) which contains the in-memory state of the program at the time it crashed. This file is often referred to as a core dump.(14) Combined with information from the symbol table produced by -g
, the core dump can be used to find the line where the program stopped, and the values of its variables at that point.
This is useful both during the development of software and after deployment--it allows problems to be investigated when a program has crashed "in the field".
Here is a simple program containing an invalid memory access bug, which we will use to produce a core file:
int foo (int *p); int main (void) { int *p = 0; /* null pointer */ return foo (p); } int foo (int *p) { int y = *p; return y; }
The program attempts to dereference a null pointer p
, which is an invalid operation. On most systems, this will cause a crash. (15)
In order to be able to find the cause of the crash later, we will need to compile the program with the -g
option:
$ gcc -Wall -g null.c
Note that a null pointer will only cause a problem at run-time, so the option -Wall
does not produce any warnings.
Running the executable file on an x86 GNU/Linux system will cause the operating system to terminate the program abnormally:
$ ./a.out Segmentation fault (core dumped)
Whenever the error message ‘core dumped’ is displayed, the operating system should produce a file called ‘core’ in the current directory.(16) This core file contains a complete copy of the pages of memory used by the program at the time it was terminated. Incidentally, the term segmentation fault refers to the fact that the program tried to access a restricted memory "segment" outside the area of memory which had been allocated to it.
Some systems are configured not to write core files by default, since the files can be large and rapidly fill up the available disk space on a system. In the GNU Bashshell the command ulimit -c
controls the maximum size of core files. If the size limit is zero, no core files are produced. The current size limit can be shown by typing the following command:
$ ulimit -c 0
If the result is zero, as shown above, then it can be increased with the following command to allow core files of any size to be written:(17)
$ ulimit -c unlimited
Note that this setting only applies to the current shell. To set the limit for future sessions the command should be placed in an appropriate login file, such as‘.bash_profile’ for the GNU Bash shell.
Core files can be loaded into the GNU Debugger gdb
with the following command:
$ gdb EXECUTABLE-FILE CORE-FILE
Note that both the original executable file and the core file are required for debugging--it is not possible to debug a core file without the corresponding executable. In this example, we can load the executable and core file with the command:
$ gdb a.out core
The debugger immediately begins printing diagnostic information, and shows a listing of the line where the program crashed (line 13):
$ gdb a.out core Core was generated by `./a.out'. Program terminated with signal 11, Segmentation fault. Reading symbols from /lib/libc.so.6...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 #0 0x080483ed in foo (p=0x0) at null.c:13 13 int y = *p; (gdb)
The final line (gdb)
is the GNU Debugger prompt--it indicates that further commands can be entered at this point.
To investigate the cause of the crash, we display the value of the pointer p
using the debugger print
command:
(gdb) print p $1 = (int *) 0x0
This shows that p
is a null pointer (0x0
) of type ‘int *’, so we know that dereferencing it with the expression *p
in this line has caused the crash.
Displaying a backtrace
The debugger can also show the function calls and arguments up to the current point of execution--this is called a stack backtrace and is displayed with the commandbacktrace
:
(gdb) backtrace #0 0x080483ed in foo (p=0x0) at null.c:13 #1 0x080483d9 in main () at null.c:7
In this case, the backtrace shows that the crash occurred at line 13 after the function foo
was called from main
with an argument of p=0x0
at line 7 in ‘null.c’. It is possible to move to different levels in the stack trace, and examine their variables, using the debugger commands up
and down
.
Setting a breakpoint
A breakpoint stops the execution of a program and returns control to the debugger, where its variables and memory can be examined before continuing. Breakpoints can be set for specific functions, lines or memory locations with the break
command.
To set a breakpoint on a specific function, use the command break function-name
. For example, the following command sets a breakpoint at the start of the main
function in the program above:
$ gdb a.out (gdb) break main Breakpoint 1 at 0x80483c6: file null.c, line 6.
The debugger will now take control of the program when the function main
is called. Since the main
function is the first function to be executed in a C program the program will stop immediately when it is run:
(gdb) run Starting program: a.out Breakpoint 1, main () at null.c:6 6 int *p = 0; /* null pointer */ (gdb)
The display shows the line that will be executed next (the line number is shown on the left). The breakpoint stops the program before the line is executed, so at this stage the pointer p
is undefined and has not yet been set to zero.
Stepping through the program
To move forward and execute the line displayed above, use the command step
:
(gdb) step 7 return foo (p);
After executing line 6, the debugger displays the next line to be executed. The pointer p
will now have been set to zero (null):
(gdb) print p $1 = (int *) 0x0
The command step
will follow the execution of the program interactively through any functions that are called in the current line. If you want to move forward without tracing these calls, use the command next
instead.
Modifying variables
To temporarily fix the null pointer bug discovered above, we can change the value of p
in the running program using the set variable
command.
Variables can be set to a specific value, or to the result of an expression, which may include function calls. This powerful feature allows functions in a program to be tested interactively through the debugger.
In this case we will interactively allocate some memory for the pointer p
using the function malloc
, storing the value 255 in the resulting location:
(gdb) set variable p = malloc(sizeof(int)) (gdb) print p $2 = (int *) 0x40013f98 (address allocated by malloc
) (gdb) set variable *p = 255 (gdb) print *p $3 = 255
If we now continue stepping through the program with the new value of p
the previous segmentation fault will not occur:
(gdb) step foo (p=0x40013f98) at null.c:13 13 int y = *p; (gdb) step 14 return y;
Continuing execution
The command finish
continues execution up to the end of the current function, displaying the return value:
(gdb) finish Run till exit from #0 0x08048400 in foo (p=0x40013f98) at null.c:15 0x080483d9 in main () at null.c:7 7 return foo (p); Value returned is $13 = 255
To continue execution until the program exits (or hits the next breakpoint) use the command continue
,
(gdb) continue Continuing. Program exited with code 0377.
Note that the exit code is shown in octal (0377 base 8 = 255 in base 10).
Shared libraries and static libraries
External libraries are usually provided in two forms: static libraries and shared libraries. Static libraries are the ‘.a’ files seen earlier. When a program is linked against a static library, the machine code from the object files for any external functions used by the program is copied from the library into the final executable.
Shared libraries are handled with a more advanced form of linking, which makes the executable file smaller. They use the extension ‘.so’, which stands for shared object.
An executable file linked against a shared library contains only a small table of the functions it requires, instead of the complete machine code from the object files for the external functions. Before the executable file starts running, the machine code for the external functions is copied into memory from the shared library file on disk by the operating system--a process referred to as dynamic linking.
Dynamic linking makes executable files smaller and saves disk space, because one copy of a library can be shared between multiple programs. Most operating systems also provide a virtual memory mechanism which allows one copy of a shared library in physical memory to be used by all running programs, saving memory as well as disk space.
Furthermore, shared libraries make it possible to update a library without recompiling the programs which use it (provided the interface to the library does not change).
Because of these advantages gcc
compiles programs to use shared libraries by default on most systems, if they are available. Whenever a static library ‘libNAME.a’would be used for linking with the option -lNAME
the compiler first checks for an alternative shared library with the same name and a ‘.so’ extension.
It is possible for the system administrator to set the LD_LIBRARY_PATH
variable for all users, by adding it to a default login script, such as ‘/etc/profile’. On GNU systems, a system-wide path can also be defined in the loader configuration file ‘/etc/ld.so.conf’.
Alternatively, static linking can be forced with the -static
option to gcc
to avoid the use of shared libraries:
$ gcc -Wall -static -I/opt/gdbm-1.8.3/include/ -L/opt/gdbm-1.8.3/lib/ dbmain.c -lgdbm
This creates an executable linked with the static library ‘libgdbm.a’ which can be run without setting the environment variable LD_LIBRARY_PATH
or putting shared libraries in the default directories:
Environment variables
The search paths for header files and libraries can also be controlled through environment variables in the shell. These may be set automatically for each session using the appropriate login file, such as ‘.bash_profile’ in the case of GNU Bash.
Additional directories can be added to the include path using the environment variable C_INCLUDE_PATH
(for C header files) or CPLUS_INCLUDE_PATH
(for C++ header files). For example, the following commands will add ‘/opt/gdbm-1.8.3/include’ to the include path when compiling C programs:
$ C_INCLUDE_PATH=/opt/gdbm-1.8.3/include $ export C_INCLUDE_PATH
and similarly for C++ programs:
$ CPLUS_INCLUDE_PATH=/opt/gdbm-1.8.3/include $ export CPLUS_INCLUDE_PATH
This directory will be searched after any directories specified on the command line with the option -I
, and before the standard default directories (such as‘/usr/local/include’ and ‘/usr/include’). The shell command export
is needed to make the environment variable available to programs outside the shell itself, such as the compiler--it is only needed once for each variable in each shell session, and can also be set in the appropriate login file.(8)
Similarly, additional directories can be added to the link path using the environment variable LIBRARY_PATH
. For example, the following commands will add ‘/opt/gdbm-1.8.3/lib’ to the link path:
$ LIBRARY_PATH=/opt/gdbm-1.8.3/lib $ export LIBRARY_PATH
This directory will be searched after any directories specified on the command line with the option -L
, and before the standard default directories (such as‘/usr/local/lib’ and ‘/usr/lib’).
With the environment variable settings given above the program ‘dbmain.c’ can be compiled without the -I
and -L
options,
$ gcc -Wall dbmain.c -lgdbm
because the default paths now use the directories specified in the environment variables C_INCLUDE_PATH
and LIBRARY_PATH
. The same compilation command with g++
would use the environment variables CPLUS_INCLUDE_PATH
and LIBRARY_PATH
.
Following the standard Unix convention for search paths, several directories can be specified together in an environment variable as a colon separated list:
DIR1:DIR2:DIR3:...
The directories are then searched in order from left to right. A single dot ‘.’ can be used to specify the current directory.(9)
For example, the following settings create default include and link paths for packages installed in the current directory ‘.’ and the ‘include’ and ‘lib’ directories under‘/opt/gdbm-1.8.3’ and ‘/net’ respectively:
$ C_INCLUDE_PATH=.:/opt/gdbm-1.8.3/include:/net/include $ LIBRARY_PATH=.:/opt/gdbm-1.8.3/lib:/net/lib
Source :
An Introduction to GCC - for the GNU compilers gcc and g++ |
Setting search paths
A common problem when compiling a program using library header files is the error:
FILE.h: No such file or directory
This occurs if a header file is not present in the standard include file directories used by gcc
. A similar problem can occur for libraries:
/usr/bin/ld: cannot find library
This happens if a library used for linking is not present in the standard library directories used by gcc
.
By default, gcc
searches the following directories for header files:
/usr/local/include/ /usr/include/
and the following directories for libraries:
/usr/local/lib/ /usr/lib/
The list of directories for header files is often referred to as the include path, and the list of directories for libraries as the library search path or link path.
The directories on these paths are searched in order, from first to last in the two lists above.(7) For example, a header file found in ‘/usr/local/include’ takes precedence over a file with the same name in ‘/usr/include’. Similarly, a library found in ‘/usr/local/lib’ takes precedence over a library with the same name in‘/usr/lib’.
When additional libraries are installed in other directories it is necessary to extend the search paths, in order for the libraries to be found. The compiler options -I
and-L
add new directories to the beginning of the include path and library search path respectively.
The following example program uses a library that might be installed as an additional package on a system--the GNU Database Management Library (GDBM). The GDBM Library stores key-value pairs in a DBM file, a type of data file which allows values to be stored and indexed by a key (an arbitrary sequence of characters). Here is the example program ‘dbmain.c’, which creates a DBM file containing a key ‘testkey’ with the value ‘testvalue’:
#include#include int main (void) { GDBM_FILE dbf; datum key = { "testkey", 7 }; /* key, length */ datum value = { "testvalue", 9 }; /* value, length */ printf ("Storing key-value pair... "); dbf = gdbm_open ("test", 0, GDBM_NEWDB, 0644, 0); gdbm_store (dbf, key, value, GDBM_INSERT); gdbm_close (dbf); printf ("done.\n"); return 0; }
The program uses the header file ‘gdbm.h’ and the library ‘libgdbm.a’. If the library has been installed in the default location of ‘/usr/local/lib’, with the header file in‘/usr/local/include’, then the program can be compiled with the following simple command:
$ gcc -Wall dbmain.c -lgdbm
Both these directories are part of the default gcc
include and link paths.
However, if GDBM has been installed in a different location, trying to compile the program will give the following error:
$ gcc -Wall dbmain.c -lgdbm dbmain.c:1: gdbm.h: No such file or directory
For example, if version 1.8.3 of the GDBM package is installed under the directory ‘/opt/gdbm-1.8.3’ the location of the header file would be,
/opt/gdbm-1.8.3/include/gdbm.h
which is not part of the default gcc
include path. Adding the appropriate directory to the include path with the command-line option -I
allows the program to be compiled, but not linked:
$ gcc -Wall -I/opt/gdbm-1.8.3/include dbmain.c -lgdbm /usr/bin/ld: cannot find -lgdbm collect2: ld returned 1 exit status
The directory containing the library is still missing from the link path. It can be added to the link path using the following option:
-L/opt/gdbm-1.8.3/lib/
The following command line allows the program to be compiled and linked:
$ gcc -Wall -I/opt/gdbm-1.8.3/include -L/opt/gdbm-1.8.3/lib dbmain.c -lgdbm
This produces the final executable linked to the GDBM library. Before seeing how to run this executable we will take a brief look at the environment variables that affect the -I
and -L
options.
Note that you should never place the absolute paths of header files in #include
statements in your source code, as this will prevent the program from compiling on other systems. The -I
option or the INCLUDE_PATH
variable described below should always be used to set the include path for header files.
Linking with external libraries
library is a collection of precompiled object files which can be linked into programs. The most common use of libraries is to provide system functions, such as the square root function sqrt
found in the C math library.
Libraries are typically stored in special archive files with the extension ‘.a’, referred to as static libraries. They are created from object files with a separate tool, the GNU archiver ar
, and used by the linker to resolve references to functions at compile-time. We will see later how to create libraries using the ar
command (see section 10 Compiler-related tools). For simplicity, only static libraries are covered in this section--dynamic linking at runtime using shared libraries will be described in the next chapter.
The standard system libraries are usually found in the directories ‘/usr/lib’ and ‘/lib’.(5) For example, the C math library is typically stored in the file‘/usr/lib/libm.a’ on Unix-like systems. The corresponding prototype declarations for the functions in this library are given in the header file ‘/usr/include/math.h’. The C standard library itself is stored in ‘/usr/lib/libc.a’ and contains functions specified in the ANSI/ISO C standard, such as ‘printf’---this library is linked by default for every C program.
Here is an example program which makes a call to the external function sqrt
in the math library ‘libm.a’:
#include#include int main (void) { double x = sqrt (2.0); printf ("The square root of 2.0 is %f\n", x); return 0; }
Trying to create an executable from this source file alone causes the compiler to give an error at the link stage:
$ gcc -Wall calc.c -o calc /tmp/ccbR6Ojm.o: In function `main': /tmp/ccbR6Ojm.o(.text+0x19): undefined reference to `sqrt'
The problem is that the reference to the sqrt
function cannot be resolved without the external math library ‘libm.a’. The function sqrt
is not defined in the program or the default library ‘libc.a’, and the compiler does not link to the file ‘libm.a’ unless it is explicitly selected. Incidentally, the file mentioned in the error message‘/tmp/ccbR60jm.o’ is a temporary object file created by the compiler from ‘calc.c’, in order to carry out the linking process.
To enable the compiler to link the sqrt
function to the main program ‘calc.c’ we need to supply the library ‘libm.a’. One obvious but cumbersome way to do this is to specify it explicitly on the command line:
$ gcc -Wall calc.c /usr/lib/libm.a -o calc
The library ‘libm.a’ contains object files for all the mathematical functions, such as sin
, cos
, exp
, log
and sqrt
. The linker searches through these to find the object file containing the sqrt
function.
Once the object file for the sqrt
function has been found, the main program can be linked and a complete executable produced:
$ ./calc The square root of 2.0 is 1.414214
The executable file includes the machine code for the main function and the machine code for the sqrt
function, copied from the corresponding object file in the library ‘libm.a’.
To avoid the need to specify long paths on the command line, the compiler provides a short-cut option ‘-l’ for linking against libraries. For example, the following command,
$ gcc -Wall calc.c -lm -o calc
is equivalent to the original command above using the full library name ‘/usr/lib/libm.a’.
In general, the compiler option -lNAME
will attempt to link object files with a library file ‘libNAME.a’ in the standard library directories. Additional directories can specified with command-line options and environment variables, to be discussed shortly. A large program will typically use many -l
options to link libraries such as the math library, graphics libraries and networking libraries.
GCC-Using library header files
When using a library it is essential to include the appropriate header files, in order to declare the function arguments and return values with the correct types. Without declarations, the arguments of a function can be passed with the wrong type, causing corrupted results.
The following example shows another program which makes a function call to the C math library. In this case, the function pow
is used to compute the cube of two (2 raised to the power of 3):
#includeint main (void) { double x = pow (2.0, 3.0); printf ("Two cubed is %f\n", x); return 0; }
However, the program contains an error--the #include
statement for ‘math.h’ is missing, so the prototype double pow (double x, double y)
given there will not be seen by the compiler.
Compiling the program without any warning options will produce an executable file which gives incorrect results:
$ gcc badpow.c -lm $ ./a.out Two cubed is 2.851120 (incorrect result, should be 8)
The results are corrupted because the arguments and return value of the call to pow
are passed with incorrect types.(6) This can be detected by turning on the warning option -Wall
:
$ gcc -Wall badpow.c -lm badpow.c: In function `main': badpow.c:6: warning: implicit declaration of function `pow'
This example shows again the importance of using the warning option -Wall
to detect serious problems that could otherwise easily be overlooked.
Thursday, August 20, 2009
How to Bulk Rename Files in Linux (Terminal or GUI)
If you have a directory of files that you would like to bulk rename, you can use the rename command from the terminal.
The syntax for the rename command is:
rename [ -v ] [ -n ] [ -f ] perlexpr [ files ]
-v means "verbose" and it will output the names of the files when it renames them. It is a good idea to use this feature so you can keep track of what is being renamed. It is also a good idea to do a test run with -n which will do a test run where it won't rename any files, but will show you a list of files that would be renamed.
The "perlexpr" part of the command is a Perl expression. Don't panic yet...
The "rename" command in action
Here is an example of the rename command:
rename -n ’s/\.htm$/\.html/’ *.htm
The -n means that it's a test run and will not actually change any files. It will show you a list of files that would be renamed if you removed the -n. In the case above, it will convert all files in the current directory from a file extension of .htm to .html.
If the output of the above test run looked ok then you could run the final version:
rename -v ’s/\.htm$/\.html/’ *.htm
The -v is optional, but it's a good idea to include it because it is the only record you will have of changes that were made by the rename command as shown in the sample output below:
$ rename -v 's/\.htm$/\.html/' *.htm 3.htm renamed as 3.html 4.htm renamed as 4.html 5.htm renamed as 5.html
The tricky part in the middle is a Perl substitution with regular expressions, highlighted below:
rename -v ’s/\.htm$/\.html/’ *.htm
Tip: There is an intro to Perl regular expressions here.
Basically the "s" means substitute. The syntax is s/old/new/ — substitute the old with the new.
A . (period) has a special meaning in a regular expression — it means "match any character". We don't want to match any character in the example above. It should match only a period. The backslash is a way to "escape" the regular expression meaning of "any character" and just read it as a normal period.
The $ means the end of the string. \.htm$ means that it will match .htm but not .html.
It's fairly basic — substitute .htm with .html:
's/\.htm$/\.html/'
The last part of the command, highlighted below, means to apply the rename command to every file that ends with .htm (the * is a wildcard).
rename -v ’s/\.htm$/\.html/’ *.htm
Other Examples
Maybe you have a digital camera that takes photos with filenames something like 00001234.JPG, 00001235.JPG, 00001236.JPG. You could make the .JPG extension lowercase with the following command executed from the same directory as the images:
rename -v 's/\.JPG$/\.jpg/' *.JPG
Here is the output of the above command:
$ rename -v 's/\.JPG$/\.jpg/' *.JPG 00001111.JPG renamed as 00001111.jpg 00001112.JPG renamed as 00001112.jpg 00001113.JPG renamed as 00001113.jpg
That is simple enough, as it is similar to the .html example earlier. You could also bulk rename them with something descriptive at the beginning like this:
Tip: Before trying more complicated renaming like in the example below, do a test run with the -n option as described at the beginning of this tutorial.
rename -v 's/(\d{8})\.JPG$/BeachPics_$1\.jpg/' *.JPG
That will change filenames that have the pattern ########.JPG (8 numbers and capital .JPG) to something like BeachPics_########.jpg (the same 8 numbers and changing the extension to lowercase .jpg). Here is a test run with the -n option:
$ rename -n 's/(\d{8})\.JPG$/BeachPics_$1\.jpg/' *.JPG 00001111.JPG renamed as BeachPics_00001111.jpg 00001112.JPG renamed as BeachPics_00001112.jpg 00001113.JPG renamed as BeachPics_00001113.jpg
Here's a quick breakdown of the Perl substitution with the regular expression above.
The highlighted section below means to count 8 digits. The parentheses mean to save those 8 digits for later because they are going to be used again in the second half of the substitution:
's/(\d{8})\.JPG$/BeachPics_$1\.jpg/'
In the highlighted section below, it adds the string BeachPics, and underscore, and then the text in parentheses from the first half of the substitution. $1 will insert the string from the first set of parentheses that it finds — in this case the 8 digits. If you have more than one set of parentheses you can access the second set with the Perl variable $2 and so on.
's/(\d{8})\.JPG$/BeachPics_$1\.jpg/'
Final Refinement
The following variation would make even cleaner-looking filenames. See if you can figure out how it works:
$ rename -n 's/\d{5}(\d{3})\.JPG$/BeachPics_$1\.jpg/' *.JPG 00000123.JPG renamed as BeachPics_123.jpg 00000124.JPG renamed as BeachPics_124.jpg 00000125.JPG renamed as BeachPics_125.jpg
Wednesday, August 19, 2009
DCP package creator -Manuals
NOM
mkcpl - Create a composition playlist
SYNOPSIS
mkcpl --kind contentkind --title "title" picture.mxf sound.mxf [picture2.mxf sound2.mxf...]
mkcpl --kind contentkind --title "title" [--annotation "annotation"] [--issuer "issuer"] picture.mxf sound.mxf
mkcpl --kind contentkind --title "title" [--ratio "aspectratio"] picture.mxf sound.mxf
mkcpl --kind contentkind --title "title" [--versionid uid] [--versionlabel label] picture.mxf sound.mxf
mkcpl --kind contentkind --title "title" [--rating agency label] picture.mxf sound.mxf
mkcpl --kind contentkind --title "title" [--norating] picture.mxf sound.mxf
mkcpl [-h | --help] [-V | --version]
DESCRIPTION
Creates a composition playlist displayed on the standard output. You need to redirect the output to create a file. See examples below.
OPTIONS
-h | --help
Show help
-V | --version
Show version information
--kind contentkind
Content kind. Could be one of the following: feature, trailer, test, teaser, rating, advertisement, short, transitional, psa, policy
--title "title"
Content title.
--annotation "annotation"
Annotation text
--issuer "issuer"
Issuer name
--ratio "aspectratio" Aspect ratio. Default is taken from the asset. This option is to override the asset aspect ratio.
--versionid uid Version UID. Default generates a new id.
--versionlabel label
Version label. Default is the versionid.
--rating agency label
Defines a rating agency and label. Default is --rating http://www.mpaa.org G
--norating
Removes the default rating.
RETURN CODES
mkcpl has the following return code:
0 Success
other return codes need to be added here |
EXAMPLE
Single reel feature with default rating
mkcpl --kind feature --title "my movie" mypicture.mxf mysound.mxf > mymoviecompo.cpl.xml
Single reel test with no rating
mkcpl --kind test --title "my test" -norating mypicture.mxf mysound.mxf > mytestcompo.cpl.xml
Single reel feature with a PG-13 rating
mkcpl --kind feature --title "my movie" --rating http://www.mpaa.org PG-13 mypicture.mxf mysound.mxf > mymoviecompo.cpl.xml
Multi reel feature
The 3 examples have the same results. The picture/sound order doesn't matter.
The mkcpl application associates the first picture in the list with the first sound in the list and so on.
mkcpl --kind feature --title "my movie" mypicture-1.mxf mysound-1.mxf mypicture-2.mxf mysound-2.mxf > mymoviecompo.cpl.xml
mkcpl --kind feature --title "my movie" mysound-1.mxf mypicture-1.mxf mypicture-2.mxf mysound-2.mxf > mymoviecompo.cpl.xml
mkcpl --kind feature --title "my movie" mypicture-1.mxf mypicture-2.mxf mysound-1.mxf mysound-2.mxf > mymoviecompo.cpl.xml
NOM
mkmap - Create an asset map
SYNOPSIS
mkmap file[file...]
mkmap [--annotation "annotation"] [--issuer "issuer"] file[file...]
mkmap[-h | --help] [-V | --version]
DESCRIPTION
Creates ASSETMAP and VOLINDEX files in the current directory. The files that can be added to the asset map are .mxf(picture or sound), composition playlists and packing lists You can add as many files as you want if they are of the previously described types.
OPTIONS
-h | --help
Show help
-V | --version
Show version information
--annotation "annotation"
Annotation text
--issuer "issuer"
Issuer name
RETURN CODES
mkpkl has the following return code:
0 Success
other return codes need to be added here |
EXAMPLE
Simple asset map
mkmap mypicture.mxf mysound.mxf mymoviecompo.cpl.xml mymoviepackage.pkl.xml
Asset map with issuer and annotation text
mkmap --issuer "me" --annotation "an annotation" mypicture.mxf mysound.mxf mymoviecompo.cpl.xml mymoviepackage.pkl.xml