Oracle Database loses its OEM configuration after a Cold Backup

This was an annoying problem that took awhile to track down. In short: after our scheduled cold backups, an Oracle (11g) database would lose its configuration in Oracle Enterprise Manager. It would present a "Metric Collection Error" that would go away after reconfiguring the database. The fix, as it turned out, was pretty simple, but it took awhile to tease out. The problem was that the trace directory (bdump in 10g) was too full. Specifically, the metric collection (the process by which OEM gathers data about the database) was timing out. We ruled out performance problems on the database side; the system is not utilized much at all. Instead, we discovered that because there were a lot of files in the trace directory (> 31k), it was taking a long time for the OEM agent to get to the alert log, which is one of the metrics that it collects. This was hinted at in the emagent.trc file:

2010-01-04 13:01:53,087 Thread-47647632 ERROR TargetManager: TIMEOUT reached in computing dynamic properties for target TESTDB,
oracle_database::compute timings were [decideIncludeDB:0-0] [SystemTablespaceNumber:0-0] [SysauxTablespaceNumber:0-0 ...
[DeduceAlertLogFile:1-1] [GetCPUCount:1-1] [EnabledFeatures:1-1] [GetOSMInstance:1-1] [GetNLSParam:1-1] [GetAdrBase:1-(1)]

So you can see above that one of the things it was trying to do was get at the alert log. It took a long time to enumerate all of the small files in the trace directory, so we shut down the instance, cleared out the trace directory, and restarted the instance. That took care of the problem. In troubleshooting this problem, we also increased the dynamic properties timeout setting (dynamicPropsComputeTimeout_oracle_database) in the emd.properties file (in [agent_home]/sysman/config), changing the value from the default (120) to a larger setting (240). That did not help, though it's a good troubleshooting step, should you run into a similar problem.

Rebuilding RPMs in RHEL

How to rebuild a RPM from source Reconfiguring and rebuilding RPMs can be immensely useful, particularly when, as in the case of AMANDA backup software, you need to make a change that is used over and over again on a lot of clients. The steps are pretty straightforward. In short: To tweak an existing package

rpm -i .src.rpm

This puts all sources into /usr/src/redhat/SOURCES/ and the .spec file into /usr/src/redhat/SPECS/ which you can change as required To change the compile settings Edit the /usr/src/redhat/SPECS/.spec file to meet your needs To build the binary Then run the following command to build the RPM:

rpm -bs /usr/src/redhat/SPECS/.spec

rebuilds the RPM from the modified source. -- How to rebuild the AMANDA source to create new install RPMs This is an important thing to do: AMANDA by default picks random high tcp ports on which to communicate with the clients. This is a problem when we cross subnets, in particular, since the firewall needs to know which ports are needed. So we recompile the source to include the --withtcpportrange= and --withudpportrange= switches. This will limit AMANDA to the appropriate port ranges. This is from http://wiki.centos.org/HowTos/AmandaBackups 1. Download the amanda-xxxxx.src.rpm file. 2. Install the source rpm: rpm -i amanda-2xxxxx.src.rpm. This will extract the contents into your rpm directory (if you're doing it as root, it'll be /usr/src/redhat/.) 3. Edit the SPECS/amanda.spec file to reflect the appropriate changes. It's a good idea to changed the Release: tag to indicate you've made changes. In addition to just helping you keep track of what you've changed from the default, this will help keep the package from being updated automatically when you patch the system. To set up AMANDA to use specific ports that we can open up on the firewall, we want to add the following to the ./configure command. Don't forget to put the trailing back-slashes at the end of the lines.

--with-tcpportrange=50000,50100 \ --with-udpportrange=700,710

4. Rebuild the rpms:

rpmbuild -bb --define "build_rhel5 1" /amanda.spec

or for RHEL4,

rpmbuild -bb --define "build_rhel4 1" /amanda.spec