Matt Gourley's TechBlog

- Allen and I have been successful this week in getting linux11 and linux12 migrated over to the BaseVol/GuestVol setup, based on our successes with linux15 last week. We need to try to get linux1, 3, 5 and 9 migrated also, and those provide their own challenges.

- Allen and I also spent some time Tuesday explaining to Ed and Brian what the architecture is, and how it works.

- Ran into some issues with UPSTREAM backup jobs on both IPO1 and IPO2. These were easy to figure out:
- The IPO1 job ZSIF003 failed because linux2 was down, and is not likely to come back.
- The IPO2 job ZSIF017 failed because 6lnx7 was having network routing issues (which were resolved by modifying /etc/sysconfig/network/routes) and because the upstream daemon did not start after the 4/2 IPL.

I suggested that ZSIF003 and ZSIF004 (linux2:/ and linux2:/opt) not run until Brian and I have determined linux2's status. Everything should be fine for this weekend.

Final steps on the BaseVol/GuestVol shared root, shared WAS system linux15:

- I've increased the size of the R/W /guestvol filesystem on lxtst00 and linux15 to 500 cyl. This was not as easy as I would have liked it to be, mostly because linux15 insisted on attempting to run fsck(1) on the mounted filesystem during boot. However, it's back on-line with a larger R/W partition.

- I discovered this morning that rebooting the system will not automatically mount linux15's 203 and 204 disks (/wasprofiles and /opt/IBM/WebSphere). I'm writing a little script to be inserted into the boot process between the completion of the boot and the starting of the applications, just to make sure any unmounted ext2 and ext3 filesystems are mounted.

- Dom is going to go through re-creating linux15 for WAS, and we're hoping to have it available by the end of the day.

- We have no plans to migrate linux11 and linux12 to the new shared root. Both lxtst00 and linux15 are running on test 3390-3 volumes, and these volumes might go back to Gary at any time.

Yesterday, using the Basevol/Guestvol shared root setup, and the shared WAS setup, I created linux15. Today, Daryl and Dom and I got WAS running on linux15. Daryl already wants to start migrating the shared root to linux11 and linux12, but I'm content to sit on this to make sure it runs well. I'm not 100% sure we've got the right amount of R/W DASD allocated, and I want to see if linux15 has issues.

Allen and I have gotten our BaseServer/GuestServer Shared Root setup. There's still a whole bunch of testing associated with it that needs to happen, including software installation, patches, etc while the GuestServers are running.

I finally, after much pain, got a clean installation of SLES9 64-bit onto LXTST00. I think it was something on my workstation's end.

Tomorrow: getting the thing setup for being a basevol in a basevol/guestvol configuration.

Done:

- Added two 3390-3s to VM7 and formatted them.
- Created the user LXTST00 in by editing USER DIRECT. Ran DISKMAP and DIRECTXA (EDIT to make sure there were no config problems; ran DIRECTXA to create.

For tomorrow:

- Get with Brian to figure out how to couple LXTST00 to the SYSTEM VSWITCH1.
- Create LNXMNT:LXTST00.PARMFILE.
- Modify linux00:/usr/local/sbin/clone.sh to clone to LXTST00's 150 disk.
- Clone LXTST00
- Boot LXTST00

I know I've been slacking on writing my updates here, for which I apologise. Here's the latest:

1) UPSTREAM backups on IPO2 are now running through Hipersockets. The change from handshaked to fixed MTUs for TCP/IP (1500) is my guess as to why this is working.

2) Today, after a hiccup on the connection between linux7 and IPO1 (mis-configured client), I got the UPSTREAM backups on IPO1 running through Hipersockets.

3) sudo(1) is now setup in production. It was determined that the db2 user on linux5 did not require sudo, and we had difficulties getting the DB2 X applications to run through it anyway.

4) Plenty of updates and software installations in production, mostly of the IBM JRE.

5) Allen and I are working on a plan for a shared root filesystem for our Linux images. I'm less concerned right now with getting a production-ready architecture than I am with getting the proof of concept running. Gary has just given us a few Mod-3s for testing, and I'm hoping to have a few roadblocks crashed through by the end of next week.

On the plate:

- Security auditing.
- Documentation on sudo(1).
- Documentation on UPSTREAM backups.
- Log rotation
- Omegamon documentation

Production jobs are submitted to Clyde.

Time to get back to working on the sudoers file to get sudo(1) working on the linux systems.

Made the Perm File requests for the GDG aliases for all of our production backup jobs.
Created the JCL in Roscoe for all 12 of our production backup jobs.
Copied the UPSTREAM PARM members for the production backup jobs.

Still to do:
Edit the UPSTREAM PARM members.
Print out the JCL for the jobs, and double-check.
Fill out Production Turnover forms for the production backup jobs, & turn into Clyde.

I've gotten UPSTREAM running on IPO2. Now it's time to get the backup system setup in production like we have it in test.

We're not done yet. Gary set up a new volume for IPO2's database files, which got moved to it. Hopefully it will be enough.

TCPIP stack stuff with Brian in the morning, then we should be ready to test.

Here's how my morning has gone:

1) Ensure UPSTREAM is APF-authorised to run on IPO2. It is (D PROG,APF on the console shows me this).

2) Setup the UPSTREAM database for IPO2. This involves creating three VSAM datasets using UPSTREAM's proprietary format, and didn't go as well as I'd have liked. I tried to allocate a 4,000 cyl dataset on a volume that only has 3,400 cyls left, and the job abended like it should. Gary is getting a mod-9 together for me, and I'm going to move all of the IPO2 datasets over to it. Considering one of the datasets has a record for every object that is backed-up and in a cataloged backup, this could get big.

3) I've double-checked USR1.NETPROCS(NET2) to make sure USR1.VTAMLT00 is in the VTAMLST. This means IPO2 has the proper VTAM application IDs defined for it.

4) Once we've gotten past step 2, I need to work on the TCPIP stack on IPO2. At a minimum, I need to add the TCP port for UPSTREAM. We do plan on getting Hipersockets setup.

Which wasn't done beforehand.

The Omegamon I did last week needs one last configuration step, but I can do that later. Now, I'm a bit curious as to what happens when I clone a system. But that's going to have to wait until after UPSTREAM is on IPO2.

Two of our jobs failed last night and this morning. The folks who are supposed to provide us with data haven't been. The frustrating thing, of course, is this is two weeks in a row the Sunday job has failed. The folks who need to contact the people who need to fix this have done so.

We did an IML this morning, and found that the golden image server was not on VM's AUTOLOG list. I also needed to get some of the rpms installed for installing WAS and IHS. Mostly tedious stuff so far.

Back to sudo this afternoon.

Whoo-hoo! Installed.

Next: sudo() changes.

But first: lunch.

The nfs updates having been completed on linux8, I find out that IBM-JRE is not installed by default on the golden image. Time to IPL back on the 103 disk, chroot() to /sles9golden, and run YaST to install this, then back to 100 to (maybe) install the client code.

I'm running into the same NFS issues on linux8 (which has the Omegamon code) that I had on linux00 (which is both the golden image and the software install server). I'm updating linux8 right now, over 109MB of updates. As Gadget would say, "Wowsers!"

1) Getting UPSTREAM running on IPO2
2) Setting up the test and production golden images with Omegamon code.
3) sudo() access for the Application Integration team.
4) Learn about the Java and DB2 tweaks done the week I was at SHARE.

I spent Monday and Tuesday getting backups setup for linux11 and linux12, and for the DB2 portion of linux5. I would like to get UPSTREAM for DB2 working, but Brian says that can wait until after the March 15th deadline.