Privacy and Security Notice

    Computer Center News

Issue 9

October 2001

Central Computing

 

Computer Center Expansion Completed

After several months of planning and execution, the expansion of the Computer Center main equipment room which began in May, has been completed successfully. The success of this project is a direct result of the combined efforts of Computer Center and many other JLab departmental staff members. The primary goal of the expansion was to add an additional data silo to bolster the storage capabilities provided for the scientific computing program. An additional benefit gained from this expansion is the increased physical space now available for the growing JLab Batch Farm, HPC effort, and central computing environment.

 

Before the expansion of the main computer equipment room could begin, it was required that several staff members be relocated from Cebaf Center to Trailer City. This relocation was necessary to provide space for equipment that would be displaced by the new data silo. Additionally, electrical requirements necessitated the installation of new electrical circuits and outlets to feed both the new data silo and relocated equipment. The timetable of the various tasks completed over the May – August time period for the expansion were as follows:

 

  • May 19 - Installation of new electrical outlets in the expansion area
  • June 6 – Removal of Computer Center offices and staff begins
  • June 20 – Expansion space finished and ready to receive equipment to be relocated

 

 

Finished Expansion Area                                                                   Populated Expansion Area

  • June 27 – Central servers jlabs1, jlabs2, ifarms1, and ifarms2 moved into new expansion area
  • July 7 – Electrical outlets in the expansion area installation completed, all farm systems and accompanying racks which needed to be relocated are moved into the new expansion area to make room for the data silo
  • July 9 – Pass through wall which joins the existing data silo and the new data silo is delivered
  • July 23 – Data silo is delivered
  • July 24 – Installation of the data silo is begun

 

Data Silos After Completion of Installation

  • August 1 –  Data silo installation completed
  • August 2 – Data silo is online and operational

 

A photographic record of the expansion can be found online at http://cc.jlab.org/general/cc_expansion/ulthm3.htm.

 

CUE Password Expiration

The CUE tool for changing your password ("jpasswd") was designed to implement our site security policies regarding system passwords. One provision of these policies currently requires that passwords be changed every 6 months.

 

The jpasswd system provides this "password aging" functionality. This feature is now being enabled. The system periodically checks all user accounts to identify those whose passwords are near or past expiration.

 

Users whose accounts are nearing expiration will receive a series of warning e-mails letting him know of the upcoming account closure. Upon expiration, these accounts will be automatically disabled and will require the user to contact the Computer Center in order to reactivate the account.

 

During the warning period (3 weeks prior to expiration), e-mail will be sent to the user, asking that he change his password and letting him know that the account will be disabled if the password is not updated soon. The warning e-mail will be repeated weekly at the start of the warning period, but will be repeated daily during the last week prior to account suspension.

 

If you have any questions or comments, please refer to the Computer Center's documentation at http://cc.jlab.org/docs/services/cue/password.html .

 

Computer Center Backup and File Recovery Procedures

The Computer Center is committed to providing reliable and redundant backups of all system and user generated data on each of the systems which it administers. The prime function of the Computer Center backups is to provide disaster recovery. This includes, for example, recovering data from occurrences of random mechanical disk failures as well as for the recovery of Jefferson Lab work and research in the event of massive physical disaster (i.e. fire, flood, and hurricane). The backups provided by the Computer Center are not designed as an archival system. If you have a need for such an archival system please contact the Computer Center to discuss these needs. Further details on backup procedures and policies can be found on the Backup and Disaster Policy page located at http://cc.jlab.org/policies/Operations/BackupDisaster.html.

 

Users are responsible to backup all information NOT on a disk managed by the Computer Center. This includes all personal computers (PC's and MACs) as well as workstations not under the aegis of the Computer Center.

 

Backups are maintained of the following centrally maintained directories that are provided from central network fileservers: home, group, apps, and site. The scratch, work, and cache directories are not backed up.

 

Backups are maintained by doing weekly images with daily incremental backups. The weekly images are kept for at least one year and the daily incrementals are kept for 45 days. This means we have the capability to restore a file from any given day for 45 days. After that, we can only restore files as they were on the weekly images.  The weekly images for user and group areas are performed on weekends. In a typical week, there is almost 800 GB of incrementals and 900 GB of images performed.

 

Support for the software that administers the backups has been discontinued at the end of this year.  This required the Computer Center to evaluate several backup solution packages to replace it.  A new package has been selected and purchased and is now in the initial implementation stages. This new software will be fully in use by the end of the year.

File Recovery

If there is a need to recover lost data from Computer Center managed filesystems or directories there are two possibilities for recovery:

 

  1. The data may be located in a snapshot directory that the user is able to access and recover the files from.
  2. The data may have to be retrieved from Computer Center tape backups.

 

Retrieving Files from Snapshot Directories

Files stored on the central fileservers have periodic "snapshots" made of each folder's contents. These snapshots include copies of files as they appeared in the past, and can be used to recover accidentally deleted or changed files without the need to go to tape backups. Every directory on the central fileservers contains a subdirectory named ".snapshot" which itself contains several folders: hourly.0, and hourly.1, … These snapshot directories can be used as a source for copies of files you may have deleted and/or corrupted.

 
Windows

As an example, suppose that you accidentally deleted the file j:\myfile on your CUE configured Windows PC. In this case, the folder J:\~snapshot\hourly.0 contains your files as they appeared at the last hourly snapshot. The folder J:\~snapshot\hourly.1 contains your files from two hours ago, and so on. To restore "myfile", you look in the ~snapshot directories for the one you want and drag it to the original location, perhaps copying J:\~snapshot\hourly.1\myfile to J:\myfile.

 

Unix

Another example would be if you had accidentally deleted the file /home/username/myfile on a CUE configured Unix system. In this case the /home/username/.snapshot/hourly.0 directory contains your files as they appeared at the last hourly snapshot. The directory /home/username/.snapshot/hourly.1 contains your files from two hours ago, and so on. To restore "myfile" you would search through the .snapshot directories that you want and copy it back to its original location. The process would be as follows:

 

  1. Change directories to the directory where the file last resided
  2. cd .snapshot
  3. cd hourly.0 (or hourly.1, hourly.2, hourly.3, nightly.0, nightly.1, nightly.2)
  4. copy desired file back to its original location or location of your choice.

 

Since the snapshot directories are maintained by the file servers for you, you cannot copy files into it. It can only be used to retrieve files, as a way to restore file that were accidentally deleted. If you delete a file, and do not notice it for a few days, it will not appear in any of the snapshot folders. If the file is needed, the Computer Center can recover it from backup tapes. Be aware that while restoring from the snapshot directory is as fast as copying a file, restoring from tape is not immediate.

 

Retrieving Files from Tape Backups

If the files that need to be restored are previous to the copies maintained in the snapshot directories a request must be made via CCPR to recover the files from tape backup. When submitting a request to recover file be sure to include the following information:

 

  • last known location of the file
  • date of the last known good version of the file
  • location to recover the retrieved file into

 

Please realize that restoration of files from tape is not immediate and can be time intensive.

 

Scientific Computing

 

SILO Changes

The second tape silo was successfully installed during the month of August.  The addition effectively doubles the number of tapes that can be kept online and available for access from 6,000 to 12,000.  These numbers are maximum values as the number of tape drives connected to the tape silo limits the actual available amount of storage space.  There is a pass-through port between the two tape silos, allowing tapes to be passed between them so that they may behave as a single large silo.

 

Along with the new tape silo came five additional T9940A tape drives.  This brings the total number of T9940A tape drives to ten.  These tape drives are to replace the older 8 Redwood tape drives. They are just as fast as the Redwoods, hold an extra 10 GBytes of data, and are Ľ the cost of a Redwood tape drive. 

 

Tape Migration

New data is no longer written to Redwoods, but goes to either T9940A or 9840 drives.  This is because the newer tape drives are far more reliable and support for the older Redwoods will end in the near future.  With support for the Redwood tape drives ending, we must migrate the data on Redwood tapes over to the new T9940A tape drives. 

 

We have already made enhancements to JASMine for doing tape migration and have begun to migrate the data from the Redwood tapes onto the new T9940A tapes.  We have placed an order for 1000 T9940A tapes to start the process of migration.  Once the tapes arrive, we will start running the migration process 24x7.  This process is expected to take many months to complete. 

 

JASMine

In addition to the tape migration enhancements to JASMine, three new user commands have been added and are being tested. These commands are jremove, jrestore, and jrename.  Once testing is completed, the commands will be made available for use.

 

The jremove command will remove a stub file from the /mss directory tree.  Since you cannot really delete a file on tape, the jremove command moves the stubfile into a hidden subdirectory called .storeattic.  A version number is also appended to the filename in case multiple stubfiles of the same name are removed over time. 

 

The jrestore command will restore a removed file to the /mss directory tree.  This is really just a move from the .storeattic subdirectory back to the original directory.  A version number must be supplied if multiple versions exist.

 

The jrename command will change the filename of a stubfile in the /mss directory tree.  It will not move stubfiles between directories.  It will only change the name of the stubfile.  This is because files are written to groups of tapes called volume sets.  The directory in which a stubfile exists or is written to determine which volume set used. 

 

Work File Servers

Three new work file servers were brought online in August.  These are Linux servers with 1 TByte of disk space and gigabit Ethernet connectivity.  The CLAS work areas were migrated to the new servers, freeing up space on the older servers for halls A and C.  These new servers are better equipped to handle the load generated by CLAS users.  Since moving the CLAS work areas to the new servers, we have not seen any problems with the CLAS work disks being unavailable because of the load on the server. 

 

Cache File Servers

The new cache file servers were also brought online in August. These are Linux servers with 900 GBytes of disk space and gigabit Ethernet connectivity.  This resulted in an increase of 2.7 TBytes to the CLAS cache areas, 900 GBytes to the HallA cache area, and 400 GBytes to the HallC cache area.  Increases were also made to the farm cache area for the pre-staging of data files from the tape silo for farm jobs.

Text Box: New 120 Farm CPU’s

Farm

The original eleven farm nodes were decommissioned in August.  They were dual Pentium II 300MHz systems.  Sixty dual Pentium III 1GHz farm nodes will replace them. These new systems have just been delivered and should be available before the end of October. They represent an increase of about 5,520 SPECint95.  This is only some 400 SPECint95 shy of doubling the processing power of the farm.  Once the new farm nodes are online, the processing power of the farm will be approximately 11,000 SPECint95.


Desktop Support

 

NCD X-terminal Warranty Service Ends

The NCD warranty service contract on all JLab NCD Xterminals has expired.

 

The warranty repair service previously contracted through NCD repair facilities is no longer be available for all remaining JLab NCD Xterminals (including HMX series and newer).  Should a failure of this brand equipment occur, the owner is responsible for its replacement. Any questions should be directed to helpdesk@jlab.org .

 

The Computer Center will stock working replacement X-terminals as far as space permits. As mentioned in previous Computer Center Newsletters the following alternatives are available as alternatives to the Xterminals:

 

  • The Thin Client solution explained below will provide either Linux or Windows Thin client terminals. 

 

  • A second alternative is to replace an Xterminal with a CUE Level 1 Linux machine. This provides the functionality of an Xterminal, plus the added benefit of easing the burden on the servers by allowing the user to run Netscape and other applications locally on their own desktop.

 

JLab Thin Clients

Due to the expiration of the NCD warranty service contract on all NCD Xterminals, the Computer Center had been searching for replacement terminals which will provide similar functionality to the NCD Xterminals. The replacement systems which have been decided upon are a range of terminals collectively known as Thin Clients. There are two different types of thin clients being recommended by the Computer Center:

 

  • a thin client for users who primarily use Windows applications
  • a thin client for users who  primarily use Unix and X windows applications.

 

The first type of client connects to a terminal server upon boot-up, and allows you to have what appears to be a complete Windows PC running on your desktop. The second type of client is a system that boots from a server and runs a Linux desktop. This configuration allows users to run Linux applications locally with the added benefit of connectivity to the terminal server to run Windows applications.

 

The Computer Center has evaluated and tested several different thin client models from Sun, Dell, NCD, IBM, ThinkNIC, and Wyse. The Sun, ThinkNIC, Dell, and NCD thin clients do provide some of the functionality of the NCD Xterminals, but have some limiting factors related to deployment and manageability. The following are the currently recommended Computer Center thin client solutions:

 

1)       Wyse Winterm 1200le - This is a Windows PC thin client which, when the system has booted (which takes less than 10 seconds from power up to login screen), presents the user a Windows 2000 login on the terminal server. X windows connectivity on the Windows 2000 desktop is provided by the Exceed Hummingbird application available by default on the terminal server. The Wyse Winterm 1200le thin client system is available for purchase from the MIS Stock application and it has a cost of approximately $400, excluding monitor.

 

2)       IBM N2800 - This thin client system boots from a server and installs RedHat Linux 7.1 locally. Users on these systems run Linux on their desktop as if they had installed it locally themselves. The applications available are those that are normally available on Linux CUE level 1 install systems. The Computer Center recommends that these systems be purchased with an additional 256MB of RAM.  This system is available from IBM Direct under a special quote for $650, excluding monitor.

 

The following client is supported and available; however it is not the client that we recommend:

 

3)       IBM N2200e - This thin client is functionally the same as the Wyse thin client mentioned above. The difference between the two systems is that Linux can be run on this system if the amount of resident memory is increased. The Linux/Windows configuration is modifiable by the Computer Center on a system by system basis.  Please note that Linux runs fairly slow on this system, and if you believe that you will be running Linux for any significant fraction of time you should choose the IBM N2800 thin client described below. This system can be purchased from IBM Direct under a special quote for $500, excluding monitor.

 

 

One of each of these systems will be installed in the public PC areas of Cebaf Center in place of the current X terminals already there.  Please drop by and check them out and provide us some feedback on these systems.

 

For more information on these systems, you can contact Kelvin Edwards at kelvin@jlab.org.

 

PC Purchasing Updates

The PC purchasing contracts that Jefferson Lab has with Dell and Micron have been updated. The current configurations have the best prices and the newest accessories for the Desktop PCs. Instructions on how to login and use of the configuration pages can be found on the Computer Center PC Purchasing pages located at http://cc.jlab.org/services/pc_purchasing/. If you have any questions or would like to see items added to the PC configurations please contact helpdesk@jlab.org.

 

Security Issues

 

Security Training for System Administrators

DOE has provided access to certified computer security courses for the Lab's system administrators. This training is provided by the SANS Institute and is one of the most highly respected certification programs available for computer security.  SANS (System Administration, Networking, and Security) has a home page at www.sans.org.

 

SANS describes the Global Information Assurance Certification (GIAC) Programs in this way:

 

“To address the lack of trained security professionals in the field, the SANS Institute has developed several core training courses, or Tracks. SANS courses were developed through a consensus process and provide education in the essential best practices and hands-on skills needed today. SANS faculty members and over one hundred security professionals contributed to the consensus process, including many of the world's most experienced front-line security and system administrators, intrusion detection analysts, consultants, auditors, and managers.”

 

The courses start with a general overview of security and then move quickly into the details of network protocols and how to secure UNIX/Linux and Microsoft NT and 2000 machines.

 

We have about a dozen individuals enlisted in the first round of this web-based training. This is a substantial commitment by those taking the course with a deadline for completion of about six months. We understand that DOE may be providing additional opportunities for free registration in these courses. If you are interested, please send a note to security@jlab.org.

 

Hoax Busters

In addition to the unsolicited email from purveyors of get-rich-quick schemes, zero-interest loans, and other less savory items, Internet mail sometimes includes messages reporting some new, extremely malicious virus that threatens world peace and prosperity or, at least, the sanctity of your computer. These messages are often characterized by an allusion to recent announcements from Microsoft, an exaggerated style (lots of exclamation points!!!, ALL UPPER CASE, etc), and instructions to "warn all your friends." Others might describe the plight of a handicapped child or person on their deathbed that deserves our money, kindness, or electronic get-well wishes.

 

Very often, this type of message is a hoax. Some of these have been circulating on the Internet for years, forwarded by well-meaning people to everyone in their address book and, thus, gaining renewed life.

 

We should all be skeptical of such messages, but it is often difficult to validate the content of a message, especially when it is a forward of a forward of a forward ...

 

Some quick help is at hand, however. One such source of information is CIAC's "Hoax Busters" page -- http://HoaxBusters.ciac.org/.  CIAC is DOE's Computer Incident Response Capability.  There is also, coincidentally, another site of nearly the same name (http://hoaxbusters.org/) that provides similar information, though perhaps a bit more general.  This second site also has a good page on "five telltale signs" of a hoax.

 

CIAC's site provides search capabilities which can be used to look for a key phase or subject line from the suspect message.  If you find a match, you can drop the message and forward the Hoax Buster reference back to the sender.

 

If no reference to the message in question is found on these pages and you feel that the message content is still important enough to send to your friends, you know you have done at least a bit to ensure that you are not a conduit for passing bad information.

 

Security Seminars

Jefferson Lab's computer security team will begin offering seminars on topics relating to computer security. Seminar topics will range from technical presentations to policy discussions. Seminars will be announced in news messages, and in the weekly Computer Center News. If you have an item you would like to see the subject of a seminar, please send email to security@jlab.org.

 

Networks

 

ARC Building Network Upgrade Coming Soon

First, Fourth, Fifth, Sixth and Seventh Floors to be upgraded

The Computer Center will be upgrading the network equipment in the ARC building during the month of November.  We will be installing new network switches on the first/fourth/fifth/sixth and seventh floors.  This upgrade should increase performance dramatically. We will be upgrading the link to the building and to each floor from 100Mb/sec to 1000Mb/sec.  The current plan is to perform the upgrade on a Saturday in November.  This should decrease the amount of network outage time needed to perform this upgrade. No media/wiring change will be needed.  We will notify users in plenty of time to plan for the Saturday outage. Questions or concerns about this upgrade can be directed helpdesk@jlab.org.

 

Test Lab, Test Lab Addition and Tech Shop Network Upgrade Nearing Completion

Network equipment and media upgrades for the Test Lab, the Test Lab Addition and the neighboring Tech Shop (building #59) have been completed.  The users can now take full advantage of the performance enhancements available to them in these areas.  As a rule of thumb all odd numbered ports (ex. 3-13) labeled in the offices are currently activated on the new networks. If you need additional ports in your area activated or network cables please submit a Computer Center Problem Report (CCPR) with the category “Network Service Request”.  We will get to them in a timely fashion.  If the need for new additional installed ports is requested, we will handle them on a case by case basis.  Once all users are converted to the new Category 6, switched, twisted pair network, we will then be coming through for the final phase – pulling out the old coax network cables and equipment to clean up.  The only area that does not require this phase is the newly erected Test Lab addition.  Any questions regarding these upgrades can be directed to helpdesk@jlab.org.

 

Due to the Lab’s rapid growth, more IP address space on the backbone subnet is needed. To reclaim IP addresses there will be a subnet change for computer systems located in the Testlab, Test Lab Addition, and Tech Shop. This subnet reconfiguration will occur at 7am November 8, 2001. Currently these building reside on our backbone subnet- 129.57.32/33/34/35.xxx, after the subnet change the address range for these buildings will be the 129.57.56/57/58/59.xxx range. The subnet change is being coordinated via Computer Center Staff and Test Lab contacts, who will be providing instructions and assistance to those individuals who will be affected.

 

Telecommunications

 

Telecommunications Cost Reduction

The Telecommunications group within the Computer Center successfully switched telephone service to a new provider on October 5th. This was the culmination of almost a year of effort researching qualified local exchange carriers and transition planning. The Procurement group within Business Services worked closely with the Telecommunications staff from the beginning to pursue opportunities to significantly reduce costs for basic and long distance telephone service. Competitive bids from qualified service providers were solicited and evaluated. The proposal from Adelphia Business Solutions was selected as providing the best value to the Lab. The transition to Adelphia took careful advance planning and required installation of new fiber optic trunks and optical multiplexing equipment at multiple locations at the lab. Starting at 6 PM on Oct. 5th, the cut over of all 3,000 numbers owned by the Lab began. These numbers were transitioned in blocks of 100 from service with Verizon to service provided by Adelphia.

 

As a result of the service change, the local dial-in number used for the Remote Access Server (RAS), which provides off-site computer access, will be changed on October 31, the toll free 1-800 number will be unchanged. The 930-2576 phone number currently in use for dial-in will be changed to 256-4000. The local dial-in number used by the AES group will be changed as well and this number will be distributed to the AES group separately.

 

New Adelphia calling cards were also issued to affected staff members to replace their existing Verizon calling cards. By completing the award for telephone service, both the Procurement department and the Telecommunications group within the Computer Center expect to save the Lab a significant amount of funding while still providing a quality service.

 

Telecommunications Helpful Hint

If you are sharing a telephone with another person; you may request a personal voice mail box. This will allow callers to leave you a message in your mail box that can only be accessed by you.

 

Any Questions? Call Telecommunications at extension 7361, or 7206 for more information.

 

Obtaining Support

From the Computer Center

Submit a problem report or request for assistance:

 

 

Note that the quickest method is to submit an electronic request, as the report is immediately assigned to a staff member and seen by many other staff.

 

After business hours, for emergencies only involving major outages or interruptions to the physics experimental program, contact the guards, who will contact our on-call staff member.

 

CUE News and Web Pages

Other sources of information in CUE are the news messages available on login, and the Computer Center’s web status and announcement pages. For news, simply type “news” to get the latest unread messages, “news a b” for a brief listing of all messages, or “news 100” to read message #100. The web page information can be found at http://cc.jlab.org/announce/status.html.

 

Dell, Gateway, Micron, and Pony Onsite Support

For users who have purchased Dell, Micron, Pony, or Gateway PCs under JLab’s ordering agreement, your machine includes 3-year onsite hardware support. You can directly contact Dell at 1-888-560-8324; you will need your 5-digit service code (a label on the back or bottom of your machine). To contact Micron, call 1-800-249-1179:  extension 59684 for Tech Support, extension 59028 for Customer Service, and extension 31205 for Sales Service. To contact Gateway, call 1-800-846-2303 with your system’s serial number.  For Pony support please call 1-888-809-1588 extension 114.

 

Computer Center Mailing List Archives

The Computer Center Computing Weekly News mail archive can be found at:

http://www.jlab.org/ccc/mail_archives/ALERTS/cc-weekly/CURRENT/.

 

The Computer Center Scientific Computing Briefs mail archive can be found at:

http://www.jlab.org/ccc/mail_archives/ALERTS/jl-scicomp/CURRENT/.

 

Newsletter Archive

The archive of previous Computer Center newsletters, as well as the current newsletter, can be found online at http://cc.jlab.org/announce/newsletter/.

 

 

 


This document is maintained by {helpdesk@jlab.org}

Copyright Jefferson Lab 2007