Softpanorama

May the source be with you, but remember the KISS principle ;-)
Contents Bulletin Scripting in shell and Perl Network troubleshooting History Humor

Troubleshooting Solaris Network Problems

News See Also Recommended Links Administering Logical Network Interfaces netstat route Snoop route command
Solaris Multipathing Solaris ndd Solaris getent Command  ping traceroute Default  Router Humor Etc

Adapted from Sun's Solaris 8 training notes. 

TIP: When the user reporting the problem first of all verify that you and the user are on the same page and that his usage of words is correct. This will eliminate problems in which the user reports a problem but uses technical terms incorrectly; for example, "my system crashes." Your version could be "a specific application terminates unexpectedly."  Attempt to locate the lowest level of the problem, for example, applications that appear to be failing may be impacted by underpinning network problems.

The ping, traceroute, ngrep and other network tools are indispensable  tools for troubleshooting networking problems. 

Network troubleshooting means recognizing and diagnosing networking problems with the goal of keeping your network running optimally. As a network administrator, your primary concern is maintaining connectivity of all devices (a process often called fault management). You may also continually evaluate and improve your network's performance. Because serious networking problems can sometimes begin as performance problems, paying attention to performance can help you address issues before they become serious.

Like in any investigation you need to avoid jumping to conclusion and calmly collect all relevant facts.  You can use famous "How to solve it "approach. Among more network specific issues:

In general, there is no one correct way to determine the root cause of a networking problem. Like any troubleshooting of complex systems this is more art then science and the success depends both on your IQ and the level of experience with the environment.  However, there are a heuristics that  you can follow:

Troubleshooting Commandments

  1. Create a backup of the faulty system before fixing anything. Backup can be done only for configuration files or for the complete system. Complete backup is important as troubleshooting is a high stress activity and it is easy accidentally to destroy some files. Such backup also can be indispensable if the fault disappears on its own: faults that fix themselves often come back on their own too. On modern servers you often can allocate /backup partition and put cpio images of other partitions into it.
     
  2. Before changing file always create a baseline of /etc partition.  Also backup each individual configuration file with the timestamp before changing it. That prevents you from the most typical mistake in troubleshooting: losing the initial configuration.
     
  3. Simplify your environment, if possible. Where possible shutdown internal firewall and try to remove routers and firewalls from the networking path affected. Often problems are introduced by the internal firewall (on both end) or other network devices in the path. 
  4. Have a testing plan. Make sure that you can replicate the reported fault at will. This is important because you should always attempt to re-create the reported fault after effecting any changes. You need to be sure that you are not changing or adding to the problem.
     
  5. Document all steps and results. This is important because you could forget exactly what you did to fix or change the problem. This is especially true when someone interrupts you as you are about to test a configuration change. You can always revert the system to the faulty state if you backed it up as suggested earlier. If you work outside office in the server room you can use tablet or netbook or even your smartphone. Remember that many changes and ideas during troubleshooting might be lost and cause problems later because they are forgotten, unless they are carefully documented. 
     
  6. Where possible, make permanent changes to the configuration settings. Temporary changes may be faster to implement, but cause tremendous confusion when the system reboots after a power failure months or even years later and the fault occurs again. At this time, nobody will remember what was done by whom.

Using ping as a Troubleshooting Tool

The ping utility sends ICMP echo request packets to the target host or hosts. Once ICMP echo responses are received, the message target is considered to be alive, where target is the hostname of the device receiving the ICMP echo requests, is displayed.

# ping problem.host.com
problem.host.com is alive

The -s option is useful when attempting to connect to a remote host that is down or not available. No output will be produced until an ICMP echo response is received from the target host. The -R option can be useful if the traceroute utility is not available.

Statistics are displayed when the ping -s command is terminated.

# ping -s problem.host.com

Another useful troubleshooting technique using ping is to send ICMP echo requests to the entire network by using the broadcast address as the target host. Using the -s option with the broadcast address provides good information about which systems are available on the network:

# ping -s 172.20.4.255

Using ifconfig as a Troubleshooting Tool

The ifconfig utility is useful when troubleshooting networking problems.  For example when you have several poert on the server and it is unlear which is connected tot he swithc it can show where traffic is.

You can use it to display an interface's current status including the settings for the following:

Be aware that there are two ifconfig commands. The two versions differ in how they use name services. The /sbin/ifconfig is called by the /etc/rc2.d/S30sysid.net startup script. This version is not affected by the configuration of the /etc/nsswitch.conf file.

The /usr/sbin/ifconfig is called by the /etc/rc2.d/S69inet and the /etc/rc2.d/S72inetsvc startup scripts. This version of the ifconfig command is affected by the name service settings in the /etc/nsswitch.conf file.

Use the plumb switch when troubleshooting interfaces that have been manually added and configured. Often an interface will report that it is up and running yet a snoop session from another host shows that no traffic is flowing out of the suspect interface. Using the plumb switch resolves the misconfiguration problem.

Using arp as a Troubleshooting Tool

The arp utility can be useful when attempting to locate network problems related to duplicate IP address usage. Determine the Ethernet address of the target host. You can do this by using the banner utility at the ok prompt, or the ifconfig utility at a shell prompt on a Sun system. Armed with the Ethernet address (also known as the MAC address) use the ping utility to determine if the target host can be reached.

Use the arp utility immediately after using the ping utility and verify that the arp table reflects the expected (correct) Ethernet address.

The following example demonstrates this technique.

Working from the system three, use the ping and arp utilities to determine if the system one is really responding to system three.

First, determine the Ethernet address of the host called one.


problem.host.com# ifconfig -a
lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000
hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST, IPv4> mtu 1500 index 6 inet 128.50.2.1 netmask ffffff00 broadcast 128.50.2.255 ether 8:0:20:76:6:b
problem.host.com#

The ifconfig utility shows that the Ethernet address of the hme0 interface is 8:0:20:76:6:b. The first half of the address, 08:00:20 shows that the system is a Sun computer. The last half of the address, 76:06:0b is the unique part of the system's Ethernet address.

Search the Internet to determine the manufacturer of devices with unknown Ethernet addresses.

2. Use the ping utility to send ICMP echo requests from system three to system problem.host.com.

ping problem.host.com problem.host.com is alive 

3. View the arp table to determine if the device that sent the ICMP echo response is the correct system, 76:06:0b.

# arp -a
Net to Media Table: IPv4
Device IP Address Mask Flags Phys Addr ------ -------------------- --------------- ----- --------------- 08: 00 : 20 : 76: 06: 0b 08: 00 : 20: 8e : ee : 18 08: 00 : 20: 7a: 0b:b8 08:00:20:78:54:90 00: 60:97:7f:4f:dd 01: 00 : 5e: 00 : 00: 00

Output from the arp utility will appear to hang if name resolution fails because the arp utility attempts to resolve names. Use the netstat -pn utility to obtain similar output.

The table displayed in step 3 proved that the correct device responded. If the wrong system responded, it could have been quickly tracked down by using the Ethernet address. Once located, it can be configured with the correct IP address.

Many hubs and switches will report the Ethernet address of the attached device, making it easier to track down incorrectly configured devices.

The first half of the Ethernet address can also be used to refine the search. The previous example showed a device, presumably a personal computer, as it reported an Ethernet address of 00:60:97:7f:4f:dd. A quick search on the Internet reveals that the 00:60:97 vendor code is assigned to the 3COM corporation.

Using snoop as a Troubleshooting Tool

The snoop utility can be particularly useful when troubleshooting virtually any networking problems. The traces that are produced by the snoop utility can be most helpful when attempting remote troubleshooting because an end-user (with access to the root password) can capture a snoop trace and email it or send it using ftp to a network troubleshooter for remote diagnosis.

You can use the snoop utility to display packets on the fly or to write to a file. Writing to a file using the -o switch is preferable because each packet can be interrogated later.

problem.host.com# snoop -o tracefile
Using device /dev/le (promiscuous mode)

You can view the snoop file by using the -i switch and the filename in any of the standard modes, namely:

Verbose is most useful when you are troubleshooting routing, network booting, Trivial File Transport Protocol (TFTP), and any network-related problems that require diagnosis at the packet level. Each layer of the packet is clearly defined by the specific headers.

View the snoop output file in terse mode and locate a packet or range of packets of interest. Use the -p switch to view these packets. For example, if packet two is of interest, type:

problem.host.com# snoop -p2,2 -v -i tracefile

Using ndd as a Troubleshooting Tool

Use extreme caution when using the Solaris ndd utility because the system could be rendered inoperable if you set parameters incorrectly. Use an escaped question mark (\?) to determine which parameters a driver supports. For example, to determine which parameters the 100-Mbit Ethernet (hme) device supports, type:

# ndd /dev/hme \?
? (read only)
trans ceiver_inuse (read only)
link_status (read only)
link_speed (read only)

... ... lance_mode
ipg0
#
(read and write) (read and write)

Routing/IP Forwarding

Many systems configured as multi-homed hosts or firewalls may have IP forwarding disabled. A fast way to determine the state of IP forwarding is to use the ndd utility.

problem.host.com# ndd /dev/ip ip_forwarding 0

This example shows that the system is not forwarding IP packets between its interfaces. The value for ip_forwarding is 1 when the system is routing or forwarding IP packets.

Interface Speed

The hme (100-Mbit Ethernet) Ethernet card can operate at two speeds, 10 or 100 Mbits per second. You can use the ndd utility to quickly display the speed at which the interface is running.

ndd /dev/hme link_speed 1

A one (1) indicates that the interface is running at 100 MBits per second. A zero (0) indicates that the interface is running at 10 MBits per second.

Interface Mode

The hme interface can run in either full-duplex or half-duplex mode. Again, the ndd utility provides a fast way to determine the mode of the interface.

# ndd /dev/hme link_mode
 

One (1) indicates that the interface is running in full-duplex mode. A zero (0) indicates that the interface is running in half-duplex mode.

Using netstat as a Troubleshooting Tool

You can use the netstat utility to display the status of the system's network interfaces. Of particular interest when troubleshooting networks are the routing tables of all the systems in question. You can use the -r switch to display a system's routing tables.

# netstat -r

Although interesting, the displayed routing table is not of much use unless you are familiar with the name resolution services, be they the /etc/hosts, NIS, or NIS+ services. The problem is that it is difficult to concentrate on routing issues when any doubt can be cast on the name services. For example, someone could have modified the name service database, and the system msbravo may no longer be the IP address that you expected. Using the -n switch eliminates this uncertainty.

# netstat -rn

# ifconfig -a

This routing table is much easier to translate and troubleshoot, especially when combined with the information from the ifconfig -a utility.

lo0: flags=849<UP,LOOPBACK,RUNNING,MULTICAST> mtu 8232
inet 127.0.0.1 netmask ff000000
 

hme0: flags=863<UP, BROADCAST, NOTRAILERS, RUNNING, MULTICAST> mtu 1500
 

inet 129.147.11.59 netmask ffffff00 broadcast 129.147.11.255 hme0: flags=863<UP, BROADCAST, NOTRAILERS, RUNNING, MULTICAST> mtu 1500
 

inet 172.20.4.110 netmask ffffff00 broadcast 172.20.4.255 #


The verbose mode switch, -v displays additional information, including the MTU size configured for the interface:

# netstat -rnv

Using traceroute as a Troubleshooting Tool

The traceroute utility is useful when you perform network troubleshooting. You can quickly determine if the expected route is being taken when communicating or attempting to communicate with a target network device. As with most network troubleshooting, it is useful to have a benchmark against which current traceroute output can be compared. The traceroute output can report network problems to other network troubleshooters. For example, you could say, "Our normal route to a host is from our router called router1-ISP to your routers called rtr-a1 to rtr-c4. Today, however, users are complaining that performance is very slow. Screen refreshes are taking more than 40 seconds when they normally take less than a second. The output from traceroute shows that the route to the host is from our router router1-ISP to your routers called rtr-a1, rtr-d4 rtr-x5, and then to rtr-c4. What is going on?"

The traceroute utility uses the IP TTL and tries to force ICMP TIME_EXCEEDED responses from all gateways and routers along the path to the target host. The traceroute utility also tries to force a PORT_UNREACHABLE message from the target host. The traceroute utility can also attempt to force an ICMP ECHO_REPLY message from the target host by using the -I (ICMP ECHO) option when issuing the traceroute command.
The traceroute utility will, by default, resolve IP addresses as shown in the following example:

# traceroute 172.20.4.110
traceroute to 172.20.4.110 (172.20.4.110), 30 hops max, 40 byte packets
1 129.147.11.253 (129.147.11.253) 1.037 ms 0.785 ms 0.702 ms
2 129.147.3.249 (129.147.3.249) 1.452 ms 1.569 ms 0.766 ms
3 * dungeon (129.147.11.59) 1.320 ms *

You can display IP addresses instead of hostnames by using the -n switch as shown in the following example. In this example, the hostname dungeon for IP address 129.147.11.59 on line 3 is no longer resolved.


# traceroute -n 172 .20.4. 110
traceroute to 172.20.4.110 (172.20.4.110), 30 hops max, 40 byte packets
1
129.147.11.253
0.954 ms
0.657 ms
0.695 ms
2
129.147.3.249
0.844 ms
0.745 ms
0.771 ms
3
129.147.11.59
0.534 ms *
0.640 ms

Common Network Problems

Following is a list of some common problems that occur:

The user statement, "My application does not work" is just a tip of an iceberg and the user often does not understand what exactly is not working by jumping to conclusions that can mislead you in troubleshooting. Never believe the user story.   You need ask the user very specific question to uncover the real story. Among questions to consider: 

Layers-based troubleshooting

When troubleshooting networks, some people prefer to think in layers, similar to the TCP/IP Model while others prefer to think in terms of functionality.

Using the TCP/IP Model layered approach, you could start at either the Physical or Application layer. Start at either end of the model and test, draw conclusions, move to the next layer and so on.

The Application Layer

A user complains that an application is not functioning. Assuming the application has everything that it needs, such as disk space, name servers, and the like, determine if the Application layer is functional by using another system.

Application layer programs often have diagnostic capabilities and may report that a remote system is not available. Use the snoop command to determine if the application program is receiving and sending the expected data.

The Transport Layer and the Internet Layer

These two layers can be bundled together for the purposes of troubleshooting. Determine if the systems can communicate with each other. Look for ICMP messages that can provide clues as to where the problem lies. Could this be a router or switching problem? Are the protocols (TFTP, BOOTP) being routed? Are you attempting to use protocols that cannot be routed? Are the hostnames being translated to the correct IP addresses? Are the correct netmask and broadcast addresses being used? Tests between the client and server can include
using ping, traceroute, arp, and snoop.

The Network Interface Layer

Use snoop to determine if the network interface is actually functioning. Use the arp command to determine if the arp cache has the expected Ethernet or MAC address. Fourth generation hubs and some switches can be configured to block certain MAC addresses.

When troubleshooting connectivity problems here are some useful questions:

The Physical Layer

Check that the link status LED is lit. Test it with a known working cable. The link LED will be lit even if the transmit line is damaged. Verify that a mdi-x connection or crossover cable is being used if connecting hub to hub.

Selected Troubleshooting Scenarios

Multi-Homed System Acts as Rogue Router

For example system A can use telnet to contact system B, but system B cannot use telnet to contact system A. Further questioning of the user revealed that this problem appeared shortly after a power failure.

For troubleshooting use the traceroute utility to show the route that network traffic takes from system B to system A. If the traceroute output reveals route that goes via additional system (let's call it system C) you have a rogue router problem.

Often that happens due to the fact that system C had been modified by an end-user. For example an additional interface was added, bit the user did not add  /etc/notrouter file to the system. In this case, after rebooting the system, it came up as a router and started advertising routes, which confuses the core routers and disrupts network traffic patterns.

Faulty Cable

For example users on network A could not reach hosts on network B even though routers R1 and E2 appeared to be functioning normally.

First you need to verify that the routers R1and R2 were configured correctly and that the interfaces are up.
They you need to verify that  systems A and B were up and configured correctly.

They you need to use the traceroute utility to discover the actual route from system A to system B.

For example the traceroute output shows that the attempted route from system A on network net-1 goes through router R1 as expected. But the trafficnever reaches router R2 though.

Investigate the router R2 log files. For example is they show that  the interface to network net-2 is flapping (going up and down at a very high rate) and corrupt routing tables you can suspect that the cable is a problem,

To solve this problem, replace the network net-2 cable to router R2. If it fixes the problem then it was faulty and causes intermittent connections.

Duplicate IP Address

Reported Problem:  Systems on network net-1 could not use ping past router R1 to a recently configured network, net-2.

You must be "root" or the sys to perform some of the other troubleshooting step in the previous examples. Suggested steps:

  1. Verify that the T1 link between the routers R1 and R2 is functioning properly.
  2. Verify that router R1 can use ping to contact router R2.
  3. Verify that system A can use ping to reach the close interface of router R2. System A cannot use ping on the far interface of router R2, though.
  4. Confirm that systems on network net-1 can use ping to reach router R1.
  5. Check that systems on network net-2 can use ping to reach router R2.
  6. Determine that the routers are configured correctly.
  7. Verify that the systems on network net-1 and network net-2 are configured correctly.
  8. Make sure the systems on network net-1 can communicate with each other.
  9. Verify that systems on network net-2 can communicate with each other.
  10. Log onto router R1 and use traceroute to display how the data is routed from router R1 to router R2.
    traceroute reported that the traffic from router
  11. R1 to router R2 was going out the network net-1 side interface of the router instead of the network net-2 side as expected. This indicates that the IP address for router R2 may also exist on network net-1.
  12. Check the Ethernet address of router R2; compare the actual address with the contents of router R1's arp cache. The arp cache revealed that the device was of a different manufacturer than expected.

To solve the problem track down the device on network net-1, system C, that has an illegal IP address (one that is the same as the network-net-1-side interface of router R2). This resulted in a routing loop as the routers had multiple best-case paths to take to the same location (which were actually in two different sites).

Correct the duplicate IP address problem on system C and make sure communications work as expected.

Duplicate MAC Address (Mostly Sun environment problem)

Reported Problem: After adding an additional Ethernet interface to your host, the system performance is very poor.

Troubleshooting (as user root):

Notice from the previous ifconfig output that all the interfaces have the same MAC address. Host C is on different subnet, so this is not a problem.  This would cause problems because packets that leave either qfe0 or qfe1 would not be guaranteed to receive a response since both interfaces are broadcasting themselves as the source for those packets.


Top Visited
Switchboard
Latest
Past week
Past month

NEWS CONTENTS

Old News ;-)

Submitted Article Network Troubleshooting Tips for the Solaris 9 OS by Ross Moffatt,

December 2006 | BigAdmin
Contents
Overview

This How-To covers some basic networking setup and troubleshooting on the Solaris 9 OS.

Two Tips for Network Performance Checking

a. Use FTP to copy a large file between hosts. Make sure you copy the file in both directions, as network performance problems can be directional. A possible cause of performance issues is autonegotiation being enabled at either the host or the router/switch. (See the Checking Network Settings section of this article for more details.)

b. Use ping with small (1Kbyte) and large (10K) packet sizes: Sometime routers in the network can have issues depending upon the size of the packet, as some use different queues within the router depending upon packet size.

Network Connectivity Troubleshooting

Here is a checklist to help you locate and resolve network connectivity problems.

1. Use ifconfig -a to check that interfaces are plumbed; that is, that they exist in the output. Also, check the network address and netmask of the interface.

To plumb an interface, run the command ifconfig <interface><instance> plumb, for example:

# ifconfig ce1 plumb
Use ifconfig to see if the interface now exists.
# ifconfig -a
lo0: 
flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> 
mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
ce0: 
flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> 
mtu 1500 index 2
        inet 444.555.666.7 netmask ffffff00 broadcast 
444.555.666.255
        ether 5:3:de:de:de:de
ce1: 
flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> 
mtu 1500 index 6
        inet 0.0.0.0 netmask 0
        ether 3:4:aa:bb:cc:dd

Give the interface its ipaddress and netmask.

ifconfig ce1 555.66.77.88 netmask 255.255.255.0 up

2. Ping the interface address; it should work!

3. Ping your router/switch. If you see => fail, then check your network settings. (See the Checking Network Settings section of this article.)

4. Ping a host on another network. If that doesn't work, check the routing table. (See the Checking Routing Settings section of this document.)

Checking Network Settings

You can check or set the status and configuration of a network interface with the ndd command. To use the ndd command, you first need to use the ifconfig command to find out what the device file is for the network interface in which you are interested.

# ifconfig -a
lo0: 
flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> 
mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
ce0: 
flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> 
mtu 1500 index 2
        inet 444.555.666.7 netmask ffffff00 broadcast 
444.555.666.255
        ether 5:3:de:de:de:de

This example shows only one interface plumbed, ce0, so the interface type is ce, and it is instance 0.

Now sometimes the ndd command uses a generic device file, e.g. /dev/ce, and you need to set the instance number first to know which instance is being interrogated. Otherwise, an instance-specific device file, e.g. /dev/bge0, is used. You can use ls -ld /dev/<interface>* to see if instance-specific device files exist, for example, ls -ld /dev/ce*.

# ls -l /dev/ce*
lrwxrwxrwx 1 root root 28 Mar 3 2006 /dev/ce ->
.../devices/pseudo/clone@0:ce

or

# ls -ld /dev/bge*
lrwxrwxrwx 1 root root 29 Oct 14 2005 /dev/bge ->
.../devices/pseudo/clone@0:bge
lrwxrwxrwx 1 root root 39 Oct 14 2005 /dev/bge0 ->
.../devices/pci@1f,700000/network@2:bge0
lrwxrwxrwx 1 root root 41 Oct 14 2005 /dev/bge1 ->
.../devices/pci@1f,700000/network@2,1:bge1
lrwxrwxrwx 1 root root 39 Oct 14 2005 /dev/bge2 ->
.../devices/pci@1d,700000/network@2:bge2
lrwxrwxrwx 1 root root 41 Oct 14 2005 /dev/bge3 ->
.../devices/pci@1d,700000/network@2,1:bge3

The following scripts print out all variables available via ndd. One is for a generic device file, and the other is for an instance-specific device file. The script is run with the instance required as a command-line option, that is, <script> 0. You would need to change the script to have the correct interface device type. For example, you may need to replace ce with eri if your interface device file is /dev/eri.

#!/bin/sh
# ndd generic device script
ndd -set /dev/ce instance $1
for p in `ndd /dev/ce \\? | awk '{print $1}' | grep -v \\?`
do echo \"$p: `ndd /dev/ce $p`\"
done

Output.
# ./script 0
"instance: 0"
"adv_autoneg_cap: 0"
"adv_1000fdx_cap: 0"
"adv_1000hdx_cap: 0"
"adv_100T4_cap: 0"
"adv_100fdx_cap: 1"
"adv_100hdx_cap: 0"
"adv_10fdx_cap: 0"
"adv_10hdx_cap: 0"
"adv_asmpause_cap: 0"
"adv_pause_cap: 0"
"master_cfg_enable: 0"
"master_cfg_value: 0"
"use_int_xcvr: 0"
"enable_ipg0: 1"
"ipg0: 8"
"ipg1: 8"
"ipg2: 4"
"rx_intr_pkts: 8"
"rx_intr_time: 3"
"red_dv4to6k: 0"
"red_dv6to8k: 0"
"red_dv8to10k: 0"
"red_dv10to12k: 0"
"tx_dma_weight: 0"
"rx_dma_weight: 0"
"infinite_burst: 1"
"disable_64bit: 0"
"accept_jumbo: 0"
"laggr_multistream: 0"


#!/bin/sh
# ndd specific device script
for p in `ndd /dev/bge$1 \\? | awk '{print $1}' | grep -v \\?`
do echo \"$p: `ndd /dev/bge$1 $p`\"
done

Output.
# ./script 0
"autoneg_cap: 0"
"pause_cap: 1"
"asym_pause_cap: 1"
"1000fdx_cap: 0"
"1000hdx_cap: 0"
"100T4_cap: 0"
"100fdx_cap: 1"
"100hdx_cap: 0"
"10fdx_cap: 0"
"10hdx_cap: 0"
"adv_autoneg_cap: 0"
"adv_pause_cap: 1"
"adv_asym_pause_cap: 1"
"adv_1000fdx_cap: 0"
"adv_1000hdx_cap: 0"
"adv_100T4_cap: 0"
"adv_100fdx_cap: 1"
"adv_100hdx_cap: 0"
"adv_10fdx_cap: 0"
"adv_10hdx_cap: 0"
"lp_autoneg_cap: 0"
"lp_pause_cap: 0"
"lp_asym_pause_cap: 0"
"lp_1000fdx_cap: 0"
"lp_1000hdx_cap: 0"
"lp_100T4_cap: 0"
"lp_100fdx_cap: 0"
"lp_100hdx_cap: 0"
"lp_10fdx_cap: 0"
"lp_10hdx_cap: 0"
"link_status: 1"
"link_speed: 100"
"link_duplex: 1"
"link_autoneg: 0"
"link_rx_pause: 1"
"link_tx_pause: 0"
"loop_mode: 0"

I have found it best to set the network devices to disable autonegotiation on both the host and the router to which the host is connected. This is done by setting the following parameters: autoneg_cap, 1000fdx_cap, 1000hdx_cap, 100T4_cap, 100fdx_cap, 100hdx_cap, 10fdx_cap, 10hdx_cap, and adv_autoneg_cap.

I use this script in the /etc/rc2 directory to set the network parameters.

smart1 # more /etc/rc2.d/S95net_tune
#!/sbin/sh
PATH=$PATH:/usr/sbin;export PATH
case "$1" in
'start')
        /usr/sbin/ndd -set /dev/bge0 adv_1000fdx_cap 0
        /usr/sbin/ndd -set /dev/bge0 adv_1000hdx_cap 0
        /usr/sbin/ndd -set /dev/bge0 adv_100fdx_cap 1
        /usr/sbin/ndd -set /dev/bge0 adv_100hdx_cap 0
        /usr/sbin/ndd -set /dev/bge0 adv_10fdx_cap 0
        /usr/sbin/ndd -set /dev/bge0 adv_10hdx_cap 0
        /usr/sbin/ndd -set /dev/bge0 adv_autoneg_cap 0
        ;;

'stop')
        ;;

*)
        echo "Usage: $0 { start | stop }"
        exit 1
        ;;
esac
exit 0

The bge interface shows you the currently running status with the following parameters: link_status, link_speed, and link_duplex.

For information on interface device drivers, look in the man pages, in Section 7: Devices and Network Interfaces.

Checking Routing Settings

To see your current routing configuration, use netstat -r, and add the -n option depending on whether you want to see DNS names or IP addresses. For example:

smart1 # netstat -rn
Routing Table: IPv4
  Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ------ ---------
222.333.444.0 222.333.444.21 U 1 104943 bge0
default 222.333.444.1 UG 163805900
127.0.0.1 127.0.0.1 UH 538851300 lo0
smart1 # netstat -r

Routing Table: IPv4
  Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ------ ---------
222.333.444.0 myhost U 1 104943 bge0
default router UG 163805927
localhost localhost UH 538851327 lo0

If the default route is missing, then use the ifconfig command to add it on the fly:

ifconfg add default 222.333.444.1
Changing the IP Address

The following startup files need to be modified to change a host's IP address.

/etc/inet/hosts: Change the IP address, file format, IP<tab>hostname.

/etc/inet/netmasks: Add a new netmask, file format, network<tab>netmask.

/etc/defaultrouter: Specify the new gateway for this subnet, file format, ipaddress.

About the Author

Ross Moffatt has been a UNIX System Administrator for more than 10 years. He can be contacted at ross.stuff@telstra.com.

Solaris- Network Troubleshooting

One of the first signs of trouble on the network is a loss of communications by one or more hosts. If a host refuses to come up at all the first time it is added to the network, the problem might lie in one of the configuration files, or in the network interface. If a single host suddenly develops a problem, the network interface might be the cause. If the hosts on a network can communicate with each other but not with other networks, the problem could lie with the router, or it could lie in another network.

You can use the ifconfig program to obtain information on network interfaces and netstat to display routing tables and protocol statistics. Third-party network diagnostic programs provide a number of troubleshooting utilities. Refer to third-party documentation for information.

Less obvious are the causes of problems that degrade performance on the network. For example, you can use tools like ping to quantify problems like the loss of packets by a host.

Running Software Checks

If the network has trouble, some actions that you can take to diagnose and fix software-related problems include:

  1. Using the netstat command to display network information.
  2. Checking the hosts database (and ipnodes if you are using IPv6) to make sure that the entries are correct and up to date.
  3. If you are running RARP, checking the Ethernet addresses in the ethers database to make sure that the entries are correct and up to date.
  4. Trying to connect by telnet to the local host.
  5. Ensuring that the network daemon inetd is running. To do this, log in as superuser and type:
    # ps -ef | grep inetd

    Here is an example of output displayed if the inetd daemon is running:

    root 57 1 0 Apr 04 ? 3:19 /usr/sbin/inetd -s root 4218 4198 0 17:57:23 pts/3 0:00 grep inetd

ping Command

Use the ping command to find out whether there is IP connectivity to a particular host. The basic syntax is:

/usr/sbin/ping host [timeout]
where host is the host name of the machine in question. The optional timeout argument indicates the time in seconds for ping to keep trying to reach the machine-20 seconds by default. The ping(1M) man page describes additional syntaxes and options. When you run ping, the ICMP protocol sends a datagram to the host you specify, asking for a response. (ICMP is the protocol responsible for error handling on a TCP/IP network. See ICMP Protocol for details.)

How to Determine if a Host Is Losing Packets

If you suspect that a machine might be losing packets even though it is running, you can use the s option of ping to try to detect the problem. On the command line, type the following command.

% ping -s hostname

ping continually sends packets to hostname until you send an interrupt character or a timeout occurs. The responses on your screen will resemble:

PING elvis: 56 data bytes
64 bytes from 129.144.50.21: icmp_seq=0. time=80. ms
64 bytes from 129.144.50.21: icmp_seq=1. time=0. ms
64 bytes from 129.144.50.21: icmp_seq=2. time=0. ms
64 bytes from 129.144.50.21: icmp_seq=3. time=0. ms
.
.
.
----elvis PING Statistics----
4 packets transmitted, 4 packets received, 0% packet loss
round-trip (ms) min/avg/max = 0/20/80   

The packet-loss statistic indicates whether the host has dropped packets.

If ping fails, check the status of the network reported by ifconfig and netstat, as described in ifconfig Command and netstat Command

ifconfig Command

The ifconfig command displays information about the configuration of an interface that you specify. (Refer to the ifconfig(1M) man page for details.) The syntax of ifconfig is:

ifconfig interface-name [protocol_family]

How to Get Information About a Specific Interface

  1. Become superuser.
  2. On the command line, type the following command.
    # ifconfig interface

    For an le0 interface, your output resembles the following:

    le0: flags=863 mtu 1500
      inet 129.144.44.140 netmask ffffff00 broadcast 129.144.44.255
    ether 8:0:20:8:el:fd
    

    The flags section just given shows that the interface is configured "up," capable of broadcasting, and not using "trailer" link level encapsulation. The mtu field tells you that this interface has a maximum transfer size of 1500 octets. Information on the second line includes the IP address of the host you are using, the netmask being currently used, and the IP broadcast address of the interface. The third line gives the machine address (Ethernet, in this case) of the host.

How to Get Information About All Interfaces on a Network

A useful ifconfig option is -a, which provides information on all interfaces on your network.
  1. # ifconfig -a interface

    This produces, for example:

    le0:  flags=49 mtu 8232
         inet 127.144.44.140 netmask ff000000 
    le0:flags=863 mtu 1500
         inet 129.144.44.140 netmask ffffff00 broadcast 129.144.44.255
    ether 8:0:20:8:el:fd
    

    Output that indicates an interface is not running might mean a problem with that interface. In this case, see the ifconfig(1M) man page.

netstat Command

The netstat command generates displays that show network status and protocol statistics. You can display the status of TCP and UDP endpoints in table format, routing table information, and interface information.

netstat displays various types of network data depending on the command line option selected. These displays are the most useful for system administration. The syntax for this form is:

netstat [-m] [-n] [-s] [-i | -r] [-f address_family]

The most frequently used options for determining network status are: s, r, and i. See the netstat(1M) man page for a description of the options.

How to Display Statistics by Protocol

The netstat -s option displays per protocol statistics for the UDP, TCP, ICMP, and IP protocols.

On the command line, type the following command.

% netstat -s

The result resembles the display shown in the example below. (Parts of the output have been truncated.) The information can indicate areas where a protocol is having problems. For example, statistical information from ICMP can indicate where this protocol has found errors.

UDP
       udpInDatagrams      =  39228     udpOutDatagrams     =  2455  
      udpInErrors         =     0
 TCP
       tcpRtoAlgorithm     =     4      tcpMaxConn          =    -1
      tcpRtoMax           = 60000      tcpPassiveOpens     =     2
      tcpActiveOpens      =     4      tcpEstabResets      =     1
      tcpAttemptFails     =     3      tcpOutSegs          =   315
   .
   .
IP
       ipForwarding        =     2      ipDefaultTTL        =   255
      ipInReceives        =  4518      ipInHdrErrors       =     0
   .
   . 
ICMP
       icmpInMsgs          =     0      icmpInErrors        =     0
      icmpInCksumErrs     =     0      icmpInUnknowns      =     0
   .
   . 
 IGMP:
 0 messages received
0 messages received with too few bytes
0 messages received with bad checksum
0 membership queries received
0 membership queries received with invalid field(s)
0 membership reports received
0 membership reports received with invalid field(s)
0 membership reports received for groups to which we belong
0 membership reports sent

How to Display Network Interface Status

The i option of netstat shows the state of the network interfaces that are configured with the machine where you ran the command. On the command line, type the following command:

% netstat -i

Here is a sample display produced by netstat -i:

Name Mtu  Net/Dest     Address   Ipkts    Ierrs Opkts    Oerrs  Collis  Queue
le0  1500 b5-spd-2f-cm tatra     14093893 8492  10174659 1119   2314178   0
lo0  8232 loopback     localhost 92997622 5442  12451748 0      775125    0

Using this display, you can find out how many packets a machine thinks it has transmitted and received on each network. For example, the input packet count (Ipkts) displayed for a server can increase each time a client tries to boot, while the output packet count (Opkts) remains steady. This suggests that the server is seeing the boot request packets from the client, but does not realize it is supposed to respond to them. This might be caused by an incorrect address in the hosts, ipnodes, or ethers database.

On the other hand, if the input packet count is steady over time, it means that the machine does not see the packets at all. This suggests a different type of failure, possibly a hardware problem.

How to Display Routing Table Status

The -r option of netstat displays the IP routing table. On the command line, type the following command.

% netstat -r

Here is a sample display produced by netstat -r run on machine tenere:

Routing tables
Destination   Gateway Flags Refcnt Use   Interface
temp8milptp   elvis   UGH   0      0 
irmcpeb1-ptp0 elvis   UGH   0      0 
route93-ptp0  speed   UGH   0      0 
mtvb9-ptp0    speed   UGH   0      0 
               .
mtnside       speed   UG    1      567 
ray-net       speed   UG    0      0 
mtnside-eng   speed   UG    0      36 
mtnside-eng   speed   UG    0      558 
mtnside-eng   tenere  U     33     190248  le0

The first column shows the destination network, the second the router through which packets are forwarded. The U flag indicates that the route is up; the G flag indicates that the route is to a gateway. The H flag indicates that the destination is a fully qualified host address, rather than a network.

The Refcnt column shows the number of active uses per route, and the Use column shows the number of packets sent per route. Finally, the Interface column shows the network interface that the route uses.

How to Log Network Problems

  1. Become superuser.
  2. Create a log file of routing daemon actions by typing the following command at a command line prompt.
    # /usr/sbin/in.routed /var/logfilename

    Caution: On a busy network, this can generate almost continuous output.

Displaying Packet Contents

You can use snoop to capture network packets and display their contents. Packets can be displayed as soon as they are received, or saved to a file. When snoop writes to an intermediate file, packet loss under busy trace conditions is unlikely. snoop itself is then used to interpret the file. For information about using the snoop command, refer to the snoop(1M) man page.

The snoop command must be run by root (#) to capture packets to and from the default interface in promiscuous mode. In summary form, only the data pertaining to the highest-level protocol is displayed. For example, an NFS packet only displays NFS information. The underlying RPC, UDP, IP, and Ethernet frame information is suppressed but can be displayed if either of the verbose options is chosen.

The snoop capture file format is described in RFC 1761. To access, use your favorite web browser with the URL: http://ds.internic.net/rfc/rfc1761.txt.

snoop server client rpc rstatd collects all RPC traffic between a client and server, and filters it for rstatd.

How to Check All Packets from Your System

  1. Type the following command at the command line prompt to find the interfaces attached to the system.
    # netstat -i

    Snoop normally uses the first non-loopback device (le0).

  2. Type snoop. Use Ctl-C to halt the process.
    # snoop
    Using device /dev/le (promiscuous mode)
         maupiti -> atlantic-82  NFS C GETATTR FH=0343
     atlantic-82 -> maupiti      NFS R GETATTR OK
         maupiti -> atlantic-82  NFS C GETATTR FH=D360
     atlantic-82 -> maupiti      NFS R GETATTR OK
         maupiti -> atlantic-82  NFS C GETATTR FH=1A18
     atlantic-82 -> maupiti      NFS R GETATTR OK
         maupiti -> (broadcast)  ARP C Who is 120.146.82.36, npmpk17a-82 ?

  3. Interpret the results.

    In the example, client maupiti transmits to server atlantic-82 using NFS file handle 0343. atlantic-82 acknowledges with OK. The conversation continues until maupiti broadcasts an ARP request asking who is 120.146.82.36?

    This example demonstrates the format of snoop. The next step is to filter snoop to capture packets to a file.

    Interpret the capture file using details described in RFC 1761. To access, use your favorite web browser with the URL: http://ds.internic.net/rfc/rfc1761.txt

How to Capture snoop Results to a File

  1. # snoop -o filename

    For example:

    # snoop -o /tmp/cap
    Using device /dev/le (promiscuous mode)
    30 snoop: 30 packets captured
    

    This has captured 30 packets in a file /tmp/cap. The file can be anywhere with enough disk space. The number of packets captured is displayed on the command line, enabling you to press Ctl-C to abort at any time.

    snoop creates a noticeable networking load on the host machine, which can distort the results. To see reality at work, run snoop from a third system, (see the next section).

    On the command line, type the following command to inspect the file.

    # snoop -i filename

    For example:

    # snoop -i /tmp/cap
     1  0.00000 frmpk17b-082 -> 224.0.0.2    IP  D=224.0.0.2 S=129.146.82.1 LEN=32, ID=0
    2  0.56104        scout -> (broadcast)  ARP C Who is 129.146.82.63, grail ?
    3  0.16742  atlantic-82 -> (broadcast)  ARP C Who is 129.146.82.76, honeybea ?
    4  0.77247        scout -> (broadcast)  ARP C Who is 129.146.82.63, grail ?
    5  0.80532 frmpk17b-082 -> (broadcast)  ARP C Who is 129.146.82.92, holmes ?
    6  0.13462        scout -> (broadcast)  ARP C Who is 129.146.82.63, grail ?
    7  0.94003        scout -> (broadcast)  ARP C Who is 129.146.82.63, grail ?
    8  0.93992        scout -> (broadcast)  ARP C Who is 129.146.82.63, grail ?
    9  0.60887        towel -> (broadcast)  ARP C Who is 129.146.82.35, udmpk17b-82 ?
    10 0.86691  nimpk17a-82 -> 129.146.82.255 RIP R (1 destinations)
    

    Refer to specific protocol documentation for detailed analysis and recommended parameters for ARP, IP, RIP and so forth. Searching the Web is a good place to look at RFCs.

How to Check Packets Between Server and Client

Establish a snoop system off a hub connected to either the client or server.

The third system (the snoop system) sees all the intervening traffic, so the snoop trace reflects reality on the wire.

  1. Become superuser.
  2. On the command line, type snoop with options and save to a file.
  3. Inspect and interpret results.

    Look at RFC 1761 for details of the snoop capture file. To access, use your favorite web browser with the URL: http://ds.internic.net/rfc/rfc1761.txt

    Use snoop frequently and consistently to get a feel for normal system behavior. For assistance in analyzing packets, look for recent white papers and RFCs, and seek the advice of an expert in a particular area, such as NFS or YP. For details on using snoop and its options, refer to the snoop(1M) man page.

Displaying Routing Information

Use the traceroute utility to trace the route an IP packet follows to some internet host. The traceroute utility utilizes the IP protocol time(to live) ttl field and attempts to elicit an ICMP TIME_EXCEEDED response from each gateway along the path, and the response PORT_UNREACHABLE (or ECHO_REPLY) from the destination host. The traceroute utility starts sending probes with a ttl of one and increases by one until it gets to the intended host or has passed through a maximum number of intermediate hosts.

The traceroute utility is especially useful for determining routing misconfiguration and routing path failures. If a particular host is unreachable, you can use the traceroute utility to see what path the packet follows to the intended host and where possible failures might occur.

The traceroute utility also displays the round trip time for each gateway along the path to the target host. This information can be useful for analyzing where traffic is slow between the two hosts.

How to Run the Traceroute Utility

On the command line, type the following command.

% traceroute destination-hostname

Example--traceroute Utility

The following sample of the traceroute command shows the 7-hop path a packet follows from the host istanbul to the host sanfrancisco along with the times for a packet to traverse each hop.

istanbul% traceroute sanfrancisco
 traceroute: Warning: Multiple interfaces found; using 172.31.86.247 @ le0
 traceroute to sanfrancisco (172.29.64.39), 30 hops max, 40 byte packets
  1  frbldg7c-86 (172.31.86.1)  1.516 ms  1.283 ms  1.362 ms
  2  bldg1a-001 (172.31.1.211)  2.277 ms  1.773 ms  2.186 ms
  3  bldg4-bldg1 (172.30.4.42)  1.978 ms  1.986 ms  13.996 ms
  4  bldg6-bldg4 (172.30.4.49)  2.655 ms  3.042 ms  2.344 ms
  5  ferbldg11a-001 (172.29.1.236)  2.636 ms  3.432 ms  3.830 ms
  6  frbldg12b-153 (172.29.153.72)  3.452 ms  3.146 ms  2.962 ms
  7  sanfrancisco (172.29.64.39)  3.430 ms  3.312 ms  3.451 ms

Recommended Links

Softpanorama hot topic of the month

Softpanorama Recommended

...



Etc

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes.   If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner. 

ABUSE: IPs or network segments from which we detect a stream of probes might be blocked for no less then 90 days. Multiple types of probes increase this period.  

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Haterís Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least


Copyright © 1996-2016 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: August 01, 2017