Understanding Trace
Cisco Unified Serviceability provides trace tools to assist you in troubleshooting issues with your voice application. Cisco Unified Serviceability supports SDI (System Diagnostic Interface) trace, SDL (Signaling Distribution Layer) trace (for Cisco CallManager and Cisco CTIManager services, applicable to Cisco Unified Communications Manager and Cisco Unified Communications Manager Business Edition only), and Log4J trace (for Java applications).
You use the Trace Configuration window to specify the level of information that you want traced as well the type of information that you want to be included in each trace file.
Unified CM and Unified CM BE only: If the service is a call-processing application such as Cisco CallManager or Cisco CTIManager, you can configure a trace on devices such as phones and gateway.
(Unified CM and Unified CM BE only: In the Alarm Configuration window, you can direct alarms to various locations, including SDI trace log files, or SDL trace log files. If you want to do so, you can configure trace for alerts in the Cisco Unified Real-Time Monitoring Tool (RTMT).
After you have configured information that you want to include in the trace files for the various services, you can collect and view trace files by using the trace and log central option in the Cisco Unified Real-Time Monitoring Tool.
Trace Configuration
You can configure trace parameters for any feature or network service that displays in Cisco Unified Serviceability. If you have clusters (Cisco Unified Communications Manager only), you can configure trace parameters for any feature or network service that is available on any Cisco Unified Communications Manager server in the cluster. Use the Trace Configuration window to specify the parameters that you want to trace for troubleshooting problems.
You can configure the level of information that you want traced (debug level), what information you want to trace (trace fields), and information about the trace files (such as number of files per service, size of file, and time that the data is stored in the trace files.) If you have clusters (Cisco Unified Communications Manager only), you can configure trace for a single service or apply the trace settings for that service to all servers in the cluster.
Unified CM and Unified CM BE only: If the service is a call-processing application such as Cisco CallManager or Cisco CTIManager, you can configure a trace on devices such as phones and gateways; for example, you can narrow the trace to all enabled phones with a directory number beginning with 555.
If you want to use predetermined troubleshooting trace settings rather than choosing your own trace fields, you can use the Troubleshooting Trace window. For more information on troubleshooting trace, see the "Troubleshooting Trace Settings" section.
After you have configured information that you want to include in the trace files for the various services, you can collect trace files by using the trace and log central option in RTMT. For more information regarding trace collection, see the "Trace Collection" section.
Troubleshooting Trace Settings
The Troubleshooting Trace Settings window allows you to choose the services in Cisco Unified Serviceability for which you want to set predetermined troubleshooting trace settings. In this window, you can choose a single service or multiple services and change the trace settings for those services to the predetermined trace settings. If you have clusters (Cisco Unified Communications Manager only), you can choose the services on different Cisco Unified Communications Manager servers in the cluster, so the trace settings of the chosen services get changed to the predetermined trace settings. You can choose specific activated services for a single server, all activated services for the server, specific activated services for all servers in the cluster, or all activated services for all servers in the cluster. In the window, N/A displays next to inactive services.
Note The predetermined troubleshooting trace settings for a Cisco Unified Communications Manager feature or network service include SDL (Cisco Unified Communications Manager and Cisco Unified Communications Manager Business Edition only), SDI, and Log4j trace settings. Before the troubleshooting trace settings get applied, the system backs up the original trace settings. When you reset the troubleshooting trace settings, the original trace settings get restored.
When you open the Troubleshooting Trace Settings window after you apply troubleshooting trace settings to a service, the service that you set for troubleshooting displays as checked. In the Troubleshooting Trace Settings window, you can reset the trace settings to the original settings.
After you apply Troubleshooting Trace Setting to a service, the Trace Configuration window displays a message that troubleshooting trace is set for the given service(s). From the Related Links drop-down list box, you can choose the Troubleshooting Trace Settings option if you want to reset the settings for the service. For the given service, the Trace Configuration window displays all the settings as read-only, except for some parameters of trace output settings; for example, Maximum No. of Files. You can modify these parameters even after you apply troubleshooting trace settings.
Trace Collection
Use Trace and Log Central, an option in the Cisco Unified Real-Time Monitoring Tool, to collect, view, and zip various service traces and/or other log files. With the Trace and Log Central option, you can collect SDL/SDI traces, Application Logs, System Logs (such as Event View Application, Security, and System logs), and crash dump files.
Tip To collect CSA logs, check the Cisco Security Agent check box in the Select System Logs tab in RTMT. To access user logs that provide information about users that are logging in and out, check the Security Logs check box in the Select System Logs tab.
Tip Do not use NotePad to view collected trace files.
Note Unified CM and Unified CM BE only: For devices that support encryption, the SRTP keying material does not display in the trace file.
For more information on trace collection, refer to the Cisco Unified Real-Time Monitoring Tool Administration Guide.
Trace Configuration and Collection Checklist
Table 6-1 provides an overview of the steps for configuring and collecting trace for feature and network services in Cisco Unified Serviceability.
Where to Find More Information
Related Topics
Additional Cisco Documentation
•Cisco Unified Real-Time Monitoring Tool Administration Guide
•Unified CM BE and Connection only: Cisco Unity Connection Serviceability Administration Guide
•Connection only: System Administration Guide for Cisco Unity Connection
Monday, August 3, 2015
Understanding Trace
Linux Permissions
Section 1: Introduction to Linux permissions
Linux is today considered the most secure operating system by many. One of key factors to system security is access permission control. All modern operating systems support this feature, which I believe first appeared in UNIX operating system. It allows file owners to restrict who can read, write, execute and otherwise change files, running processes ('tasks') and other parts of the system.
Linux, as every UNIX-like OS, has a built-in file permission control system. It assigns the following attributes to every file on its file system:
Owner - user who owns the file, has unlimited control over it and can change other file attributes.
Group - user group that the file belongs to.
UNIX permissions - a set of rules defining who can do what to the file. Fear not, it is discussed below.
You can see what user and group you are by issuing the following command in a terminal emulator (try gnome-terminal or konsole):
id -a
uid will tell you who you are (as if you didn't already know this), gid is your "effective" group, and groups - all other groups your user belongs to. If a particular access permission is granted to one of the groups your user belongs to then you will be allowed access too. The effective group id is significant when creating files and directories, as explained below. For the record, when you login your effective group (and your "real" group) is set to your "primary" group -- the one group associated with your login in /etc/passwd.
Used terms:
file system - an on-disk structure holding descriptions of files (such as the attributes mentioned above, file modification date etc.) and the files' contents themselves. File systems are contained in disk partitions (also called slices). Most popular file systems today are ext3, xfs and reiserfs. If you run Debian, you probably use ext3. Worth mentioning is the fact that directories ('folders') are also considered files, simply containing other files. Therefore, permissions apply to directories, too.
user group - in UNIX-like systems, every user is assigned to some group. Users in the same group may share rights, for example a file's permissions may be set so that all users in a group can modify its contents.
Section 2: UNIX permissions explained
Having learnt the theory, it's time to pass on to practice - what do UNIX file permissions look like and how to use them? First of all, let us examine the permissions of an example file. By issuing the following command in Linux console or a terminal emulator:
stat /etc/hostname
you will see a list of file's attributes. It includes file type (it could also be a directory, a symlink, etc.), file size et cetera and a line like the one quoted below, which is the item of our interest:
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Obviously, the file is owned by the root user (system administrator) and belongs to the root group. After the slash, numeric user IDs are shown - that's the way they are stored in the filesystem, in order to conserve disk space.
Access field contains an octal number and its human-readable representation (I personally consider the numeric one to be more readable). It is crucial to know what the permission number means. It consists of four digits, ranging from 0 to 7. For now, we shall skip the first one and focus on the last three, as they are used most commonly on every system. In our example, those are 644. Each digit may be a sum of 4, 2 and 1, but not every component has to be included, giving a possible range from 0 to 7. Below is the meaning of the sum components, with Subject being user, group or others, as discussed below.
4 - read permission. Subject is allowed to read the contents of the file or list the content of a directory. Directory content is more than the names of the files, sub-directories, and the other filesystem objects held within each directory; it also includes other meta-information such as whether the filesystem object is a file or directory, the permissions associated with the object, and so forth.
2 - write permission. Subject may modify file content. With directories, it allows the subject to modify what's recorded as being in the directory. This means being able to create files within the directory, or equivalently, move files into the directory; delete files from the directory, or equivalently, move files out of the directory; and, because the directory is where there's a record of the directory content, to modify the permissions of each file or sub-directory contained in the directory.
1 - execute permission. Subject may execute the file. Any file may be marked executable; when the content won't execute an error is reported at runtime. Likewise any file may have execution privileges removed; files with executable content but no permission to execute can't be run. In the case of directories, execute permission lets the subject traverse through the directory into sub-directories. Note that directory traversal does not require read permission. See the note on path handling below.
Therefore, number 5, for example, would mean: a permission to read and execute, but not to write.
The digits define respectively: owner, group and others' permissions. Therefore, we can see that, in our example, file owner (root) may write to the file and read its contents, while group 'root' and other users (not being root nor a member of group 'root') are given the right to read the file.
Now, compare it to file permissions of /etc/shadow (use 'stat' again). This file has 0 as the third meaningful digit, so users not being root nor in group 'shadow' may not even read the file. You can easily confirm that by running a text editor and trying to open /etc/shadow - you, as a regular user, should not be allowed to see its contents as it contains system-wide passwords (and this is beyond the scope of this little How To).
Human-readable form
Several system tools and graphical programs recognize the idea of a human-readable form - a string of 10 consecutive characters. To see an example, issue the command below:ls -l /etc
The -l flag tells ls to display file permissions in the left column of output. The full sequence that you might encounter is as follows (although you probably won't find such files in /etc):-rwxrwxrwx+
Now, let's divide this into parts. The first character defines node type, which is - for normal file, d for a directory, l for symbolic link, c for a character device, p for a pseudo-terminal and b for a block device. You will find files, directories and links commonly throughout the filesystem, while devices and pseudo-terminals should only appear in /dev. Then we have 3 chunks, 3 characters each: rwx rwx rwx. They directly correspond to respective digits of permissions: if the permission is enabled, you get a letter, and if not, you get - in place of that letter. In this case, the first rwx means 7 to owner, the second is also 7 for owner's group, and the third is the world (others) permission. Thus, for example, 640 translates to:rw-r-----
(rw- for owner, r-- for group, --- for others). The last column is the + sign. You are unlikely to see it while listing a directory now (it will appear empty), but it means that extended access rules are in effect, so the file's real permissions are not only what the file access mode says - you can read about ACL below in this howto.
A note on path handling
To access any path in the filesystem, the user (which the particular process is running as) needs at least execute privilege for all its parent directories. Therefore, if you try to access an example file /etc/security/limits.conf, even though it has a mode of 0755 (for the sake of example), it does not necessarily mean you are free to read it. To read the file, you have to be able to 'execute' all of its parent directories, so you need execute permission on /etc and /etc/security. If either /etc or /etc/security has permissions set so that you are not allowed to execute it (1), then reading /etc/security/limits.conf will fail. This rule applies anywhere in the filesystem.
The defaults for new files and directories
This section is included primarily for reference and to aid understanding. Default permissions and group assignment are not often changed so it is safe to skim over this section and come back to re-read it for more detail should the need arise.
The permissions associated with newly created files and directories are, for the most part, determined by something called a umask. The umask is a 4 digit octal number that is subtracted from 0777 to produce the default permission associated with objects newly created in the filesystem. The umask of a 'stock' Debian system is 0022 which makes the default permissions be 0755 -- the owner has all permissions, the group read and execute but not write, and everybody else can read and execute but not write. One would then expect all newly created files to be marked executable but this is prevented because the system call that creates files defaults to creating files that are not executable. Directories on the other hand do have their execute bit set, umask permitting, and so by default can be traversed by all.
The shell umask command can usually be used (without any arguments) to display the current default umask. The umask is set globally by the system administrator in one of various ways; the most elegant is probably the use of the ?PAM pam_umask module in/etc/pam.d/common-session. The system-wide umask may be overridden by each user. This is usually done in ~/.bashrc on a per-user basis, with the shell umask command on a per-process basis, or using the umask(2) system call from within a program.
The user uid (user id number) associated with a newly created file or directory is that of the running process effective uid. In most cases this is the uid of the user who logged in and started the process.
The group associated with a newly created file or directory is the effective group of the running process. This is normally the group named with the username of the logged in user, but can be manually changed (along with the "real" group) on a per-process basis with the newgrpcommand although this is rarely done.
The first of the 4 octal digits which represent permissions contains the setuid and setgid bits. These can be used to override some of the defaults described above but it is not worth getting into details other than to note that the user private groups project collaboration idiom (see below) depends on the behavior of the setgid bit.
Section 3: Modifying file permissions
This section shows, using an example, the very basic usage of chmod command. Chmod is one of sysadmin's best friends and the standard tool for manipulating file permissions in various Unices (also works with *BSD and Solaris!). Let's begin... First of all, create a file for demonstration purposes. In the example, I will be using name testfile. Commands below are to be executed in a terminal emulator or Linux console. You can just copy and paste, and see how it works.
# first of all, create the file using touch command (see 'man touch' for details) touch testfile # now, let's see its permissions stat testfile # modify the file so that group members and other users can write to it chmod 666 testfile # see the new permissions stat testfile
Have the file permissions changed? You can verify that it actually worked by starting a new session and logging on to another user account, or issuing su username. If you only have one user account, create a new one for testing:
su (your root password here, to log on to root account and add a test user) adduser demo # you can remove this user when you've finished: deluser demo
Now, log on to demo, open testfile (in your regular user's home directory) and type something in it. Save, and then check with your own user's account that it contains whatever you may have written. Voila! You may now want to check it with various different permissions. Try chmod with arguments like 644, 640 and so on.
Section 4: Example scenarios involving chmod
You now know how to change file permissions. However, how can they be useful in real life besides letting your buddy leave you a random message in your own text files?
Case 1: Family photos
Situation: You store family photos in directory Photos on your user account. Several other family members use the computer and you want them to be able to access the photos.
Question: How to set directory permissions so that other users can see your files and their content?
Answer: Set the directory to 755 and all files under it to 644:
chmod 755 Photos # Photos/* means all files in Photos directory chmod 0644 Photos/*
Case 2: Software and data files for your department at work
Note on below: ~ means your home directory.
Situation: In your home directory you have a program in ~/AppSoftware/program.bin . It stores your department-specific data files in~/OurData. The system operator has assigned you and other people in your department a user group 'mydept'. You want other people from your department to be able to run the provided software and to write the data files. At the same time, other people from outside the groupshould be allowed to run the software but not to modify the data. For simplicity's sake, we skip things like logging who added/removed what in terms of data (logging is a necessity in real life), focusing only on appropriate permissions.
Question: How to allow execute access for a group to one file (program binary) and read-write access to other directory for the same group, while denying world (other users) access?
Answer: In our example, this would be:
# below: -R flag, affects the directory and files/subdirs inside chmod -R 0755 ~/AppSoftware chmod -R 0770 ~/OurData
In case files have a wrong group attribute set, you can correct it by first running chgrp -R mydept files, where 'mydept' is the group name, 'files' is file path, and -R switch tells chgrp to run recursively (see above code example). Chgrp changes files' group to the one given.
Case 3: Classified files
Question: How to protect files that are to be kept secret?
Answer: A very basic protection can be achieved by chmodding the sensitive files/directories to 0600. However, remember that the system administrator (root) can still access them, regardless of set file permissions. Therefore, besides locking down file permissions, it is highly advisable that you encrypt the files using strong encryption software (try gpg encryption via programs like KGpg, or see ccrypt - symmetric cryptography).
Case 4: Special Bits
Question: How to unset special bits?
Answer: To remove special bits from a file after they have been set, it is convenient to use the symbolic names. For instance if you accidentally set a directory to 6755 permission you can use the symbolic name(s) of the special bit(s) to unset them. When using octal numbers with leading 0, your shell may require you to quote the number '0755' to apply the permissions correctly. chmod u-s,g-s /path/to/dir chmod '0755' /path/to/dir
Group file sharing scenarios and the limits of basic UNIX permissions
Examples above show the usefulness of UNIX file permissions. You can grant users from your group access to your files, expose them to the whole world or have them only for yourself. However, there are use cases in which this access control model is not enough. Assume that you are on a large system (perhaps a server) and, together with several dozen users you are members of group 'users'. Now, you want to make some of your files available to just one of them so that the others can not read it. How can UNIX permissions benefit you? You could use the user private groups directory sharing idiom; a common solution to this problem. But the user private groups idiom pushes the UNIX permission system to its limits and there are cases, even simple file sharing cases between 2 people, where the idiom is simply not suitable.
When the limits of basic UNIX file permissions are reached it is time to make use of...
Access Control Lists in Linux
Access Control Lists (called ACL) are an extended means of defining access rights to files and objects. They allow you to specify file permissions in a more fine-grained way, assigning any user or group (besides owner and file's set group) different privileges. For instance, you may share a file with just one specific user, no matter what group they are in. How to make use of this new, powerful feature?
First, make sure your system supports ACL. Several criteria must be met before you can enable ACL for your files. Check your kernel version. If it is anything later than 2.6.18, then chances are you already have ACL support built-in. (I'm not quite certain at which version Debian kernels received the ACL patch). The next thing is acl package, required for ACL attribute manipulation. You can install it by issuing:
# if you are not logged on as root, use 'su' first apt-get install acl
Alternatively, you can use Synaptic package manager, or another package manager, to get and install the package. If you are not the system administrator, ask your sysadmin to enable ACL on your machine.
Once you have installed acl, you can try and see if your file system supports it. Example command (I assume that file 'testfile' exists):
setfacl --modify user:demo:5 testfile
If setfacl complains about an error, you probably need to mount your filesystem with acl option. Assuming that the filesystem 'testfile' is located on is / , execute the below as root:
mount -o remount,acl /
Try setfacl again. If successful, a call to:
getfacl testfile
should show, among others, a line like this:
user:demo:r-x
Here, rx means 'read, execute' permission, which is equivalent to 5. To see if Access Control Lists work, set the file permissions on testfile to 700 using chmod and try to open it from 'demo' user account. If successful, ACL did override UNIX permissions indeed. Your file system is now ready for granular access control with ACL!
Note: To enable ACL permanently for certain filesystems, you should include acl option in /etc/fstab. Please refer to fstab(5) manual page for instructions.
Example uses of setfacl to manage file permissions
setfacl -R -m user:josh:6 filedir # sets read-write permissions for josh on filedir and all its contents setfacl -m group:junior-sys-admins:4 /var/log/apache2/error.log # let group members of junior-sys-admins read Apache2 error log file setfacl -m user:evilcraig:0 my_notes.txt # prevent user evilcraig from accessing my_notes.txt
Default (inherited) ACL
Note: a bug in coreutils commands cp and mv limits the scope of the below to pure file creation, e.g. with touch: with copy and move, the "Default mask" of the target parent directory won't be inherited as the "Access mask" for the copied/moved file/directory: http://debbugs.gnu.org/db/85/8527.html
Default ACL are an invaluable tool when making a directory that you want to share for reading or writing among users. This hint is inspired by this thread on the Debian forums: http://forums.debian.net/viewtopic.php?f=10&t=53591
Default ACL are access control entries that get inherited by all sub-items of a directory (recursion deeper is allowed!). Thus, if you want to create a directory for bob and fred so that both can work on each other's files, the below should suffice (notice the -d flag to setfacl, it sets a default ACL):
mkdir common_workspace setfacl -m u:bob:7 common_workspace setfacl -d -m u:bob:7 common_workspace setfacl -m u:fred:7 common_workspace setfacl -d -m u:fred:7 common_workspace
Note to the above: a default ACL is inherited by all child nodes as an ACL entry and default ACL, but a default ACL on its own does not take any action permission-wise - hence the double command. The first call gives user 'bob' the right to write, read and execute the directory, and the second one sets up the default ACL which will be inherited.
Now, whenever a file gets created, it retains its original owner and group, but should automatically get assigned the above ACL. This is, for example, useful when you have users co-working on website development. You can use Apache or PHP running as www-data, write a script to change file ownership upon creation to www-data (inotify helps!), and all files are still writable by bob and fred, your Web developers.
Appendix: Some hints
On Debian systems, every user is traditionally assigned their own group. File sharing may be accomplished by adding one user to other's group, as shown below (only to be done as root):
adduser me otherguy # adds user 'me' to group 'otherguy'
Then, 'otherguy' can just set their files to 0750 or whatever permissions they want you to have. However, this is the old-fashioned approach to granular file permissions and should be avoided whenever possible in favour of user private groups or ACLs.
Konqueror (at least in Debian Squeeze) supports ACL out-of-the-box when filesystems are mounted with acl option. It allows for easy, graphical management of extended access rights, similar to that of Microsoft Windows.
Sometimes you have to mount a filesystem that does not support ACL. For example, a NFS volume exported by a central storage solution, or an userspace zfs diskset. In these situations, you can try one simple solution: http://cintrabatista.net/nfs_with_posix_acl.html
You can find a wonderful but pretty old (still current, though) ACL guide here: http://www.vanemery.com/Linux/ACL/linux-acl.html
https://wiki.debian.org/Permissions
Cisco Unified Communications -- Isolating Point(s) of Failure
After collecting information on the symptoms and behavior of the problem, to narrow the focus of your efforts you should:
Identify the specific devices involved in the problem.
Check the version of software running on each device.
Determine if something has changed in the network.
Verify the integrity of the IP network.
1 Identify Devices Involved in the Problem
2 Check Software Release Versions for Compatibility
3 Determine if Network Changes Have Occurred
4 Verify the IP Network Integrity
Identify Devices Involved in the Problem
In large- to medium-sized networks, it is crucial to identify the specific phones, routers, switches, servers and other devices that were involved in a reported problem. Isolating these devices allows you to rule out the vast majority of equipment within the network and focus your time and energy on suspect devices. To help you isolate which devices were involved in a problem, two types of information can prove invaluable:
Network topology diagrams: It is strongly recommended that you have one or more diagrams that show the arrangement of all Cisco Unified Communications products in your network. These diagrams illustrate how these devices are connected and also capture each device's IP address and name (you may want to also have a spreadsheet or database of the latter information). This information can help you visualize the situation and focus on the devices that may be contributing to the reported problem.
Call flow diagrams: Cisco equipment, including Unified Communications Manager servers, typically provide detailed debug and call trace log files. To interpret these log files, however, it is useful to understand the signaling that occurs between devices as calls are set up and disconnected. Using the network topology and call flow diagrams in conjunction with the log files, you can trace how far a call progressed before it failed and identify which device reported the problem.
Check Software Release Versions for Compatibility
After you have identified which devices may be involved in the problem, verify that the version of software running on each device is compatible with the software running on every other device. As part of Cisco Unified Communications release verification, Cisco Systems performs interoperability and load testing on simulated network environments running specific software versions. The Release Matrix lists the combination of software releases that were tested. However, if the combination of releases installed in your network does not match the values in the Release Matrix, it does not necessarily mean the combination is invalid.
To check interoperability for a specific device and software release, locate and review its Release Notes. Release Notes contain up-to-date information on compatibility between the product and various releases of other products. This document also describes open caveats, known issues that may cause unexpected behavior. Before beginning extensive troubleshooting work, examine the Release Notes to determine if you are experiencing a known problem that has an available workaround. Known problems are available in the Bug Toolkit.
Tip: The Bug Toolkit requires that you are a Cisco partner or a registered Cisco.com user with a Cisco service contract. Using the Bug Toolkit, you can find caveats for any release.
Determine if Network Changes Have Occurred
Before focusing on the particular device or site where the problem occurred, it may be useful to determine if a change was made to surrounding devices. If something has been added, reconfigured or removed from elsewhere in the network, that change may be the source of the problem. It is recommended that you track changes to the IP telephony network such as:
New user phones added
Modifications to Cisco Unified Communications Manager call routing settings, such as new directory numbers, route patterns and dial rules to support new sites or devices
Changes to port configurations on switches, routers or gateways (new equipment, wiring changes or new port activation)
Changes to IP addressing schemes (such as adding new subnets) that may have affected route tables
Verify the IP Network Integrity
Always remember that Cisco Unified Communications equipment relies on a backbone IP network. Many connectivity problems are not caused by configuration errors or operational failures on Cisco devices, but rather by the IP network that interconnects them. Problems such as poor voice quality are typically due to IP network congestion, while call failures between locations may be the result of network outages due to disconnected cables or improperly configured IP route tables.
Before assuming that call processing problems result from Cisco Unified Communications devices themselves, check the integrity of the backbone IP network. Keep the OSI model in mind as you perform these checks. Start from the bottom, at the physical layer, by checking that end-to-end cabling. Then verify the status of Layer 2 switches, looking for any port errors. Move from there to confirm that the Layer 3 routers are running and contain correct routing tables. Continue up the OSI stack to Layer 7, the application layer. To resolve problems occurring at the top levels of the stack, a protocol analyzer (or "sniffer") may be useful. You can use sniffer to examine the IP traffic passing between devices and also decode the packets. Sniffers are particularly useful for troubleshooting errors between devices that communicate using Media Gateway Control Protocol (MGCP) or Session Initiation Protocol (SIP).
http://docwiki.cisco.com/wiki/Cisco_Unified_Communications_--_Isolating_Point(s)_of_Failure
Unified Communications System Troubleshooting
| General Problem Area | General Symptom | Specific Indicators | Possible Culprit |
|---|---|---|---|
| Broad statement of the problem (for example, calls failing) | More specific problem case #1 (for example, calls not reaching a connected state | How the problem is reported and/or where to get more details on the problem (for example, alarm/alert notifications, error messages in log files, and so forth) | Device(s) that may be misconfigured or not operating properly (for example, incorrect routing instructions in Unified Communications Manager). Device-specific diagnostic tools and troubleshooting steps will be presented in separate product articles with links from this table. |
| More specific problem case #2 (for example, calls routed to wrong agent in a call center environment) | How the problem is reported and/or where to get more details on the problem (for example, alarm/alert notifications, error messages in log files, and so forth) | Device(s) that may be misconfigured or not operating properly (for example, bad ICM scripts). Device-specific diagnostic tools and troubleshooting steps will be presented in an article with links from this table. |
In a Cisco Unified Communications network, problems are typically discovered and reported by one of the following types of users:
- External users trying to reach employees within the company
- External customers dialing into a call center to order products, obtain customer service, and so forth.
- Internal agents receiving incoming calls from a call queue or initiating outbound collection calls to customers.
- Internal users using administrative phones to call employees in other company locations or PSTN destinations, and perform basic actions such as call transfers and dialing into conferences.
Network administrators must collect sufficient information from these users to allow them to isolate the problem. Detailed, accurate information will make this task easier. The following table lists recommended questions to ask users when they report a problem. As you turn up your network, you may consider putting these questions in an on-line form. A form will encourage users to provide more details about the problem and also put them into the habit of looking for particular error messages and indicators. Capturing the information electronically will also permit you to retrieve and re-examine this information in the future, should the problem repeat itself.
| Ask this Question... | To Determine... |
|---|---|
| Did something fail or did it simply perform poorly? | Whether the issue relates to system degradation or a connectivity failure. An example of a failure is when a user dials a phone number and hears fast busy tone. An example of a performance problem is when a user dials into a conference call and hears "choppy" audio when other parties speak. Quality of service or performance issues require a different approach than connectivity or operational problems. You must still isolate the potential sources of the problem, but you will typically use performance management tools instead of log files. |
| What device were you trying to use? | The device type, model and version of software installed. It is also critical to capture the IP address assigned to the device, as well as its MAC address. If the case of Cisco Unified IP phones, determining the phone's active Cisco Unified Communications Manager server is also important. On Cisco Unified IP phones, these important network values can be displayed by pressing the Settings button and choosing the Network Configuration option from the menu. |
| Did it ever work? | If a device was recently installed and the problem occurred while making it work for the first time, or if the device was operating normally before the problem occurred. If the device was newly installed, the problem is most likely due to improper configuration or wiring of that particular device. Problems with devices that are already up and running can typically be traced back to one of two causes: (a) the user modifying their device, such as changing their configuration or upgrading software, or (b) a change or failure elsewhere in the network. |
| Exactly what action(s) did you perform? | The steps that led up to the problem, including which buttons were pressed and in which order. Capturing this information in detail is important so that you can consistently reproduce the problem. |
| What error message(s) appeared or announcements did you hear? | The visual and audio indicators of the problem. Ask users to provide the exact text that appears and any error codes in either an E-mail or on-line form. If the error indication was audible, ask the user to write down the announcement they heard, the last menu option they were able to successfully choose or the tone they heard when the call failed. |
| What time did the problem occur? | The date and time to compare against entries in log files. If the problem occurred on a Cisco Unified IP phone, make certain the user provides the timestamp that appears on their phone's display. Several Cisco components in a network may capture the same problem event in separate log files, with different ID values. In order to correlate log entries written by different components, you must compare the timestamps to find messages for the same event. Cisco Unified IP phones synchronize their date and time with their active Cisco Unified Communications Manager server. If all Cisco components in the network use Network Time Protocol (NTP) to synchronize with the same source, then the timestamps for the same problem messages will match in every log file. |
| What is the number of the phone you used and what was the phone number you called? | If the problem relates to a WAN or PTSN link, or a Cisco Unified Communications Manager dial plan issue. Ask the user the phone number he or she dialed (called number) and determine if the destination was within his or her site, another site within the corporate network, or a PSTN destination. Because the calling number (the number of the phone used) also affects call routing in some cases, capture this number as well. |
| Did you try to perform any special actions, such as a transfer, forward, call park, call pickup, or meet-me conference? Is the phone set up to automatically perform any of these actions? | If the problem is not directly related to the calling number or called number but rather to the supplementary service setup on Unified Communications Manager or the problem is at the destination phone the user tried to reach by transferring or forwarding the call. |
| Did you attempt the same action on another device? | If the problem is isolated to that user's device or represents a more widespread network problem. If the user cannot make a call from his or her phone, ask the user to place a call to the same destination using a phone in a nearby office. |
Cisco Unified Communications
Cisco Unified Communications
Cisco Unified Communications Systems securely integrate voice, video, and other collaborative data applications into intelligent network communications solutions. These systems, which include IP telephony, voice messaging, rich-media conferencing, IP video broadcasting, and customer contact solutions, take full advantage of all of the power, resiliency, and flexibility of an IP network.
This wiki offers a collaborative environment as part of the overall Cisco Unified Communications System documentation suite, which includes the system-level Cisco Unified Communications Technical Information Sites as well as documentation covering Cisco Unified Communications products.
Topics will be added and updated on a regular basis, with an emphasis on applying, expanding, and clarifying the content contained in the documentation highlighted above. Articles will be clearly marked with the system release(s) to which the information applies. For instructions on editing or adding articles, refer to About DocWiki.
Current Cisco Unified Communications system-level topics are:
Subscribe to:
Posts (Atom)