Common Grep Keywords for Logs

This article highlights common keywords that are used for filtering log file contents.  Using grep with the keywords mentioned can help find specific activity in log files relevant to the issue you are troubleshooting.  

Grepping the Logs

1. SSH into the applicable server as root
2. Run the command cat PATH/TO/LOG/ | grep yourkeywords or sudo cat PATH/TO/LOG | grep yourkeywords
   REPLACE: PATH/TO/LOG/ with the path of the log file.
   REPLACE: yourkeywords with the text you want to search for.
   TIP: Add the -i flag to the grep command to make the keywords case insensitive.
   grep -i YoUrKeYwOrDs
   TIP: The keywords must be surrounded by quotations if there are spaces in them.
   grep "your keywords" or grep 'your keywords'
   

IMPORTANT: If at any point the command you're attempting to run with sudo prompts for a password, please ctrl+c (or equivalent to your OS) and review the command that you attempted to run. This prompt either means the command you attempted to run was incorrect or that you do not have access as your current user to the file you're trying to parse.

Apache Error Logs - /usr/local/apache/logs/error_log

NOTE: Website errors, commonly shown as 500 errors, but some other error codes will have more specific errors logged in the Apache error logs.

1. Specific Error Codes
   
   1. Some commonly found errors
      1. File does not exist
      2. Client denied by server configuration
      3. Connection refused
      4. Request exceeded the limit of 10 internal redirects due to probable configuration error
      5. ModSecurity: Access denied with code 406
         EXAMPLE:[Wed Nov 16 12:54:14.351132 2022] [proxy_fcgi:error] [pid 1065:tid 47354923230976] [client 185.119.81.106:58836] AH01071: Got error 'Primary script unknown'
         To have gotten this error, the following keywords may have been grepped.
         proxy_fcgi
Primary script unknown
AH01071
         Grep for specific error codes when a suspected error is occurring, but needs to be confirmed.
2. Time Stamps
   EXAMPLE: [Wed Nov 16 12:54:14.351132 2022] [proxy_fcgi:error] [pid 1065:tid 47354923230976] [client 185.119.81.106:58836] AH01071: Got error 'Primary script unknown'
   To have gotten this timestamp, the following may have been grepped.
   Nov 16 12:5
   Using Nov 16 12:5 would return all log entries on Nov 16th from 12:50-12:59.
   Likewise, using Nov 16 12: would return all log entries from 12:00-12:59.
3. Domain Names
   EXAMPLE: [Wed Nov 16 14:19:17.734094 2022] [proxy_fcgi:error] [pid 1820022:tid 23050852902656] [client 208.113.196.5:0] AH01071: Got error 'Primary script unknown', referer: http://domain.com
4. IP Addresses
   EXAMPLE: [Wed Nov 16 12:54:14.351132 2022] [proxy_fcgi:error] [pid 1065:tid 47354923230976] [client 185.119.81.106:58836] AH01071: Got error 'Primary script unknown'

Exim Main Log - /var/log/exim_mainlog

NOTE: In the exim_mainlog, incoming and outgoing emails are logged, as well as outgoing (SMTP) connections (logins).

1. Email Addresses (both sending and receiving)
2. Time Stamps
3. Subject Lines of Emails
4. Error Codes
   1. Some commonly found errors are
      1. TLS error on connection
      2. SSL verify error: certificate name mismatch
      3. dovecot_login authenticator failed
5. Exim Transaction IDs
   EXAMPLE: 2022-11-16 14:56:10.617 [2286604] 1ovOW1-009aqi-65 <= user@domain.com H=helo.domain.com [123.123.123.123]:48910 I=[123.123.123.123]:25 P=esmtps L.- X=TLS1.2:ECDHE-RSA-AES256-GCM-SHA384:256 CV=no S=40845 M8S=0 RT=0.190s id=001E2A619DF84E488B83846AE0D1502A@gxbk82 T="This is the email's subject" from <user@domain.com> for user@domain.com
   In this example, the transaction ID is shown in bold as 1ovOW1-009aqi-65

Mail Log - /var/log/maillog 

NOTE: In the maillog, IMAP/POP connections (logins) are logged.

1. Email Addresses
2. Time Stamps
3. IP Addresses of a client connecting via IMAP or POP3
4. Connection Type (POP3 or IMAP)

PHP Error Log - /home/userna5/logs/domain_com.php.error.log (default path)

1. Time Stamps
2. Error Codes
   1. Some commonly found errors are
      1. Failed opening required
      2. Call to undefined function
      3. failed to open stream: No such file or directory
      4. Class *** not found in

      5. Allowed memory size exhausted

3. Domain Names
4. Function Names

Secure Log - /var/log/secure

NOTE: In the secure log, SSH connections are logged, as well as SFTP connections. Commands run as root/sudo are also logged here.

1. Time Stamps
2. Usernames/cPanel Usernames
3. IP Addresses of the client trying to connect to the server

Messages Log - /var/log/messages

NOTE: In the messages log, FTP and FTPS connections are logged.

1. Time Stamps
2. IP Addresses of the client trying to connect to the server
3. Usernames/cPanel Usernames
4. Domain Names

Fail2Ban - /var/log/fail2ban.log

NOTE: In the Fail2Ban Log, the cause of an IP block on the server is logged.

1. IP Addresses that were blocked by Fail2Ban
2. Service Name
   1. Some common services are
      1. exim
      2. apache-modsecurity
      3. sshd
      4. dovecot
      5. mysqld-auth

TIP: If you are unsure of how to use any of these keywords, you can view a log entry to get an example of how the keyword is formatted in the log. Use cat /path/to/log to view the log, and note the format used by the log to format your keyword in the same way.