Using Perl, Shell, and Python to Perform Web Master Tasks

I work on a lot of sites, and I have to perform a lot of system maintenance. Sometimes I  have to keep everyone in the loop on the status of let’s say servers 500 errors, and sometimes I have to find and edit the same line in multiple unknown files. Ah the wonderful world of knowing how. Here are just a few tips and tricks that I keep around to make my life easier.

Using Perl in a shell script

for line in $(< filenames.txt);do      
    filenames=$line     
    perl -pi -e "s/input type=\"hidden\" name=\"mainItem\".*$/$&>/g;" $filenames
done

The above code goes threw a list of file names and searches in those files for a line that includes [ input type="hidden" name="mainItem" ], excluding the brackets, and replaces the WHOLE line with itself and include a new > at the end. In this case $& is the perl variable for the result of the search.

The ” perl -pi -e” portion is how you can run perl code from the command line, without writing a perl file. Makes for some super fast and easy scripts.

NOTE: I made filenames.txt by running a grep for a snippet of code that I knew was included on every file that included the line that needed the “>” added to the end of the line. I ran the grep results to a file, took the file and imported it into a spreadsheet and grabbed the first column, which was filenames and paths, and put them into a file called filenames.txt. The only reason I didn’t write  a program to buddy with this script to run the grep, make the list, and pipe the list to the perl script I did write, is because I was on a deadline and couldn’t buy enough time. It was a one time fix, and it worked. If you come up with the rest of the program, please feel free to share it.

Using Python and Cronjobs to email you, and others, information

I wrote a Python module called ybEmail that I find to be AWESOME and very use full. To send emails automatically, it really is as easy as this:

from ybEmail import sendEmail
 
# mailing list for this report
mailing_list = ['me@example.com','mobile@example.com','boss@example.com']
#mailing_list = ['test@example.com'] # I like to include this always just incase testing needs to be done quickly
 
# send the email
for who in mailing_list:
    sendEmail(who, 'computer@whatever.tld', subject, message, ATTACH='raw_data_log.csv')

The previous email script, would be the last in the procedural part of the script. obviously, you can make subject, message, etc whatever you want. But just to help you understand what you can do with this (for those of you new to programming), I’m going to give you a jumpstart. Let’s say you are putting together a 500 internal server error report, and you have several web servers pooled together. This means you have several logs, which means you should probably have that stuff in a database.

I am going to assume you received and parsed the data already, you wrote the raw data a file to be attached to the email, and you made some summary type data for the body of the message. A VERY IMPORTANT NOTE, is to make sure you add context to EVERY report you create, especially if you have it automated like the intent of this type of report. The following is how I put together the subject and message of the email:

#Subject line for the email
subject = '%s - 500 Internal Server Error Report' % yesterday
 
# Body of the email
message = "Date: %s \n" % yesterday
message += 'Total Errors: %s' % total
message += '%s \n\n' % file_errors
message += '*Attached is a .csv file of the errors counted\n\n'

Ok, so now you have this script that does stuff, and formats a nicely laid out summary email, and includes some raw data for the data miners in your upper chain. Let’s make this happen every day at noon. This part also assumes you are using Linux, btw:

crontab -e

this will open your cronjobs list and from here you can add commands you want to be executed in any sort of automated way. annually, daily, monthly, every 5 seconds, 1 minute, every hour on 15 after, etc. Add this line to the end of the crontab:

0 12 * * * python /path/to/your/new/script.py

Save and exit, and your all done. Sit back and work on automating another part of your job.

Python Quick DB Queries

I want to point out that I find it very nice to have a common.py file handy for all of my custom python scripts to use, with great little functions/classes that are not something you want to have a stand alone module for. Such as a function to query your database that includes a password. For this snippet, I am going to use cx_Oracle, and assume you already imported it.

def query_db(query):
    con = cx_Oracle.connect('username/password@ip.or_address.server/ID')
    cur = con.cursor()
    cur.execute(query)
    return cur
    cur.close()
    con.close()

I know that isn’t the best written module for that type of work, but I needed something quick, and I have the ability to test and verify my queries before I use them in my scripts. Regardless, this is a great function to have around when your creating quick on the fly reports to email yourself to know what’s going on before your boss asks you a question.

Bye for now

Posts like this generate a lot of thoughts about what one uses and has in their toolbox. If you have something you use a lot, that someone else might find use full, feel free to share it.