Parse From Log File In Python

I have a log file with arbitrary number of lines and json strings. All I need is to extract is one json data from the log file BUT ONLY AFTER '_____GP D_____'. I do not want any ot

Solution 1:

You can read your file line by line until you encounter _____GP D_____ at the end of the line, and when you do pick up just the next line:

found_json = Nonewithopen("input.log", "r") as f:  # open your log filefor line in f:  # read it line by lineif line.rstrip()[-14:] == "_____GP D_____":  # if a line ends with our string...
            found_json = next(f).rstrip()  # grab the next linebreak# stop reading of the file, nothing more of interest

Then you can do with your found_json whatever you want, including parsing it, printing it, etc.

UPDATE - If you want to continuously 'follow' your log file (akin to the tail -f command) you can open it in read mode and keep the file handle open while reading it line by line with a reasonable delay added between reads (that's largely how tail -f does it, too) - then you can use the same procedure to discover when your desired line occurs and capture the next line to process, send to some other process or do whatever you plan to do with it. Something like:

import time

capture = False# a flag to use to signal the capture of the next line
found_lines = []  # a list to store our found lines, just as an examplewithopen("input.log", "r") as f:  # open the file for reading...whileTrue:  # loop indefinitely
        line = f.readline()  # grab a line from the fileif line != '':  # if there is some content on the current line...if capture:  # capture the current line
                found_lines.append(line.rstrip())  # store the found line# instead, you can do whatever you want with the captured line# i.e. to print it: print("Found: {}".format(line.rstrip()))
                capture = False# reset the capture flagelif line.rstrip()[-14:] == "_____GP D_____":  # if it ends in '_____GP D_____'..
                capture = True# signal that the next line should be capturedelse:  # an empty buffer encountered, most probably EOF...
            time.sleep(1)  # ... let's wait for a second before attempting to read again...

Solution 2:

import json from ast import literal_eval

KEY_STRING = '''_____GP D_____'''

text = """INFO:modules.gp.helpers.parameter_getter:_____GP D_____
{'from_time': '2017-07-12 19:57', 'to_time': '2017-07-12 20:57', 'consig_number': 'dup1', 'text': 'r155', 'mobile': None, 'email': None}
ERROR:modules.common.actionexception:ActionError: [{'other': 'your request already crossed threshold time'}]
{'from_time': '2016-07-12 16:57', 'to_time': '2016-07-12 22:57', 'consig_number': 'dup2', 'text': 'r15', 'mobile': None, 'email': None}"""



lines = text.split("\n") # load log text into a list. # for loading from log would be more  like# with open("/var/log/syslog.log", 'r') as f:#     lines = f.readlines()# set "gate" flag to False
flag = Falsefor loop in lines:
        line = loop.strip()
        if flag:  # "gate" opened# depends how's the dictionary streamed to log# you could use json.loads(line), but if it is not sent to log with json.dumps than you have pythonic dictinary and use # literal_eval to load that dictionary to a variable# .. a 
                target_json = literal_eval(line)
                print json.dumps(target_json, indent=4)
        if KEY_STRING in line:
                flag = True# KEY_STRING found open "gate"else:
                flag = False# close "gate"
~                               

Output:

{"consig_number":"dup1","text":"r155","email":null,"mobile":null,"to_time":"2017-07-12 20:57","from_time":"2017-07-12 19:57"

}