How to parse this custom log file in Python – Stack Overflow
I recently had a similar task of parsing the log records, but along with exception tracebacks for further analysis. Instead of banging my head against home brewed regular expressions, I used two wonderful libraries: parse for parsing records (this is actually a very cool library, practically an inverse function to stdlib’s) and boltons for parsing tracebacks. Here is a sample code I extracted from the impl of mine, adapted to the log in question:
import datetime
import logging
import os
from pathlib import Path
from boltons. tbutils import ParsedException
from parse import parse, with_pattern
LOGGING_DEFAULT_DATEFMT = f”{fault_time_format}, %f”
# TODO better pattern
@with_pattern(r”\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d, \d\d\d”)
def parse_logging_time(raw):
return rptime(raw, LOGGING_DEFAULT_DATEFMT)
def from_log(file: thLike, fmt: str):
chunk = “”
custom_parsers = {“asctime”: parse_logging_time}
with Path(file)() as fp:
for line in fp:
parsed = parse(fmt, line, custom_parsers)
if parsed is not None:
yield parsed
else: # try parsing the stacktrace
chunk += line
try:
yield om_string(chunk)
except (IndexError, ValueError):
pass
if __name__ == “__main__”:
for parsed_record in from_log(
file=””,
fmt=”{asctime:asctime} – {module} – {levelname} – {message}”):
print(parsed_record)
When executed, this yields
ParsedException(‘NameError’, “name ‘numFilesDownloaded’ is not defined”, frames=[{‘filepath’: ‘
Notes
If you are specifying the log format using the { style, chances are high that you can simply pass the logging format string to parse and it will just work. In this example, I had to improvise and use a custom parser for timestamps to match the question’s requirements; if the timestamps would be of a common format, e. g. ISO 8601, one could just use fmt=”{asctime:ti} – {module} – {levelname} – {message}” and throw out parse_logging_time and custom_parsers from the example code. parse supports several common timestamp formats out of the box; check out the “Format Specification” section in its readme.
The sults are dict-like objects, so parsed_record[“message”] returns the parsed message etc.
Notice the ParsedException object printed – this is the exception parsed from the traceback.
Log File Parsing In Python | Pythonicways
In this tutorial you will learn how to create log file parser in python. Parsing a log file or any type of text file in order to extract specific information is not that hard if you know a bit of python and regex. Python itself is perfect for this kind of use and does not require any third party modules. Also, if you are new to python world, I wrote a quick python setup on windows tutorial in which you can look into if you want to quickly setup python environment.
In my day job, I was working on testing Skype for Business iOS application and it came to the point where I had to go through iOS application’s log files in order to see all the requests and received responses. I had to look for specific properties like:
Usually they were buried under bunch of other not so important text. While I was looking for specific things in those log files, I realized that it’s going to be really time consuming to go through log files manually. In order to save time I wrote the following python script. If you want to test this script with similar text file, you can download SfB iOS log file here.
Open Log file
First of all, we have to open the log file, read each line and look for specific text in that line using regex. It might be confusing and a bit scary to pick up regex, but believe me it’s not that complicated. Usually I like to go to regexr site and just play around with different expressions until I find something that matches the text that I wanted to match. If that doesn’t work for you try googling, that’s what I did when I first started using regex.
with open we’re opening log_file_path file with read-only type “r” and assigning file’s data to file variable. for line in file gives us access to each line in file. You can add print line under that for loop and run the script, it will print each line of the text file. But in order to match only specific text, we have to use one more for loop.
for match in nditer(regex, line, re. S) is looking for text which matches regex in each line and then the corresponding text is assigned to match variable as an object.
In order to test that you have matched the text you wanted to match, you can use () which will group all regex groups and print them.
You can change () to (2) in order to print the second regex group. Groups in regex are organized using (). This way you can extract specific variables of the text that you are trying to match. In the end we’re adding matched text to the match_list in order to use these values later in the script.
import re
log_file_path = r”C:\ios logs\”
